Detector channel fix, SIS3300/SIS3301 fixes, and file copy, partial merge

CHANGELOG.md

@@ -0,0 +1,21 @@
+# Change Log
+## 1.1.2
+### New features
++ Add `events.bad_events` metadata to merged HDF5 output.
++ Mark runs as broken when required map-defined GRAW stacks are missing.
++ Add `transpose_frib` config option to restore the historical transposed FRIB matrix layout when needed.
+### Behavior changes
++ Existing YAML configs remain compatible; `transpose_frib` defaults to `false`.
++ The default FRIB dataset layout is unchanged unless `transpose_frib` is enabled.
+
+## 1.1.1
++ Update to hdf5-metno 0.12.3, compatiable with hdf5 2.0.0.
+
+## 1.1.0
+### New features
++ Option for copying files to local storage before merging.
++ Option for merging only ATTPC or silicon data.
++ Add overange flag to events from SIS3300/SIS3301 module.
+### Bug fixes
++ Fix bug in wrong assignment of channel (column 4) in AGET matrix.
++ Fix bug in decoding data from SIS3301 module.

Cargo.toml

@@ -1,7 +1,7 @@
 [workspace]
 members = [
     "libattpc_merger",
-    "attpc_merger", 
+    "attpc_merger",
     "attpc_merger_cli",
 ]
 resolver="2"
@@ -11,7 +11,7 @@ authors = ["Gordon McCann"]
 edition = "2021"
 license = "MIT"
 license-file = "LICENSE.txt"
-version = "0.3.1"
+version = "1.1.2"
 repository = "https://github.com/ATTPC/attpc_merger"
 readme = "README.md"
 
@@ -21,7 +21,7 @@ bit-set = "0.8.0"
 bitvec = "1.0.1"
 byteorder = "1.5.0"
 fxhash = "0.2.1"
-hdf5 = { package = "hdf5-metno", version = "0.10.1" }
+hdf5 = { package = "hdf5-metno", version = "0.12.3" }
 human_bytes = "0.4.3"
 ndarray = "0.16.1"
 serde = { version = "1.0.219", features = ["derive"] }

README.md

@@ -82,10 +82,16 @@ experiment.
 - GRAW directory: Specifies the full-path to a directory which contains the AT-TPC
 GETDAQ .graw structure (i.e. contains subdirectories of the run_# format). If online
 is checked, this field is not available.
+- Merge AT-TPC data: Enables GET pad-plane streams.
+- Merge Silicon data: Enables silicon streams.
 - EVT directory: Specifies the full-path to a directory which contains the FRIBDAQ EVT
 structure (i.e. contains subdirectories of the run# format)
+- Transpose FRIB matrices: If enabled, FRIB waveform datasets are written in the historical
+transposed layout. If disabled, the current layout is used.
 - HDF5 directory: Specifies the full-path to a directory to which merged HDF5 (.h5)
 files will be written
+- Copy directory: Optional local staging directory used before merging.
+- Delete copied files after merging: Remove staged files after a successful merge.
 - Channel map (Optional): Specifies the full path to a CSV file which contains the
 mapping information for AT-TPC GET detectors and electronics. If set to default
 (clicking the Default button), it will use a channel map that has been bundled with the
@@ -104,19 +110,34 @@ YAML format of a configuration file is as follows:
 
 ```yml
 graw_path: None
+online: false
+experiment: ''
+merge_atttpc: true
+merge_silicon: true
+transpose_frib: false
 evt_path: None
 hdf_path: None
+copy_path: null
+delete_copied: true
 channel_map_path: null
 first_run_number: 0
 last_run_number: 0
-online: false
-experiment: ''
 n_threads: 1
 ```
 
 Note that if the `channel_map_path` field is set to `null`, the bundled default map
 will be used.
 
+### Versioning
+
+The software package version and the HDF5 format version are tracked separately.
+
+- The package/crate version is defined in the workspace `Cargo.toml`, and matched the git tag versions.
+- The HDF5 format version is defined in `libattpc_merger/src/hdf_writer.rs` as `FORMAT_VERSION`, currently usually is v2.
+
+This means a feature release can bump the project version without necessarily changing
+the HDF5 format version.
+
 ### Channel Map Format
 
 The channel map is a CSV file with *no* whitespaces. The columns are as follows:
@@ -152,7 +173,7 @@ The data format used in the HDF5 data is as follows:
 
 ```text
 run_0001.h5
-events - min_event, max_event, min_get_ts, max_get_ts, frib_run, frib_start, frib_stop, frib_time, version
+events - min_event, max_event, min_get_ts, max_get_ts, frib_run, frib_start, frib_stop, frib_time, version, bad_events(optional)
 |---- event_#
 |    |---- get - id, timestamp, timestamp_other
 |    |    |---- pads(dset)
@@ -163,9 +184,15 @@ events - min_event, max_event, min_get_ts, max_get_ts, frib_run, frib_start, fri
 |    |---- frib_physics - id, timestamp
 |    |    |---- 907(dset) (coincidence register)
 |    |    |---- 1903(dset) (SIS3300 8 channels 12 bits)
-|    |    |---- 1904(dset) (SIS3301 8 channeks 14 bits)
+|    |    |---- 1904(dset) (SIS3301 8 channels 14 bits)
 |    |    |---- 1905(dset) (SIS3301 8 channels 14 bits)
 |    |    |---- 1906(dset) (SIS3316 16 channels 14 bits)
 scalers - min_event, max_event
 |---- event_#(dset) - start_offset, stop_offset, timestamp, incremental
 ```
+
+`bad_events`, when present, stores real GET event IDs that were found to be incomplete
+for the required stream set.
+
+The merger will also fail a run as broken if a required `(cobo, asad)` stack defined by
+the active channel map is missing from the input data.

attpc_merger/README.md

@@ -1,4 +1,4 @@
-# attpc_merger_cli
+# attpc_merger
 
 Part of the attpc_merger crate family.
 
@@ -25,10 +25,18 @@ The following configuration controls are available in the GUI:
 - GRAW Files from Online Source Checkbox: If checked, will try to access GRAW files from the AT-TPC DAQ itself. Should typically be unchecked. Only for use when running an experiment.
   - Experiment: Only available when online checkbox is checked. Name of the experiment, matching the AT-TPC DAQ experiment name.
 - GRAW directory: Specifies the full-path to a directory which contains the AT-TPC GETDAQ .graw structure (i.e. contains subdirectories of the run_# format). If online is checked, this field is not available.
+- Merge AT-TPC data: Enables GET pad-plane streams.
+- Merge Silicon data: Enables silicon streams.
 - EVT directory: Specifies the full-path to a directory which contains the FRIBDAQ EVT structure (i.e. contains subdirectories of the run# format)
+- Transpose FRIB matrices: Write FRIB waveform datasets in the historical transposed layout when required by downstream analysis.
 - HDF5 directory: Specifies the full-path to a directory to which merged HDF5 (.h5) files will be written
+- Copy directory: Optional local staging directory used before merging.
+- Delete copied files after merging: Remove staged files after a successful merge.
 - Pad map: Specifies the full path to a CSV file which contains the mapping information for AT-TPC pads and electronics
 - First Run Number: The starting run number (inclusive)
 - Last Run Number: The ending run number (inclusive)
 
 Configurations can be saved using File->Save and loaded using File->Open
+
+If a required `(cobo, asad)` stack from the active channel map is missing, the GUI merge
+will stop and report the run as broken.

attpc_merger/src/app.rs

@@ -1,6 +1,6 @@
 use std::fs::File;
 use std::io::Write;
-use std::path::Path;
+use std::path::{Path, PathBuf};
 use std::sync::mpsc;
 use std::thread::JoinHandle;
 
@@ -10,7 +10,7 @@ use rfd::FileDialog;
 use libattpc_merger::config::Config;
 use libattpc_merger::error::ProcessorError;
 use libattpc_merger::process::{create_subsets, process_subset};
-use libattpc_merger::worker_status::WorkerStatus;
+use libattpc_merger::worker_status::{BarColor, WorkerStatus};
 
 fn render_error_dialog(show: &mut bool, ctx: &eframe::egui::Context) {
     eframe::egui::Window::new("Error")
@@ -67,7 +67,13 @@ impl MergerApp {
                 // Spawn it
                 let conf = self.config.clone();
                 let tx = self.worker_tx.clone();
-                self.worker_statuses.push(WorkerStatus::new(0.0, 0, idx));
+                let bar_color = if self.config.need_copy_files() {
+                    BarColor::GREEN
+                } else {
+                    BarColor::CYAN
+                };
+                self.worker_statuses
+                    .push(WorkerStatus::new(0.0, 0, idx, bar_color));
                 self.workers.push(std::thread::spawn(move || {
                     process_subset(conf, tx, idx, subset)
                 }))
@@ -218,20 +224,30 @@ impl eframe::App for MergerApp {
                     }
                     ui.end_row();
                 }
+                ui.checkbox(&mut self.config.merge_atttpc, "Merge AT-TPC data");
+                ui.checkbox(&mut self.config.merge_silicon, "Merge Silicon data");
+                ui.end_row();
 
                 //EVT directory
-                ui.label(format!("EVT directory: {}", self.config.evt_path.display()));
+                ui.label(format!(
+                    "EVT directory: {}",
+                    self.config
+                        .evt_path
+                        .clone()
+                        .unwrap_or(PathBuf::from("None"))
+                        .display()
+                ));
                 if ui.button("Open...").clicked() {
-                    if let Some(path) = FileDialog::new()
+                    self.config.evt_path = FileDialog::new()
                         .set_directory(
                             std::env::current_dir().expect("Couldn't access evt directory"),
                         )
-                        .pick_folder()
-                    {
-                        self.config.evt_path = path;
-                    }
+                        .pick_folder();
                 }
                 ui.end_row();
+                ui.label("Transpose FRIB matrices");
+                ui.checkbox(&mut self.config.transpose_frib, "");
+                ui.end_row();
 
                 //HDF directory
                 ui.label(format!(
@@ -250,6 +266,27 @@ impl eframe::App for MergerApp {
                 }
                 ui.end_row();
 
+                // Copy file
+                ui.label(format!(
+                    "Copy directory: {}",
+                    self.config
+                        .copy_path
+                        .clone()
+                        .unwrap_or(PathBuf::from("None"))
+                        .display()
+                ));
+                if ui.button("Open...").clicked() {
+                    self.config.copy_path = FileDialog::new()
+                        .set_directory(
+                            std::env::current_dir().expect("Couldn't access runtime directory"),
+                        )
+                        .pick_folder();
+                }
+                ui.end_row();
+                ui.label("Delete copied files after merging");
+                ui.checkbox(&mut self.config.delete_copied, "");
+                ui.end_row();
+
                 //Pad map
                 let map_render_text: String = match &self.config.channel_map_path {
                     Some(p) => p.to_string_lossy().to_string(),
@@ -309,12 +346,28 @@ impl eframe::App for MergerApp {
                     .size(18.0),
             );
             for status in self.worker_statuses.iter() {
-                ui.add(ProgressBar::new(status.progress).text(format!(
-                    "Worker {} : Run {} - {}%",
-                    status.worker_id,
-                    status.run_number,
-                    (status.progress * 100.0) as i32
-                )));
+                let msg = match status.color {
+                    BarColor::GREEN => "Copying",
+                    BarColor::CYAN => "Merging",
+                    _ => "",
+                };
+                let color = match status.color {
+                    BarColor::GREEN => Color32::from_rgb(57, 158, 90),
+                    BarColor::CYAN => Color32::from_rgb(35, 123, 160),
+                    BarColor::MAGENTA => Color32::from_rgb(184, 51, 106),
+                    BarColor::RED => Color32::RED,
+                };
+                ui.add(
+                    ProgressBar::new(status.progress)
+                        .text(format!(
+                            "Worker {} : {} run {} - {}%",
+                            status.worker_id,
+                            msg,
+                            status.run_number,
+                            (status.progress * 100.0) as i32
+                        ))
+                        .fill(color),
+                );
             }
 
             ctx.request_repaint_after(std::time::Duration::from_secs(1));