Skip to content

Commit 4b0c1f9

Browse files
ximionximion
committed
Add dedicated documentation for common stream metadata
1 parent b0ef71e commit 4b0c1f9

5 files changed

Lines changed: 123 additions & 37 deletions

File tree

‎content/docs/common-stream-metadata.md‎

Lines changed: 88 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,88 @@
1+
---
2+
title: Common Stream Metadata
3+
weight: 15
4+
---
5+
6+
Every Syntalos data stream carries a small key/value map of static metadata
7+
alongside the actual samples. Producers set the keys that are meaningful
8+
for their data; consumers default gracefully when a key is absent.
9+
10+
The keys themselves are not enforced types - they are conventions shared
11+
across modules. This page is the canonical list. Module pages refer back
12+
to it where they handle a specific key.
13+
14+
## Routing metadata
15+
16+
These keys are set automatically by Syntalos on every output stream and
17+
are intended for downstream provenance / data-naming. Modules don't
18+
normally read or write them directly.
19+
20+
| Key | Type | Description |
21+
|----------------------|--------|------------------------------------------------------------------|
22+
| `src_mod_type` | String | Type (unique identifier) of the source module. |
23+
| `src_mod_name` | String | User-defined name of the source module instance. |
24+
| `src_mod_port_title` | String | Title of the source module's output port. |
25+
| `data_name_proposal` | String | Proposed name for the dataset that stores data from this stream. |
26+
27+
28+
## Signal / sample streams
29+
30+
For one-dimensional time-series streams (`SignalBlockU16`,
31+
`SignalBlockI32`, `SignalBlockF32`).
32+
33+
| Key | Type | Default | Description |
34+
|----------------|----------------|----------------|--------------------------------------------------------------------------------------------------------------|
35+
| `sample_rate` | Double (Hz) | — | Sampling frequency. Required when `time_unit` is `index`, recommended otherwise. |
36+
| `time_unit` | String | `milliseconds` | One of `index`, `seconds`, `milliseconds`, `microseconds`. Describes the per-sample timestamps. |
37+
| `signal_names` | List\[String\] | — | Per-channel labels. The list length must match the channel count. |
38+
| `data_unit` | String | — | Physical unit of a sample *after* the affine transform below is applied (e.g. `µV`, `mPa`, `°C`). |
39+
| `data_scale` | Double | `1.0` | See `data_offset`. |
40+
| `data_offset` | Double | `0.0` | Together with `data_scale`, defines the affine relation between raw samples on the wire and physical values. |
41+
42+
43+
### The `data_unit` / `data_scale` / `data_offset` contract
44+
45+
Consumers that want physical-unit values apply
46+
47+
```
48+
physical_value = data_scale * raw_value + data_offset
49+
```
50+
51+
and interpret the result as being in the unit named by `data_unit`. The
52+
defaults (`scale = 1.0`, `offset = 0.0`) make this a no-op for streams
53+
that already carry physical-unit samples, so producers that emit
54+
pre-scaled data only need to set `data_unit`.
55+
56+
Two patterns are common:
57+
58+
* **Pre-scaled** (e.g. [Intan RHX]({{< ref "modules/intan-rhx" >}})): samples arrive
59+
as `float32` already in µV. The module sets `data_unit = "µV"` and
60+
leaves `data_scale` / `data_offset` unset (i.e. identity).
61+
* **Raw + transform** (e.g. [Open Ephys AcqBoard]({{< ref "modules/open-ephys-acq" >}})):
62+
samples stay as `uint16` to minimize bandwidth and storage. The
63+
module sets `data_unit = "µV"`, `data_scale`, and
64+
`data_offset`.
65+
66+
Writers ([Zarr Writer]({{< ref "modules/zarrwriter" >}}), [JSON Writer]({{< ref "modules/jsonwriter" >}}))
67+
persist the keys alongside the raw data so the conversion is recoverable
68+
even after a recording is closed.
69+
70+
71+
## Image / frame streams
72+
73+
For `Frame` streams.
74+
75+
| Key | Type | Description |
76+
|-------------|-------------|--------------------------------------------------------|
77+
| `framerate` | Double (Hz) | Frame rate. |
78+
| `size` | `MetaSize` | Frame dimensions in pixels (width × height). |
79+
| `has_color` | Bool | True if frames carry colour data, false if monochrome. |
80+
81+
82+
## Table-row streams
83+
84+
For `TableRow` streams.
85+
86+
| Key | Type | Description |
87+
|----------------|----------------|-------------------------------------------------------------------|
88+
| `table_header` | List\[String\] | Column names. The list length must match each row's column count. |

‎content/docs/modules/_index.md‎

Lines changed: 4 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -18,26 +18,7 @@ their usage instructions and programming interface information.
1818

1919
## Common Stream Metadata
2020

21-
Some metadata is always available on output streams:
22-
23-
<table class="list-table" width="100%">
24-
<thead>
25-
<tr>
26-
<th style="width: 15%;">Direction</th>
27-
<th style="width: 85%;">Metadata</th>
28-
</tr>
29-
</thead>
30-
<tbody>
31-
<tr>
32-
<td>Out🠺</td>
33-
<td>
34-
<ul>
35-
<li><code>src_mod_type</code>: String, type (unique name) of the source module.</li>
36-
<li><code>src_mod_name</code>: String, name (user-defined) of the source module.</li>
37-
<li><code>src_mod_port_title</code>: String, title of the source module's port.</li>
38-
<li><code>data_name_proposal</code>: String, proposed name for data storage of data generated by source module.</li>
39-
</ul>
40-
</td>
41-
</tr>
42-
</tbody>
43-
</table>
21+
Every output stream carries a small set of standard metadata keys
22+
(routing/provenance, signal units, frame size, table headers, …). See
23+
[Common Stream Metadata]({{< ref "/docs/common-stream-metadata" >}}) for the canonical
24+
reference.

‎content/docs/modules/jsonwriter.md‎

Lines changed: 8 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -92,7 +92,12 @@ making them distinguishable.
9292

9393
## Stream Metadata
9494

95-
No output streams are generated, but input streams of type `SignalBlockF32`/`SignalBlockI32` must have
96-
`time_unit`, `data_unit` and `signal_names` set.
95+
No output streams are generated. Input signal streams of type `SignalBlockF32`/`SignalBlockI32` must have
96+
[`time_unit`, `data_unit` and `signal_names`]({{< ref "/docs/common-stream-metadata" >}}) set.
9797

98-
For `TableRow` data, the `table_header` metadata entry has to be set.
98+
If the source advertises a raw→physical affine transform via
99+
`data_scale` / `data_offset` (see "[common stream metadata]({{< ref "/docs/common-stream-metadata" >}})"),
100+
those values are persisted as `data_scale` / `data_offset` attributes on the resulting EDL dataset so
101+
consumers can recover physical units from the recording.
102+
103+
For `TableRow` data, the `table_header` metadata entry must be set.

‎content/docs/modules/plot-timeseries.rst‎

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -56,5 +56,10 @@ Ports
5656
Stream Metadata
5757
===============
5858

59-
No output streams are generated, but input streams must have ``time_unit``, ``data_unit`` and ``signal_names`` set.
60-
If the time unit is `index`, ``sample_rate`` also has to be set on the incoming channel.
59+
No output streams are generated. Input streams must have ``time_unit``, ``data_unit`` and ``signal_names`` set
60+
(see `Common Stream Metadata <{{< ref "/docs/common-stream-metadata" >}}>`_). If the time unit
61+
is ``index``, ``sample_rate`` also has to be set on the incoming channel.
62+
63+
If the source stream advertises a raw→physical affine transform via ``data_scale`` and ``data_offset``,
64+
the canvas applies it on the fly (``displayed = data_scale * raw + data_offset``) and shows samples in
65+
the unit named by ``data_unit``. Streams that do not set these keys are displayed unchanged.

‎content/docs/modules/zarrwriter.md‎

Lines changed: 16 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -31,12 +31,19 @@ read the array data.
3131

3232
## Stream Metadata
3333

34-
No output streams are generated, but for input streams of type `SignalBlockF32`/`SignalBlockI32` the
35-
following metadata will be handled explicitly:
36-
37-
| Key | Type | Required? | Description |
38-
|----------------|--------------|-------------|---------------------------------------------------|
39-
| `signal_names` | List<String> | Recommended | List of signal names contained in each data block |
40-
| `sample_rate` | Double | Recommended | Sampling rate in samples per second. |
41-
| `time_unit` | String | No | Unit of the data block timestamps. |
42-
| `data_unit` | String | No | Unit of the signal block values. |
34+
No output streams are generated, but for input signal streams the
35+
following [stream metadata](../common-stream-metadata)
36+
keys are read and persisted as attributes on the resulting Zarr array:
37+
38+
| Key | Type | Required? | Description |
39+
|----------------|--------------|-------------|------------------------------------------------------------------------|
40+
| `signal_names` | List<String> | Recommended | List of signal names contained in each data block. |
41+
| `sample_rate` | Double | Recommended | Sampling rate in samples per second. |
42+
| `time_unit` | String | No | Unit of the data block timestamps. Persisted on the timestamps array. |
43+
| `data_unit` | String | No | Unit of the signal block values after the affine transform below. |
44+
| `data_scale` | Double | No | See [the affine contract](../common-stream-metadata) for `data_scale`. |
45+
| `data_offset` | Double | No | See `data_scale`. |
46+
47+
When the upstream module emits raw samples plus an affine scale/offset, the writer preserves the raw samples in the
48+
Zarr array and stores `data_scale` / `data_offset` as array attributes, so the conversion to physical units is
49+
recoverable from the recording alone.

0 commit comments

Comments
 (0)