Skip to content

Follow chunk_dim rather than assuming "time" - #11

Merged
cboulay merged 1 commit into
mainfrom
cboulay/chunk-dim-and-fingerprint
Sep 4, 2026
Merged

cboulay merged 1 commit into
mainfrom
cboulay/chunk-dim-and-fingerprint

Conversation

@cboulay

@cboulay cboulay commented Sep 4, 2026

Copy link
Copy Markdown
Member

AxisArray.chunk_dim (ezmsg 3.10) names the dimension messages accumulate along. Three places here were guessing "time" instead, and one of the guesses is wrong in a way that reaches the display.

The shmem sink buffers the wrong dimension

ShMemCircBuffSettings.axis defaulted to "time". The ring is a history of the stream, so it has to be the dimension messages accumulate along — time on a raw signal, win downstream of a windowing stage.

Nothing rejected the old default on a windowed stream, because time is present in a (win, time, ch) message. It just buffered the wrong thing:

buffered axis = 'time'
   n_win=4: frame_shape=(4, 3)  frames written=10  srate=100
   n_win=7: frame_shape=(7, 3)  frames written=10  srate=100

buffered axis = 'win'
   n_win=4: frame_shape=(10, 3) frames written=4   srate=10
   n_win=7: frame_shape=(10, 3) frames written=7   srate=10

Two consequences:

  • the window count lands inside frame_shape, so the buffer is reallocated whenever it jitters — which for a windowing stage is most messages;
  • the sample rate published to the viewer is the within-window rate. For a 10-sample window that is a 10× wrong time base on the display.

axis now defaults to None, meaning "follow the message". An explicit setting still wins so an operator can drive a producer that declares nothing, and without either the old "time" fallback stands — so no existing configuration changes behaviour. The resolved dimension is held in state and the buffer is torn down if it changes, since a windowing stage inserted upstream leaves the ring describing the old layout.

The viewer and sigmon sweeps had the same assumption

A sweep keyed on "time" draws each window's interior along the x-axis and treats the windows as channels, reading an offset that doesn't advance with the stream. Both now go through a new describe.stream_axis, which prefers the declaration and falls back to the old guess. A declaration naming a dimension the message no longer has is ignored rather than trusted.

chunk_dim on the wire

The aux blob carries it, recorded separately from buffered_axis so a consumer can tell the source's declaration from an operator's override. The mirror exposes both.

Adding a key deliberately does not bump AUX_FORMAT_VERSION. A reader that predates it ignores what it doesn't know; a newer reader defaults it when an older writer omits it. Both directions keep working, which is the mixed-version pairing this plain-dict format exists to support — bumping would break exactly that. The module docstring now states the policy, and there's a test for a blob written before the key existed.

_axis_equal asks for the fingerprint first

It's on the publisher's per-message path. Its identity shortcut misses precisely when a producer rebuilds its axes — and that's where CoordinateAxis.fingerprint, which every ezmsg source now primes at construction, turns an O(bytes) array_equal into a comparison of two tuples. Absent on older ezmsg and None for an undigestable dtype, so it stays a pure fast path: it can make the check cheaper, never wrong.

Its docstring is also corrected. The CoordinateAxis.__eq__ MRO bug it describes was fixed in ezmsg 3.10 — but the explicit field-by-field comparison stays, because the two halves of a shmem link need not share an ezmsg version and a writer on 3.9 is still one this has to be correct for.

Testing

91 passed, up from 52 before this branch (72 mine, plus 19 from #10 which landed while I was working). Mutation-checked:

mutation tests failed
sink ignores chunk_dim 1
blob drops chunk_dim 3
fingerprint path answers unconditionally 3
stream_axis ignores the declaration 1
stream_axis trusts a stale declaration 1

Dependency

ezmsg>=3.10.0b2, a pre-release pin until 3.10.0 ships. Pinned directly rather than left transitive: uv only enables pre-releases for a package named with a pre-release marker in this file.

🤖 Generated with Claude Code

`ShMemCircBuffSettings.axis` defaulted to `"time"`, and the ring is a history
of the stream, so it has to be the dimension messages accumulate along. That is
`time` on a raw signal and `win` downstream of a windowing stage, and only the
producer reliably knows which.

Nothing rejected the old default on a windowed stream, because `time` *is*
present in a `(win, time, ch)` message. It just buffered the wrong thing:

    buffered axis = 'time'
       n_win=4: frame_shape=(4, 3)  frames written=10  srate=100
       n_win=7: frame_shape=(7, 3)  frames written=10  srate=100

    buffered axis = 'win'
       n_win=4: frame_shape=(10, 3) frames written=4   srate=10
       n_win=7: frame_shape=(10, 3) frames written=7   srate=10

The window *count* lands inside `frame_shape`, so the buffer is reallocated
whenever it jitters, and the rate published to the viewer is the within-window
rate -- a 10x error in its time base for a 10-sample window.

`axis` now defaults to None, meaning "follow the message". An explicit setting
still wins, so an operator can drive a producer that declares nothing; without
either, the old `"time"` fallback stands. The buffered dimension is resolved per
message and held in state, and the buffer is torn down if it changes -- a
windowing stage inserted upstream leaves the ring describing the old layout.

The blob gains `chunk_dim`, recorded separately from `buffered_axis` so a
consumer can tell the source's declaration from an operator's override, and the
mirror exposes both. Adding a key deliberately does not bump
`AUX_FORMAT_VERSION`: a reader that predates it ignores what it does not know
and a newer reader defaults it, so a mixed-version link -- the pairing this
plain-dict format exists to support -- keeps working. Bumping would break
exactly that. There is a test for a blob written without the key.

`_axis_equal` asks for `CoordinateAxis.fingerprint` before reading bytes. It is
on the publisher's per-message path, and its identity shortcut misses precisely
when a producer rebuilds its axes -- where the cached digest, which every ezmsg
source now primes, turns an O(bytes) comparison into a comparison of two
tuples. Absent on older ezmsg and None for an undigestable dtype, so it stays a
pure fast path. Its docstring is also corrected: the `CoordinateAxis.__eq__`
MRO bug it describes was fixed in ezmsg 3.10, but the explicit comparison
stays, because the two halves of a link need not share a version.

The same wrong assumption was in the viewer and sigmon plot paths, where a
sweep keyed on `time` draws each window's interior along the x-axis and treats
the windows as channels. Both now go through `describe.stream_axis`, which
prefers the declaration and falls back to the old guess. A declaration naming
a dimension the message no longer has is ignored rather than trusted.

Requires ezmsg 3.10.0b2 for both fields.

72 passed, up from 52. Mutation-checked: ignoring `chunk_dim` in the sink fails
1, dropping it from the blob fails 3, making the fingerprint path answer
unconditionally fails 3, and both failure modes of `stream_axis` fail 1 each.
@cboulay
cboulay merged commit 8dfe43f into main Sep 4, 2026
11 checks passed
@cboulay
cboulay deleted the cboulay/chunk-dim-and-fingerprint branch September 4, 2026 16:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant