Skip to content

CoordinateAxis.fingerprint: compute when pickling, widen to 64 bits - #274

Draft
cboulay wants to merge 3 commits into
cboulay/protocol-hot-pathfrom
cboulay/fingerprint-on-pickle
Draft

cboulay wants to merge 3 commits into
cboulay/protocol-hot-pathfrom
cboulay/fingerprint-on-pickle

Conversation

@cboulay

@cboulay cboulay commented Oct 1, 2026 •

Copy link
Copy Markdown
Member

Restacked: now based on #268. Adds a commit chaining CoordinateAxis.__getstate__ to super().__getstate__(): with #269 below, ArrayWithNamedDims.__getstate__ exists (it makes strided data contiguous), and returning self.__dict__ bypassed it for coordinate axes (#269's test_coordinate_axis_shares_the_contiguity_fix caught it). Reaching it through the MRO also makes super() safe on Python 3.10.

Stacked on #271.

Why

CoordinateAxis.fingerprint is cached in the instance __dict__, so it already survived pickling, but only if the producer had touched it. A producer that never did shipped its axes without a fingerprint. On the far side of a process boundary every message unpickles into a new axis object, so the first consumer that hashes the axis (e.g. any baseproc BaseStatefulTransformer) recomputed it for every message.

Change

CoordinateAxis.__getstate__ touches self.fingerprint and returns self.__dict__. Pickling (and deepcopy, used for leaky subscribers) now always carries the fingerprint, whichever unit built the axis. Producers no longer need to touch it themselves.

It returns self.__dict__ rather than super().__getstate__() because object.__getstate__ only exists from Python 3.11 and ezmsg still supports 3.10. Worth revisiting once 3.10 is dropped.

Cost

Serializing a (10, 256) float32 message with a 256-label channel axis:

before after
reused axis object (typical) 8.7–9.0 µs 8.8 µs
fresh axis object per message 9.5–9.6 µs 10.3 µs

The ~0.7 µs for a per-message axis is the fingerprint computation plus pickling the cached tuple. The first consumer after the hop no longer pays it.

Follow-up

Once this is released, producers that touch .fingerprint at construction (blackrock, simbiophys, orion, parts of lsl/neo/nwb/xdf/sigproc) can stop doing so and leave it lazy: nothing is computed if no unit ever needs it. Touching it at construction saves at most ~1 µs for a small axis, and only when the cache is cold (256 labels: 0.45 µs hot vs 1.39 µs after evicting 64 MiB; 1024 structured records: 5.88 vs 5.96 µs). And when producers reuse their axes, that's per configuration change, not per message.

Tests

test_pickling_computes_it_if_untouched covers plain pickle and MessageMarshal. Full suite: the only failures are the 10 test_settings_json_schema / test_inspect_command failures that also fail on the base branch.

Related: #272, ezmsg-org/ezmsg-baseproc#17.

Also: 64-bit digest

The fingerprint's digest is now (crc32 << 32) | adler32 instead of crc32 alone. A collision used to mean missing one reconfiguration; with #276 the fingerprint also keys axes elided on the wire, where a collision would silently swap one axis's values for another's. crc32 and adler32 are structurally different, so equal-length contents must collide in both. The fingerprint stays a 5-tuple with one int digest.

Cost per axis object on a structured (label, x, y, z) axis: 0.36 → 1.0 µs at 256 channels, 1.3 → 3.8 µs at 1024 (blake2b would be ~10 / 39 µs). test_a_crc32_collision_is_still_told_apart constructs a genuine crc32 collision (CRC is linear, so one can be solved for) and checks the fingerprints still differ.

@cboulay
cboulay marked this pull request as draft October 1, 2026 03:28
@cboulay cboulay changed the title Compute CoordinateAxis.fingerprint when pickling CoordinateAxis.fingerprint: compute when pickling, widen to 64 bits Oct 1, 2026
The cached fingerprint already rode along when a producer had touched it,
but a producer that never did shipped axes without one, so the first
consumer past a process boundary recomputed it for every message.
__getstate__ now materializes it first. A reused axis costs a dict lookup;
a per-message axis moves the one computation to the publisher.

Returns self.__dict__ rather than super().__getstate__(): object only
gained __getstate__ in Python 3.11, and ezmsg still supports 3.10.
crc32 alone was enough while the fingerprint only decided state resets,
where a collision means missing one reconfiguration. The transport is about
to send axes a receiver already holds as a token derived from the
fingerprint, and a collision there would silently swap one axis's values
for another's. Fold adler32 in beside crc32: a structurally different
checksum, so equal-length contents must collide in both. The digest stays a
single int, so the fingerprint's shape is unchanged.

Cost on a structured (label, x, y, z) axis: 1.0 vs 0.36 us at 256 channels,
3.8 vs 1.3 us at 1024 -- once per axis object, and ~10x cheaper than
blake2b. The new test constructs a genuine crc32 collision and checks the
fingerprints still differ.
Stacked on #269, ArrayWithNamedDims now defines __getstate__ (making strided
data contiguous before pickling), and returning self.__dict__ here bypassed
it for coordinate axes. super().__getstate__() reaches it through the MRO,
so it is also safe on Python 3.10, where object has no __getstate__.
@cboulay
cboulay force-pushed the cboulay/fingerprint-on-pickle branch from 014f440 to 8f43997 Compare October 1, 2026 16:22
@cboulay
cboulay changed the base branch from cboulay/json-schema-settings-stack to cboulay/protocol-hot-path October 1, 2026 16:22

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant