Conversation
cboulay
marked this pull request as draft
October 1, 2026 03:28
This was referenced Oct 1, 2026
The cached fingerprint already rode along when a producer had touched it, but a producer that never did shipped axes without one, so the first consumer past a process boundary recomputed it for every message. __getstate__ now materializes it first. A reused axis costs a dict lookup; a per-message axis moves the one computation to the publisher. Returns self.__dict__ rather than super().__getstate__(): object only gained __getstate__ in Python 3.11, and ezmsg still supports 3.10.
crc32 alone was enough while the fingerprint only decided state resets, where a collision means missing one reconfiguration. The transport is about to send axes a receiver already holds as a token derived from the fingerprint, and a collision there would silently swap one axis's values for another's. Fold adler32 in beside crc32: a structurally different checksum, so equal-length contents must collide in both. The digest stays a single int, so the fingerprint's shape is unchanged. Cost on a structured (label, x, y, z) axis: 1.0 vs 0.36 us at 256 channels, 3.8 vs 1.3 us at 1024 -- once per axis object, and ~10x cheaper than blake2b. The new test constructs a genuine crc32 collision and checks the fingerprints still differ.
Stacked on #269, ArrayWithNamedDims now defines __getstate__ (making strided data contiguous before pickling), and returning self.__dict__ here bypassed it for coordinate axes. super().__getstate__() reaches it through the MRO, so it is also safe on Python 3.10, where object has no __getstate__.
cboulay
force-pushed
the
cboulay/fingerprint-on-pickle
branch
from
October 1, 2026 16:22
014f440 to
8f43997
Compare
cboulay
changed the base branch from
cboulay/json-schema-settings-stack
to
cboulay/protocol-hot-path
October 1, 2026 16:22
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #271.
Why
CoordinateAxis.fingerprintis cached in the instance__dict__, so it already survived pickling, but only if the producer had touched it. A producer that never did shipped its axes without a fingerprint. On the far side of a process boundary every message unpickles into a new axis object, so the first consumer that hashes the axis (e.g. any baseprocBaseStatefulTransformer) recomputed it for every message.Change
CoordinateAxis.__getstate__touchesself.fingerprintand returnsself.__dict__. Pickling (anddeepcopy, used for leaky subscribers) now always carries the fingerprint, whichever unit built the axis. Producers no longer need to touch it themselves.It returns
self.__dict__rather thansuper().__getstate__()becauseobject.__getstate__only exists from Python 3.11 and ezmsg still supports 3.10. Worth revisiting once 3.10 is dropped.Cost
Serializing a
(10, 256)float32 message with a 256-label channel axis:The ~0.7 µs for a per-message axis is the fingerprint computation plus pickling the cached tuple. The first consumer after the hop no longer pays it.
Follow-up
Once this is released, producers that touch
.fingerprintat construction (blackrock, simbiophys, orion, parts of lsl/neo/nwb/xdf/sigproc) can stop doing so and leave it lazy: nothing is computed if no unit ever needs it. Touching it at construction saves at most ~1 µs for a small axis, and only when the cache is cold (256 labels: 0.45 µs hot vs 1.39 µs after evicting 64 MiB; 1024 structured records: 5.88 vs 5.96 µs). And when producers reuse their axes, that's per configuration change, not per message.Tests
test_pickling_computes_it_if_untouchedcovers plainpickleandMessageMarshal. Full suite: the only failures are the 10test_settings_json_schema/test_inspect_commandfailures that also fail on the base branch.Related: #272, ezmsg-org/ezmsg-baseproc#17.
Also: 64-bit digest
The fingerprint's digest is now
(crc32 << 32) | adler32instead of crc32 alone. A collision used to mean missing one reconfiguration; with #276 the fingerprint also keys axes elided on the wire, where a collision would silently swap one axis's values for another's. crc32 and adler32 are structurally different, so equal-length contents must collide in both. The fingerprint stays a 5-tuple with one int digest.Cost per axis object on a structured
(label, x, y, z)axis: 0.36 → 1.0 µs at 256 channels, 1.3 → 3.8 µs at 1024 (blake2b would be ~10 / 39 µs).test_a_crc32_collision_is_still_told_apartconstructs a genuine crc32 collision (CRC is linear, so one can be solved for) and checks the fingerprints still differ.