Hold the hash witness's axes by weak reference - #18
Merged
Merged
Conversation
The witness kept the axis objects of the message that set the hash, for an identity fast path. Across a process boundary their data are zero-copy views into the channel's shared memory, so the first message's segment stayed exported for the transformer's lifetime, and a publisher growing its SHM killed the subscriber's channel (ezmsg-org/ezmsg#272). The witness now holds weakref.ref(axis): the fast path still hits when a producer reuses its axes, nothing is pinned, and a dead reference falls through to the value comparison. A dead reference is never mistaken for a dimension that lost its axis. Axis types that cannot be weakly referenced are compared by value every time rather than held strongly.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #17.
Problem
_build_witnesskept the axis objects of the message that set the hash, for an identity fast path (incoming is kept). The witness is only rebuilt when the hash changes, so in a steady stream it held the first message's axes for the transformer's lifetime. For a message from another process, those axes'dataare zero-copy views into the channel's shared memory. When the publisher grew its SHM, the subscriber's channel couldn't close the old segment and died for good (ezmsg-org/ezmsg#272).Change
The witness holds
weakref.ref(axis)instead:Noneand the check falls through to the value comparison (fingerprint, or gain/offset), which is what already happens on every message across processes.Copying the kept axes (the issue's option 3) wouldn't keep the fast path: a copy is a new object, so
isnever matches. Dropping the shortcut entirely (option 1) works too, but costs ~160 ns per transformer per in-process message.Cost
_message_hashwitness hit,(10, 256)message with a 256-label channel axis, best of 5:Tests
TestTheWitnessHoldsNothingintests/test_hash_witness.py:(time, ch)validator and the generic one (both fail without this change);The issue's repro now reports every segment closable while the witness is set.
One failure in the full suite, which also fails on
dev:test_producer_handles_missing_graph_server_gracefullyfinds a GraphServer if one is already listening on the default port.Related: ezmsg-org/ezmsg#272 (channel-side fix, draft #273), ezmsg-org/ezmsg#274 (fingerprint computed at pickling).