Skip to content

Surface a dead channel to subscribers; refuse attaching unlinked SHM (#272) - #273

Draft
cboulay wants to merge 1 commit into
cboulay/wire-elisionfrom
fix/272-shm-grow-retained-views
Draft

cboulay wants to merge 1 commit into
cboulay/wire-elisionfrom
fix/272-shm-grow-retained-views

Conversation

@cboulay

@cboulay cboulay commented Sep 30, 2026 •

Copy link
Copy Markdown
Member

Restacked: ported onto #268. A channel can now fail in _reattach_shm (which aborts) or in frames_available (caught by FramedProtocol._drain_frames, kept as _close_exc, then aborted); both end in ChannelProtocol.connection_lost, which records the failure, and _on_disconnected notifies subscribers. New test test_a_channel_crash_reaches_the_subscriber makes delivery raise and checks the subscriber gets ChannelFailed chained from it.

Part of #272. Draft, pending review (Griff). Stacked on #276.

The uncontroversial parts of the original #273 moved to #275 (tolerant SHM close, wait_closed leak, read-only SHM messages, and the grow-race fix). What's left here implies policy choices, so it's held for review.

Changes

  • Surface channel death. A crashed _publisher_connection records the exception and notifies its clients. Subscriber.recv_zero_copy() raises a new ChannelFailed, chained from the cause, instead of waiting forever. In a unit, handle_subscriber logs it at ERROR and keeps serving the stream's other publishers.
  • Refuse attaching an unlinked segment. SHMInfo marks itself unlinked when its last lease ends (and never unlinks twice). The GraphServer forgets such entries, both on a late SHM_ATTACH and as a sweep on SHM_CREATE. A late attach is then refused (ValueError, which the channel already treats as a stale generation) instead of succeeding on the server and failing on the client with FileNotFoundError.
  • Channel treats FileNotFoundError on attach as stale too, for older GraphServers.

With #275's deferred close, a publisher no longer retires a segment a channel still needs, so the last two are defensive.

Open questions

  1. What ChannelFailed should do in a unit: currently log and keep serving other publishers. Alternatives: end the subscriber when none remain, mark the shutdown unclean, or reconnect the channel.
  2. Stale-generation drops: when a channel does see a stale segment, it drops that message (existing behaviour). Is that acceptable, or should the publisher guarantee delivery across generations?

Tests

  • test_subclient.py::test_recv_zero_copy_raises_channel_failed
  • test_shm.py::test_attaching_an_unlinked_segment_is_refused (fails without the GraphServer change)

Full suite: only the 10 test_settings_json_schema / test_inspect_command failures that also fail on the base branch. test_shm_resize_race_repro_completes and the grow tests pass 6/6. The earlier hang came from losing the grow race, which #275 now fixes.

@cboulay
cboulay force-pushed the fix/272-shm-grow-retained-views branch from 8db92a0 to d5aa26a Compare October 1, 2026 06:47
@cboulay
cboulay changed the base branch from cboulay/json-schema-settings-stack to cboulay/wire-elision October 1, 2026 06:47
@cboulay cboulay changed the title Survive SHM grow while subscribers retain zero-copy views (#272) Surface a dead channel to subscribers; refuse attaching unlinked SHM (#272) Oct 1, 2026
@cboulay
cboulay force-pushed the fix/272-shm-grow-retained-views branch 2 times, most recently from d5aa26a to 7e3cec0 Compare October 1, 2026 06:51
cboulay added a commit that referenced this pull request Oct 1, 2026
Split from #273: the parts that stand on their own, without the
ChannelFailed policy that is still under discussion.

- SHMContext: if closing the mapping raises BufferError because views into
  it are still alive (a unit kept a zero-copy array), drop the handle and
  let the mmap be unmapped when the last view is collected, instead of
  killing the channel when the publisher grows its segment.
- SHMContext.wait_closed() also closes the segment: a monitor task
  cancelled before it first ran never executed its finally, leaking the
  mapping (the "Exception ignored in SharedMemory.__del__" noise).
- Channel: SHM-delivered messages are read-only, matching the TCP path, so
  a subscriber cannot write into memory shared with the publisher and
  every other subscriber.
- Tests: closing with a live view, and a cross-process grow with a
  receiver that retains every message.
@cboulay
cboulay force-pushed the cboulay/wire-elision branch from 62ffa8e to c16bff6 Compare October 1, 2026 16:22
@cboulay
cboulay force-pushed the fix/272-shm-grow-retained-views branch from 7e3cec0 to 3871eea Compare October 1, 2026 16:22
@cboulay
cboulay force-pushed the fix/272-shm-grow-retained-views branch from 3871eea to 2bd2882 Compare October 1, 2026 16:33
…272)

What remains of the #272 work after the uncontroversial parts moved to

- Channel: a crashed publisher connection records its exception and
  notifies its clients; Subscriber.recv_zero_copy raises ChannelFailed
  (chained from the cause) instead of waiting forever. handle_subscriber
  logs it at ERROR and keeps serving the stream's other publishers.
- GraphServer: a segment whose last lease has ended is marked unlinked and
  forgotten, so a late SHM_ATTACH is refused (ValueError, which channels
  already treat as a stale generation) instead of handing back a name the
  client cannot open. Unlinked entries are also pruned on SHM_CREATE, and a
  segment is never unlinked twice.
- Channel: FileNotFoundError on attach is treated as stale too, for older
  GraphServers.
@cboulay
cboulay force-pushed the fix/272-shm-grow-retained-views branch from 2bd2882 to b37b82a Compare October 1, 2026 17:20

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant