feat(ipc): control socket stage 3 - a state stream replaces polled cache keys

[ChuckBuilds]

Stage 3 of the display control socket (#706, #720, #723): a state stream, so the web interface reads what the display is doing from the socket and stops polling cache files. The design follows the stage plan in docs/IPC_CONTROL_SOCKET.md.

Protocol (still version 1, two new commands)

• state.get {since?, epoch?} returns a versioned snapshot: {schema, version, epoch, pid, served_at, changed, loop, state}.
  • state has five sections:
    • display: what display_current_state holds;
    • on_demand: what display_on_demand_state holds;
    • brightness: {brightness, panel_brightness, dimmed};
    • plugins: the plugin_runtime_snapshot;
    • loop: the render loop's heartbeat age, measured in memory when the display answers.
  • version goes up only when a section really changes. Timestamps such as last_updated, remaining and published_at don't count. epoch changes with each run of the display process.
  • With since and epoch still current, the answer is the short {changed: false, ...} form with no state in it.
• state.subscribe answers with the same snapshot, then pushes events on that connection:
  • {"event": "state"}, a full snapshot of the latest version, whenever the state changes. A reader that falls behind skips versions; nothing is queued up for it.
  • {"event": "tick"}, carrying loop, at least every 5 s.
  • Events have event where responses have ok.
• A stage-2 display answers unknown_command, and the web falls back.
• A snapshot too big for a 64 KiB message is sent without plugins (truncated: ["plugins"]), and readers use the cache for that section only.

Display side.

• StateHub (in src/ipc/server.py) holds the state in memory, with no disk writes.
• The render thread publishes from the places it already published the cache keys: _publish_current_mode_state[_if_changed] and _publish_on_demand_state. The plugin runtime publisher publishes on its 5 s tick.
• Publishing is a hand-off. The lock is held only to swap a reference and compare it. Every socket write runs on the subscriber's own thread.
• Subscribers give back their request slot and take one of 4 subscriber slots, so they never starve commands. A subscriber that stops reading is dropped after the 2 s IO timeout.
• display_controller.py edits are limited to the state-publish helpers, _start_control_server (plus _start_state_stream) and cleanup. I didn't touch run() or the Source helpers (coordination with run-loop stage 2).

Readers that switched

web_interface/display_state.py holds one subscription per web process and answers from memory. Before the subscription has a snapshot, it makes one state.get (0.5 s timeout). When neither works, readers use the cache keys and heartbeat file as before.

Reader	Socket	Fallback
GET /display/current-status	state.display	display_current_state
GET /display/on-demand/status	state.on_demand (remaining recomputed)	display_on_demand_state
/plugins/installed runtime, /plugins/state, /plugins/state/reconcile, startup reconciliation	state.plugins + state.loop	plugin_runtime_snapshot + heartbeat file
/health checks.display_loop	state.loop	heartbeat file

• Each answer carries source (socket / cache / heartbeat_file).
• fix(status): runtime status agrees with the heartbeat; current-status republishes on wake #726's rules apply to the socket path too:
  • stalled when the loop heartbeat is at least 60 s old;
  • stale when the snapshot is past stale_after;
  • a missing beat leaves the snapshot judged on its own;
  • a display section unrefreshed for 120 s reads as unknown, like the cache key's max_age.
• The SSE display stream reads the preview PNG, not cache keys, so it is unchanged.
• The error snapshot and font usage stay cache-only; they were out of scope.

SD writes

The cache keys are still written for one release. While the socket is serving readers (a subscriber is connected, or a state.get came in the last 60 s), the display writes two of them less often:

• display_current_state: written every 60 s and on a flag change, instead of on every mode change.
• plugin_runtime_snapshot: refreshed every 120 s instead of 60 s. The snapshot advertises stale_after 360.

Why it is safe:

• The relaxed rate applies only while readers are on the socket.
• When they leave, the next publish after the reader window writes a changed mode at once.
• A fallback copy is never older than the readers' 120 s max_age.

Measured with a fake-clock harness (test_cache_writes_per_minute_with_and_without_socket_readers, rotation of 15 s screens):

Key	writes/min before	with socket readers
display_current_state	4.0	1.0
plugin_runtime_snapshot	1.0	0.5
total	5.0	1.5 (-70%, ~7,200 → ~2,200 writes/day)

display_on_demand_state is event-only and unchanged. The heartbeat is on tmpfs, so it was never an SD write; it stays, because auto_update_verify reads it.

Tests

test/test_ipc_state_stream.py (44) and test/test_state_stream_readers.py (35) cover:

• Stream correctness and versioning:
  • volatile keys don't bump the version;
  • the since/epoch short form;
  • a new epoch per run;
  • changes pushed in order;
  • a 200-publish burst coalesced to the latest version;
  • keepalive ticks;
  • a restarted display picked up through its new epoch.
• Fallback: with no socket (or socket off), the routes use the cache and heartbeat file. In the end-to-end test a real server serves the routes; after it closes they report source: cache.
• fix(status): runtime status agrees with the heartbeat; current-status republishes on wake #726 consistency: stalled at the 60 s threshold (and not at 59.5 s), time since arrival counted, no beat judged alone, stale publisher, stale display section.
• Concurrency: 3 concurrent subscribers all receive a change. A 4th is refused busy. ping still gets through with max_clients=2.
• A slow subscriber never blocks the render thread:
  • a raw subscriber with a 4 KiB receive buffer never reads;
  • the publisher keeps going for 3 s with ~13 KB snapshots, worst publish < 50 ms;
  • the stuck subscriber is dropped;
  • a reading subscriber stays current.
  • Also, 8 threads waiting and snapshotting while 500 publishes stay under 50 ms.
• Display side: the publish points feed the hub. Without a hub, writes are unchanged. Relaxed writes behave as described above, and return to normal when readers leave. The runtime publisher pushes to the hub unthrottled.
• Mutation check: I broke 10 things one at a time in a copy, and each was caught (1-3 new tests fail per mutation):
  • a subscriber keeps its request slot;
  • no truncation of oversized snapshots;
  • volatile keys bump the version;
  • the socket view ignores the loop age;
  • writes are never relaxed;
  • the runtime refresh is never relaxed;
  • a stale display section is trusted;
  • the routes ignore the socket;
  • publish holds the lock;
  • no keepalive ticks.
• WSL (Ubuntu, Python 3.12), as user and as root: the two new files pass 79/79 as chuck and as root. The wider IPC, runtime, health and socket-route set passes 396 (2 skipped) both ways.
• Cross-user (WSL): a root display served a root:chuck 0660 socket. chuck got state.get and a subscription that saw a pushed change. nobody was refused at connect.
• Golden traces (test/test_run_loop_golden.py): 16 passed, byte-identical (the hub is None without a socket, so the traced cache writes are unchanged).
• Full suite (Windows) against an origin/main baseline worktree, same Pillow (12.3.0 before and after): baseline 61 failed / 6 errors / 8003 passed. Branch 62 failed / 6 errors / 8072 passed (+69 new tests).
  • The FAILED/ERROR IDs are identical except test_display_watchdog.py::TestCheckInPoints::test_the_dwell_sleep_checks_in. It is a real-time sleep test of _sleep_with_plugin_updates, which this PR does not touch.
  • That run overlapped a 3-minute WSL mutation run. The test passes alone, and the whole file passed 5 runs in a row afterwards.
• scripts/check_types.py with mypy 1.20.2 (pure-Python build): clean, 93 modules.

Not done here

• No new src/ module (the hub is in src/ipc/server.py, the subscription in src/ipc/client.py), so no follow-up is needed for the monorepo's core-module table. The one new module is web_interface/display_state.py.
• No rig was touched. Checks needed on ledpi:
  1. Deploy, restart both services, and confirm Control socket listening in the journal. Confirm ls -l /run/ledmatrix/control.sock is still srw-rw---- root ledmatrix.
  2. curl -s localhost:5000/api/v3/display/current-status should show "source": "socket" and follow the panel through a rotation. /api/v3/health should show display_loop.source: socket with an age under ~5 s. /api/v3/plugins/installed should show runtime.source: socket and status: live.
  3. On-demand start then stop: on-demand/status should show the outcome at once, with remaining counting down.
  4. Fallback: sudo systemctl stop ledmatrix should switch the routes to source: cache (with nulls once the 120 s max_age passes). Start it again and they should return to socket within about 30 s (reconnect backoff). sudo systemctl restart ledmatrix-web.service should make the subscription come back, with no errors in the display journal.
  5. Write rate: for 10 min with the web service up, then 10 min with it stopped, sample the mtimes of the display_current_state and plugin_runtime_snapshot files in /var/cache/ledmatrix once a second and count the changes. Expect about 1 + 0.5 per minute versus about 4 + 1 per minute for 15 s screens.
  6. Frame timing: python3 scripts/frame_soak.py --preview against alternated main arms (the default gate already fails on plain main). This checks that the per-pass in-memory publish adds no late frames.
  7. Optional: a plugin whose display() sleeps 90 s should make runtime.status stalled and display_loop stalled over the socket, as fix(status): runtime status agrees with the heartbeat; current-status republishes on wake #726 does over the files.

🤖 Generated with Claude Code

Summary by CodeRabbit

• New Features

  • Display and plugin status now use live state updates when available, with cache and heartbeat-file fallbacks.
  • Status and health responses identify where their state information comes from.
  • Runtime health reporting includes the age of the display-loop heartbeat.

• Improvements

  • Reduced cache update frequency while live state readers are active.
  • Added documentation for the state stream, status sources, and device checks.

[coderabbitai]

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

• Configuration used: Organization UI
• Review profile: CHILL
• Plan: Advanced
• Run ID: 286f3a0d-3266-45e1-9803-8d664bae8a2c

📥 Commits

Reviewing files that changed from the base of the PR and between 6bf7c3f and e0c15b0.

📒 Files selected for processing (16)

• CHANGELOG.md
• docs/IPC_CONTROL_SOCKET.md
• docs/REST_API_REFERENCE.md
• src/display_controller.py
• src/display_watchdog.py
• src/ipc/client.py
• src/ipc/contract.py
• src/ipc/server.py
• src/plugin_system/plugin_runtime.py
• test/test_ipc_state_stream.py
• test/test_state_stream_readers.py
• web_interface/app.py
• web_interface/blueprints/api_v3/__init__.py
• web_interface/blueprints/api_v3/display.py
• web_interface/blueprints/api_v3/misc.py
• web_interface/display_state.py

Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.

---

📝 Walkthrough

Walkthrough

The pull request adds versioned state snapshots and subscriptions to the control socket. Display and plugin publishers send state to an in-memory hub. Web-interface readers prefer socket data and retain cache and heartbeat-file fallbacks.

Changes

Control Socket State Stream

Layer / File(s)	Summary
Define and serve state snapshots src/ipc/contract.py, src/ipc/server.py, test/test_ipc_state_stream.py, docs/IPC_CONTROL_SOCKET.md	The protocol adds state.get and state.subscribe. The server versions snapshots, streams state and keepalive events, and limits active subscribers. Tests cover snapshot handling, subscription behavior, and server shutdown.
Publish display and plugin state src/display_controller.py, src/plugin_system/plugin_runtime.py, src/display_watchdog.py, test/test_state_stream_readers.py, docs/IPC_CONTROL_SOCKET.md, CHANGELOG.md	The display controller and plugin runtime publisher send state to the hub. Cache refresh intervals are relaxed while socket readers are active. RenderWatchdog.liveness() reports heartbeat age.
Read socket state in web routes src/ipc/client.py, web_interface/display_state.py, web_interface/blueprints/api_v3/*, web_interface/app.py, test/test_state_stream_readers.py, docs/REST_API_REFERENCE.md, docs/IPC_CONTROL_SOCKET.md	The IPC client adds one-shot reads and a reconnecting subscription. Web readers prefer socket state and fall back to cache keys or the heartbeat file. Responses identify their source.

Priority: ➖ Normal

Estimated code review effort: 4 (Complex) | ~45 minutes

Change: Feature

Sequence Diagram(s)

sequenceDiagram
  participant WebRoute
  participant DisplayState
  participant StateSubscription
  participant ControlServer
  participant StateHub
  participant Cache
  WebRoute->>DisplayState: Request display or runtime state
  DisplayState->>StateSubscription: Read latest snapshot
  StateSubscription->>ControlServer: Subscribe to state stream
  ControlServer->>StateHub: Read snapshot and wait for changes
  StateHub-->>ControlServer: Return state and keepalive events
  ControlServer-->>StateSubscription: Send snapshot and events
  DisplayState-->>WebRoute: Return socket state when available
  DisplayState->>Cache: Read fallback when socket state is unavailable

Loading

Merge Risk: ⚪ Minimal · up to e0c15

No identified issue blocks merging after normal checks. Validation on ledpi remains outstanding.

Security Architecture Review

Security architecture risk: 🔵 Low · up to e0c15

The state stream is bounded, uses connection-level access controls, and retains file-based fallbacks. No introduced security issue was established in the inspected paths. Actual deployment permissions and the complete pre-change exposure comparison remain unverified.

Retained concerns
No architecture-level concerns identified.

Security review details

Security Blast Radius

• inferred — The supported attack surface is the installation's local control socket and existing HTTP state consumers. Subscription access reaches runtime snapshots, not a new state-mutation command. Effective reachability depends on shared-group membership and optional web-login configuration; no cross-tenant or cross-environment expansion was established.

Security Findings and Attack Paths

• inferred — A peer admitted by the socket policy can occupy the bounded subscriber pool. The inspected path does not establish unbounded subscriber retention or authorization bypass: admission precedes streaming, command capacity is separate, slow sends terminate, and terminal cleanup releases reservations. No introduced security concern was established for this candidate.

Trust Boundaries and Controls

• observed — Socket-first reading does not replace the application's optional authentication hook. When login is enabled, sessions or tokens and established local/setup exceptions govern access. Unauthenticated health callers receive only overall health status rather than detailed socket-derived checks.

Resilience and Maintainability Implications

• observed — Connection failures mark the client copy disconnected and close the socket before bounded-backoff retry. Server shutdown wakes waiting subscribers; their cleanup releases ownership, while an already blocked send relies on its I/O timeout. These mechanisms contain stale-reader authority and reservation leakage.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 35.33% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 184 functions across 13 files. (3 skipped…	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly identifies the control socket state stream as the primary change and notes its replacement of polled cache keys.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

Full details: Docstring Coverage

Explanation

Docstring coverage is 35.33% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 184 functions across 13 files. (3 skipped: 3 unsupported.)

• Fix all pre-merge checks with AI

✨ Finishing Touches 💡 1

📝 Generate docstrings 💡

• Commit to this branch
• Create a new PR

🧪 Generate unit tests (beta)

• Commit to this branch
• Create a new PR

• Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[codacy-production]

Up to standards ✅

🟢 Issues 0 issues

Results:
0 new issues

View in Codacy

🟢 Metrics 197 complexity

Metric	Results
Complexity	197

View in Codacy

_{NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.}