Skip to content

fix(status): runtime status agrees with the heartbeat; current-status republishes on wake - #726

Merged
ChuckBuilds merged 2 commits into
mainfrom
claude/fix-status-reporting
Oct 2, 2026
Merged

ChuckBuilds merged 2 commits into
mainfrom
claude/fix-status-reporting

Conversation

@ChuckBuilds

Copy link
Copy Markdown
Owner

Fixes two places where the status the web interface reports disagrees with what the display is doing.

1. Plugin runtime said "live" while the render loop was stalled

Root cause. The plugin runtime snapshot from #690 is written by the plugin-runtime-publisher daemon thread. That thread keeps ticking when the render thread hangs inside a plugin. So /api/v3/plugins/installed reported data.runtime.status: "live" while /api/v3/health reported checks.display_loop: "stalled" from the #687 heartbeat. After a watchdog kill, systemd removes /run/ledmatrix (RuntimeDirectory=), so no heartbeat is left behind. The dead process's last snapshot then stayed "live" until its 180 s stale_after ran out.

Fix: on the reading side (src/plugin_system/plugin_runtime.py). There are no new files and no extra writes. The heartbeat is already written to tmpfs.

  • read_plugin_runtime() also reads the render-loop heartbeat (display_watchdog.read_heartbeat). The status becomes stalled when the snapshot is fresh, the heartbeat came from the same pid, and the heartbeat is at least HEARTBEAT_STALE_SECONDS (60 s) old. That is the same threshold /health uses. A stalled view reports no per-plugin facts, just like stale.
  • A heartbeat that is missing (dev server, emulator, Windows, a display still starting up) or comes from another pid (a restarted display) carries no information. In that case the snapshot is judged on its own.
  • A running snapshot whose pid no longer exists is stale at once. The check is POSIX kill(pid, 0), which sends no signal. It returns EPERM when the web user is not root and the display process is alive, and that counts as alive. On Windows the check is skipped, because os.kill terminates the process there. This closes the window after a watchdog kill.
  • data.runtime gains heartbeat_age_seconds. docs/REST_API_REFERENCE.md and the route docstring are updated.

I chose the reader option over having the publisher include render-loop liveness. The publisher option would mean another write, or the publisher thread tracking the render thread. The reader already has both signals, and doing the check there keeps the /health and runtime verdicts on one threshold. Reconciliation sees stalled as not-live and skips observed checks, which is the same handling as stale.

2. current-status kept is_display_active stale for up to 30 s after a wake

Root cause. _publish_current_mode_state_if_changed() republished display_current_state only when the mode name changed or CURRENT_STATE_REFRESH_SECONDS (30 s) had passed. Waking from scheduled-off, blanking, or an on-demand session starting or ending usually leaves the mode name unchanged. The blank path publishes once and then sleeps through _service_pending_changes(). On wake, the next publish point saw the same mode and a recent publish, so it skipped. This is what ledpi showed.

Fix (src/display_controller.py, the publish helpers only, away from the plugin-reload code that #723 touches): record (is_display_active, on_demand_active) at each publish, and republish as soon as either one changes. When nothing changes there are no extra writes.

Tests

  • test/test_plugin_runtime_snapshot.py:
    • a stale heartbeat with a live snapshot reads as stalled and reports no plugin facts;
    • the threshold is the one the health check uses;
    • a missing or foreign-pid heartbeat, or one with no time in it, leaves the snapshot live;
    • a stale snapshot stays stale;
    • a dead pid reads as stale;
    • process_exists;
    • route-level checks through the real shared cache: a hung loop gives stalled, a fresh heartbeat gives live.
  • test/test_display_controller_shutdown_and_state.py:
    • a wake, a blank, or on-demand starting or ending with the same mode is republished on the next publish;
    • a wake inside the blank's sleep is published by _service_pending_changes();
    • unchanged flags add no writes.
  • Mutation check: with src/ reverted to origin/main, 15 of the new tests fail (the route-level stalled test among them).
  • Golden traces (test/test_run_loop_golden.py): 16 passed, byte-identical.
  • Full suite against an origin/main baseline worktree (Windows, Pillow 12.3.0 before and after): the FAILED/ERROR test IDs are identical (61 failed, 6 errors, all pre-existing). The only diff is three captured log lines whose line number moved from 272 to 273 after a one-line docstring edit. There are 17 more passes.
  • scripts/check_types.py with mypy 1.20.2 (pure-Python build): clean, 93 modules.

🤖 Generated with Claude Code

… republishes on wake

/api/v3/plugins/installed said the display's plugins were "live" while
/api/v3/health said display_loop: stalled. The runtime snapshot is written
from its own thread, which keeps ticking while the render loop is hung.
The reader now also checks the render loop's heartbeat: a fresh snapshot
whose process's heartbeat is HEARTBEAT_STALE_SECONDS old is "stalled",
with no per-plugin facts. A running snapshot from a process that no longer
exists (after a watchdog kill systemd removes the heartbeat directory) is
"stale" at once. Reading side only: no new files or writes.

display_current_state was republished only on a mode change or every
CURRENT_STATE_REFRESH_SECONDS, so waking from scheduled-off (or on-demand
starting or ending) with the same mode left is_display_active /
on_demand_active up to 30 s out of date. A change to either flag now
republishes at once.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 35 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Advanced

Run ID: 7323276f-7a3c-4a17-9869-537f93a36a76

📥 Commits

Reviewing files that changed from the base of the PR and between 8136a2d and e87fe63.

📒 Files selected for processing (7)
  • CHANGELOG.md
  • docs/REST_API_REFERENCE.md
  • src/display_controller.py
  • src/plugin_system/plugin_runtime.py
  • test/test_display_controller_shutdown_and_state.py
  • test/test_plugin_runtime_snapshot.py
  • web_interface/blueprints/api_v3/plugins.py
  • Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Autopilot is currently an internal CodeRabbit preview.


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@codacy-production

Copy link
Copy Markdown

Up to standards ✅

🟢 Issues 0 issues

Results:
0 new issues

View in Codacy

🟢 Metrics 21 complexity

Metric Results
Complexity 21

View in Codacy

NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.

@ChuckBuilds
ChuckBuilds merged commit c14002e into main Oct 2, 2026
13 checks passed
@ChuckBuilds
ChuckBuilds deleted the claude/fix-status-reporting branch October 2, 2026 21:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant