Skip to content

test(rooms-node): trace crash/rejoin recovery without changing WAL - #334

Draft
0thernet wants to merge 11 commits into
mainfrom
codex/rooms-rejoin-recovery-20261014
Draft

0thernet wants to merge 11 commits into
mainfrom
codex/rooms-rejoin-recovery-20261014

Conversation

@0thernet

@0thernet 0thernet commented Oct 14, 2026 •

Copy link
Copy Markdown
Member

Scope and hold

Test/evidence-only PR; keep draft without auto-merge. Current head 90e361fe5ebc7e9846f723654e13c89372cbcc47 changes only crates/vhalla-rooms-node/src/unix/{tests.rs,rejoin_trace.rs}. Bounded test observers record decoded sync requests/replies and verified validator peer keys, whitelist the restarted h3 WAL startup branch, and append only redacted numeric facts to the CI job summary after all recovery assertions pass. No runtime, WAL, journal, dependency, lockfile, or integration-owned plan change.

Evidence and limits

Local Rust 1.98.1 locked/offline trace tests 6/6 and rejoin-plan test 1/1, strict all-target/all-feature rooms-node Clippy, workspace fmt, and focused diff check passed. Local socket bind returned EPERM; no local socket verdict. Exact-head Rust run 38632796561 passed security, Required and rooms-node; the Linux job log reports crash/rejoin PASS (95.947 s) at 90e361fe. Exact-head CodeQL run 38632784978 passed. Passing-test stdout was captured by nextest and does not appear in the job log; the bounded job-summary trace has not been independently read. Thus neither the observed WAL branch, proof-verified peer lifecycle nor the h3–h4 page exchange is established by these receipts. Do not promote on a green run alone. Earlier run 38628896097 failed a test-only WAL tracer that assumed a nonexistent node span; its failure log showed n3 receiving a request from n1 for h3–h4 and n1 receiving decoded [3,4]. That was not WAL or peer-proof qualification.

Remaining acceptance

Obtain and independently inspect a bounded redacted socket-capable Linux trace (job summary or equivalent) and verify the observed WAL path (valid empty, replayed nonempty, or delayed replay superseded by a committed sync certificate), proof-verified source/victim peers, and source request / victim decoded-page height correspondence; verify stopped same-home n1 had no preloaded h3–h4 bodies and recovered h4 with preserved journal prefix, peer state and planned effects. Source/victim correspondence is by peer and height; libp2p inbound/outbound request IDs are local, not a shared wire ID. The original post-main restarted-h4 timeout remains unexplained even if this new run passes. Diagnose a demonstrated fault before any runtime change; rerun exact-tree gates and review before promotion. Never pad timeouts or rewrite/delete WAL or journal.

@vercel

vercel Bot commented Oct 14, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
valhalla Ready Ready Preview Oct 14, 2026 8:31pm UTC

Request Review

@0thernet

Copy link
Copy Markdown
Member Author

Exact-head Linux receipt (agent-decided, owner may revisit): Rust run 38514134141 passed Required at f5db76e406dca1374c3b5f7649d209f8c138fb0c; the rooms-node job passed 106/106 (6 skipped long poles), including crashed_validator_recovers_and_rejoins in 70.651 s and the four-validator control in 23.369 s. This single green socket-capable run does not identify why post-main run 38501531414 failed at restart, validate WAL replay/peer connections/decided sync, or independently prove exactly-once application. Nextest captures successful-test stdout, so no failure snapshots were emitted here. Keep this PR draft without auto-merge while the main recovery gate remains causally unresolved; acquire a bounded failure trace before changing runtime or promoting the red receipt.

@0thernet

0thernet commented Oct 14, 2026 •

Copy link
Copy Markdown
Member Author

Delivery watch (2026-10-14): I reran the existing exact-head Rust workflow once to look for the intermittent recovery failure. Run 38514134141, attempt 2 passed Required; rooms-node job 115696092658 passed 106/106, including crash/rejoin (40.979s) and four-validator control (20.718s). No failure snapshots were emitted, so the earlier post-main height-4 timeout remains unexplained. PR remains a diagnostic draft with no auto-merge. Next: bounded socket-capable causal instrumentation/trace, then a cause-directed fix and exact-head/post-main qualification; do not treat two green attempts as an exactly-once or WAL/peer-sync proof.

@0thernet

Copy link
Copy Markdown
Member Author

Exact-head CI receipt (2026-10-14; diagnostic draft, not a recovery verdict): pushed test-only a4d7702. Rust run 38617974519 passed security, Required, and Workspace tests (rooms-node) on this head; nextest passed crashed_validator_recovers_and_rejoins (97.231s) and four_validators_commit_planned_batches (22.714s). CodeQL run 38617964034 succeeded on the same head. The success log has no bounded peer-authentication / replay-completion / decided-page-request trace, so the earlier restarted-h4 timeout remains causally unexplained. Local socket bind returns EPERM. Keep draft, no auto-merge until the evidence and same-home recovery acceptance are met; the integration-owned promotion plan was not committed.

cursor Bot pushed a commit that referenced this pull request Oct 15, 2026
<!-- CURSOR_AGENT_PR_BODY_BEGIN -->
## Metric

This PR moves the worker-direction metric: whether autonomous agents
read a written, dated current priority before picking work.

| | Before | After |
| --- | --- | --- |
| Written current priority in files workers read | 0 (two #303 comments
on 2026-10-13 asked for three items; workers shipped #328–#334 instead)
| 1 dated section in `AGENTS.md` and `docs/north-star.md`, plus a
pointer in `docs/north-star-progress.md` |

It does not change `supervisor.lock` flake rate, #280, or the
two-machine journey. Those remain the only in-scope product work.

## What changed

Docs and agent instructions only. No source or test changes. Open PRs
#303, #280, and #334 are untouched.

Inspected for task selection: `AGENTS.md` (always applied), `CLAUDE.md`
(pointer to `AGENTS.md`), `.algal/` (product manifests, not agent
config), `docs/north-star.md`, `docs/north-star-progress.md`,
`docs/main-policy.md` (PR enforcement, not task pick), and repo
skills/routines (none pick the next task).

Added a short **Current priority (set 2026-10-15)** section:

1. Root-cause and fix the intermittent `supervisor.lock` refusal on
Linux (#303), with a same-inode holder trace. A passing rerun is not a
diagnosis.
2. Rerun and land or close #280 (retire control and rooms TUI for
headless surfaces).
3. Record a two-machine north-star journey pass in
`docs/north-star-progress.md`.

Done means measured evidence: flake rate before and after over a stated
number of CI runs, #280 resolved, and a dated two-machine journey pass
recorded. Every later PR must state which metric it moved, with
before/after numbers. Private-relay and other test hardening, marketing,
and site work are paused; Dependabot and security fixes may continue.

<!-- CURSOR_AGENT_PR_BODY_END -->

<div><a
href="https://cursor.com/agents/bc-5203ab5a-0750-5412-a67a-37f2139b2317?cursor_ref=pr_footer&cursor_cta=open_in_web"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://cursor.com/assets/images/open-in-web-dark.png"><source
media="(prefers-color-scheme: light)"
srcset="https://cursor.com/assets/images/open-in-web-light.png"><img
alt="Open in Web" width="114" height="28"
src="https://cursor.com/assets/images/open-in-web-dark.png"></picture></a>&nbsp;<a
href="https://cursor.com/background-agent?bcId=bc-5203ab5a-0750-5412-a67a-37f2139b2317&cursor_ref=pr_footer&cursor_cta=open_in_cursor"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://cursor.com/assets/images/open-in-cursor-dark.png"><source
media="(prefers-color-scheme: light)"
srcset="https://cursor.com/assets/images/open-in-cursor-light.png"><img
alt="Open in Cursor" width="131" height="28"
src="https://cursor.com/assets/images/open-in-cursor-dark.png"></picture></a>&nbsp;</div>

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: hraness <0thernet@users.noreply.github.com>

This branch was successfully deployed

1 active deployment
Preview — 90e361fe Deployed Oct 14, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant