Skip to content

Continue the chat a person picks; keep the retry card; honest waits on a slow relay; trust-proxy for every relay limit - #2089

Merged
AbirAbbas merged 7 commits into
devfrom
fix/relay-l3-findings
Oct 31, 2026
Merged

AbirAbbas merged 7 commits into
devfrom
fix/relay-l3-findings

Conversation

@AbirAbbas

@AbirAbbas AbirAbbas commented Oct 31, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes what the two-network relay test (lane L3, re-check on 72a2792e6) found on the client side. One PR, split by concern: D1, P, the continue fixes (D6, D3, D5, D7), a fix round, and the change entry.

What each defect was, and what changed

D6: the chat a person picked was not the chat that opened. A bare codeaf chat continues the newest chat another computer let go of. While that automatic move ran, every explicit Enter or confirm on Home answered a chat is already continuing here; wait for it to open, and the older chat opened. Now a confirmed different chat waits with the holder the person saw (continuing <Y> next; <X> is still on its way here). The automatic move finishes, is let go without opening (through the path a closing window uses, so a shared session is never closed), and then the picked chat is taken and opened. Confirming the arriving chat joins its move. Only a confirm abandons the automatic move; opening a card and answering "leave it there" does not. An abandoned move is bounded (2 min, then cancelled the way Escape used to). With no input, a bare start behaves as before.

D3: a false "another device" refusal, and no retry card. ErrLeaseHeld went to a footer-only path that skipped #2078's card. The likely race: after a move, the receiving computer's writer re-acquires, which raises the ownership number, and a slow relay's stale listing gives the other side a card with the old number. Now every refusal keeps the 1 retry / 2 leave it there card and the draft. A fresh reading separates the same holder reopening (the card asks again, with what it now shows) from a real different device (the real refusal, on the card). Consent still binds holder and ownership number. Reproduced with the two-home rig in TestAReturnAfterTheWriterReopensIsNotALostRace.

D5: Home gave up on a slow relay, and the first message after a move stayed unsent. Home's listing had 5 s. The send check ran its own 5 s directory read while the arriving writer was still starting, and it printed Home's timeout sentence into the chat. Now Home's listing waits up to 25 s, shows waiting for <host>; still checking at 5 s, and keeps its rows. A real outage (refused, DNS, TLS, 5xx) still shows at once and retries on the old schedule. A send right after this window's own move waits up to 30 s for its writer, off the UI loop, then sends once. Displacement or another window's hold still refuses. A send that cannot leave says <host> has not answered; your message is still in the box, press enter to send it.

D1: a directly exposed codeaf relay trusted any X-Forwarded-For. Registration and dial limits counted a caller-supplied header. Now relay.Server counts the socket peer unless it is given a policy, and --trust-proxy (default off) applies one policy to registrations, dials, the pairing mailbox and link pairing. A relay behind a proxy must now pass --trust-proxy, or all its callers share one allowance. Pairing already behaved that way without the flag. Flag help, docs/relay.md and the relay manual page say so.

P (from the Cloudflare audit): a chat this computer never held came over serial object batches. SyncEngine.Holds read the store-wide publication ledger, and new chats share one store, so every new chat looked warm and skipped its frame plan. Holds now asks about this chat (its adoption record or a recorded turn), and a return into a fresh staging folder asks about the chat's existing folder. In the fixture, a cold 2 MiB chat in a used store went from 7 object POSTs to 1 frame GET, and a warm one-file return still moves 5.9 KB. Correctness rests on the unchanged completeness loop. Known perf-only edge cases (follow-up): after a failed take whose staging was removed, the next take re-downloads frames, and chat metadata restored into an empty store reads warm (today's behaviour).

D4 (found by the re-measure of this PR): a continue failed when another chat was saving its secrets. On rd, 2 of 5 picks were refused with could not continue this chat here because the chat could not be put in this folder. In chat.log: handoff: resource temporarily unavailable, with the hooks phase at 5 ms. A continue's last hook pulls the vault under state-sync.lock without waiting, and every running chat's drive pushes the vault under that same lock, for seconds over a slow relay. The pick fix made it more frequent, because picks that used to be refused as busy now reach the take while the bare start's engines are still saving. The continue's pull now waits up to 30 s, then fails as before; background exchanges still skip a busy lock. Proof: TestAContinueWaitsForAnotherWindowsVaultExchange holds the lock from a second open file through the move; without the wait it fails with exactly that error.

D7: slow empty-folder moves. The cause is not established. Each continue now writes one chat.log line with per-phase durations. First data, from the hand walk on a local relay with the laptop under build load: time spread across directory (0.3–7.3 s), restore (0.3–11 s) and staging (1–7 s). No fixed timer showed up; it looks disk- and relay-bound under load.

Proof

  • Hand walk, D6, local relay, real binary in tmux (B bare start auto-continues X while the person picks Y on Home and confirms):
    • BEFORE (72a2792e6): Y opened 1 of 3. X opened 2 of 3 with the busy line, and X was left running on B.
    • AFTER (round 1 head): Y opened 5 of 5 with no busy line. The waiting line showed, and X was let go each time (released, not running).
    • AFTER (final head): Y opened 6 of 6 (4 of 4 in a clean run, plus 2 of 2 before a run whose harness CLI read timed out under laptop load). The waiting line showed whenever the automatic move was still running, and X ended released every time.
  • New tests (each maps to a contract item in the lane's spec): 45 new tests across internal/tui3, internal/syncsetup (real two-home rig), internal/cellsync, internal/cellstore, internal/relay, internal/relayserve, internal/remote and internal/manual, each written from a contract item. 5 existing tests changed only where the contract changed (the footer-only lost race, the 5 s sentences, the stale listing read out of askChats' new batch). The D3, D5 and P tests were each shown to fail on the pre-fix code first. A read-only review round and one fix round ran before merge (shared-session cleanup, a bound on an abandoned move, Escape, inspect vs confirm, a bound on recovery, progress state, displacement during the writer wait).
  • make test-laws: green (43 packages). On head 55945ba9c (rebased onto teams: opt-in team sharing between paired computers #2080; one conflict in internal/syncsetup/take.go, where teams: opt-in team sharing between paired computers #2080's team rebind met this PR's named return, kept both): check is green (touched tui3, codeaf, rest) and the Full check is green. After each rebase the overlap tests pass locally (A displaced window keeps its unsent turns, and a taken chat can be taken again #2091's drive tests, the native two-home rig, D4, P, D5 and the whole manual package).
  • rd (the Cloudflare dev relay) re-measure by the test lane, on 0ee6ce39e (before the D4 fix), with A and B on one laptop:
    • D3: another device continued this chat first appeared 0 times (0 of 5 small returns, 0 in the D6 setup). The card asked again with l3b1 picked this chat up again; press enter to continue it here 6 times, and one more Enter moved the chat each time. The ownership-number move behind D3 still happens; it is now worded honestly.
    • D5: the relay answered within 5 s, so neither sentence was shown. The first message right after Moved from sent and was answered 10 of 10, with 0 kept-in-box notices.
    • D6: on rd, 3 of 5 picks opened the chosen chat, with no older chat and no already continuing here; the other 2 hit D4 (fixed above). On the local relay with older chats: this build 5 of 5; old 72a2792e6 1 of 5.
    • D2: 50 MiB / 2,000 files A→B in 51.6 s with all hashes OK (was 0 of 5 within 180 s on 72a2792e6, 159 s earlier). Phases: directory 6.1 s, fetch 25.4 s, restore 8.7 s, staging 31.4 s, total 51.4 s. Small A→B median 18.6 s (was 38.4 s); B→A median 44.8 s; a local relay does about 3 s.
    • D9: one pairing, 7.0 s (was 8.8 s median). The Worker's cost is a follow-up.
  • D4 re-check by the test lane on d22a0718a (dev relay, A and B on one laptop, older chats present): the chosen chat opened 5 of 5, with 0 could not be put in this folder and 0 already continuing. The trigger did occur: before picks 4 and 5, B's bare start parked its automatic continue on needs a choice (.git/COMMIT_EDITMSG), and both picks still opened with no handoff error (on 0ee6ce39e, 2 of 4 such picks failed with EAGAIN). Confirm-to-open took 37–79 s on the dev relay, which is the Worker slowness (D2), a follow-up.

Not in this PR (follow-ups)

  • Worker slowness (D2/D9, including the 37–79 s confirm-to-open on the dev relay): every store request on the hosted relay runs two Raft authorize passes, each a device_seen commit plus a barrier, about 2.7 s per request on the dev relay. Ranked causes and the Worker follow-ups are for the relay owners: measure one request in wrangler dev --local, drop the post-I/O device_seen commit, coalesce device_seen, and name the unavailable() branch in 503 bodies. No Worker change in this PR.
  • The automatic continue declines whenever B holds an older copy: continuation needs a choice for changed path: .git/COMMIT_EDITMSG (seen in L3's runs and in this lane's first walk). .git internals probably should not count as local changes.
  • The duplicate directory reads in a take (≈2 requests per move) and the P edge cases above.

Stability

Nothing changes for a person who never races a continue, never hits a slow relay, and does not run a self-hosted relay behind a proxy. No change to internal/clusterstate, wire or stored formats, relay/hosted/, or workflows.

Hello's Mac check (macOS, quoted):

PR #2089 PASS @55945ba9c (on dev b42fdf1): build, all 12 test lines, the handcheck (matching dev at the same wait) and the pick-over-auto check pass. make build PASS (9s).
(1) Tests PASS, all 12 lines with tree clean after each: tui3 x2 (incl. TestContinueTakeReplacesAnOutstandingListing); syncsetup x4 (the 3 #2080-neighbour tests, the 2 vault-wait tests, the stale-refusal set and the return/logs/warm/unused-take set); cellsync; cellstore; relay; relayserve; remote; manual '.'.
(2) Handcheck PASS, 0 FAIL lines, compared with dev b42fdf1 at the same wait: B's Home was read 14s after the last --once on the PR and 13s on dev, both inside the 90s lease. Both show "● alpha" and "running on alpha now". All 5 continues PASS on both, doctor a and b say "nothing wrong" on both, and quits are EXIT=0. Relay cells 3/3, nothing reappeared.
(3) Pick-over-auto PASS, on the first attempt. A delaying proxy in front of B held X's frame reads for 4.8s. Footer: "continuing a brief and simple exchange of words next; random questions and simple answers discussion is still on its way here". Y opened ("Moved from alpha in 700ms") and its follow-up answered KIWI. X's words never appeared on B. A's Y window: "beta continued this chat; this window now only shows it". X: B's directory calls were acquire 200, then release 204 one second later. cell list --all shows X with holder beta and status "-" on both B and A (released). Y reads "running on beta" on A. state-sync.lock: no wait. The take's hooks phase took 15.4ms for X and 15.9ms for Y. The X total was 6.11s, 4.8s of it my hold. Y took 724ms. No team rebind: sharing was off, and no "teams could not be updated" appeared.

An earlier interim (6623e35) showed alpha offline and a lapsed hold after chat --once. The same repro on dev shows the same after the 90 s lease, because --once does not release its hold. That is unchanged by this PR, and how a finished --once chat should look is an open question for the owner.

🤖 Generated with Claude Code

AbirAbbas added a commit that referenced this pull request Oct 31, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
AbirAbbas added a commit that referenced this pull request Oct 31, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@AbirAbbas
AbirAbbas force-pushed the fix/relay-l3-findings branch from efa3098 to 0ee6ce3 Compare October 31, 2026 05:45
AbirAbbas added a commit that referenced this pull request Oct 31, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@AbirAbbas
AbirAbbas force-pushed the fix/relay-l3-findings branch from 14676ae to 6623e35 Compare October 31, 2026 06:12
@AbirAbbas
AbirAbbas marked this pull request as ready for review October 31, 2026 06:19
AbirAbbas and others added 7 commits October 31, 2026 03:19
A self-hosted codeaf relay took the first X-Forwarded-For address as the
caller's network for its registration and dial limits whenever a request
carried one, so a relay exposed directly to the internet let any caller
mint a fresh allowance by naming its own address. --trust-proxy already
existed but reached only the pairing limits.

relay.Server now counts the socket peer unless it is handed a peer policy,
and relayserve hands the pipe the same socket-or-forwarded policy the
pairing mailbox and link pairing use. A relay behind a proxy must now pass
--trust-proxy, or every caller shares the proxy's allowance; the flag help,
docs/relay.md and the relay manual page say so.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
SyncEngine.Holds read the store-wide published.<ledger> file, and new chats
share one store, so after any earlier exchange with a relay every new chat
looked warm and Fetcher.prime skipped its frame plan. A large chat then
came over one serial object batch after another instead of its frames,
eight at a time; on the hosted relay each batch costs seconds.

Holds now asks whether this computer holds history for this chat (its
adoption record or a recorded turn), and a return landing in a fresh
staging folder asks about the chat's existing folder, so a warm return
still fetches only its delta. The completeness loop is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… slow relay says so

The public-network test found four ways continuing a chat between two
computers went wrong:

- D6: a bare start continued the newest chat another computer let go of,
  and while that move ran every explicit pick on Home answered "a chat is
  already continuing here" and the older chat opened. An automatic move now
  gives way: a confirmed different chat waits with the holder the person
  saw, the automatic move settles and its unused hold is released with a
  fresh bounded context, then the picked chat is taken and opened.
  Confirming the arriving chat joins its move. With no input a bare start
  behaves as before.
- D3: a lost-race refusal skipped #2078's retry card and only said
  "another device continued this chat first" in the footer, and a holder
  whose own writer reopened after a move (raising its ownership number) was
  blamed as another device. Every refusal now keeps the card and the draft;
  a fresh reading tells the same holder reopening (the card asks again with
  what it now shows) from a real different device.
- D5: Home gave up on a slow relay after 5 s, and the first message after a
  successful move was left in the box under Home's timeout sentence. Home's
  listing now waits up to 25 s, says "waiting for <host>; still checking"
  at 5 s and keeps its rows; a real outage still shows at once. A send
  right after this window's own move waits up to 30 s for its writer to
  start, then sends once; a send that cannot leave says the message was
  kept and names the relay.
- D7: each continue writes one chat.log line with the time spent in each
  phase, so a slow empty-folder move can be read instead of guessed.

Consent still binds holder and ownership number, and nothing in
internal/clusterstate, the wire or stored formats changed. The manual
pages say what the code now does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… move

From the review of the continue fixes:

- An abandoned automatic arrival no longer closes its conversation; it
  leaves through the path a departing window takes, so another window on
  the same session keeps working and the engine releases the chat only
  when the last window goes. AlreadyHere skips cleanup.
- An abandoned automatic move is cancelled the way Escape used to cancel
  it if it has not answered within two minutes, so a waiting pick always
  runs. A second Escape navigates again instead of being swallowed.
- Opening another chat's card no longer abandons the automatic arrival;
  only confirming it, or the resume chord, does.
- The already-here recovery retries silently once, then keeps the card.
- A send's own reading keeps Home's outstanding listing progress and host.
- A writer wait ends at once on displacement, with the existing words,
  instead of waiting 30 s and blaming the relay.
- The manual says what Escape does to an automatic continue; the release
  retry reuses cellsync.ReleaseRetryEvery.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
askChats now carries its five-second progress tick in a batch, so
a.askChats()() returned the batch rather than the answer, and the test ran
the "old" read only after the take, against the new holder. It now takes
the listing out of the batch and reads it first, which is the in-flight
answer the test was written about; what it asserts is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ling

D4 in the two-network re-measure: 2 of 5 picks on the dev relay failed
with "could not continue this chat here because the chat could not be put
in this folder", and chat.log said "handoff: resource temporarily
unavailable" with the hooks phase at 5 ms. A continue's last hook pulls
the vault under state-sync.lock, taken without waiting, while every
running chat's drive pushes the vault under the same lock; over a slow
relay that push holds it for seconds, so the pull failed at once and the
chat was handed back. The pick fix made this more frequent: picks that
used to be refused as busy now reach the take while the bare start's
engines are still saving.

The continue's pull now waits up to 30 s for the lock, then fails as
before. Background exchanges still skip when the lock is busy, because
the holder is already carrying the same state.
TestAContinueWaitsForAnotherWindowsVaultExchange holds the lock from a
second open file through the move's restore; without the wait it fails
with the exact error L3 saw.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@AbirAbbas
AbirAbbas force-pushed the fix/relay-l3-findings branch from d22a071 to 55945ba Compare October 31, 2026 07:19
@AbirAbbas
AbirAbbas merged commit 4e42800 into dev Oct 31, 2026
22 checks passed
@AbirAbbas
AbirAbbas deleted the fix/relay-l3-findings branch October 31, 2026 09:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant