Heartbeat the worldtree control connection - #52
Open
sonkehahn-shopify wants to merge 1 commit into
Open
Conversation
worldtreed reaps a control-plane connection idle for its `idle_timeout` (300 s in every listener today), and the client only learns at its next request — which then dies mid-evaluation with `worldtree: daemon closed the connection` / EPIPE. Tecnix reads source bytes from the FUSE projection rather than the socket, so a wide evaluation routinely leaves the control connection untouched for far longer than that window: one 678 s gap was enough to kill a whole-World eval. Keep it warm instead. `WorldtreeConn` now runs a heartbeat that, every `tectonix-worldtree-keepalive-interval-ms` (default 120 s, 0 disables), sends one cheap request under the same mutex as real traffic. The daemon has no ping verb, so the probe is `scoped.resolve_ref` of HEAD: a pure overlay-then-base ref lookup that touches no working tree and is a verb the daemon accounts for by name. Any well-formed reply proves liveness and has already reset the daemon's timer, so a typed error is fine; a transport failure retires the loop and leaves the next real request to report it (this seam cannot re-dial).
sonkehahn-shopify
marked this pull request as ready for review
August 26, 2026 17:21
| * leaves the next real request to surface the error with its own context. | ||
| */ | ||
| void keepalive(std::chrono::milliseconds interval) | ||
| { |
Collaborator
There was a problem hiding this comment.
I'm a bit worried about this thread handling signals in line with the rest of nix, e.g. should this use ReceiveInterrupts as an additional trigger to stop?
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
worldtreed's control-plane RPC server reaps any connection that has been idle for
300 s (
ServeConfig::idle_timeout, applied to the host socket and every scopedlistener). tecnix opens exactly one
WorldtreeConnperEvalStateand nevertouches it again between requests — no keepalive, no health check, no reconnect.
A long evaluation that goes quiet for more than five minutes therefore loses its
connection and the next request dies with
worldtree: daemon closed the connection/write(): Broken pipe. Becausetectonix-worldtree-socketdeliberately has no libgit2 fallback, that is a hardeval failure.
Evidence: a whole-World
tec architect get-all-targets --stream(
trig_be7c0e688f7af4a1807f3c38636229b0, 30m32s, 3895 records, 85 outputbursts) had exactly one inter-burst gap over 300 s — 678.5 s — immediately
followed by the EPIPE. The 181 s / 80 s / 66 s gaps in the same run all survived.
Fix
Heartbeat the control connection from the client.
WorldtreeConnspawns akeepalive thread that issues a cheap request every
tectonix-worldtree-keepalive-interval-ms(default 120 000,0disables andspawns no thread). Any traffic resets the daemon's per-connection idle timer, so
the reaper never fires on an otherwise-idle eval.
The probe is
scoped.resolve_refofHEAD:wt-proto::methodis onlyhost.*,scoped.*,tecnix.*).resolve_refis a pure overlay-then-base RefStore lookup — no working-treewalk, no side effects, and it's accounted under its own metric label.
Rejected alternatives:
connection survives
Error::Unsupported), butmetric_labelmaps it to"unknown", so every beat records a failed op.tecnix.zone_tree_shaswith empty zones — triggers aws.porcelain.dirty_zones()sweep.CANCELframe — free and legal, but gets no reply, so it provesnothing about liveness.
idle_timeoutserver-side — owned by//system/scm/worldtree,and leaves every other client exposed.
Details:
RpcErroris swallowed;only a transport failure retires the loop, leaving the next real request to
surface the error (this seam cannot re-dial).
long request isn't late — that request's own traffic already reset the timer.
Testing
src/libexpr-tests/worldtree-keepalive.ccruns an in-process fake worldtreed onan AF_UNIX socket speaking the real length-delimited protobuf wire and records
the methods seen per connection:
heartbeats_the_idle_control_connection— with a 25 ms interval, an idleconnection accumulates
scoped.resolve_refbeats.keepalive_can_be_disabled— with0, it does not.Full
nix-expr-testssuite green (510 passed).