Repository navigation
feat(daemon): keep servers running across a daemon restart or upgrade - #41
Merged
Merged
Conversation
The menu bar filter matched on server names and URLs that no row displays, so a row could appear with nothing marked. The filter now matches only the labels a row shows, and highlighting marks each match in the original text, so characters whose lowercase form changes length no longer shift the marks.
…launchd's job list
…check, full probe timeout, and still-starting timeouts
…ts, per-agent hook records, and lock pause guarantees
A test shell that loops with no end and outlives a killed `make test` run kept forking for hours. The run now names and kills any process whose command line names its run root, and fails. Each test shell loop stops after a minute, and the next run reaps shells left by a dead run.
… ensure restarting an unhealthy server
`directa wait <name> --stopped` returned as soon as a server reported failed, even when a process was still running for it. That happens when another server took the port: the first server keeps its process, so a stop was still needed. Now the wait holds until no process is alive, and it returns once `directa stop` has taken the process down. When such a wait times out, the message names `directa stop <name>` as the next step.
The after-tool hook announced servers it had already reported whenever Claude Code subagents ran beside their parent, or an agent moved between a project's checkouts, because they all shared one remembered picture. Each agent (a subagent counts as its own) now has its own record, holding one picture per project, so unchanged servers stay quiet and a project not yet seen by that agent gets its full summary once. Records written by an older version keep loading. Their single picture has no project to file under and is dropped, so each project speaks its summary once more. Tool calls that finish together now take turns on the record with a file lock, so only one of them speaks; a hook that cannot get the lock within a few seconds goes ahead without it.
When the daemon restarts and launchd's job list does not answer, a server whose process survived is left alone until the list can be read. Until now a later status read made that server look stopped with no process, and "directa ensure" then started a second copy beside it or failed on the port it still held. Now the server keeps showing the phase and pid it had, its port still counts as taken for other servers, and start, restart, up, stop, down, unregister, and a pausing lock are refused with daemon-starting, naming the server and its pid. The daemon re-reads launchd's job list in the background with growing delays for up to five minutes. The first answer decides the survivor the way boot restore does: a listed job is taken back over, anything else is stopped and started fresh. A list that never answers leaves the server as it is for the next launch. Also stops a re-register test from racing its own row read by giving the new run its own gate.
Run from a directory that no longer exists (a removed worktree), every project-scoped command walked a relative path upward forever at full CPU. The current directory now reads as gone up front: the command fails usage (exit 2) saying so, with hint `run: directa status --all`. Session hooks and the statusline stay silent, and `lock` refuses because its guarded command needs a directory to run in. Every ancestor walk now ends on any input: a relative path with no current directory comes back as written, and the walks are bounded by the path's own depth.
Asking for a server the project lacks answered with hint `directa status --json`, which listed nothing. The error now looks for the name in the other registered projects: none keeps `directa status --all`, one names that project with `directa status <name> --project '<path>'`, and several list every path with `directa status --all <name>`. A name is never resolved into another project on its own. The linked-worktree message also says that `--project` with the main checkout's path acts on the main checkout's server instead. Covers status, open, monitor, restart, deep links, and the daemon's own start, ensure, stop, logs, and restart refusals.
A shared session capped every HTTP health check at five seconds whatever
timeoutMs said, so a slow cold boot that answered at six or seven seconds
was reported as a timeout and the server stayed "starting". timeoutMs now
bounds each check in full: an answer that arrives inside it passes, however
slow, and a check that outlasts it ends at the deadline.
A check now falls back from IPv4 to IPv6 only when the IPv4 connection was
refused. A timeout is no longer retried over IPv6, which doubled the wait.
Each check also reports why it failed ("HTTP 500", "connection refused",
"timed out after 10s"), and the supervisor keeps the latest result for the
current run as lastHealthCheck on the server's status: failures in a row,
whether it passed, and the reason. It is cleared when the run ends and absent
for servers with no healthcheck.
A running server whose latest health check failed was described as "running and healthy". The phase stays running until enough checks fail in a row, so a server answering 500 read as fine. `directa why` now says "running; N health checks failed in a row (HTTP 500)" for such a server, and "running and healthy" only while the latest check passed. The session context lists a running server with failing checks ahead of the quiet ones, names the failing check, and points at `directa why`. A new health check result alone never makes a tool hook speak.
When ensure, wait, up, restart, or switch timed out while a server was still starting, the output read "fell short (timeout)", which looks like a crash. It now says the server is still starting, not failed, and names `directa wait <name> --timeout <seconds>` to keep waiting. The exit code is still 1 and the JSON is unchanged. The one-line description of a server (status, ensure, up) also names a failing latest health check, so a running server answering errors no longer looks healthy there either.
`directa daemon restart` and `directa daemon install` over a running daemon now hand the servers to the next daemon instead of stopping them. Under the background agent the old daemon exits without stopping anything, keeps every restore intent, and the next daemon adopts the still-running launchd child jobs. A SIGTERM that races the exit does not drain either. A daemon run with --foreground, `daemon stop`, uninstall in every form, logout, and machine shutdown still drain first. `daemon.shutdown` gains an optional `handoff` param (append-only; an older daemon ignores it and drains). A restart or install also waits for the new daemon to finish restoring before it re-ensures the servers, so that re-ensure is no longer refused with daemon-starting.
The command that `directa lock` runs now sees a `DIRECTA_LOCKS` environment variable listing the locks held around it. Each entry is the project's full path, `::`, and the resource name, comma-separated and added after any value the caller already had, so a tool with a lock of its own for the same resource can tell it is already protected. The lock key is now built in one place shared by the daemon and the command line.
`directa lock --pause` used to count a server as paused even when its stop gave up, and to start the command as soon as the server's main process exited, while a worker it started could still be holding the file. The command now starts only after every paused server and every process it started has exited. A server whose stop gives up, or leaves processes running, fails the lock with `resource-locked`, names that server, and points at `directa why <name>`. Servers already paused are resumed and the command never runs. The command line fails at once on that refusal instead of waiting out the acquire timeout. Every stop (restart, watch, stop) now also waits briefly for the processes it killed to exit before it returns, so a respawn does not race a worker still releasing its port.
A restart could report the new run healthy while nothing of its own was listening: a process left over from the old run's tree still held the port for a moment and answered the first health check. Every transition that stops a run and starts the next (restart, the watch sweep's restart, and start or ensure over a server that failed for its port) now waits a bounded time after the stop for the server's claimed ports to have no listener. A listener that stays is named in the server's log (pid, command, and whether it belonged to the run just stopped) and the transition fails with port-held instead of starting a run it cannot vouch for. The server's log also notes when a health check passed but a listener scan afterward found nothing on the server's port.
…status read A freshly restarted server could read as running a stale config, or could start without the port directa injects into its environment. A status read landing in the short stopped moment between a restart's stop and its start replaced the spec the restart had prepared with the raw committed one. The check that the server is idle and the replacement are now one step, and a prepared spec stays in force until the next run starts.
…-held at why When a restart or start was refused after directa had prepared the server's launch settings, the server stayed down but kept showing those settings, and ignored later devservers.json edits until it next ran. The prepared settings are now released on every refusal path (restart, start or ensure over a port failure, and up). The port-held refusal after a stop now tells the reader to run `directa why <server>`, which shows the listener's pid and command, instead of suggesting a kill.
Starting a server whose port an unmanaged process holds now suggests `directa why <server>`, which shows the listener's pid and command, instead of a kill command for a process directa does not own. The refusal message still names the pid and command. The listener lookup the post-healthy check uses is now the one every caller shares, and the test health checker follows the health result type.
A daemon-starting refusal can arrive just before restore ends. When daemon.info then reads as serving, the CLI repeats the request once at once and fails only if that attempt is refused again.
A server whose surviving process boot restore could not decide about stayed refused after that process exited, answered `wait` from an idle supervisor, and was left running by a daemon shutdown. - A deferred survivor whose run is gone is settled on each retry tick and when a request meets the refusal, with no job list needed. - Once the retry window gives up, the refusal says the job list never answered and points at `directa daemon restart`. - `wait` is refused like the other requests for a deferred survivor. - A drain stops deferred survivors through the identity-checked teardown and keeps their restore intent. - A settle that finishes after its project was forgotten acts on nothing.
…le tree A `--pause` acquire stopped declarers one at a time with the key still free, so a `start` or `ensure` of a declarer, or a second acquire, could slip in mid-pause. It also trusted the phase alone when a stop it joined was still waiting for the run's tree to exit. - The key is reserved before the first await. A declarer's start or ensure (including one waiting out a stop) and a second acquire are refused as contended. The reservation is released on every refusal and resumes what was already paused. - A pause that joins a stop in flight, or meets a declarer whose last stop is still waiting on its tree, waits for that stop's own completion. - StopCompletion cases are alphabetized.
The two mid-pause tests ended with the stuck run still writing, which left a temporary tree behind and failed the make test cleanup check.
…nce, refuse unlistable names
A health probe reused a cached 200 from the server's previous run when the server sent a Cache-Control max-age, so a server that was not listening could read as healthy. The probe session now keeps no HTTP cache, and every probe request ignores cached answers.
App Transport Security refuses plain http to `*.localhost` with or without an embedded Info.plist, so the comment now gives that as the reason.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This branch works through the reported bugs and keeps your servers running while the daemon restarts or upgrades. It is stacked on #40, so review that one first; this diff starts where #40 ends.
What changes for you
Servers survive a daemon restart or upgrade.
directa daemon restartanddirecta daemon installused to stop every server. Now the old daemon steps aside, the new one picks each server up where it is, and output written during the swap still reaches the logs. Open browser tabs, long builds, and in-memory state come through intact.daemon stop,uninstall, logout, and shutdown still stop everything first. The menu bar app does the same when it has to re-register the background agent, and it now logs why it did so.Health reporting you can trust
Cache-Control: max-age) could read healthy right after a restart, before it was listening.directa ensurerestarts a server that wentunhealthyinstead of handing it back broken.directa wait <name> --stoppedreturns only once no process is left.lsofcalls.Locks
directa lock --pausestarts your command only once every paused server and its children have exited.DIRECTA_LOCKS, and a nested lock on the same resource runs at once instead of waiting on its own parent.,or::is refused, both on request and indevservers.json.Clearer messages
--projectwhere it matters.Agent hooks. A new session or a compaction gives a full summary again for each checkout, and long agent ids no longer produce file names the system refuses.
Platform
Other changes in this branch
brew stylerequires, so the cask check passes again.make testcleans up its run directory even when it finds stray processes.Verification
On the merged branch:
make buildmake testmake dead-codescripts/test-narrow-pool.shscripts/smoke.shscripts/smoke-cask.shEach new behavior has a test that failed before its fix.
Root cause of the restart-then-healthy report, from the server's own event log: the new run read healthy about 0.2 seconds after it started, a cold start normally takes 35 to 110 seconds, and the health URL answered
cache-control: max-age=5.