Skip to content

fix(test): wait for writers before removing temp homes to end ENOTEMPTY flakes (lr-c3e80a) - #486

Merged
clagentic-merger[bot] merged 5 commits into
mainfrom
fix/lr-c3e80a-enotempty-teardown
Oct 23, 2026
Merged

clagentic-merger[bot] merged 5 commits into
mainfrom
fix/lr-c3e80a-enotempty-teardown

Conversation

@clagentic-builder

@clagentic-builder clagentic-builder Bot commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

TASK: lr-c3e80a (folds lr-efe464)

What changed (final state, test-only)

One shared helper, test/helpers/remove-temp-home.js (removeTempHome), removes every temp home that a spawned process or in-process async writer may still write into. Order is the fix, not a retry: (0) the caller quiesce step stops in-process timer or debounce writers through their own API; (1) bounded wait for every listed child (ChildProcess or bare pid), failing with the pid if one outlives the bound; (2) stop any live process carrying the harness token (an orphaned grandchild included), SIGTERM then SIGKILL, failing with the pid if one survives; each pid is re-verified (alive, same /proc start time, still carrying the token) before every signal, so a recycled pid is never signalled; (3) let in-flight fs.promises requests finish; (4) bounded removal passes with one shared option set (maxRetries 10, retryDelay 100; the retry is only the last step), then confirm the home stayed gone. No sleep or timeout was lengthened. Without a listable /proc (macOS) the helper does not throw: it still waits for listed children, skips the token scan and logs once (util.debuglog remove-temp-home, or the log option).

Routed through it: test/message-sender-record.test.js (claude + codex loop tests; its world quiesce stops tracked scheduled-input services and flushes every session buffer), test/automation-sender.test.js (loop timers cancelled through quiesce), test/fixtures/daemon-harness.js cleanup, test/shutdown-entry-points-lr-8f12ca.test.js reapDaemon, test/access-harness.js stop (replaced its sleep-and-retry loop; it already destroys the relay before removal). Also fixed waitForExit in the shutdown test and the harness: a signal-killed process has exitCode null, so it hung once cleanup awaited the exit.

Root causes (reproduced on main first)

  1. Loop iteration test (claude/codex): lib/store writes in several fs.promises steps (mkdir loops, writeFile, rename). The test removed the home with a synchronous rmSync, which blocks the event loop so the chain cannot finish, while a request already on the threadpool lands a file mid-removal. Baseline 2 failures in about 175 runs of the file.
  2. web-UI shutdown_server test: the daemon starts a claude CLI child. When the daemon exits first the child is re-parented (ppid=1) and keeps writing .claude.json under the test home. Baseline 5 failures in 13 runs of the file. This is product behavior; no product code changed. Followup below.

Timer-scheduled writers (fold round 1)

The drain sees fs requests that exist; a timer that has not fired has none. Sweep of in-process writers into a test home: lib/store.js is a promise chain only (covered by the drain); lib/config.js:326 and :398 are connection timeouts, not writers; lib/project-loop.js saveLoopState is called from retry, watchdog and stop timers; lib/sessions.js:478 is a 50 ms append debounce with the existing flushSessionBuffer API; lib/scheduled-input/service.js:80 re-arms a timer, with the existing stop(). The limit (a timer-scheduled write must be quiesced by the caller) is in the helper JSDoc and in test/remove-temp-home.test.js with a regression test.

Round 2 (review of db4d353)

  • Tests that need a real /proc (orphaned token writer, stillOwned identity) carry a node:test skip with a stated reason, decided once at load. The stubbed no-/proc test runs everywhere. No other new test assumes /proc.
  • The stop path is driven directly: stopOwned takes an optional injection of scan, identity check and signal functions. New tests assert that a pid whose identity fails between SIGTERM and the SIGKILL step receives no SIGKILL, and that a still-owned pid is escalated and reported as a survivor. Mutation check: removing the pre-SIGKILL filter makes the first test fail (an extra SIGKILL:4242 was sent); restored.
  • Not changed: stillOwned and startTimeOf stay exported for the helper test (test-only module, not shipped API); the /proc literal is a kernel interface, not a host path.

Race demonstrated, then fixed

test/remove-temp-home.test.js shows the old shape failing deterministically (a late-writing child recreates the removed home; an in-flight store write recreates it; a live orphan writer beats a bare rmSync), then the helper passing each case, plus the bounded-wait failure naming the pid.

Sweep

git grep -nE "rmSync|fs.rm(|rmdir" -- test : 100+ hit lines across 95 files. Only sites where a process or async writer can outlive the test were routed. The rest remove dirs no live writer touches; the boot-smoke style sites already report a removal failure as a diagnostic and were left alone. Class sweep: git grep removeTempHome in test, 7 files, all reviewed.

Tests

The repo profile has no test block (clagentic-loadout test exits 72), so the repo admitted commands were used. Isolated runs, each file alone via node --test, 5 consecutive: test/remove-temp-home.test.js 5/5, test/message-sender-record.test.js 5/5, test/automation-sender.test.js 5/5, test/shutdown-entry-points-lr-8f12ca.test.js 5/5, test/access-gate-integration.test.js 5/5, test/daemon-harness-cleanup-lr-e85fec.test.js 5/5. Full npm test on the head: 3603 tests, 3603 pass, 0 fail, 0 skipped; failing-test-ID set: empty.

Out of scope, observed

  • The restart-to-apply-update stale-install test failed in 2 of the 13 baseline shutdown-file runs, alongside the ENOTEMPTY of the previous case; no failure in the post-fix runs (not established as separate). The send-failure test (a step it chose could not be sent) was not reproduced. Both stay open on lr-c3e80a.
  • Followup: the daemon leaves its claude child running after it exits (product). Not fixed here per the ROCI ruling.

No CREW_SOP section 6 record needed: no hook, gate, merge step or test runner changed; tests only, over the repo daemon harness.

🤖 Generated with Claude Code

clagentic-builder Bot and others added 2 commits October 23, 2026 14:05
…-flight writes (lr-c3e80a)

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…ENOTEMPTY flakes (lr-c3e80a)

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@clagentic-security

Copy link
Copy Markdown

BOBBIE — clean (0 finding(s), 1 dropped)
verdict: CLEAN
range: 06b5660..9a9cb9c
engine: carrier: gpt-5.6-sol

Dropped candidates (1):

  • test/helpers/remove-temp-home.js:32 [bobbie.bleed.4] The helper hardcodes the absolute host path /proc, violating the released-tool rule against absolute host paths and making cleanup Linux-specific. (reason: /proc is a standard Linux kernel interface, not a machine-specific path like /home/user; this is a test-only helper with no security exposure, and bobbie.bleed.4 is a nit in the rulebook, not blocking)
{"reviewer": "bobbie", "review_status": "clean", "head_sha": "9a9cb9c8d373fb701804326c79a4cb9916b336c9", "pr_number": 486, "scanners_run": [{"scanner": "gitleaks", "status": "ran"}, {"scanner": "trufflehog", "status": "ran"}, {"scanner": "semgrep", "status": "ran"}, {"scanner": "osv-scanner", "status": "not_applicable", "reason": "no packages found to scan"}], "fence_schema_version": 3, "dropped": [{"file": "test/helpers/remove-temp-home.js", "line": 32, "rule_id": "bobbie.bleed.4", "message": "The helper hardcodes the absolute host path /proc, violating the released-tool rule against absolute host paths and making cleanup Linux-specific.", "reason": "/proc is a standard Linux kernel interface, not a machine-specific path like /home/user; this is a test-only helper with no security exposure, and bobbie.bleed.4 is a nit in the rulebook, not blocking"}], "range": {"basis": "base..head", "base": "06b56605cc7661cf5c694be37f8738922c9f0ada", "head": "9a9cb9c8d373fb701804326c79a4cb9916b336c9"}, "engines": [{"engine": "carrier", "model": "gpt-5.6-sol", "chunks": 1}], "verdict_outcome": "CLEAN", "verdict_reasons": [], "evaluator_version": "2", "policy_base_sha": "06b56605cc7661cf5c694be37f8738922c9f0ada", "evaluated_findings": []}

@clagentic-reviewer

Copy link
Copy Markdown

PEACHES — blocking (2 finding(s), 1 dropped)
verdict: BLOCKED
range: 06b5660..9a9cb9c
engine: carrier: gpt-5.6-sol

Verdict reasons (1):

  • test/helpers/remove-temp-home.js:104:amos.path-choice.1 blocks (computed severity high)

Blocking findings (1):

  • test/helpers/remove-temp-home.js:104 [amos.path-choice.1] The drain sees only currently active FS requests; delayed in-process writes are missed, so cleanup can return before they recreate the removed home.
    failure sequence: An adapter schedules a home write on a timer and shutdown calls removeTempHome before the timer fires -> the timer writes and recreates the home after cleanup returns
    computed severity: high; blocking
    possible duplicate of: test/helpers/remove-temp-home.js:73:amos.path-choice.1
    stakes: exposure=exposed (public repository), data_class=regulated (unstated (worst case)), merge_control=uncontrolled (unstated (worst case)), CODEOWNERS: no CODEOWNERS file (context only)
    To clear it, add this entry to .clagentic/loadout/dispositions.yaml on the base branch (an entry present only in the change under review does not apply):
    dispositions:
    - path: "test/helpers/remove-temp-home.js"
    rule_id: "amos.path-choice.1"
    kind: by_design
    rationale: ""
    by: ""
    at: 2026-10-23

Advisory findings (1):

  • test/helpers/remove-temp-home.js:73 [amos.path-choice.1] SIGKILL uses stale PIDs without re-verifying their token; a PID recycled after SIGTERM can make cleanup kill an unrelated same-user process.
    computed severity: medium; advisory
    possible duplicate of: test/helpers/remove-temp-home.js:104:amos.path-choice.1

Dropped candidates (1):

  • test/helpers/remove-temp-home.js:36 [amos.code-craft.10] The helper hardcodes /proc/..., violating the released-tool ban on absolute host paths and coupling cleanup to a Linux-only filesystem. (reason: The procfs path /proc/PID/stat is a kernel interface, not a hardcoded host path or machine identifier, so amos.code-craft.10 does not reach it)
{"reviewer": "peaches", "review_status": "blocking", "head_sha": "9a9cb9c8d373fb701804326c79a4cb9916b336c9", "pr_number": 486, "findings_open": [{"id": "f-a50542383e18a7c2", "rule_id": "amos.path-choice.1", "head": "9a9cb9c8d373fb701804326c79a4cb9916b336c9"}], "fence_schema_version": 3, "failure_sequences": [{"file": "test/helpers/remove-temp-home.js", "line": 104, "rule_id": "amos.path-choice.1", "failure_sequence": "An adapter schedules a home write on a timer and shutdown calls removeTempHome before the timer fires -> the timer writes and recreates the home after cleanup returns"}], "dropped": [{"file": "test/helpers/remove-temp-home.js", "line": 36, "rule_id": "amos.code-craft.10", "message": "The helper hardcodes `/proc/...`, violating the released-tool ban on absolute host paths and coupling cleanup to a Linux-only filesystem.", "reason": "The procfs path /proc/PID/stat is a kernel interface, not a hardcoded host path or machine identifier, so amos.code-craft.10 does not reach it"}], "range": {"basis": "base..head", "base": "06b56605cc7661cf5c694be37f8738922c9f0ada", "head": "9a9cb9c8d373fb701804326c79a4cb9916b336c9"}, "engines": [{"engine": "carrier", "model": "gpt-5.6-sol", "chunks": 1}], "verdict_outcome": "BLOCKED", "verdict_reasons": ["test/helpers/remove-temp-home.js:104:amos.path-choice.1 blocks (computed severity high)"], "evaluator_version": "2", "policy_base_sha": "06b56605cc7661cf5c694be37f8738922c9f0ada", "evaluated_findings": [{"file": "test/helpers/remove-temp-home.js", "line": 73, "rule_id": "amos.path-choice.1", "message": "SIGKILL uses stale PIDs without re-verifying their token; a PID recycled after SIGTERM can make cleanup kill an unrelated same-user process.", "fingerprint": "fp2:77d7b7d3315f8568", "facts": {"attacker_precondition": "none", "category": "concurrency", "cwe": "CWE-367", "durability": "ephemeral", "evidence": "Ownership is checked only while building `pids`; after waiting up to half of `ms`, `signalAll(pids, \"SIGKILL\")` trusts that stale identity.", "impact": {"availability": "partial", "confidentiality": "none", "integrity": "partial"}, "reachable": "yes"}}, {"file": "test/helpers/remove-temp-home.js", "line": 104, "rule_id": "amos.path-choice.1", "message": "The drain sees only currently active FS requests; delayed in-process writes are missed, so cleanup can return before they recreate the removed home.", "fingerprint": "fp2:45158bb2b317e719", "facts": {"attacker_precondition": "none", "category": "concurrency", "durability": "durable", "evidence": "`fsPromiseRequestsInFlight` checks only current `FSReqPromise` resources; it cannot detect a timer or debounce that will start a filesystem request later.", "impact": {"availability": "partial", "confidentiality": "none", "integrity": "partial"}, "reachable": "yes"}}]}

clagentic-builder Bot and others added 2 commits October 23, 2026 14:31
… to the temp-home helper (lr-c3e80a)

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…temp-home removal (lr-c3e80a)

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@clagentic-security

Copy link
Copy Markdown

BOBBIE — clean (0 finding(s), 1 dropped)
verdict: CLEAN
range: since 9a9cb9c
engine: carrier: gpt-5.6-sol

Dropped candidates (1):

  • test/remove-temp-home.test.js:216 [bobbie.bleed.4] The released-tool test hard-codes the absolute host path /proc, violating the repository rule that shipped files contain no absolute host paths. (reason: Upheld drop per HOLDEN ruling: /proc is a kernel interface, not a host path)
{"reviewer": "bobbie", "review_status": "clean", "head_sha": "db4d353700b9f3b966177b571a1e722d75acf48a", "pr_number": 486, "scanners_run": [{"scanner": "gitleaks", "status": "ran"}, {"scanner": "trufflehog", "status": "ran"}, {"scanner": "semgrep", "status": "ran"}, {"scanner": "osv-scanner", "status": "not_applicable", "reason": "no packages found to scan"}], "fence_schema_version": 3, "dropped": [{"file": "test/remove-temp-home.test.js", "line": 216, "rule_id": "bobbie.bleed.4", "message": "The released-tool test hard-codes the absolute host path `/proc`, violating the repository rule that shipped files contain no absolute host paths.", "reason": "Upheld drop per HOLDEN ruling: /proc is a kernel interface, not a host path"}], "range": {"basis": "since", "since": "9a9cb9c8d373fb701804326c79a4cb9916b336c9", "head": "db4d353700b9f3b966177b571a1e722d75acf48a"}, "engines": [{"engine": "carrier", "model": "gpt-5.6-sol", "chunks": 1}], "verdict_outcome": "CLEAN", "verdict_reasons": [], "evaluator_version": "2", "policy_base_sha": "06b56605cc7661cf5c694be37f8738922c9f0ada", "evaluated_findings": []}

@clagentic-reviewer

Copy link
Copy Markdown

PEACHES — clean (3 finding(s), 1 resolved)
verdict: CLEAN
range: since 9a9cb9c
engine: carrier: gpt-5.6-sol

Advisory findings (3):

  • test/remove-temp-home.test.js:197 [amos.path-choice.1] This test unconditionally requires /proc although the helper supports macOS; startTimeOf returns null there, so every non-Linux test run fails.
    computed severity: medium; advisory
  • test/remove-temp-home.test.js:190 [amos.code-craft.4] The PID-reuse test only calls stillOwned; it never drives stopOwned or observes signals, so removing the pre-SIGKILL guard would still pass.
    computed severity: none; advisory
  • test/helpers/remove-temp-home.js:207 [amos.code-craft.12] startTimeOf and stillOwned are /proc implementation helpers exported solely to test internals, exposing them as callable module API.
    computed severity: none; advisory

Resolved (1):

  • test/helpers/remove-temp-home.js:104 [amos.path-choice.1] The drain sees only currently active FS requests; delayed in-process writes are missed, so cleanup can return before they recreate the removed home.
{"reviewer": "peaches", "review_status": "clean", "head_sha": "db4d353700b9f3b966177b571a1e722d75acf48a", "pr_number": 486, "resolved": [{"file": "test/helpers/remove-temp-home.js", "line": 104, "rule_id": "amos.path-choice.1", "message": "The drain sees only currently active FS requests; delayed in-process writes are missed, so cleanup can return before they recreate the removed home.", "by": "reviewer"}], "range": {"basis": "since", "since": "9a9cb9c8d373fb701804326c79a4cb9916b336c9", "head": "db4d353700b9f3b966177b571a1e722d75acf48a"}, "engines": [{"engine": "carrier", "model": "gpt-5.6-sol", "chunks": 1}], "verdict_outcome": "CLEAN", "verdict_reasons": [], "evaluator_version": "2", "policy_base_sha": "06b56605cc7661cf5c694be37f8738922c9f0ada", "evaluated_findings": [{"file": "test/remove-temp-home.test.js", "line": 197, "rule_id": "amos.path-choice.1", "message": "This test unconditionally requires /proc although the helper supports macOS; startTimeOf returns null there, so every non-Linux test run fails.", "fingerprint": "fp2:a199491fbbb44bf6", "facts": {"attacker_precondition": "none", "category": "correctness", "durability": "ephemeral", "evidence": "startTimeOf reads only /proc/<pid>/stat, while ownedProcesses explicitly treats an unavailable /proc as a supported platform case.", "impact": {"availability": "partial", "confidentiality": "none", "integrity": "none"}, "reachable": "yes"}}, {"file": "test/remove-temp-home.test.js", "line": 190, "rule_id": "amos.code-craft.4", "message": "The PID-reuse test only calls stillOwned; it never drives stopOwned or observes signals, so removing the pre-SIGKILL guard would still pass.", "fingerprint": "fp2:b2058467fdd02a1f", "facts": {"attacker_precondition": "none", "category": "test_gap", "durability": "ephemeral", "evidence": "Assertions cover the predicate only; no test invokes the stop/signalling path whose behavior the fix changes.", "impact": {"availability": "none", "confidentiality": "none", "integrity": "none"}, "reachable": "yes"}}, {"file": "test/helpers/remove-temp-home.js", "line": 207, "rule_id": "amos.code-craft.12", "message": "startTimeOf and stillOwned are /proc implementation helpers exported solely to test internals, exposing them as callable module API.", "fingerprint": "fp2:2fd1233552f2359e", "facts": {"attacker_precondition": "none", "category": "api_contract", "durability": "durable", "evidence": "test/remove-temp-home.test.js imports both new exports directly; neither is part of the cleanup helper's caller-facing operation.", "impact": {"availability": "none", "confidentiality": "none", "integrity": "none"}, "reachable": "yes"}}], "fence_schema_version": 3}

…drive the pre-SIGKILL identity guard (lr-c3e80a)

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@clagentic-security

Copy link
Copy Markdown

BOBBIE — clean (0 finding(s))
verdict: CLEAN
range: since db4d353
engine: carrier: gpt-5.6-sol

{"reviewer": "bobbie", "review_status": "clean", "head_sha": "8f8d0d572a88d508a87fcbf2fd118db112d4d4af", "pr_number": 486, "scanners_run": [{"scanner": "gitleaks", "status": "ran"}, {"scanner": "trufflehog", "status": "ran"}, {"scanner": "semgrep", "status": "ran"}, {"scanner": "osv-scanner", "status": "not_applicable", "reason": "no packages found to scan"}], "fence_schema_version": 3, "range": {"basis": "since", "since": "db4d353700b9f3b966177b571a1e722d75acf48a", "head": "8f8d0d572a88d508a87fcbf2fd118db112d4d4af"}, "engines": [{"engine": "carrier", "model": "gpt-5.6-sol", "chunks": 1}], "verdict_outcome": "CLEAN", "verdict_reasons": [], "evaluator_version": "2", "policy_base_sha": "06b56605cc7661cf5c694be37f8738922c9f0ada", "evaluated_findings": []}

@clagentic-reviewer

Copy link
Copy Markdown

PEACHES — clean (0 finding(s), 2 dropped)
verdict: CLEAN
range: since db4d353
engine: carrier: gpt-5.6-sol

Dropped candidates (2):

  • test/helpers/remove-temp-home.js:210 [amos.code-craft.12] Exporting stopOwned exposes the private scan/ownership/signal injection seam to callers; keep the cleanup implementation behind removeTempHome's public surface. (reason: Settled by HOLDEN ruling A3 for lr-c3e80a: a test helper under test/ exporting internals to its own test is not shipped API, so amos.code-craft.12 does not apply.)
  • test/remove-temp-home.test.js:19 [amos.code-craft.10] The new capability probe hardcodes /proc paths in a released-tool path, violating the repository's ban on absolute host paths. (reason: Upheld HOLDEN drop for lr-c3e80a: /proc paths are the kernel interface for the probe, not a host path, so amos.code-craft.10 does not apply.)
{"reviewer": "peaches", "review_status": "clean", "head_sha": "8f8d0d572a88d508a87fcbf2fd118db112d4d4af", "pr_number": 486, "dropped": [{"file": "test/helpers/remove-temp-home.js", "line": 210, "rule_id": "amos.code-craft.12", "message": "Exporting stopOwned exposes the private scan/ownership/signal injection seam to callers; keep the cleanup implementation behind removeTempHome's public surface.", "reason": "Settled by HOLDEN ruling A3 for lr-c3e80a: a test helper under test/ exporting internals to its own test is not shipped API, so amos.code-craft.12 does not apply."}, {"file": "test/remove-temp-home.test.js", "line": 19, "rule_id": "amos.code-craft.10", "message": "The new capability probe hardcodes /proc paths in a released-tool path, violating the repository's ban on absolute host paths.", "reason": "Upheld HOLDEN drop for lr-c3e80a: /proc paths are the kernel interface for the probe, not a host path, so amos.code-craft.10 does not apply."}], "range": {"basis": "since", "since": "db4d353700b9f3b966177b571a1e722d75acf48a", "head": "8f8d0d572a88d508a87fcbf2fd118db112d4d4af"}, "engines": [{"engine": "carrier", "model": "gpt-5.6-sol", "chunks": 1}], "verdict_outcome": "CLEAN", "verdict_reasons": [], "evaluator_version": "2", "policy_base_sha": "06b56605cc7661cf5c694be37f8738922c9f0ada", "evaluated_findings": [], "fence_schema_version": 3}

@clagentic-merger
clagentic-merger Bot merged commit a0a8d02 into main Oct 23, 2026
6 checks passed
@clagentic-merger

Copy link
Copy Markdown
Contributor

Merged via clagentic-loadout v0.2.0

Field Value
Gated HEAD SHA 8f8d0d572a88d508a87fcbf2fd118db112d4d4af
Merged SHA 8f8d0d572a88d508a87fcbf2fd118db112d4d4af
Reviews clagentic-reviewer[bot], clagentic-security[bot]
Verdicts peaches: fence CLEAN, gate CLEAN
bobbie: fence CLEAN, gate CLEAN
CI status combined_state='pending' (ignored: 0 commit statuses, so the check-runs decide) (0 status(es), 6 run(s); source=check-runs)
task_id lr-c3e80a

@clagentic-merger
clagentic-merger Bot deleted the fix/lr-c3e80a-enotempty-teardown branch October 23, 2026 19:08
@clagentic-release-bot

Copy link
Copy Markdown
Contributor

This issue has been resolved in version 1.11.0-beta.2.

To update, run:

npx @clagentic/console@1.11.0-beta.2

@clagentic-release-bot

Copy link
Copy Markdown
Contributor

This issue has been resolved in version 1.11.0.

To update, run:

npx @clagentic/console@1.11.0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

0 participants