fix(review): time out hung git fetch and state-blob requests - #1175
Open
SebTardif wants to merge 1 commit into
Open
fix(review): time out hung git fetch and state-blob requests#1175SebTardif wants to merge 1 commit into
SebTardif wants to merge 1 commit into
Conversation
Review blob hydration's origin fetch used spawnSync without a timeout, so a stalled git remote blocked the review worker. State blob POST retries used fetch without AbortSignal, so a hung Worker blocked each attempt. Bound git fetch to 180s (local git to 60s) and each blob request to 15s while keeping the existing four-attempt backoff. Signed-off-by: Sebastien Tardif <sebtardif@ncf.ca>
Contributor
|
🦞👀 Pull request received. I will update this pull request when review starts. |
clawsweeper
Bot
added
merge-risk: 🚨 availability
Contributor
|
Codex review: needs real behavior proof before merge. Reviewed August 24, 2026, 4:12 AM ET / 08:12 UTC. ClawSweeper reviewWhat this changesThe PR adds deadlines to review-blob Git commands and state-blob Worker POST retries, with focused timeout tests. Regression provenancePossible regression — suspected (reviewed change). No predecessor PR is attributed. Merge readiness⛔ Blocked until real behavior proof from a real setup is added - 5 items remain Keep open: the focused timeout fix remains useful, but the branch conflicts with current main and does not cover the newer exact-checkout Git fetch path, so its central availability claim is incomplete. Priority: P2 Review scores
Verification
Live VerificationCommand: Result: FAIL (partial) — step 2 Assertions:
How this fits togetherReview workers first obtain the exact pull-request commits and hydrate selected Git blobs before inspection, then publish signed state blobs to the canonical Worker/R2 storage path. A hang at either I/O boundary can hold a review worker until an outer timeout. flowchart LR
Review[Review worker] --> Checkout[Exact commit checkout]
Checkout --> GitFetch[Git fetch]
GitFetch --> Hydration[Changed-blob hydration]
Hydration --> Inspection[Restricted review inspection]
Review --> Publisher[Signed state-blob publisher]
Publisher --> Worker[Canonical Worker and R2]
Before merge
Findings
Agent review detailsSecurityNone. Review metrics
Merge-risk optionsMaintainer options:
Technical reviewBest possible solution: Rebase onto current main, preserve the exact-head safeguards from #1170, apply bounded timeouts to both checkout and hydration network fetches, and provide redacted real-boundary recovery proof. Do we have a high-confidence way to reproduce the issue? Yes: current-main source shows both the unsigned deadline-free state-blob fetch and the later exact-checkout Git fetch without a timeout, although this read-only review did not execute a live stall. Is this the best way to solve the issue? No: the state-blob change is directionally correct, but the branch must cover the newer current-main checkout fetch and be rebased before it solves the worker-hang class comprehensively. Full review comments:
Overall correctness: patch is incorrect AGENTS.md: found and applied where relevant. Codex review notes: model internal, reasoning high; reviewed against ce250708c1ea. LabelsLabel justifications:
EvidenceWhat I checked:
Likely related people:
Rank-up movesOptional improvements that raise the rating; they are not merge blockers.
Rating scale
Overall follows the weaker of proof and patch quality. Workflow
HistoryReview history (1 earlier review cycle)
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What Problem This Solves
Fixes an issue where a review worker could hang forever on two I/O paths. Blob hydration runs
git fetch origin --stdinthroughspawnSyncwith notimeout, so a stalled origin never returns. The state-blob client retries a Worker POST up to four times with noAbortSignal, so a hung Worker blocks each attempt without a deadline.Why This Change Was Made
Review blob hydration now gives every
gitspawnSynca deadline: 180s for the network fetch, 60s for localls-tree/rev-list/cat-file. The existinghydrated: falsepath still applies when Node returnsETIMEDOUT. State-blob POSTs now passsignal: AbortSignal.timeout(15000)on each attempt and keep the current four-attempt backoff. Media-proofffmpeg/ffprobestays on its own PR.User Impact
A stalled git origin or Worker no longer pins a review worker until an outer job timeout. Hydration fails closed (
hydrated: false). A hung blob publish fails after four timed attempts instead of blocking the process.OpenClaw Bay Impact
Unaffected. This change is review-worker I/O deadlines only. It does not alter Bay lifecycle, queue, status, telemetry, or dashboard data contracts.
Documentation Lifecycle
No documentation lifecycle changes.
Evidence
Before this change,
src/clawsweeper-review-blobs.tspassedcwd/encoding/maxBuffertogit fetchwith notimeout, andsrc/state-blob-client.tscalledfetchwith nosignal. After the change, a livenodeimport of the compiled helpers prints fetch 180000 / local 60000.sleep 30withspawnSynctimeout 80ms returnsETIMEDOUTin 82ms.AbortSignal.timeout(80)firesTimeoutError. A hung state-blobfetchImplthat waits onsignal.abortis called four times with requestedMs=15000 and rejects withThe operation was aborted.in 1713ms.Related: unbounded
git fetchlanded in #982 (2026-07-31). State-blob POST without a signal landed in #936 (2026-07-29). Repair git helpers already have this bound in #508. Media-proof spawn timeout is a separate surface in #1173. Sibling Worker POSTs already useAbortSignal.timeout(5s to 20s).Real behavior proof
Behavior or issue addressed: Review-worker blob hydration could hang forever on
git fetch origin --stdinbecausespawnSynchad notimeout. State-blob publish could hang forever on each Worker POST becausefetchhad noAbortSignal.Real environment tested: macOS, Node v26.7.0, worktree
/tmp/oc-impl-clawsweeper-io-timeoutat the patched commit.Exact steps or command run after this patch:
The script imported
reviewBlobGitSpawnTimeoutsForTestandpublishStateBlobfrom the compiled helpers, ranspawnSync("sleep", ["30"], { timeout: 80 }), waited onAbortSignal.timeout(80), then calledpublishStateBlobwith afetchImplthat never resolves and only rejects wheninit.signalaborts. The client still requestedAbortSignal.timeout(15000)per attempt; the script shortened that timer to 50ms so the hang path could be observed.Evidence after fix: terminal output from the live
nodecommand above. Compiled review-blob timeouts arefetchMs: 180000andlocalMs: 60000. Hungsleep 30returnsstatus: null,signal: SIGTERM,error.code: ETIMEDOUTin 82ms.AbortSignal.timeout(80)reportsTimeoutError/The operation was aborted due to timeout. The hung Worker POST is attempted four times withrequestedMs=15000and ends withThe operation was aborted.in 1713ms.Observed result after fix: Node kills the hung
sleepinstead of waiting 30 seconds. The state-blob client no longer waits on a fetch that never settles. It aborts each attempt and keeps the existing four-attempt backoff, then throws.What was not tested: A live hung
git fetchagainst a real origin and a live hung Cloudflare Worker. Media-proofffmpeg/ffproberemains on fix(media-proof): time out hung ffmpeg and ffprobe #1173.Keep Allow edits from maintainers enabled.