Skip to content

Make the git clone/fetch/checkout timeouts configurable and state the LFS behavior - #78

Merged
AbirAbbas merged 3 commits into
mainfrom
fix/76-configurable-git-timeouts
Sep 21, 2026
Merged

AbirAbbas merged 3 commits into
mainfrom
fix/76-configurable-git-timeouts

Conversation

@AbirAbbas

Copy link
Copy Markdown
Contributor

The review path's git subprocess timeouts were hardcoded, and the tightest one sat on the step
that does the most work: git checkout -B pr-review FETCH_HEAD, which materializes the working
tree, had a 30-second ceiling. On a repo whose checkout is inherently large — the reporter's is a
Unity project with tens of GB of LFS-tracked assets — every review dies there, with no
documented way to move the wall:

Command ['git', '-C', '/workspaces/<repo>-pr<N>', 'checkout', '-B', 'pr-review', 'FETCH_HEAD']
timed out after 30 seconds

Configurable ceilings

Every git command on the review path now resolves its timeout from the environment, with the
precedence per-operation variable → PR_AF_GIT_TIMEOUT_SECONDS → built-in default. An
unparsable or non-positive value logs one line and falls back to the default rather than
disabling the ceiling.

variable operation default
PR_AF_GIT_CLONE_TIMEOUT_SECONDS git clone 600
PR_AF_GIT_FETCH_TIMEOUT_SECONDS workspace refresh + PR-head fetch 600 (PR-head fetch was 300)
PR_AF_GIT_CHECKOUT_TIMEOUT_SECONDS PR working-tree checkout 600 (was 30)
PR_AF_GIT_DIFF_TIMEOUT_SECONDS git diff Python: none (unchanged); Go: 120
PR_AF_GIT_TIMEOUT_SECONDS any of the above with no own value —

The checkout default moves to 600s to match the clone ceiling: with --no-checkout on the
clone, the checkout is the tree materialization, so it deserves the same order of ceiling. The
Python node and the Go node read the same names with the same precedence and defaults, and a
test in each suite pins that contract independently so the two cannot drift.

Git LFS, stated rather than inferred

pr-af now exports GIT_LFS_SKIP_SMUDGE=1 into the environment of every git command it runs,
unless the operator has set that variable themselves (any value, including 0, is passed
through). So LFS-tracked files land in the review workspace as pointer files and no LFS content
is downloaded during a review — which is enough, since the review reads the diff and the source
text. This was already true by accident (the runtime image installs git with
--no-install-recommends, and the Debian git package neither depends on nor recommends
git-lfs); it is now true by construction, independent of the base image, and documented in the
README, .env.example and the package manifest.

An operator who does want real LFS content installs git-lfs in a derived image, sets
GIT_LFS_SKIP_SMUDGE=0, and raises PR_AF_GIT_CHECKOUT_TIMEOUT_SECONDS to cover the download.

Two things the investigation turned up

The node went unresponsive during repo resolution. _resolve_repo — clone, fetch, checkout —
ran inline inside the async review reasoner, so a multi-minute clone blocked the event loop and
the agent stopped answering /health; that is the Agent pr-af marked inactive after 4 consecutive failures in the issue. It now runs through asyncio.to_thread. Raising the checkout
ceiling to 600s would have made that window six times longer, so it had to move.

A git timeout did not actually stop the work. Measured against a real LFS repo with smudge
enabled, a 30s ceiling took 46.2s to surface, and git-lfs filter-process was still running —
still downloading — after the reasoner had already failed: subprocess.run(timeout=…) kills only
the direct git child, and the grandchild keeps the captured pipes open. Git now runs in its own
process group and the whole group is killed when the ceiling is hit (Go: Setpgid + Cancel,
with WaitDelay as the backstop), so the failure is prompt and nothing is left behind eating
bandwidth and disk in the container.

Validation

Repro first: against a workspace whose checkout goes through a slow smudge filter — the
mechanism git-lfs uses — _checkout_pr_branch on main dies at exactly 30.0s with the
reporter's error string. Then, on this branch:

scenario result
45s checkout, no configuration completes in 45.0s, tree on the PR head (died at 30s on main)
same, PR_AF_GIT_CHECKOUT_TIMEOUT_SECONDS=10 fails at 10.0s — the ceiling is the configured one
Unity-Technologies/BoatAttack PR #216 (real Unity repo, 204 LFS files / 948 MB at the head), pr-af defaults clone + checkout in 1.5s, 42.5 MB worktree, Assets/Gizmos/NavMeshLink Icon.png is a 130-byte pointer — no LFS content fetched
same repo, git-lfs installed and GIT_LFS_SKIP_SMUDGE=0, ceiling 30s the reporter's failure, surfacing at 31.2s (46.2s before the process-group fix) with no git-lfs process left behind
same, PR_AF_GIT_CHECKOUT_TIMEOUT_SECONDS=900 checkout completes in 135.1s, 990.8 MB worktree with real binaries — the reporter's scenario, with a raised ceiling

Gates (the CI jobs, run locally): ruff check src/ scripts/ clean; pytest 125 passed (the one
failure, test_hitl_review_gate.py::test_create_request_fails_fast_when_hax_wedges, fails the
same way on origin/main under Python 3.10 and passes on CI's 3.11); go build ./... && go vet ./... && go test ./... clean and gofmt -l empty. The docker-build job was not run locally —
no Dockerfile changed.

An adversarial review pass of the first draft is what turned up the compose passthrough, the
af install manifest, and the stale-lock retry failure; all three are fixed here.

Fixes #76

🤖 Generated with Claude Code

AbirAbbas and others added 3 commits September 21, 2026 13:39
The Python node's git subprocesses had hardcoded timeouts, and the
tightest one sat on the step that does the most work: `git checkout -B
pr-review FETCH_HEAD` materializes the whole working tree with a 30
second ceiling. A repo whose checkout is inherently large — a Unity
project with tens of GB of LFS-tracked assets — cannot finish in 30s, so
every review of it dies there with no way to move the wall.

Each git operation now resolves its ceiling from the environment
(per-operation variable, then the PR_AF_GIT_TIMEOUT_SECONDS umbrella,
then the built-in default; an unparsable or non-positive value logs one
line and falls back). The checkout default becomes 600s, matching the
clone ceiling it shares work with, and the PR-head fetch joins it.

Two things follow from raising those ceilings:

- Repo resolution moves off the event loop. `_resolve_repo` ran inline
  inside the async `review` reasoner, so a multi-minute clone stopped the
  node answering /health and the control plane marked the agent inactive.

- A timeout now actually stops the work. `subprocess.run(timeout=…)`
  kills only the direct git child, so a `git-lfs filter-process`
  grandchild kept the captured pipes open (a 30s ceiling measured 46.2s
  against a real LFS repo) and went on downloading after the reasoner had
  given up. Every git command runs in its own session and the whole
  process group is killed, with the index/shallow lock files the dead
  command left behind removed so the next review of that workspace is not
  blocked by a stale lock.

git-lfs smudge is now skipped explicitly: pr-af exports
GIT_LFS_SKIP_SMUDGE=1 to every git command unless the operator set that
variable themselves, so LFS-tracked files land as pointer files
regardless of what the image happens to ship.

Fixes #76

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The Go node ported app.py's literals, including the 30 second checkout
ceiling. It now reads the same environment variables with the same
precedence and the same defaults as the Python node — a test in each
suite pins the names and defaults independently, so the two cannot drift
— and exports GIT_LFS_SKIP_SMUDGE=1 unless the operator set it.

Timed-out git commands are killed as a process group (Setpgid plus a
Cancel that signals the group, behind a unix build tag, with WaitDelay
as the backstop) and the index/shallow lock files left behind are
removed, so a filter helper cannot hold the pipes open past the ceiling
or block the next review of that workspace.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The new variables are only useful if they reach the node. Both compose
files listed their environment explicitly, so nothing set in the .env
next to them was forwarded; they now pass the git timeout variables,
GIT_LFS_SKIP_SMUDGE and the two budget variables through, and run with
`init: true` so a killed git helper reparented to PID 1 is reaped. Both
package manifests declare the same variables — including go/, which is
the manifest `af install` actually uses — and deliberately give the
per-operation timeouts no manifest default, since a materialized 600
would shadow PR_AF_GIT_TIMEOUT_SECONDS.

The README now states the LFS behavior instead of leaving it to be
inferred from the Dockerfile: the workspace is a shallow clone, pr-af
sets GIT_LFS_SKIP_SMUDGE=1 for every git command, LFS-tracked files are
checked out as pointer files (which is all a review needs, since it
reads the diff and the source text), and the runtime image ships git
without git-lfs anyway. Wanting real LFS content means installing
git-lfs in a derived image, setting GIT_LFS_SKIP_SMUDGE=0, and raising
PR_AF_GIT_CHECKOUT_TIMEOUT_SECONDS to cover the download.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@AbirAbbas
AbirAbbas merged commit 421fbd2 into main Sep 21, 2026
4 checks passed
@AbirAbbas
AbirAbbas deleted the fix/76-configurable-git-timeouts branch September 21, 2026 20:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

git checkout hardcoded 30s timeout fails on large (Unity) repos; LFS handling unclear

1 participant