Skip to content

Ephemeral previews: capture the worktree diff, guard the silent tool-loop exit, withhold web_search on confined runs - #1692

Merged
Evanfeenstra merged 2 commits into
mainfrom
preview-diff-capture-and-loop-guard
Sep 15, 2026
Merged

Evanfeenstra merged 2 commits into
mainfrom
preview-diff-capture-and-loop-guard

Conversation

@Evanfeenstra

@Evanfeenstra Evanfeenstra commented Sep 15, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

propose_code_change preview runs (Hive → /repo/agent with ephemeral: true) were returning completed with the model's narration as the answer and no diff, even though the edit had landed in the worktree. Confirmed from repo2graph.sphinx logs on swarm38, 2026-09-15.

Root cause chain:

  1. The Bifrost gateway rewrites the web_search_20250305 server tool to a newer version on the way to Anthropic (acknowledged by a Bifrost dev on Discord, 2026-07-07: "tool version is the difference").
  2. With the newer web tools the API implicitly offers code_execution; Sonnet answered with a server_tool_use for it (text-editor str_replace).
  3. @ai-sdk/anthropic only tolerates that when it sent web_search_20260209 itself, so ai marked the call invalid and emitted a tool-error without providerExecuted.
  4. The loop's client call/output counts disagreed (1 vs 2), so streamText exited: no error, no stop condition, finish reason tool-calls.
  5. get_context saw a last step with tool calls, skipped the stall nudge, fell to extractFinalAnswer's all-text fallback, and reported success.

Changes

tools.ts — withhold web_search on confined runs. create_pr worktrees and ephemeral previews edit files; they have no use for web search, and with no server tool in the request there is nothing for a gateway to rewrite.

agent.ts / utils.ts — guard the silent exit. unresolvedToolCalls finds calls on the last step that are invalid, unregistered, or without an output. get_context retries once with those calls stripped from the transcript (stripToolCallParts; the API rejects a tool_result for a server tool it never ran) plus a nudge naming the unavailable tool, and throws if the retry ends the same way. Also on this path: a mid-run abort now surfaces as AbortError instead of a completed run, and exhausted continuations set incomplete: { reason } on the ContextResult.

git_pr.ts / index.ts — capture the diff from the worktree. captureWorktreeDiff runs git add -A (new files included; git diff HEAD alone omits them) then git diff --cached, with landChange's caps and gitleaks fail-closed scan. The non-stream terminal result reports it as preview, the counterpart of pr on create_pr runs, with the same shape: { ok: true, diff, filesChanged } or { ok: false, failure, error } where failure is one of no_changes, change_too_large, secrets_detected, git_failed. Callers no longer depend on the model pasting a diff into its final message.

Terminal result additions (ephemeral runs)

preview?:    { ok: true, diff, filesChanged } | { ok: false, failure, error }
incomplete?: { reason: "stall" | "length" | "error" }

Testing

  • src/repo/**/*.test.ts: 583 passed
  • New: unresolved_tools.test.ts (includes the exact production step shape), worktree_diff.test.ts (real git fixtures), and confinement tests asserting web_search is absent on ephemeral and prMode runs
  • tsc --noEmit clean on touched files

Notes

…loop exit, withhold web_search on confined runs

propose_code_change preview runs were coming back "completed" with the
model's narration as the answer and no diff, while the edit had actually
landed in the worktree. Root cause, confirmed from swarm38 logs: the
Bifrost gateway rewrites the web_search_20250305 server tool to a newer
version, the Anthropic API then implicitly offers code_execution, Sonnet
answers with a server_tool_use the SDK never registered, `ai` marks the
call invalid with a tool-error that lacks providerExecuted, the loop's
client call/output counts disagree, and streamText exits with no error and
no stop condition. get_context saw a last step with tool calls, skipped
the stall nudge, fell through to extractFinalAnswer's all-text fallback,
and reported success.

Three changes, each sufficient on its own for the observed failure:

- tools.ts: web_search is withheld on confined runs (create_pr worktrees
  and ephemeral previews). Those runs edit files and have no use for it,
  and with no server tool in the request there is nothing for a gateway
  to rewrite.

- agent.ts / utils.ts: get_context detects a last step with unresolved
  tool calls (unresolvedToolCalls: invalid, unregistered, or without an
  output), retries once with those calls stripped from the transcript
  (stripToolCallParts — the API rejects a tool_result for a server tool
  it never ran) and a nudge naming the unavailable tool, and throws if
  the retry ends the same way. Two related holes closed on the same
  path: a mid-run abort now surfaces as AbortError instead of a completed
  run, and exhausted stall/length continuations set `incomplete` on the
  ContextResult so a caller that needs a deliverable can refuse it.

- git_pr.ts / index.ts: captureWorktreeDiff stages the ephemeral worktree
  (git add -A, so new files are included — `git diff HEAD` alone omits
  them) and returns `git diff --cached`, with landChange's caps and
  gitleaks fail-closed scan. The non-stream terminal result carries it as
  `diff` / `diff_files` / `diff_error`, next to `pr`, so the caller no
  longer depends on the model pasting a diff into its final message.

Tests: unresolved_tools.test.ts (helpers, including the exact production
step shape), worktree_diff.test.ts (real git fixtures), and confinement
tests asserting web_search is absent on ephemeral and prMode runs. The
streaming path is untouched; Hive uses the non-stream path.
Same ok / failure / error shape as the create_pr result, one field instead
of three, so the two run types read alike on the terminal result.
Evanfeenstra added a commit to stakwork/hive that referenced this pull request Sep 15, 2026
… refuse incomplete runs (#5308)

* propose_code_change: take the diff from the swarm's worktree capture, refuse incomplete runs

Preview runs were coming back with narration instead of a diff while the
edit had landed in the swarm's worktree (stakwork/stakgraph#1692 has the
root cause: a gateway tool-version rewrite made the AI SDK's tool loop
exit silently, and the swarm reported success). The swarm now reads the
diff from the ephemeral worktree itself and returns it as `diff`, with
`diff_error` when it looked and found a reason there is none, and
`incomplete` when the run never reached a proper termination.

- Prefer the swarm's `diff` over anything in the model's text. It is
  ground truth and includes new files; the model's pasted diff was a
  transcription that `git diff HEAD` would have left new files out of.
- Refuse `incomplete` runs outright: the text is narration and the
  worktree may hold a half-applied change, so no proposal card.
- Believe `diff_error` (no_changes, change_too_large, secrets_detected)
  over a diff the model pasted, with a specific message for each.
- Keep the regex-on-text extraction as the fallback for swarms that
  predate the capture, and make that fallback stage first
  (`git add -A && git diff --cached`) so it too includes new files.
- Surface the swarm's error text when the run fails instead of a generic
  "timed out or returned an error".

Tests cover each source and refusal, the error passthrough, and the
prompt change. Backward compatible with older swarms.

* Read the swarm's preview diff from `preview`, mirroring `pr`

The swarm reports the ephemeral worktree diff as one `preview` field with
the create_pr result's ok / failure / error shape, instead of three
top-level fields.
@Evanfeenstra
Evanfeenstra merged commit 1e97d95 into main Sep 15, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant