Skip to content

Run QA synthesizer through the coding harness - #155

Merged
AbirAbbas merged 4 commits into
Agent-Field:mainfrom
i2020github:fix/qa-synthesizer-harness
Oct 20, 2026
Merged

AbirAbbas merged 4 commits into
Agent-Field:mainfrom
i2020github:fix/qa-synthesizer-harness

Conversation

@i2020github

Copy link
Copy Markdown
Contributor

What changed and why

run_qa_synthesizer was the only reasoner calling router.ai() (the SDK's direct LiteLLM path). Every other agent goes through router.harness(). As a result:

  • the synthesizer needs its own provider API key, even when the harness is authenticated another way. On a claude_code deployment using a Claude subscription (claude setup-token / CLAUDE_CODE_OAUTH_TOKEN, or a logged-in CLI) with no ANTHROPIC_API_KEY, every call fails;
  • with claude_code model ids such as claude-sonnet-5-5, LiteLLM rejects the id outright: Invalid model spec: 'claude-sonnet-5-5'. Must include provider prefix.

Each failure is swallowed, and the coding loop falls back to the tests-passed/review-approved heuristic on every flagged-path iteration. That silently disables the synthesizer, including its stuck detection.

This PR routes the synthesizer through router.harness() like the other agents. It uses the same provider, permission_mode and check_fatal_harness_error, with read/write file tools and the worktree as cwd. It reads result.parsed and keeps the existing heuristic fallback and its error note.

Behaviour changes to orchestration

  • The synthesizer uses the runtime's harness credentials and model ids, as qa_synthesizer_model already implied.
  • A fatal harness error (e.g. credit exhaustion) now propagates from the synthesizer as it does from the other agents.
  • No change to the fallback decisions or to the QASynthesisResult shape.

Test evidence

  • tests/test_qa_synthesizer_direct_schema.py updated to mock router.harness. A synthesizer decision still wins over the heuristic, and an unparseable result still falls back loudly.
  • Full suite on this branch (rebased on main @ 4d1f11c): 1336 passed, 1 skipped.
  • Verified live on a claude_code node with subscription auth. Given passing tests and a blocking review flagging SQL injection, the synthesizer returned fix with concrete remediation steps instead of the heuristic result.

The Go node's port notes (go/docs/port/port-work-breakdown.md, T3.R2) describe the same router.ai-equivalent path, so it likely needs the matching change.

🤖 Generated with Claude Code

run_qa_synthesizer was the only reasoner calling router.ai() (LiteLLM),
so it needed its own provider API key and failed on deployments that
authenticate the harness another way (e.g. a Claude subscription via
claude_code). It now uses router.harness() like every other agent, with
the same runtime credentials and the existing heuristic fallback.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@CLAassistant

CLAassistant commented Oct 10, 2026 •

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

ZeroPoint95
ZeroPoint95 previously approved these changes Oct 14, 2026

@ZeroPoint95 ZeroPoint95 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this @i2020github

AbirAbbas and others added 3 commits October 20, 2026 09:08
Translate legacy Anthropic and OpenAI routing prefixes only for their native harnesses, while preserving OpenCode model IDs. Cover runtime defaults, overrides, judge tools, heuristic failures, and fatal propagation, and make the fallback regression test require a harness call.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Use the recommended MiniMax-M3.1-Flash direct-provider model from the newly merged main branch in the synthesizer routing matrix.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

@AbirAbbas AbirAbbas left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @i2020github, this is a good catch: the synthesizer was the last reasoner still on the direct LiteLLM path.

I pushed three small commits on top before merging:

  • Model IDs. Per-role overrides written in LiteLLM form (anthropic/claude-…, openai/gpt-…) would have gone to the native Claude Code / Codex CLIs unchanged. Those prefixes are now stripped only for the matching native harness. OpenCode keeps its provider-qualified IDs.
  • Tests. The unparseable-result test passed even if the harness was never called, so it now asserts the call. I also added coverage across every runtime default, reviewer-equivalent tools (the synthesizer gets Read/Write/Glob/Grep, a subset of the reviewer's set; Write is what the harness needs for structured output), the fallback decisions, and fatal-error propagation through the coding loop.
  • Main. Merged current main, and the MiniMax routing case now uses the refreshed preset.

make check on 3.12 passes (1,379 passed) and the Go job is green. The Go node keeps its own direct-AI synthesizer, which is unaffected.

@AbirAbbas
AbirAbbas merged commit 536b6a1 into Agent-Field:main Oct 20, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants