Repository navigation
Run QA synthesizer through the coding harness - #155
Merged
AbirAbbas merged 4 commits intoOct 20, 2026
Merged
Conversation
run_qa_synthesizer was the only reasoner calling router.ai() (LiteLLM), so it needed its own provider API key and failed on deployments that authenticate the harness another way (e.g. a Claude subscription via claude_code). It now uses router.harness() like every other agent, with the same runtime credentials and the existing heuristic fallback. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ZeroPoint95
previously approved these changes
Oct 14, 2026
ZeroPoint95
left a comment
Contributor
There was a problem hiding this comment.
Thanks for this @i2020github
Translate legacy Anthropic and OpenAI routing prefixes only for their native harnesses, while preserving OpenCode model IDs. Cover runtime defaults, overrides, judge tools, heuristic failures, and fatal propagation, and make the fallback regression test require a harness call. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Use the recommended MiniMax-M3.1-Flash direct-provider model from the newly merged main branch in the synthesizer routing matrix. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
AbirAbbas
approved these changes
Oct 20, 2026
AbirAbbas
left a comment
Collaborator
There was a problem hiding this comment.
Thanks @i2020github, this is a good catch: the synthesizer was the last reasoner still on the direct LiteLLM path.
I pushed three small commits on top before merging:
- Model IDs. Per-role overrides written in LiteLLM form (
anthropic/claude-…,openai/gpt-…) would have gone to the native Claude Code / Codex CLIs unchanged. Those prefixes are now stripped only for the matching native harness. OpenCode keeps its provider-qualified IDs. - Tests. The unparseable-result test passed even if the harness was never called, so it now asserts the call. I also added coverage across every runtime default, reviewer-equivalent tools (the synthesizer gets Read/Write/Glob/Grep, a subset of the reviewer's set; Write is what the harness needs for structured output), the fallback decisions, and fatal-error propagation through the coding loop.
- Main. Merged current main, and the MiniMax routing case now uses the refreshed preset.
make check on 3.12 passes (1,379 passed) and the Go job is green. The Go node keeps its own direct-AI synthesizer, which is unaffected.
This was referenced Oct 20, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed and why
run_qa_synthesizerwas the only reasoner callingrouter.ai()(the SDK's direct LiteLLM path). Every other agent goes throughrouter.harness(). As a result:claude_codedeployment using a Claude subscription (claude setup-token/CLAUDE_CODE_OAUTH_TOKEN, or a logged-in CLI) with noANTHROPIC_API_KEY, every call fails;claude_codemodel ids such asclaude-sonnet-5-5, LiteLLM rejects the id outright:Invalid model spec: 'claude-sonnet-5-5'. Must include provider prefix.Each failure is swallowed, and the coding loop falls back to the tests-passed/review-approved heuristic on every flagged-path iteration. That silently disables the synthesizer, including its
stuckdetection.This PR routes the synthesizer through
router.harness()like the other agents. It uses the sameprovider,permission_modeandcheck_fatal_harness_error, with read/write file tools and the worktree ascwd. It readsresult.parsedand keeps the existing heuristic fallback and its error note.Behaviour changes to orchestration
qa_synthesizer_modelalready implied.QASynthesisResultshape.Test evidence
tests/test_qa_synthesizer_direct_schema.pyupdated to mockrouter.harness. A synthesizer decision still wins over the heuristic, and an unparseable result still falls back loudly.main@ 4d1f11c):1336 passed, 1 skipped.claude_codenode with subscription auth. Given passing tests and a blocking review flagging SQL injection, the synthesizer returnedfixwith concrete remediation steps instead of the heuristic result.The Go node's port notes (
go/docs/port/port-work-breakdown.md, T3.R2) describe the samerouter.ai-equivalent path, so it likely needs the matching change.🤖 Generated with Claude Code