Integrate PR Review into Dev Lead CLI

[don-petry]

A number of our code review agents support a CLI based invocation which can be performed within a Dev Lead's session to quickly gain valuable feedback prior to pushing commits. This could improve the speed to merge and reduce token usage by avoiding unnecessary loops.

Research, review, evaluate and pilot the use of CLI pr reviews:

1. coderabbit review --agent - https://docs.coderabbit.ai/cli?utm_source=ghpr
2. codeant review - https://docs.codeant.ai/cli/review
3. codex exec review --uncommitted - https://zenn.dev/tdksk/articles/44efbe35d23870?locale=en
   etc...

[github-actions[bot]]

Research Enhancement — CLI Pre-Push Review Integration

Problem Framing (sharpened)

The dev-lead agent's current workflow has a structural gap between self-review (Phase 5, "Rubber Duck Review" — the same Claude model re-reads its own diff) and external review (post-push, via the 9-bot advisory council in reviewer-sources.tsv). That gap is where cost and latency accumulate:

• Fix-reviews cycles are expensive. Each push → bot review → fix-reviews session → re-push cycle costs a full agentic session worth of tokens. Most PRs need 1-3 cycles before bot findings are cleared.
• The advisory council is degraded. Issue Tracking: advisory review bots degraded (billing/trial/usage limits) #1914 tracks 4+ bots hitting billing/trial/rate limits (CodeRabbit "Review limit reached", Codex "usage limits", Qodo "trial ended"). Only 3 of 9 registered sources are in the advisory approval-wait set (Lower the advisory quorum to 5: drop the three reviewers that produce no countable signal (re-file of #1894 AC #3) #1997).
• O(n²) re-review amplification. Issue Merge protections are unsatisfiable at N>1 open PRs: one merge to main forces a full re-review of every other PR — ~154 forced re-reviews in 48h, zero merges #1864 documents that each merge to main forces a full re-review of every open PR — so every avoidable post-push finding multiplies across the queue.
• Phase 5 self-review has inherent blind spots. Claude reviewing Claude's own output is limited by the same model's failure modes — it catches surface issues but misses the cross-perspective insights that external tools provide.

Running CLI reviews pre-push, inside the dev-lead session would catch findings before they become PR comments, reducing cycles, token spend, and the re-review amplification cascade.

Prior Art & Competitive Signal

Tool	CLI Command	Output Format	Pricing	Headless/CI Ready	Limitations
CodeRabbit	cr review --agent	NDJSON (structured, one JSON object/line)	Free: 200 files / 4 reviews/hr; Paid: $24/dev/mo; Usage add-on: $0.25/file	Yes — API key auth, --plain or --agent flags	Cloud-dependent (sends diff to CodeRabbit servers)
CodeAnt	codeant review --headless	JSON to stdout	Free 14-day trial; $24/user/mo	Yes — --fail-on <level> exit codes	Max 10 files / 5,000 lines per file per session
Codex CLI	codex exec review	Text	Codex subscription	No — auto_review only works in interactive mode (tracked: openai/codex#23284)	Not headless-ready; non-starter for agent use today
Claude Code	/code-review	Terminal text / structured findings	Included in Claude Code usage	Yes — runs in any Claude Code session	Same-model self-review (Claude reviewing Claude)
Macroscope	macroscope review	Structured findings	$0.05/KB of diff	Yes — plugins for Claude Code, Codex, Cursor	Newer entrant; review-fix-verify autopilot loop

Industry trend: "Shift-left review" — running AI review before push rather than after PR creation — is a clear 2026 trend. CodeRabbit, CodeAnt, and Macroscope all launched dedicated CLI products in 2026 specifically for this use case. The AI code review market is growing 30-40% annually.

Key finding: CodeRabbit CLI already has a Claude Code native plugin (/coderabbit:coderabbit-review), which means integration into the dev-lead's runtime may be lower-friction than expected. Its --agent NDJSON output is designed exactly for machine consumption by agentic coding tools.

Options Table

Option	Description	New Cost	Cross-Engine Value	Integration Effort	Risk
A. Claude Code /code-review	Add /code-review low to Phase 5 as enhanced self-review	Zero (included)	None — same model	Low (prompt change only)	Limited value over existing Phase 5 rubber duck
B. CodeRabbit CLI	Install cr, run cr review --agent --committed pre-push, parse NDJSON findings	$0.25/file or $24/mo	High — 40+ analysis sources, different AI backbone	Medium (install, auth, NDJSON parser)	Cloud dependency; rate-limit risk on free tier
C. CodeAnt CLI	Install codeant, run codeant review --headless, parse JSON	$24/mo	High — two-phase agentic analysis	Medium (npm install, auth, JSON parser)	10-file cap may be too low for larger changes
D. Hybrid	Phase 5a: /code-review (free, fast). Phase 5b: one external CLI for cross-engine validation (only when Phase 5a finds 0 issues — the "too clean" heuristic)	External CLI cost, but invoked selectively	Adaptive — external only when self-review is suspiciously clean	Medium-High	Complexity; needs a "too clean" threshold
E. engine.sh native	Add a run_cli_review function to engine.sh that abstracts over CLI tools the way run_duck abstracts over cross-engine adversarial review	Depends on tool chosen	Configurable via AI_ENGINES	High (new engine.sh function + model chain)	Scope creep; engine.sh is already complex

Recommendation: Start with Option A, graduate to B.

Option A is zero-cost and zero-risk — it enhances Phase 5 with a structured /code-review pass at effort level low (fast, high-confidence findings only). This is strictly better than the current "re-read your own diff" Phase 5 because /code-review applies a dedicated review recipe rather than an informal self-check.

Option B (CodeRabbit CLI) is the strongest cross-engine candidate because: (a) the org already has a CodeRabbit relationship (coderabbitai is in reviewer-sources.tsv); (b) NDJSON output is purpose-built for machine parsing; (c) the $0.25/file usage add-on model aligns with per-PR billing rather than per-seat; and (d) the existing Claude Code plugin suggests tight integration is on CodeRabbit's roadmap.

Risks & Unknowns

1. Session time budget. CLI review adds wall-clock time to the dev-lead session. At the low effort level, Claude Code /code-review completes in seconds; CodeRabbit CLI typically runs 30-90 seconds. The dev-lead session has a hard timeout (Action budget, slice 4: make a timed-out dev-lead session diagnosable — stream the engine's turn-by-turn record to a file that survives the kill #1996), so the review step must be time-bounded.
2. Authentication in Actions. External CLI tools need API keys stored as Actions secrets. The org already manages GH_TOKEN, ANTHROPIC_API_KEY, and Gemini keys in engine.sh — adding a CODERABBIT_API_KEY follows the same pattern.
3. Finding triage overhead. Not every CLI finding is worth fixing pre-push. The dev-lead needs a severity filter (e.g., fix CRITICAL/MAJOR, skip MINOR/INFO) to avoid rabbit holes. CodeAnt's --fail-on and CodeRabbit's severity levels both support this.
4. Duplication with post-push review. If the same tool reviews both pre-push (CLI) and post-push (bot), it will report the same findings twice. Mitigation: use the CLI for pre-push and let the bot run confirm zero new findings — which is actually the ideal outcome (clean PR on first push).
5. Codex CLI is not ready. The discussion lists codex exec review --uncommitted, but headless auto-review in codex exec is still a feature request (Supports auto-review for CLI exec subcommand openai/codex#23284). Deprioritize this path until it ships.

Phased Validation / Pilot Path

Phase 1 — Baseline measurement (1 week)
Instrument Phase 5 and the fix-reviews flow to measure: (a) how many findings does the current self-review catch? (b) how many fix-reviews cycles does each PR need? (c) what is the per-PR token spend on fix-reviews sessions? The token-metrics.sh infrastructure (#1955) already records per-call data — extend it with a kind:"self_review" record.

Phase 2 — Claude Code /code-review pilot (2 weeks)
Add a # shellcheck disable= style structured /code-review low invocation to the fix-issue prompt's Phase 5. Measure whether the structured review catches findings the informal rubber-duck misses. Compare fix-reviews cycle counts before/after.

Phase 3 — CodeRabbit CLI cross-engine pilot (2 weeks)
Install cr in the dev-lead runner, configure API key auth, add cr review --agent --committed after the /code-review step. Parse NDJSON findings, filter to CRITICAL/MAJOR, and address them before pushing. Measure the incremental finding rate (findings caught by CodeRabbit that Claude Code missed).

Phase 4 — Evaluate and decide (1 week)
Compare: (a) fix-reviews cycles before vs. after, (b) net token spend (CLI review cost vs. saved fix-reviews sessions), (c) time-to-merge, (d) whether the cross-engine step justified its cost. Decide whether to promote to permanent, try Option D (hybrid/selective), or drop.

Concrete Success Metrics

Metric	Baseline (measure in Phase 1)	Target
Fix-reviews cycles per PR	TBD	≥30% reduction
Post-push bot findings addressed by dev-lead	TBD	≥40% reduction
Net token spend per PR (including CLI review cost)	TBD	≤10% increase (ideally net decrease)
Time-to-merge (PR open → merge)	TBD	≥20% reduction
Pre-push CLI review duration	N/A	≤120 seconds

Connection to Existing Work

• Evaluate additional AI code-review services (benchmark + rate-limit + cost analysis) #1348 (Evaluate additional AI code-review services) — this pilot would contribute benchmark data for the CLI variants of tools already evaluated as PR bots.
• RFC: Deterministic pre-agentic review gate to cut token burn (draft-until-green) #743 (Deterministic pre-agentic review gate) — pre-push CLI review is a complementary approach; RFC: Deterministic pre-agentic review gate to cut token burn (draft-until-green) #743 gates on CI, this gates on review findings.
• 💡 LSP-Grounded Finding Verification for PR Review False-Positive Reduction #594 (LSP-Grounded Finding Verification) — LSP tooling could reduce false positives in CLI review findings, making pre-push review more precise.
• run_duck in engine.sh — the existing cross-engine adversarial review pattern; CLI review extends this pattern to the pre-push phase.

[don-petry]

@petry-projects/solution-architect - Input requested. What about our other Code Review tools like Graphite?

[Tomosu-AI]

Adding one option that fits your Phase 3, with a disclosure: this is the Tomosu account.

Two of your problems are related: fix-review cycles run until every bot's findings clear, and the bots producing those findings are hitting rate limits. Tomosu goes after both. Instead of another set of line comments to work through, it gives each PR one scored verdict: a 0-100 Production Reliability Index and Ready / Not Ready to Merge at 70, with before/after code for each finding. The fix-reviews session gets a concrete list of changes and a defined exit (get the PR to Ready) instead of looping until nine sources go quiet. It also doesn't add a tenth bot to the council, because it can run inside the CodeRabbit review you're already piloting.

How: CodeRabbit supports external MCP servers in its PR reviews, and Tomosu has one. One person connects it once under CodeRabbit's Integrations → MCP Servers, and from then on CodeRabbit's reviews include the Tomosu verdict, score breakdown and fixes. No config files or API keys.

Two caveats. First, this applies to CodeRabbit's PR reviews. I haven't seen it documented for coderabbit review --agent, so it won't help Phase 5's pre-push step, and a full Tomosu scan wouldn't fit your 120-second budget anyway. Second, CodeRabbit only enables MCP servers on its paid plans.

Tomosu is free for one repo, so the cheapest test is running Phase 3 twice, CodeRabbit alone and CodeRabbit plus Tomosu, and comparing fix-review cycles per PR. If the verdict doesn't cut cycles, drop it.