💡 Copilot Code Review Resolution Feedback Loop & Signal Quality Standard

[github-actions[bot]]

Summary

Leverage GitHub Copilot's new resolution reasons (Addressed / Won't fix / Incorrect — shipped Aug 27, 2026) to build a continuous feedback loop tracking AI code review signal quality across the fleet. Collect resolution reason data per-repo and per-reviewer (Copilot vs CodeRabbit), compute false positive rates, and surface weekly signal quality trends in the org status digest. This provides the missing input data for the multi-tool review arbitration standard (Discussion #976).

Market Signal

GitHub shipped Copilot code review resolution reasons on Aug 27, 2026, enabling structured feedback on review quality for the first time. Users can now select "Addressed," "Won't fix," or "Incorrect" when resolving a Copilot review comment. Independent benchmarks show a significant quality gap between AI review tools: CodeRabbit scores 51.5% F1 vs Copilot's 44.5% on code review benchmarks, with CodeRabbit at 52.5% recall vs Copilot's 36.7% (morphllm.com, deepsource.com). Both tools consistently miss business logic errors, auth flow flaws, and race conditions. The org uses both tools but has no mechanism to measure which produces actionable signal vs noise for its specific codebase.

User Signal

The org's AGENTS.md mandates: "Address CodeRabbit and Copilot review comments the same way you address human reviewer comments — fix or explicitly justify skipping with a reply." But without signal quality data, agents treat all AI review comments with equal weight, wasting time on false positives and potentially dismissing genuine findings. Discussion #976 (Multi-Tool AI Code Review Arbitration & Signal Routing Standard) proposed intelligent routing between review tools but lacks the input data to implement routing decisions. The pr-auto-review workflow already processes Copilot reviews and could capture resolution reasons with minimal extension.

Technical Opportunity

Copilot review comments are already visible via the GitHub GraphQL API (pullRequest.reviews). Resolution reasons are exposed as a new field on review thread resolution events. The pr-auto-review workflow (.github/scripts/pr-auto-review/) already queries PR review state via the ready-check.sh pure function. Adding resolution reason aggregation requires extending the existing data model with a few additional GraphQL fields. The daily-org-status digest (daily-org-status.yml) already aggregates fleet health data — adding signal quality metrics (% Addressed, % Won't fix, % Incorrect per tool) is architecturally aligned.

Proposed signal quality metrics:

Metric	Definition	Use
addressed_rate	% of comments resolved as "Addressed"	Measures actionable signal
incorrect_rate	% of comments resolved as "Incorrect"	Measures false positive rate
wont_fix_rate	% of comments resolved as "Won't fix"	Measures style-vs-substance balance
tool_signal_ratio	Addressed / (Incorrect + Won't fix)	Comparative tool quality score

Assessment

Dimension	Score	Rationale
Feasibility	high	API exists, pr-auto-review already queries review state, minimal extension
Impact	high	Enables data-driven review tool optimization; feeds into #976 arbitration
Urgency	med	No immediate deadline, but the longer resolution reasons go uncollected, the more training data is lost

Adversarial Review

Strongest objection: The org has only 3-4 active repos and ~30 PRs/month. The sample size for resolution reason analytics is too small to draw statistically meaningful conclusions. This is premature optimization of a non-problem.

Rebuttal: Small sample size is actually an advantage: with ~30 PRs/month, every false positive costs disproportionate agent time (an agent investigating a false positive consumes the same Claude tokens as investigating a real bug). Even directional data — "CodeRabbit's security findings are 80% Addressed vs Copilot's 40%" — enables smarter routing decisions without statistical significance. The cost to collect is near-zero (one additional GraphQL field per review thread resolution); the cost of not collecting is permanent blindness to review quality. The data also compounds: 6 months of collection yields ~180 data points per tool, more than enough for trend analysis.

Suggested Next Step

Add resolution reason tracking to the pr-auto-review workflow's review-state query. Emit a weekly summary in the org status digest showing per-tool signal quality metrics. Define thresholds for "signal quality degradation" alerts (e.g., if incorrect_rate exceeds 30% for any tool over a 4-week window).