Skip to content

feat: review every changed file and stop dropping real findings - #32

Merged
hyochan merged 4 commits into
mainfrom
feat/review-coverage
Nov 18, 2026
Merged

hyochan merged 4 commits into
mainfrom
feat/review-coverage

Conversation

@hyochan

@hyochan hyochan commented Nov 18, 2026

Copy link
Copy Markdown
Member

Why

Gitdog found none of the 74 issues that Opus and Fable later found on PRs 25–30. Every review said "No issues found", and most changed files were "not fully reviewed", even on 3-file follow-ups. A single review preloaded 8 changed files and relied on the model to read the rest. A replay of the reviewer on those commits, with a candidate trace, showed three causes:

  • most files were never read;
  • the model calibrated to the 0.9 confidence bar and proposed almost nothing;
  • verification discarded real defects it was merely unsure about.

What changes

  • Coverage passes. Changed files the main review could not hold are reviewed in fresh requests of their own, implementation files first, each with its own verification. A file is reported as not fully reviewed when:
    • its source can't be read;
    • it is too large for a pass;
    • a candidate on it failed validation, so a defect the model saw can never turn into an approval;
    • the passes stopped at the account limit, a provider or GitHub error, or 4 minutes after the action started. The cutoff keeps even the slowest request inside the 10-minute action limit.
      The finished main review is never discarded because of a pass, and batched reviews keep covering files through their groups.
  • Calibration. The prompt asks for every distinct P1/P2 defect, and the confidence floor drops from 0.9 to 0.8. Suggestions still need 0.95.
  • Verification. A candidate is dropped only on a refutation that quotes its own files beyond its cited evidence. A candidate that is merely "not retained" is kept.
  • Mentions. A finding that mentioned @types/node or a bot command used to fail the whole review. Now a word joiner follows every @name and every & that starts a character reference. I checked with GitHub's markdown API that this survives escapes, backtick runs, fences, HTML and @-style references, and that no one is notified.
  • Recall check. bun run test:e2e:recall replays the reviewer on 4 past commits (e2e/recall/cases.json, 30 known issues) on the development deployment. An optional trace hook shows what each step proposed and what verification discarded.

Measured

On the replay of those 4 commits, before this change Gitdog found 0 of the 30 known issues. After it, all 7 published findings are real known issues, with no false positives:

Case Findings
PR 28 @ eef23f8 bot-prefix heuristic running commands from sentences; failure comments without the old limit-notice checks
PR 26 @ ed6452d full review of a reviewed commit failing after "started" (P1); mid-sentence and HTML-comment mentions; resolve over-reporting
PR 26 @ ab98760 command comments going silent on GitHub read failures
PR 25 @ ab36a20 failed approval dismissals forgotten

Only README.md, which is also a truncated guide, stayed "not fully reviewed" in each case. Runs vary, and over several replays Gitdog surfaced 9 distinct known issues.

Production impact

  • Large single reviews make more requests. Each pass reserves at most 150K request bytes, and passes stop at the account limit.
  • Batched reviews are unchanged.
  • No schema, auth or permission change.

Verification

  • bun run lint, typecheck, build, test (2073 tests), format:check, check:api, typecheck:cli, test:cli
  • Three rounds of independent Opus and Fable review. They found 7 blockers (mention bypasses, approval after a dropped candidate, deadline, quota hold, error handling, entity mentions, read failures), all fixed. The exact head was confirmed clean, including GitHub markdown rendering of every mention and entity form.

🤖 Generated with Claude Code

A single review preloaded eight changed files and relied on the model to
read the rest, so most files went unread and reviews reported no issues.
Files the main review cannot hold are now reviewed in coverage passes of
their own, implementation first, until the account limit or a time
budget. Reviewers are asked for every distinct defect with a confidence
floor of 0.8, verification drops a candidate only on a quoted
refutation, and @-names in findings become code instead of failing the
review. A recall replay of past commits shows 7 of 30 known issues found,
from none before.
Mentions get a word joiner after @, which no markdown escape, backtick
run, fence or HTML can undo. A file whose coverage candidate fails
validation stays not fully reviewed, so a seen defect never becomes an
approval. Provider and billing errors end the optional passes instead of
discarding the finished review, coverage requests start only within five
minutes of the action's start, and batched groups skip coverage passes.
A refutation must quote the candidate's own files beyond its evidence.
Character references that GitHub decodes into a mention (@, @,
@o) get a word joiner after the ampersand. A GitHub read failure in
a coverage pass ends the passes instead of failing the review, the time
budget is rechecked right before each request with a four-minute cutoff,
a candidate without a string path keeps the whole pass unreviewed, and
uncertain candidates are filtered before the cap.
Joiners the model wrote itself are removed before mentions are quieted,
so length limits count real text. The recall runner records a case the
CLI gave up on and keeps the fixture funded for the remaining cases.
@gitdogai

gitdogai Bot commented Nov 18, 2026 •

Copy link
Copy Markdown

Reviewed 0e11cbd · see the review

@hyochan hyochan left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

7 issues found while reviewing this change, all fixed in 0e11cbd

Not a merge approval. Tests were not run. Gitdog

Comment thread convex/pullRequests/review.ts Resolved
Comment thread convex/pullRequests/workflow.ts Resolved
Comment thread convex/pullRequests/workflow.ts Resolved
Comment thread convex/pullRequests/workflow.ts Resolved
Comment thread convex/pullRequests/workflow.ts Resolved
Comment thread convex/pullRequests/workflow.ts Resolved
Comment thread convex/pullRequests/workflow.ts Resolved
@github-actions

Copy link
Copy Markdown

Visit the preview URL for this PR (updated for commit 0e11cbd):

https://codeyou-df251--pr32-feat-review-coverage-19yrziff.web.app

(expires Wed, 25 Nov 2026 01:47:29 GMT)

🔥 via Firebase Hosting GitHub Action 🌎

Sign: 844dad9c62537d028cabc21008e4e2e4031ce4e5

@gitdogai gitdogai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found in 0e11cbd

9 of 14 changed files were not fully reviewed.

Not a merge approval. Tests were not run. Gitdog

@hyochan
hyochan merged commit 904106b into main Nov 18, 2026
9 checks passed
@hyochan
hyochan deleted the feat/review-coverage branch November 18, 2026 01:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant