Skip to content

feat: check a refutation again before dropping a finding - #38

Merged
hyochan merged 3 commits into
mainfrom
feat/refutation-appeal
Nov 19, 2026
Merged

hyochan merged 3 commits into
mainfrom
feat/refutation-appeal

Conversation

@hyochan

@hyochan hyochan commented Nov 18, 2026 •

Copy link
Copy Markdown
Member

Summary

  • A refuted finding gets a second check before it is dropped.
    • Verification drops a candidate when it quotes source that refutes it.
    • A refuted candidate now goes to one more verification request, which sees the candidate and the first refutation's quote. It is dropped only if that check refutes it again with a valid quote.
    • If the check cannot run (time cutoff, outage, unusable answer), the first decision stands, so nothing gets worse than today.
    • In the recall replay, verification dropped a real P2 ("fork authors could run full reviews on the owner's tokens") on a single quoted refutation.
  • The recall replay reports a failure's sanitized details (stage, request, HTTP status), as production records them, instead of only "Review request failed".

Measurement

Measured locally with GPT-6.2 Sol, the model production runs on while Anthropic credit is out. The four recall cases (PR 25, both PR 26 commits, PR 28) were replayed with main's code and with this branch.

  • The second check never ran. In all 8 runs, Sol's verification refuted nothing, so this change does not alter Sol reviews. The refutations it targets came from Claude Opus, which cannot be replayed until Anthropic credit returns.
  • Sol reviews on current main find many issues. The replays produced 8–11 findings per case, including several known issues (for PR 26 ed6452d: resolve misreporting, commands in code blocks, full review failing after "started", non-atomic command recording, missing permission checks). Precision was not scored.
  • One main-code run failed: "Finding contains unsupported evidence or mentions" on PR 26 ed6452d. The same case succeeded on this branch, so it is nondeterministic.
  • Fixed from Gitdog's review of this PR: a second check whose refutations is missing or not an array now leaves the first decision in place, instead of restoring every dropped finding.

Verification

  • bun run lint, bun run typecheck, bunx prettier --check .
  • bun run test: 2113 passed, including the second check upholding a refutation, overturning one, and keeping the first decision when it cannot run

🤖 Generated with Claude Code

@hyochan
hyochan force-pushed the feat/refutation-appeal branch from 115cc9a to 141e23c Compare November 18, 2026 14:23
@hyochan

hyochan commented Nov 18, 2026

Copy link
Copy Markdown
Member Author

@gitdogai review

@gitdogai

gitdogai Bot commented Nov 18, 2026

Copy link
Copy Markdown

Review not started

Gitdog reviews open pull requests that are ready for review.

…refutation-appeal

# Conflicts:
#	convex/pullRequests/workflow.ts
@hyochan
hyochan marked this pull request as ready for review November 19, 2026 00:01
@gitdogai

gitdogai Bot commented Nov 19, 2026 •

Copy link
Copy Markdown

Review cancelled for 97c6970

The pull request changed or was closed before the review finished.

Comment thread convex/pullRequests/workflow.ts Resolved
@gitdogai

gitdogai Bot commented Nov 19, 2026 •

Copy link
Copy Markdown

Review cancelled for 4508e55

The pull request changed or was closed before the review finished.

@hyochan
hyochan merged commit 2f42161 into main Nov 19, 2026
7 checks passed
@hyochan
hyochan deleted the feat/refutation-appeal branch November 19, 2026 00:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant