CodeGovindz · Oct 26, 2026 · Oct 26, 2026 · Oct 26, 2026
diff --git a/docs/PLAN.md b/docs/PLAN.md
@@ -106,6 +106,33 @@ Other env vars: `AWS_REGION=us-east-1`; `AWS_ACCESS_KEY_ID`/`AWS_SECRET_ACCESS_K
 - Concurrency limit for verification calls: 4.
 - Record which model produced each `ClaimResult` (`modelUsed`); `Report.modelsUsed` aggregates for the footer, appending e.g. "(2 via gpt-oss-120b)" when fallback fired.
 
+**Discovery for P2 (recorded during P1):** `next.config.ts` sets `cacheComponents: true`. Any page component that reads `searchParams` (the `?pr=` share link on `src/app/page.tsx`) must be wrapped in a `<Suspense>` boundary or `next build` fails — with `cacheComponents`, `searchParams` access must be deferred to a client/suspended piece of the tree.
+
+**P1 real-PR validation (2026-10-26, all < 60 s target):**
+
+| PR | Kind | Grade | Runtime |
+|---|---|---|---|
+| [CapSoftware/Cap#3412](https://github.com/CapSoftware/Cap/pull/3412) | failing CI (Format (Biome), Vercel) + agent-authored claims | **FAIL · trust 67%** (8/12 verified, 2 contradicted) | ~27 s |
+| [facebookincubator/dispenso#129](https://github.com/facebookincubator/dispenso/pull/129) | description claims test results, adds NO test files; CI still running | **MIXED · trust 42%** (5/12 verified) | ~42 s |
+| [github/docs#46488](https://github.com/github/docs/pull/46488) | docs-only typo fix, merged | **MIXED · trust 67%** (2/3 verified) | ~11 s |
+
+Fixtures pinned under `fixtures/real-*.json` (bundle at head SHA). Highlights: "Validation: all 41 SDK tests pass" in the Cap PR was CONTRADICTED by the failing Biome/Vercel checks; dispenso's "1287/1287 pass" claims became UNVERIFIED via the CI-pending rule (correction 5) because many check runs were queued/in-progress at head SHA.
+
+**P1 honest-observations (recorded for future phases):**
+- FAST extraction reliability (qwen3-coder-next): solid on the three recorded runs (12/12/3 claims with verbatim quotes). One observation run (hmcts/pcs-api#3121, unfilled Jira-template body) returned `{"claims":[]}` and missed the single subtle claim ("Minor label change ... to 'Submit application'"). Decision: KEEP qwen as FAST; monitor.
+- Extraction can turn a PR's problem-statement sentence ("the SDK stays in its recording phase...") into a claim about the new behavior; the Cap run judged that sentence CONTRADICTED. Semantically debatable — reviewers should read CONTRADICTED verdicts on description prose carefully.
+- Verdicts on open PRs drift while CI is pending: the dispenso run's trust moved 50%→42% between two runs ~15 min apart as check runs completed. Fixtures pin the bundle snapshot; re-running against a live open PR re-fetches current CI state.
+- A long audit-style PR body (dispenso) hit the 12-claim cap and made the FAST call take ~23 s of the 42 s total — acceptable, but P2's UI should stream progress so the wait is legible.
+
+**P1 precision review (both Cap CONTRADICTED verdicts above were false):**
+1. Problem statements are not claims. Extraction now only emits statements about the PR's outcome; a described bug becomes one `bug_fixed` outcome claim. Claims carry `scope: ci | local | unspecified`.
+2. Non-test CI failures no longer contradict test claims. Every check is classified by kind (`test | lint | typecheck | build | deploy | bot | other`); `tests_pass` is CONTRADICTED only by a failed check of the kind the claim talks about, and a VERIFIED one gets the passing checks of that kind attached as evidence.
+3. Skipped/neutral/stale checks are ignored instead of making CI look "pending" forever.
+4. `sanitizePrBody()` strips bot sections (e.g. Greptile's `<!-- greptile_comment -->` block), HTML comments and tags before extraction; the quote guard runs on the sanitized text.
+5. CONTRADICTED below confidence 0.75 is downgraded to UNVERIFIED.
+6. `npm run eval` runs every `fixtures/real-*.json` through the live models against `fixtures/expectations.json`. After the fixes: docs#46488 PASS, Cap#3412 MIXED (was a false FAIL), dispenso#129 MIXED, no CONTRADICTED anywhere. Extraction is not deterministic on Bedrock even at temperature 0, so expectations on single claims are "if extracted", not "required".
+- CLI quirk (Windows PowerShell): `npm run verify -- <url> --fresh --dump-fixture x` drops the unknown flags in npm's argument forwarding (the URL still works, and the flag VALUES become positionals!). `npx tsx scripts/verify.ts <url> --json --fresh --dump-fixture x` works everywhere — document it in the README (P4).
+
 ## 6. Signals (deterministic, computed once per PR, no LLM)
 
 - **testFilesChanged:** test files added/modified; count of added test cases (regexes: `it(`, `test(`, `describe(`, `def test_`, `func Test`, `@Test`).

diff --git a/eslint.config.mjs b/eslint.config.mjs
@@ -5,6 +5,15 @@ import nextTs from "eslint-config-next/typescript";
 const eslintConfig = defineConfig([
   ...nextVitals,
   ...nextTs,
+  {
+    rules: {
+      // Test stubs intentionally declare unused (command, options) parameters.
+      "@typescript-eslint/no-unused-vars": [
+        "warn",
+        { argsIgnorePattern: "^_", varsIgnorePattern: "^_" },
+      ],
+    },
+  },
   // Override default ignores of eslint-config-next.
   globalIgnores([
     // Default ignores of eslint-config-next:

diff --git a/fixtures/expectations.json b/fixtures/expectations.json
@@ -0,0 +1,62 @@
+{
+  "$comment": "Regression-eval expectations for the recorded real-PR fixtures (scripts/eval.ts). claimVerdict: every claim whose text contains textIncludes (case-insensitive) must have the verdict; required=true means the claim must exist. noClaimVerdict: no matching claim may have that verdict (empty textIncludes = all claims). contradictedMustCiteKind: every CONTRADICTED claim of the category must cite a check run classified as that kind. These encode the P1 precision review's invariants: pre-fix problem prose must not be contradicted, tests-pass claims are not refuted by non-test CI failures, and no test-kind evidence means UNVERIFIED.",
+  "real-docs-only": {
+    "url": "https://github.com/github/docs/pull/46488",
+    "expectations": [
+      {
+        "$comment": "The typo-fix outcome claim (the section title correction) must be VERIFIED against the diff hunk.",
+        "type": "claimVerdict",
+        "textIncludes": "editing generated files",
+        "verdict": "VERIFIED",
+        "required": true
+      },
+      { "type": "noClaimVerdict", "textIncludes": "", "verdict": "CONTRADICTED" },
+      { "type": "gradeIsNot", "grade": "FAIL" }
+    ]
+  },
+  "real-failing-ci": {
+    "url": "https://github.com/CapSoftware/Cap/pull/3412",
+    "expectations": [
+      {
+        "$comment": "The pre-fix problem statement ('the SDK stays in its recording/paused phase...') must never be judged CONTRADICTED again.",
+        "type": "noClaimVerdict",
+        "textIncludes": "recording/paused phase",
+        "verdict": "CONTRADICTED"
+      },
+      {
+        "$comment": "'All 41 SDK tests pass' must be UNVERIFIED: the only test-kind checks were skipped in CI; the failing checks (Format (Biome), Vercel) are lint/deploy, not tests.",
+        "type": "claimVerdict",
+        "textIncludes": "tests pass",
+        "verdict": "UNVERIFIED",
+        "required": true
+      },
+      {
+        "$comment": "A tests_pass contradiction is legitimate only if it cites a test-kind check.",
+        "type": "contradictedMustCiteKind",
+        "claimCategory": "tests_pass",
+        "kind": "test"
+      },
+      { "type": "gradeIsNot", "grade": "FAIL" }
+    ]
+  },
+  "real-tests-claim-no-tests": {
+    "url": "https://github.com/facebookincubator/dispenso/pull/129",
+    "expectations": [
+      {
+        "$comment": "Nothing in this PR's claims can be contradicted: local test-run claims are UNVERIFIED, not CONTRADICTED.",
+        "type": "noClaimVerdict",
+        "textIncludes": "",
+        "verdict": "CONTRADICTED"
+      },
+      {
+        "$comment": "'ctest -LE flaky: 1287/1287 pass' is a local-machine claim (CI has no matching executed test-kind check) -> UNVERIFIED. Not required: extraction picks up to 12 claims and is not deterministic on Bedrock even at temperature 0.",
+        "type": "claimVerdict",
+        "textIncludes": "1287/1287",
+        "verdict": "UNVERIFIED",
+        "required": false
+      },
+      { "type": "claimVerdict", "textIncludes": "0/60", "verdict": "UNVERIFIED", "required": false },
+      { "type": "gradeIsNot", "grade": "FAIL" }
+    ]
+  }
+}