Commit the application review benchmark and freeze the holdout rule (#908)

[pengfei-threemoonslab]

Why

#868 closes at 10 Q2 cases, and the reader issues (#874, #909–#913) are about to move that count. The 49-PR development corpus lived in a session scratchpad, and the OS's /tmp cleanup deleted its pins on 2026-09-30, so they had to be re-pinned from PR numbers. A count that cannot be reproduced is not a measurement. A count taken only on the PRs the readers were fixed against also overstates how the next stranger's PR will fare, which is why #908 asks for a holdout.

What this commits: benchmark/application-q2/

File	What it is
development.json	The 49 PRs: repository, number, URL, title, state, created date, merge base, head, the frameworks the engine read for each (not what the repository imports: two members import a repository-local agents package) and the search that admitted them. 46 merged, 3 open.
run.py	One full-object clone per repository. A member's pins are kept under refs/q2/<slug>/{base,head} so garbage collection cannot drop them. Completeness is checked over every object of both pins, with lazy fetching disabled. --fetch is the only network step, with explicit refspecs, timeouts and no credential prompts. Then diff --application --base <merge_base> --head <head> --json runs with no --scope, as for a new user. Each member's outcome (ok, refused, timeout, unavailable) and reason go to runs.json. Earlier outputs and runs.json are removed before anything else, so a failed rerun never reports an earlier build's answer. It exits non-zero unless every member answered.
summarize.py	Mechanical counts from a run directory: statuses, rows by kind, and sides naming an outbound call or an unfollowed hop. It joins the hand scores and never derives a level. A score names the answer_id it judged: the digest of the answer without the engine's own version, Python, platform and build. A byte-identical answer from another machine or release keeps its score; a moved answer is listed as stale and not counted.
results/2026-09-30-6ced6f70.{md,scores.json}	The 2026-09-30 ledger: the counts, plus one row per PR with status, row counts, Q0/Q1/Q2 and a rationale.
pool.json, pool.py	The holdout's repository pool, recorded once: 8,126 repositories from 160 GitHub code-search requests (2 queries × 8 file-size shards × 10 pages, since search serves at most 1,000 results a query). It is a sample, never regenerated.
holdout.json	The holdout's record: the sha256 of the frozen definitions and rule, and of pool.json. No pins yet; see below.
README.md	The scoring protocol (#868's levels verbatim, disagreement resolution, worked examples at Q2 and not-Q1), the definitions (binding change, a row appearing, and therefore that a relevant PR with no row fails Q1), the holdout rule, and the release step.

docs/release-runbook.md records Q2: n/49 development, m/≥30 holdout in step 1 of § Cutting the release, and in the advisory list. Every release so far went through the advisory list, which joins the main path at step 4, after step 1. A test fails any docs/changelog/<version>.md after 1.2.0 that lacks the line.

Acceptance (#908)

• The committed runner reproduces the 2026-09-30 numbers from the committed pins. On build 6ced6f70, from fresh clones made with --fetch, all 49 answered ok. Statuses, rows and comparison_ids are identical to the 2026-09-30 outputs: 1 compared, 42 partial, 6 not_established; 9 PRs with rows; 60 changed, 8 added, 230 not established; Q2 = 1 (Memory Bank: Vertex AI working memory, and fix the import recursion it exposed jpka/attest#3).
• Holdout pins committed before Application diff: resolve tool lists passed into agent-building functions and constructors #874 / Application diff: resolve tools lists built by spread, concatenation, conditionals and comprehensions #909–Tool reach: name effects beyond HTTP — databases, subprocesses, files, cloud SDKs and messaging #913 merge. Not possible yet: the window (PRs created on or after 2026-10-02) cannot supply 30 cases for weeks. On 2026-10-01 the owner chose to freeze what decides membership before any reader PR merges, instead of holding the reader work. That is the definitions, the rule (window, pool, candidates, relevance, stop at 30, pins) and the pool itself, each held to its digest by the tests. Nobody who has seen the new readers' behaviour picks a member: the rule does, from a pool recorded in advance, and every candidate walked is recorded with its reason. The box stays open until the pins land. I'll record this PR's merge commit on Q2 measurement: commit the acceptance corpus, freeze a holdout, and report Q1/Q2 on every advisory release #908.
• The runbook step exists, and 1.2.1 or 1.3.0 carries the line. The step exists on both release paths and the test above enforces the line. The release itself is still to come.
• No public success-rate or market claim is made from either corpus. The README, the ledger and the CHANGELOG say both are development evidence.

What the review changed

• Scores. Re-read against source: Q0 37/49, Q1 2/49, Q2 1/49.
  • Twelve members are not Q0. Eight change agents of other frameworks: LiveKit Agents (2), ZeroRuntime (2), rustic-ai (1), a course's own Agent class (2) and a hand-written ReAct loop (1). Four change no application agent's wiring: a model-string update, library docstrings, and two agent-framework internals.
  • MIS_TALENT#6 is Q1. Finance_Agent's construction and every tool it binds are unchanged, so the spread list the reader cannot resolve hides no row.
  • All 49 stay in the corpus, so the published counts reproduce.
• Runner. Two P2 findings in the inline review (a failed rerun reusing an earlier build's answer; the exit status ignoring failures) were fixed in 1a0acb7 on the original runner. aa11aea replaces that runner with the per-repository design above, after a second review found three more defects: blobless clones were never fully hydrated, a bare console-script --engine failed, and git calls had no timeout. The eight regression cases from 1a0acb7 are ported to it, with two more for an interrupted run and a git timeout.
• The holdout rule now reads relevance from the source at the pins, before the engine runs, rather than "from the diff alone", which cannot show which agent binds an edited list. It excludes the frameworks' own repositories, forks and copies by their package files (src/agents/function_schema.py, src/google/adk/runners.py) rather than by judgement. And it freezes the definitions it uses together with the steps.

Choices for you to check

• The holdout excludes repositories already in the development corpus (15 of the 31 are in the pool), because readers were tuned against their constructions. This is my addition to "same criteria"; say if you'd rather drop it.
• The pool was recorded on 2026-10-02 at 20:15 UTC, about 20 hours into the window. It is fixed by digest before any reader PR merges. The queries were set in pool.py before it ran, and no window PR was looked at.
• One corpus repository was renamed (contributory/starchatter-telegram → starfall-orb/starchatter-telegram). It is recorded under its canonical name, with renamed_from.

Validation

• tests/test_application_q2_benchmark.py (20 tests):
  • The corpus is pinned and unique.
  • Each ledger's table and counts are recomputed from its scores; Q2 ⇒ Q1 ⇒ Q0, and Q1 needs a stated change.
  • The 2026-09-30 counts reproduce.
  • The summarizer is mechanical, and a score survives another machine or release but not a moved answer.
  • The README quotes Application-agent PR review without prior setup: exact-ref comparison, tool bindings, and 10 verified cases #868 verbatim.
  • The definitions-and-rule digest and the pool digest hold, and a pinned holdout follows its rule.
  • The runbook and the release records carry the line.
  • Ten runner cases: reruns, refusals, timeouts, missing commits, clone failures, mixed members, an interrupted run and a git timeout.
• summarize.py over the reproduced run with the committed scores: 49 scored, 0 stale, Q0 37, Q1 2, Q2 1.
• The full local suite (-m "not perf", plus the separately run CI files) passed on the review-fix tree; the benchmark, privacy and docs-link tests pass on the final one. ruff check . is clean.

Refs #868, #830. The holdout and release boxes stay open, so this does not close #908.

🤖 Generated with Claude Code

[pengfei-threemoonslab]

Reviewed commit fba503421b30800116dce84c8ac7f54c46c1b1d2.

The committed pins, explicit hand scoring, retained non-Q0 development cases, and separation of development evidence from a future holdout make the measurement easier to audit. I found two runner defects that can misrepresent a failed measurement, described inline; I recommend fixing them before this runner becomes the release-runbook path.

Validation: the benchmark, documentation-link, privacy and public-surface suites passed (607 tests), and Ruff passed on the new Python files. I exercised the runner's timeout, missing-commit and nonzero-engine paths with a synthetic one-member corpus and controlled subprocess results. A timeout retained the prior compared JSON and returned exit 0; the summarizer counted that old row as current. An engine exit 2 also left the runner's exit at 0. These cases are not covered by the new tests.

The head's GitHub CI and both Agents Shipgate workflows are successful. Local advisory verification of the pinned base/head returned control_state=complete, decision=passed. I did not independently rerun all 49 external repositories or rescore their source; the published Q2 count remains the committed ledger's claim. The holdout has no pins yet, as explicitly recorded in this PR.

Posted by Codex at the repository owner's request.