Q2 measurement: commit the acceptance corpus, freeze a holdout, and report Q1/Q2 on every advisory release

[pengfei-threemoonslab]

PM sequencing note — 2026-10-01

P1 measurement prerequisite for #868. Commit the development pins, runner and scoring protocol first. The specified holdout contains >=30 PRs created after 2026-10-01, with pins frozen before #874/#909–#913 merge. As of this review, that future sampling window cannot supply a completed holdout; no freeze is claimed here.

Record the accountable owner, sampling window and actual freeze commit before reader merges. If the window cannot provide the required cases in time, record the shortfall and an explicit methodology/scheduling decision; do not silently use exposed development cases, backdate pins, lower the bar or call a development rerun holdout evidence. Oct 14 remains a checkpoint, not a guarantee that the sample or ten-case outcome exists.

This is independent of #830's released-build outreach gate. The existing counts and acceptance below are unchanged.

---

Evidence base (2026-09-30). Main 6ced6f70 (#864, #865, #875, #876 and #872 merged; #874 open) was run as diff --application --json with a derived scope (no --scope), which is the default a new user gets. The corpus is the #868 set of 49 real PRs that modify an existing OpenAI Agents SDK or Google ADK agent's tools= / sub_agents= / handoffs= in non-test code (46 merged, 3 open, all from the last 90 days). Each run is pinned from merge-base to head. Every row was scored by hand against the source using #868's Q1 and Q2 definitions.

	main a430e81a (2026-09-25)	main 6ced6f70 (2026-09-30)
PRs producing any row	3 / 42	9 / 49
Q2 (a useful, correct and complete answer on an existing agent)	0	1 (jpka/attest#3)
False complete answers (compared while a binding changed unseen)	1 (kkmiecik-coder/CRM#5)	0

40 PRs produce no row. Every one of them is honestly partial or not_established, and names what it could not read. The remaining work is therefore coverage and usefulness, not honesty.

---

Outcome

Every advisory release reports, on a corpus we did not tune against, how many real agent-changing PRs get a correct, complete and useful answer. "Is it useful yet?" then becomes a number we can check, not an impression.

Why now

• Application-agent PR review without prior setup: exact-ref comparison, tool bindings, and 10 verified cases #868 closes at 10 Q2 cases. Today we have 1 (see below). Several reader issues are about to move that number, and we need to see which ones actually do.
• The 49-PR corpus lived in a session scratchpad under /tmp. On 2026-09-30 the OS's daily /tmp cleanup deleted its pin list and the clones' Git metadata, and it had to be re-pinned from PR numbers. A measurement we cannot reproduce is not a measurement.
• We have fixed readers against these same 49 PRs. A count on them alone overstates how the next PR a stranger opens will fare. We need a holdout set that no reader change has seen.

Scope

1. Commit the development corpus in a benchmark directory:
   • one row per PR: repository, number, state, merge-base SHA, head SHA and the selection criteria it met;
   • the runner (full-object clones; diff --application --json with the derived scope; no network during the diff);
   • a per-release ledger with, for each PR: status, row counts by change kind, sides naming a reach or a named unresolved hop, the Q0/Q1/Q2 score and a one-line rationale.
2. Freeze a holdout corpus of at least 30 PRs:
   • chosen with the same criteria from a later window (PRs created after 2026-10-01);
   • pins committed before any of the reader issues below land;
   • scored only at release time and never used to debug a reader.
3. Scoring protocol in the benchmark README:
   • Application-agent PR review without prior setup: exact-ref comparison, tool bindings, and 10 verified cases #868's Q0/Q1/Q2 definitions, verbatim;
   • "any material omission or false row disqualifies the case";
   • how a disagreement is resolved;
   • a worked example per level (Memory Bank: Vertex AI working memory, and fix the import recursion it exposed jpka/attest#3 at Q2, contract form hot fix Tiendat2703/MIS_TALENT#7 at not-Q1).
4. Release step: docs/release-runbook.md § Cutting the release gains one step. Each advisory release's record (docs/changelog/<version>.md) states Q2: n/49 development, m/≥30 holdout and the change since the previous release.

Acceptance criteria

• The committed runner reproduces the 2026-09-30 numbers from the committed pins:
  • 1 compared, 42 partial, 6 not_established;
  • 9 PRs with rows;
  • Q2 = 1.
• Holdout pins are committed before Application diff: resolve tool lists passed into agent-building functions and constructors #874 or the reader issues (Application diff: resolve tools lists built by spread, concatenation, conditionals and comprehensions #909–Tool reach: name effects beyond HTTP — databases, subprocesses, files, cloud SDKs and messaging #913) merge. Record the commit in this issue.
• The runbook step exists, and 1.2.1 or 1.3.0 (whichever ships first) carries the line.
• No public success-rate or market claim is made from either corpus. Both are development evidence, per Application-agent PR review without prior setup: exact-ref comparison, tool bindings, and 10 verified cases #868.

Non-goals

• Automating the hand scoring.
• Growing the corpus beyond what scoring can sustain per release.

Metric this moves

It defines the metric: Q1 and Q2 counts on development and holdout. Related: #868, #830.

[pengfei-threemoonslab]

Holdout prerequisite recorded (acceptance item 2, and the owner's 2026-10-01 decision).

#926 merged as 0180c9bada4ef767c9b90cc8e2277c499cbc7d2e on 2026-10-02, before any of #874 or #909–#913 merged. It commits benchmark/application-q2/:

• the 49-PR development corpus with pinned merge bases and heads, the runner and summarizer, and the 2026-09-30 ledger. The ledger was reproduced from fresh clones: 1 compared, 42 partial, 6 not_established, 9 PRs with rows. Re-read against source, the levels are Q0 37/49, Q1 2/49 and Q2 1/49.
• the holdout's definitions and selection rule, frozen by digest (holdout.json rule_sha256 3de8311d…);
• the repository pool, recorded once from GitHub code search: pool.json, 8,126 repositories, pool_sha256 936568dd….

The window (pull requests created on or after 2026-10-02T00:00:00Z) cannot yet hold 30 qualifying pull requests. The owner chose to freeze the rule and the pool rather than hold the reader work, so the rule decides membership, never someone who has seen the new readers. The pins land in holdout.json once the window fills, with every candidate walked and its reason, so this box stays open until then. The release box stays open until 1.2.1 or 1.3.0 records Q2: n/49 development, m/≥30 holdout.