Skip to content

docs(bench): publish every claim as a 12-field evidence card (#951) - #1086

Merged
Birfy merged 1 commit into
mainfrom
docs/951-evidence-cards
Nov 15, 2026
Merged

Birfy merged 1 commit into
mainfrom
docs/951-evidence-cards

Conversation

@HugoXK

@HugoXK HugoXK commented Nov 14, 2026 •

Copy link
Copy Markdown
Collaborator

Partial: #951 — the evidence-card schema and eight new cards. The algorithm-port rows on docs/results.md are still uncarded, so this does not close the issue.

What changed

  • The 12-field evidence-card schema is published as a machine-checkable artefact: tools/evidence_card.py --check validates every card in reports/evidence/ and fails on a missing or unknown field instead of skipping it; a missing value is written not reviewed, never omitted.
  • Eight new cards in reports/evidence/: one-call path, bring-your-own-agent, efficiency, equal-budget HotpotQA, the merge-cost lever, judge bias across five workloads, ground-truth repair, and the rule-search residual. Each states its held-out set, matched budget, cost, reliability, code pointer and the rest of the twelve fields.
  • reports/evidence/README.md indexes all 11 cards; docs/results.md ends by pointing a reader at the directory instead of repeating numbers.
  • Two cards are recomputed in CI from committed runs (equal-budget medians plus separates(); the matrix card's 1425 s / 1.39× / 1.92×). The six live-measurement cards say "not reproducible offline" on the card itself rather than implying a reproduction that does not exist.

Size: 12 files, +291/−7 (T1 by path: reports/**, tests/**, docs/**).

Verification at the tip

Tip SHA: ac68dcf2c485be1ba4a7c5d4fe0d90cecb684eb3

Ran: every command below from the repo root of a clean extraction of this tip, PYTHONPATH=., pytest with -o addopts="".

  • python3 -m tools.evidence_card --check → 11 cards, exit 0.
  • python3 -m pytest -q -o addopts="" tests/test_evidence_card_reports.py → 46 passed.
  • python3 -m pytest -q -o addopts="" tests/test_evidence_card.py tests/test_evidence_card_reports.py tests/test_docs_examples.py tests/test_efficiency.py tests/test_matrix_report.py tests/test_baselines.py tests/test_audit_attach.py tests/test_audit_diagnose_script.py → 167 passed.
  • python3 -m tools.changelog_fragments check → 34 changelog fragment(s) OK.
  • Red at base 0c9e936d9fd48c7712ef2ab2360413fdc927e311: this PR's test file with the eight new card files absent → 33 failed / 13 passed, exit 1.

Not run:

  • Full pytest -q; the two pytest commands above are the suites this change touches.
  • mkdocs build --strict → skipped, not run at this tip; mkdocs is absent from this box's python, and no earlier run is claimed for this tree.
  • The six live-measurement cards are not reproducible offline — each needs its model endpoint and the run behind it, and each card says not reproducible offline on the card itself.

Eight claims the results and audit pages already publish had no card, so a
reader could not tell a measured field from an unmeasured one. Each card names
the page it came from, the number and the command or test behind it, and says
'not reproducible offline' where the number needs a live model.

tests/test_evidence_card_reports.py pins every card against its page and
recomputes the two bench cards from the committed runs; reports/evidence/README.md
indexes all of them and docs/results.md points a reader at the directory.
@HugoXK
HugoXK marked this pull request as ready for review November 14, 2026 09:49
@HugoXK
HugoXK marked this pull request as draft November 14, 2026 10:06
@github-actions github-actions Bot added the risk:T1 Risk tier T1 (tests, examples, changelog.d), set by the controller from the diff label Nov 14, 2026
@HugoXK
HugoXK marked this pull request as ready for review November 15, 2026 04:12
@Birfy

Birfy commented Nov 15, 2026

Copy link
Copy Markdown
Collaborator

Verdict: PASS

Tip SHA: ac68dcf2c485be1ba4a7c5d4fe0d90cecb684eb3 · Risk tier: T1 (risk_gate.py from main c94e9d67)

Ran:

  • python -m tools.evidence_card --check on the tip → 11 cards, all valid.
  • pytest tests/test_evidence_card_reports.py → pass.
  • Spot-checked results-efficiency.json against docs/results.md lines 99-126: 5.8× / 1.1× / 1.8× / 1.7× / 2.65× / 3.6 s → 1.2 s all match the page; the retired 7.1× / 5.9× / 2.4× / 3.0× / 193.6 s figures are named as not re-runnable, as the card says.
  • Required CI green on this tip.

Not run: the live-API reruns the non-bench cards point at (they say so on the card). Partial for #951, so it correctly does not close the issue.

@Birfy
Birfy merged commit 1fd3b77 into main Nov 15, 2026
9 checks passed
@Birfy
Birfy deleted the docs/951-evidence-cards branch November 15, 2026 04:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

risk:T1 Risk tier T1 (tests, examples, changelog.d), set by the controller from the diff

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants