Disclose why 330 frozen households record near-zero latency and two models record none

[MaxGhenis]

Audits the frozen US run's household latencies (the Gemini 4 Argon finding, #221) and corrects every published explanation of them. No published number changes. Whether latencySeconds should leave out replayed households is Max's call, queued as d1542.

What the audit found

The full write-up is docs/latency_replay_audit.md, and every household is in docs/latency_replay_audit/households.csv.

• Every household that sums to under a second is a disk-cache replay, not a provider call. All 330 such households sum to under 0.12 s, while the fastest live household takes 1.75 s. 328 are traced: their first response existed before the request was sent. The evidence is the response in the main .policybench_cache with an earlier store time, the creation time embedded in OpenRouter (gen-<seconds>) or OpenAI Responses ids, cache_hit: true in the board run's spend ledger, or the same id served live in a smoke. Two (Grok 4.3 scenario_000, Claude Haiku 4.5 scenario_009) come from June runs that left no timestamps, cache or ledger. Another 33 households are partial replays: the first call was replayed and later calls went live, so their times omit that first call.
• Qwen3.7-max (93), GLM-5.2 (80), Kimi K2.6 (75), MiniMax M3 (53). rerun_openweight.sh and grind_model.sh re-ran each model's full range several times, and each pass's cache replayed what earlier passes had finished. Their published medians are 0.003 to 0.033 s.
• Other replays. Claude Sonnet 5 (9) replays its smoke and two restarted chunks. Muse Spark 1.3, Muse Wonder, Claude Opus 5.5 and Claude Haiku 5.5 replay onboarding smokes. DeepSeek V4 Pro and GPT-5.6 Sol, Luna and Terra replay requests sent minutes to an hour before their runs. Dropping these would raise no median by more than 2.5 s.
• GPT-5.5 ran through OpenAI's batch API: one batch, 716 requests. Batch rows record no time by design.
• Claude Fable 5 ran synchronously (2026-07-01 22:26 to 07-02 01:53 UTC), before the batch adapter first landed (Add batch-API eval mode (Anthropic, OpenAI, Gemini) #88, 10:03 UTC on 07-02). Its board rows were rebuilt from release dashboard-data-20260701 without usage, timing or raw columns (57d26f1). The original per-call rows are that release's claude-fable-5-predictions.csv.gz, and they reproduce the published $54.11, 3,850,174 tokens and 97.68 s exactly.

What the published figures did

• modelStats.latencySeconds (the dashboard JSON) reproduces exactly from the frozen rows, except Fable 5's carried fields. Replays count as calls, so the four models above publish cache-lookup times. The site shows no latency (dropped in Drop the latency column from the leaderboard #100).
• The paper already showed the four as dashes, but it explained the dashes as "batch or asynchronous dispatch whose logged times reflect submission rather than generation". It also said Fable 5's usage was "recorded at the run level". Neither fit any model. The snapshot manifest and five release drivers said Fable 5 "ran through the Anthropic batch adapter", which was also false.
• costUsd is not misleading. A replay keeps the cost recorded when its response was generated: on all ten ledger-era replays, the board cost equals the ledger's cached_response_cost_usd.

Changes

• Paper (paper/index.qmd). The caption and paragraph now name each dashed model and its real cause, computed from the frozen data (new PaperResults accessors). The render asserts that every dash in the table is one the caption explains. The paper is re-rendered and re-pinned (freeze_snapshot.py --rendered-only). The rendered text changes only in the caption, the paragraph, page numbers and the "Scoring code SHA-256 prefix", which hashes policybench/analysis.py. No table value changes.
• Benchmark card. The cost basis is rewritten, and a new latency basis section is added.
• Corrections. The manifest's Fable 5 note (in both freeze_snapshot.py and manifest.json), the release drivers' CARRIED_USAGE comments, docs/gpt61sol/design.md, and a paper_results docstring.
• Code. analysis.household_latency_seconds and REPLAYED_HOUSEHOLD_SECONDS; model_cost_latency now uses the helper, with identical output. scripts/audit_latency_replays.py rebuilds the evidence CSV from local caches and ledgers, and reproduced it byte for byte on a second run.
• Tests. Replaces the Argon-era card test, which asserted that Claude Haiku 5.5's scenario_001 (0.118 s) was not a replay; its board ledger records cache_hit: true.

Invariants (tested)

Property tests (Hypothesis, tests/test_cost_latency.py):

1. A household's latency is the sum of its calls, whatever the split of each call across its rows.
2. Published latency is the median over timed households. Untimed households are left out, not counted as zero, and an all-untimed model publishes none.
3. Lowering any household's time never raises the median, so replays can only bias it down.
4. With replays under a second and calls above it, the median is under a second exactly when more than half the households replayed. This is the paper's dash rule.
5. Cost is additive over rows.

Frozen-data and differential tests (tests/test_latency_replay_audit.py):

• The audit CSV lists exactly the sub-second households plus the partial replays, and each row is the household it names.
• Full replays are under 1 s and partial ones are not; unlisted households all exceed 1.5 s.
• The id-time evidence recomputes from the frozen ids and is complete. On every other household with a timed id, the creation time sits within the margin of the request start.
• Every published cost and latency equals model_cost_latency over the frozen rows, except Fable 5's four carried fields.
• Every dash in the rendered table is one the text explains, and the rendered paragraph states the computed counts.
• tests/test_disclosures.py checks the card's counts against the frozen data, and its false-claim patterns now reject the old phrases in the paper source, HTML and PDF.

axiom: n/a: benchmark disclosure and test changes; no policy encoding.

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
policybench-site	Ready	Preview	Oct 14, 2026 1:21pm UTC

[MaxGhenis]

#230 merged first, as 3c2d928. As agreed between the two sessions, docs/latency_replay_audit.md's "model_cost_latency itself returns $0 …" sentence is now this PR's to fix. Suggested text:

model_cost_latency returned $0 for a model whose rows carry no tokens when a price override exists, as Fable 5's do, until #230; it now reports no cost, and the release drivers carry the published value.

Taking main into this branch will conflict in six files:

• scripts/finish_{argon,gpt61sol,haiku55,musewonder}.py: keep this PR's rebuilt-from-20260701 account, ending "so export_full_run reports none of its usage."
• tests/test_cost_latency.py: both sides append tests. Keep both, with two blank lines between.
• docs/gpt61sol/design.md: keep this PR's text, but change "so export_full_run reports costUsd $0 and omits the rest." to "so export_full_run reported costUsd $0 and omitted the rest (since Give a model with no recorded tokens no cost, not $0 #230 it reports none of them)."

paper/index.qmd merges cleanly. #230 seeds every board model in load_us_cost_latency and adds a render-time assertion after it. This PR's cost_dash == r.zero_cost_models caption assertion depends on that seeding: without it, Fable 5 drops out and shows a cost dash.

Measured on a scratch merge of #230 473ebb3 with this PR's f1caa70:

• test_cost_latency, test_latency_replay_audit, test_carried_usage, test_disclosures, test_paper_results and test_finish_gpt61sol: 374 passed, 4 skipped.
• The paper's cost table, summary sentence and this PR's caption, with its dash assertions, are identical to this PR alone.

A resolver script that applies all of the above is at ~/reviews/policybench-cost-override-needs-usage/resolve_228_conflicts.py <checkout> on Max's machine. It was written for merging this branch into #230's, so it treats #230's side as "ours". If you merge main into this branch, git swaps ours and theirs, so read the script as a recipe rather than running it as is.

pytest -n 4 on main needs PYTHONHASHSEED=0 until tests/test_reference_audit_pins.py stops parametrizing over sets; that fix has its own task.

🤖 Generated with Claude Code