Give a model with no recorded tokens no cost, not $0

[MaxGhenis]

The bug

analysis.model_cost_latency fills a missing total_cost_usd from PRICE_OVERRIDES_PER_1M using the model's summed prompt and completion tokens. It counted an unrecorded (NaN) sum as 0. Claude Fable 5's 1,984 rows in the frozen snapshot (paper/snapshot/20260501/runs/us_full_run_20260612_policyengine_4_16_1_populace/predictions.csv.gz) record no cost, tokens or latency, and Fable 5 has an override. So the function returned costUsd: 0.0, costPerHousehold: 0.0 for it instead of nothing.

Nothing published shows the $0:

• The release drivers overwrite Fable 5's four usage fields from the base release (CARRIED_USAGE).
• paper/index.qmd load_us_cost_latency treats a cost of 0 or less as missing.

An uncarried export (a PARTIAL stage, or any ad-hoc export_full_run) did show the $0 in the app. ModelLeaderboard renders a cost whenever costUsd != null, so Fable 5 appeared as "$0.00 total". Context: docs/latency_replay_audit.md in #228, "What the published figures do with them".

The fix

• policybench/analysis.py: the override fill now prices only the rows that record both prompt and completion counts. eval_no_tools._reconstruct_token_cost already returns no cost for a row missing either count, so the fill now follows the same rule. A row missing either count adds nothing; it is not priced at $0 for the missing side. A model with no row to price gets no cost, and Fable 5, which records nothing, gets no entry at all. When every row records both counts, the arithmetic is unchanged, and so are the results, to the bit.
• paper/index.qmd: load_us_cost_latency looped only over the models model_cost_latency returns. With Fable 5 gone from that output, Fable 5 would have lost its modelStats fallback ($0.541 per household, 98 s). It would also have dropped out of the summary sentence, which names it the priciest model. The loader now seeds every board model first. A render-time assertion checks that every model the board prices shows a cost; it fails with ['claude-fable-5'] without the seeding loop. CI's paper job runs it.
• Driver tests: every fake export_full_run (tests/test_finish_{argon,gpt61sol,haiku55,musewonder,adds0928}.py, tests/test_freeze_{argon,gpt61sol,haiku55,musewonder}.py) now drops all four Fable 5 fields, as the real exporter does now, instead of reporting a $0 cost. The finish tests assert that the carried row keeps the published key order.

Invariants

Each is a property that holds for every input. Each has a test.

#	Invariant	Test
1	costUsd and costPerHousehold are both present or both absent	test_cost_is_present_exactly_when_recorded_usage_supports_it (Hypothesis, 300 examples)
2	A model has a cost exactly when one of its rows recorded total_cost_usd, or when it has an override and one of its rows recorded both prompt and completion counts. A row missing either count is not priced. So no cost rests on a count nobody recorded, and a model with no usage gets no cost, not $0.	same, against a plain-Python reference. The generator draws whole uncosted and untokened models, so each override branch gets 5 to 68 cases per run. Plus test_price_override_without_recorded_tokens_gives_no_cost (the Fable 5 case), test_price_override_needs_both_token_counts, test_price_override_needs_both_counts_on_one_row, test_price_override_prices_only_rows_with_both_counts, test_recorded_zero_cost_and_recorded_zero_tokens_stay_zero
3	A recorded cost is the sum of the recorded row costs, and the override is not consulted. A filled cost is the override's prices applied to the token sums of the rows that recorded both counts.	same
4	costPerHousehold = costUsd ÷ the input's distinct households	same
5	Differential test against the published figures: every model's published costUsd, costPerHousehold, latencySeconds and totalTokens, except Fable 5's, equal model_cost_latency on the frozen predictions, to the bit and with the same fields present. Fable 5's are carried, not computed.	test_frozen_usage_reproduces_every_published_model_but_fable_5, test_frozen_fable_5_usage_is_carried_not_computed
6	Every driver carries the same four fields in one order. The published Fable 5 row ends with those fields in that order.	test_every_driver_carries_fable_5s_four_fields_in_one_order, test_the_published_fable_5_row_ends_with_the_carried_fields
7	release_20261006.export_payload's real carry restores the published row byte for byte whether the export omits Fable 5's usage or reports a $0 cost, and leaves every other row alone	test_the_carry_restores_the_published_row_whichever_way_the_export_reports_it (Hypothesis)
8	(slow, local) The committed snapshot's US payload rebuilds byte for byte from its committed inputs through the real export_full_run and the carry. The raw export reports none of Fable 5's usage.	test_the_committed_payload_rebuilds_through_export_and_the_carry

Mutation checks:

• origin/main's analysis.py fails 8 of the cost tests and the slow rebuild.
• This PR's first version, which priced model-level sums, fails the 3 tests on split and mixed rows.
• Swapping two fields in release_20261006.CARRIED_USAGE fails tests 6 and 7.
• Removing the paper's seeding loop fails its render-time assertion.

No published number changes

Before = origin/main bf845cb, after = this branch, run on the frozen snapshot (the live release's 50-model board):

Check	Before	After
model_cost_latency entries for the 49 models other than Fable 5 (values as repr, key order, model order)	—	identical
Fable 5's model_cost_latency entry	{costUsd: 0.0, costPerHousehold: 0.0}	absent
Paper cost table rows and the cost/latency summary sentence	—	identical
Paper's Fable 5 entry	$0.541/household, 98 s (plus an unused costUsd: 0.0)	$0.541/household, 98 s
Raw export_full_run Fable 5 row	costUsd: 0.0, costPerHousehold: 0.0	no usage fields
export_full_run + release_20261006.export_payload carry vs the committed data.json.gz	byte-identical	byte-identical

Scripts and outputs are in ~/reviews/policybench-cost-override-needs-usage/evidence/ (before-after-diff.txt, capture_cost_latency.py, export_carry_check.py; the row-level fill's outputs are in after-rowlevel/ and export-after-rowlevel/, identical to after/). In the frozen predictions, the override path is reached by three models only:

• Fable 5, with no counts at all;
• gemini-3.6-flash and grok-build-0.1, with both counts on all 1,984 rows.

Driver audit

Path	How it carries / gates	With the export now omitting Fable 5's cost
release_20261006.export_payload, release_20261013b.export_payload	Assigns the four fields from the base. verify_payload requires the same key order and the same STABLE_STATS bytes.	Assignment adds the missing keys in CARRIED_USAGE order, which is where the published row has them, because costUsd/costPerHousehold were the exporter's last keys. Shown byte-identical on real data.
finish_{gpt61sol,haiku55,musewonder,argon}.build_payload	Assigns the four fields, then runs a JSON-byte drift gate. gpt61sol gates the carried export itself; the later three gate the scope export (release_20261006.export_payload, which carries itself).	Same. One test needed changing. gpt61sol's drift-gate property mutated the exported Fable 5 row's costUsd, which the export no longer has. At target 44 its two cost mutations now model an export that reports a cost after all; the carry overwrites it with no drift. @example pins those cases so they don't depend on sampling.
freeze_{gpt61sol,haiku55,musewonder,argon}.rebuild_payload	Re-runs build_payload and requires the staged bytes	The carried bytes are identical either way, so a stage exported before this change freezes cleanly after it
finish_adds0928.export / replay_base_references	Assigns the four fields; dict-equality drift check	Order-insensitive; fine
freeze_snapshot.py (row["costUsd"] for every row)	Reads the frozen, carried payload	Unaffected
partial builds	No carry	Fable 5 now shows no cost rather than $0

No driver compares Fable 5's cost to 0. The total <= 0 checks in the finish drivers are about run_state.json totals.

Not changed

• Partial row coverage: a recorded cost sums the rows that recorded one (_sum_or_nan, unchanged), and a filled cost sums the rows that recorded both counts. Either way the cost covers those rows only. No override-filled model in the snapshot has partial coverage.
• The published paper's "Scoring code SHA-256 prefix" row hashes analysis.py at render time. This PR doesn't re-render, so that row reflects the last render until the next one. Past non-release PRs that changed analysis.py didn't re-render either (ea67093, 8f3c44c, e9e91df, fba3cfc); releases do. No displayed number changes.
• The model-level fill prices input and output only, with no cache rates, as before.

Tests

• PYTHONHASHSEED=0 pytest -m "not slow" -n 4 at 473ebb3: 5,160 passed, 58 skipped. The fixed hash seed works around the collection-order issue under Related; CI runs serially.
• tests/test_carried_usage.py, slow test included, at 473ebb3: 9 passed.

Review

• r1, an in-session Opus 5.5 reviewer at 460cad1: AGREE.
  • Blocker: the gpt61sol KeyError, fixed in 460cad1.
  • Should-fix: split-count rows were priced at $0 for the missing side, fixed in 473ebb3 with the row-level fill.
  • Nits, fixed: the design doc's tense, the paper assertion, the generator bias and the docstring.
  • Nit, disclosed: the paper's hash row (see Not changed).
• r2, GPT-6.1 Sol at ultra effort (subfleet run --task review --tier hard, job 20261014-104306-pb230-r2, lane codex-4) at 473ebb3: AGREE, no findings. It read the implementation, consumers, driver paths and tests. It did not execute anything, and it relied on the evidence above for bytes, rendered numbers, CI and the Disclose why 330 frozen households record near-zero latency and two models record none #228 merge.

Related

• Fable 5 wording: the new docstrings state only that Fable 5's frozen board rows record no usage. They don't say why: Disclose why 330 frozen households record near-zero latency and two models record none #228's audit finds the old batch-adapter account wrong and corrects the driver comments. The finish drivers' "export_full_run reports $0" comments now say it reports none of its usage, and docs/gpt61sol/design.md puts its "$0" sentence in the past tense.
• Checked with Disclose why 330 frozen households record near-zero latency and two models record none #228 merged in (a scratch merge of 473ebb3 and Disclose why 330 frozen households record near-zero latency and two models record none #228's f1caa70, conflicts resolved by ~/reviews/policybench-cost-override-needs-usage/resolve_228_conflicts.py): test_cost_latency, test_latency_replay_audit, test_carried_usage, test_disclosures, test_paper_results and test_finish_gpt61sol give 374 passed, 4 skipped. The paper's cost table, its summary sentence and Disclose why 330 frozen households record near-zero latency and two models record none #228's new caption (with its dash assertions) are identical to Disclose why 330 frozen households record near-zero latency and two models record none #228 alone. This PR's seeding loop is what keeps Disclose why 330 frozen households record near-zero latency and two models record none #228's assertion that cost dashes are only zero-cost models true, so the two PRs depend on each other there.
• Disclose why 330 frozen households record near-zero latency and two models record none #228's doc says model_cost_latency "returns $0" for Fable 5. Its owner session has been told. Whichever PR lands second fixes the sentence.
• pytest -n fails at collection on main: tests/test_reference_audit_pins.py parametrizes over sets. This was worked around here with PYTHONHASHSEED=0 and split into its own task.

axiom: n/a: benchmark cost accounting, no policy rule.

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
policybench-site	Ready	Preview	Oct 14, 2026 1:43pm UTC