Disclose why 330 frozen households record near-zero latency and two models record none

app/public/paper/web/index.html

@@ -250,7 +250,7 @@ <h3 id="frozen-snapshot-and-open-set-status" class="anchored">Frozen snapshot an
 </tr>
 <tr class="even">
 <td>Scoring code SHA-256 prefix</td>
-<td>baaf2506800b</td>
+<td>e75a972a2f98</td>
 </tr>
 <tr class="odd">
 <td>US run label</td>
@@ -299,7 +299,7 @@ <h3 id="frozen-snapshot-and-open-set-status" class="anchored">Frozen snapshot an
 </figure>
 </div>
 </div>
-<div id="7f7f2048" class="cell" data-execution_count="3">
+<div id="076a6925" class="cell" data-execution_count="3">
 <div class="cell-output cell-output-display cell-output-markdown" data-execution_count="3">
 <div id="tbl-model-runs" class="quarto-float quarto-figure quarto-figure-center anchored">
 <figure class="quarto-float quarto-float-tbl">
@@ -1096,15 +1096,15 @@ <h3 id="uncertainty" class="anchored">Uncertainty</h3>
 <p>Sampling uncertainty is substantial at this sample size. In the frozen snapshot, 6 of the other 49 models have an exact-match 95% interval that overlaps the leader’s. Small differences in the leaderboard ordering should therefore be read as ties rather than as a strict ranking, and the rank-range column makes the set of plausible positions explicit. These intervals are a manuscript artifact: the public leaderboard and app report point estimates only, because the benchmark sample, not run-to-run model variation, is the dominant source of uncertainty here.</p>
 <h3 id="cost-and-latency" class="anchored">Cost and latency</h3>
 <p>Accuracy is one axis; the price and speed of a no-tools answer are another. <a href="#tbl-us-cost-latency" class="quarto-xref">Table&nbsp;7</a> reports each model’s estimated cost to compute one household’s full output vector and its median per-household latency, next to the headline exact rate. Per-household cost spans $0.001 (Claude Haiku 5.5) to $0.541 (Claude Fable 5), and median per-household latency 3 s (Gemini 3.5 Flash-Lite) to 1023 s (Kimi K3). Neither dimension cleanly tracks accuracy: the top-scoring model, GPT-6 Sol at 95.2% exact, sits mid-pack at $0.027 per household and 23 s.</p>
-<div class="cell" data-execution_count="8">
-<div id="tbl-us-cost-latency" class="cell quarto-float quarto-figure quarto-figure-center anchored" data-execution_count="8">
+<div id="a62d2fd0" class="cell" data-execution_count="8">
+<div class="cell-output cell-output-display cell-output-markdown" data-execution_count="8">
+<div id="tbl-us-cost-latency" class="quarto-float quarto-figure quarto-figure-center anchored">
 <figure class="quarto-float quarto-float-tbl">
 <figcaption class="quarto-float-caption-top quarto-float-caption quarto-float-tbl" id="tbl-us-cost-latency-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
-Table&nbsp;7: Estimated cost and median latency per household for the frozen US snapshot, alongside the headline exact-match rate. Each row uses its recorded per-call cost: reconstructed from token counts at the list price configured at request time, or the provider-reported charge where no reconstruction was available. List-price overrides apply at request time, not retroactively to recorded costs. Models without per-call costs use the frozen release-metadata cost. Latency is the median across households of each household’s summed request time. A dash marks usage or wall-clock latency that was not measured per call.
+Table&nbsp;7: Estimated cost and median latency per household for the frozen US snapshot, alongside the headline exact-match rate. Each row uses its recorded per-call cost: reconstructed from token counts at the list price configured at request time, or the provider-reported charge where no reconstruction was available. List-price overrides apply at request time, not retroactively to recorded costs. Claude Fable 5’s cost and latency come from its original run’s per-call rows, which its board rows do not carry. Latency is the median across households of each household’s summed request time. A cost dash marks a provider-reported charge of zero on every call (GLM-5.3-Flash (preview)). A latency dash marks rows that record no request time (GPT-5.5, through OpenAI’s batch API) or a median household answered from the harness’s response cache rather than by the provider (GLM-5.2, Kimi K2.6, MiniMax M3, and Qwen3.7-max; see text).
 </figcaption>
 <div aria-describedby="tbl-us-cost-latency-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
-<div class="cell-output cell-output-display cell-output-markdown" data-execution_count="8">
-<table class="cell caption-top">
+<table class="caption-top">
 <thead>
 <tr class="header">
 <th>Model</th>
@@ -1417,11 +1417,12 @@ <h3 id="cost-and-latency" class="anchored">Cost and latency</h3>
 </tbody>
 </table>
 </div>
-</div>
 </figure>
 </div>
 </div>
-<p>Cost preserves each call’s recorded total: a request-time reconstruction from token counts where one was available, and the provider-reported charge otherwise. Configured list-price overrides do not reprice recorded calls. Rows without per-call costs use the frozen release-metadata cost. Latency is wall-clock request time as observed by the harness, summed within each household and taken as the median across the 100 households. The per-call timer spans any provider-side retry or rate-limit backoff, so the median is reported in preference to the mean, which a few throttled calls inflate. Latency is not comparable across serving regimes: Claude-family models issue chunked serial calls that lengthen wall-clock time, several models ran through batch or asynchronous dispatch whose logged times reflect submission rather than generation (shown as a dash), and Claude Fable 5’s usage and latency are recorded at the run level rather than per call. Both cost and latency are properties of this run’s providers, prompts, and serving configurations at evaluation time rather than fixed model constants.</p>
+</div>
+<p>Cost preserves each call’s recorded total: a request-time reconstruction from token counts where one was available, and the provider-reported charge otherwise. Configured list-price overrides do not reprice recorded calls. Claude Fable 5’s board rows carry no usage or timing fields, because they were rebuilt from release dashboard-data-20260701 without those columns; its cost, tokens and latency come from the per-call rows of its original run, which that release publishes. Latency is wall-clock request time as observed by the harness, summed within each household and taken as the median across the 100 households. The timer spans one request as the provider handles it, including anything that happens inside that call. When the harness retries a failed request, only the attempt that succeeded is timed, without the failed attempts or the wait between them. The median is reported in preference to the mean, which a few slow calls inflate. Latency is not comparable across serving regimes. A model served one or three outputs per request (<a href="#tbl-model-runs" class="quarto-xref">Table&nbsp;2</a>) sends a household’s calls one after another, and its household time is their sum. GPT-5.5 ran through OpenAI’s batch API, which records no per-call request time, so it shows a dash.</p>
+<p>The harness also caches provider responses on disk, keyed on the request and the prompt contract version. A request identical to an earlier one is answered from the cache in hundredths of a second, and the timer records that as if it were a call. A replayed response’s cost is computed from that response when its row is written: its token counts at the configured list price, or the provider’s charge embedded in it where no reconstruction exists. A household’s cost therefore counts each of its responses once, whether or not the board run paid for it. GLM-5.2, Kimi K2.6, MiniMax M3, and Qwen3.7-max were finished over several passes, each of which replayed the earlier passes’ successful responses. Each of them records a replay on 53 to 93 of its 100 households, so its median falls under a second and it shows a dash as well. In 11 other models, 29 households sum to under a second. The audit traces 27 of them to an earlier request on the same household, such as an onboarding smoke or a restarted chunk; the other two come from runs that predate per-call timestamps and left no cache or ledger behind. Dropping them would raise no model’s median by more than 2.5 seconds. A further 33 households replay their first recorded call and then reach the provider, so their times leave out that first call. The benchmark card summarizes the replays, and the repository’s latency replay audit lists each household. Both cost and latency are properties of this run’s providers, prompts, and serving configurations at evaluation time rather than fixed model constants.</p>
 <h3 id="simple-baselines" class="anchored">Simple baselines</h3>
 <p>Simple baselines help interpret score levels in a zero-heavy benchmark. <a href="#tbl-baselines" class="quarto-xref">Table&nbsp;8</a> reports an always-zero response and a median-reference-by-output response on the same frozen household sample, scored with the household-impact weighting. These are not model competitors; they show how much of each metric can be earned without household-specific policy calculation. The always-zero baseline scores 69.7% on the headline exact-match rate — below the 85% of outputs that are exact zeros because the weighting concentrates mass in high-dollar output groups that are rarely zero (<a href="#sec-headline-metric" class="quarto-xref">Section&nbsp;3.1</a>). The top model clears the baseline by 25.5 points, and the within-1% companion shows the same separation with near-miss tolerance.</p>
 <div class="cell" data-execution_count="9">

app/src/paperSnapshot.json

@@ -1,6 +1,6 @@
 {
-  "pdfVersion": "d0b236367fb3",
+  "pdfVersion": "d16eb03ce07d",
   "responseWindow": "June 12 and October 13, 2026",
   "snapshotDate": "2026-10-14",
-  "webVersion": "533a0cb79b6a"
+  "webVersion": "6cb2fe1ad2b3"
 }

docs/benchmark_card.md

@@ -420,23 +420,64 @@ the site's types and leaderboard already filter on `condition`.
 
 Each frozen row uses its recorded per-call cost: reconstructed from token counts
 at the list price configured at request time, or the provider-reported charge
-where no reconstruction was available. List-price overrides apply at request time, not retroactively to
-recorded costs. Models without per-call costs use the frozen release-metadata
-cost. Published model costs retain these recorded totals rather than repricing
-past calls at today's rates.
+where no reconstruction was available. List-price overrides apply at request
+time, not retroactively to recorded costs. Published model costs retain these
+recorded totals rather than repricing past calls at today's rates. GPT-5.5 ran
+through OpenAI's batch API, and its costs are reconstructed at standard
+synchronous rates. Claude Fable 5's board rows carry no usage fields: they were
+rebuilt from release dashboard-data-20260701 without them. Its published cost,
+tokens and latency come from the per-call rows of its original run, which that
+release publishes as `claude-fable-5-predictions.csv.gz`.
 
 The harness caches provider responses on disk, keyed on the request and the
-prompt contract version (`policybench/cache.py`). A board request identical to
-one sent earlier, such as an onboarding smoke on the same household, returns
-the cached response: the row records that response's answers, tokens and cost,
-and a latency of about zero. Each prediction row records its call's latency
-divided among the call's outputs, so a household's latency is the sum over its
-rows. The Muse Spark 1.3 row replays its four smoke households (scenario_000,
-scenario_001, scenario_099 and scenario_119), each with the smoke's provider
-response id and a household latency of 0.03 seconds or less. In the frozen
-predictions other rows also have households under 0.05 seconds by that sum,
-among them scenario_000 and scenario_001 for Muse Wonder and Claude Opus 5.5
-and scenario_000 for Claude Haiku 5.5.
+prompt contract version (`policybench/cache.py`). A request identical to an
+earlier one returns the cached response, and its row records that response's
+answers and tokens. Its cost is computed from the replayed response when the
+row is written: its token counts at the configured list price, or the
+provider's charge embedded in it where no reconstruction exists. Each
+household's cost counts its responses once, whether or not the board run paid
+for them, and leaves out spend on attempts that did not reach the board.
+
+## Latency basis
+
+Each prediction row records its household's request time divided among the
+household's rows, so a household's latency is the sum over its rows
+(`analysis.household_latency_seconds`). A household that needs several calls
+(one or three outputs per request, a repair, or a re-request with a larger
+completion budget) makes them one after another and records their sum. A
+request the harness retries after a provider error is timed only on the
+attempt that succeeded. Published `latencySeconds` is the median
+across households. A household with no recorded time is left out rather than
+counted as zero, so GPT-5.5, whose batch rows record none, has no latency.
+
+A cached response is answered in hundredths of a second, and its row records
+that lookup as the call's latency. In the frozen predictions, 330 households
+sum to under a second, all under 0.12 s; the fastest live household takes
+1.75 s. The audit (`docs/latency_replay_audit.md`, with every household in
+`docs/latency_replay_audit/households.csv`) traces 328 of them to an earlier
+request on the same household. The other two (Grok 4.3's scenario_000 and
+Claude Haiku 4.5's scenario_009) come from runs that predate per-call
+timestamps and left no cache or ledger. Another 33 households replay their
+first recorded call and then reach the provider, so their times leave out that
+call.
+
+- Qwen3.7-max (93 households), GLM-5.2 (80), Kimi K2.6 (75) and MiniMax M3
+  (53) were finished over several passes, each of which replayed the earlier
+  passes' successes. Their published medians, 0.003 to 0.033 seconds, are
+  cache lookups, not latency, and the paper shows them as a dash.
+- Claude Sonnet 5 (9 households) replays its smoke and two restarted chunks.
+  Muse Spark 1.3 (4) and Muse Wonder, Claude Opus 5.5 and Claude Haiku 5.5
+  (2 each) replay their onboarding smokes. DeepSeek V4 Pro and GPT-5.6 Sol,
+  Luna and Terra (2 each) replay requests sent minutes to an hour before
+  their runs. Replays can only lower a median. Dropping these households would
+  raise no published median by more than 2.5 seconds (DeepSeek V4 Pro, 169.9
+  to 172.4 s).
+- The 33 partial replays are in GLM-5.3 (18), GLM-5.3-Flash (preview) (4),
+  Kimi K2.6 (4), Claude Sonnet 5 (2), Kimi K3 (2), and GLM-5.2, MiniMax M3 and
+  Qwen3.7-max (1 each).
+
+The leaderboard does not display latency. Whether published medians should
+leave out replayed households is an open methodology decision.
 
 ## Country data paths
 

docs/gpt61sol/design.md

@@ -101,9 +101,10 @@ and `test_every_committed_adjudication_decides_a_case_the_seed_judged` run
 anywhere, and `prepare`, `bind-seed` and every later step check the seed and
 the stage's binding against it.
 
-The exporter still needs the Fable 5 usage carry-over. Fable 5 ran through the
-Anthropic batch adapter, and its committed rows carry no cost, token or latency
-fields, so `export_full_run` reports `costUsd` $0 and omits the rest. The live
+The exporter still needs the Fable 5 usage carry-over. Fable 5's committed rows
+were rebuilt from release 20260701 without their cost, token and latency
+fields (an earlier version of this note said it ran through the Anthropic
+batch adapter; it ran synchronously, see `docs/latency_replay_audit.md`), so `export_full_run` reports `costUsd` $0 and omits the rest. The live
 payload has $54.11, 3,850,174 tokens and 97.68 s. The driver copies those four
 values from the base, and the drift gate then checks every field and the key
 order. Ox Alpha's `costUsd` of $0.0 comes out of the exporter unchanged, so it