Record whether each prediction row's responses were live calls or cache replays

[MaxGhenis]

Follow-up to the latency replay audit (#228). That audit found 328 households in the frozen US run that are full disk-cache replays and 33 that are partial replays, and that their rows record the cache lookup as elapsed_seconds. The spend ledger already records cache_hit per call, but prediction rows did not, so analysis could not tell a replay from a call. Now every prediction row records where its responses came from.

The fields

Column	Meaning
live_calls	responses behind the row that the provider answered
cache_hit_calls	responses behind the row that the LiteLLM disk cache replayed
live_elapsed_seconds	the live calls' share of elapsed_seconds

The counts cover the request group whose time the row's elapsed_seconds apportions. For run_no_tools_eval that is the whole household: every chunk, repair round, budget escalation and explanation repair. For single-output runs and row repairs it is the one output, and for batch rows the one request. Every row of a group repeats the counts. live_elapsed_seconds is divided among the rows like elapsed_seconds. All three come from _response_cache_hit, the flag the spend ledger already used. LiteLLM sets _hidden_params["cache_hit"] = True on every disk-cache replay (litellm/caching/caching_handler.py, read for this PR).

Every writer

• Sync (eval_no_tools): _request_predictions_once and _request_explanations_once record each answered request. _aggregate_request_results and the chunked branch of run_single_no_tools sum them. Both row builders write them: household rows divide the live time by batch_size, and single-output rows write it unchanged. Their exception rows record zero calls. Unknown in means unknown out: a result without provenance makes the aggregate empty rather than undercounted.
• Batch (batch_eval.rows_from_unit): cache_hit_calls is 0, because batch requests never read the disk cache. live_calls is 1 when a response answered the row, and the live time is empty, like elapsed_seconds.
• Row repair: repaired rows carry their own request's provenance, and its exception row records zero calls. _append_rows used to append under whatever header the file had. It now widens an older header first and keeps every existing cell's text, so a repair round resumed across this change cannot shift cells. It also replaces a headerless empty file, which it previously appended to without a header. A merge that applies an older repair, written without the columns, empties them on the row it replaces: the source's values described the replaced request and could otherwise exceed the repair's time.
• Retry (retry_eval) and the merges (chunked_eval, fold_board, supervisor.combine) concatenate frames, so the columns pass through and older rows stay empty. Tests cover the retry and row-repair merges.
• Run store: the columns are response-level (_RESPONSE_COLUMNS). Their exact text is stashed for a byte-identical export, and typed copies go in responses.usage_json for SQL queries. A CSV written before the columns existed exports without them, byte for byte.
• Supervisor: _scenario_complete still accepts a scenario file that predates the columns, so a supervised run resumed across this change does not refuse its finished scenarios. A file with only some of the three columns is rejected.
• Analysis: call_provenance.household_call_provenance classifies each household as live, partial_replay, full_replay, no_response or unknown, with its summed live time. It is a new module, so analysis.py (whose hash the paper prints as the scoring code) is untouched.

Nothing published changes

latencySeconds, model_cost_latency and usage_summary_by_model are unchanged, and nothing published reads the new fields. A property test checks that adding the columns leaves both functions' output identical. Whether replays leave the median is decision d1542, which is still open, so it is not implemented here. The frozen board predates the fields, so all of its households classify as unknown, and a test reads the snapshot header to confirm it.

Invariants

These hold for every row a writer emits. tests/test_call_provenance.py executes each one:

1. live_calls and cache_hit_calls are non-negative integers. Their sum is every response behind the row's time, live or replayed.
2. 0 <= live_elapsed_seconds <= elapsed_seconds, and the two are empty together.
3. A row whose responses were all replays has live_calls == 0 and live_elapsed_seconds == 0 exactly.
4. A row with no replays has live_elapsed_seconds == elapsed_seconds exactly. Both sums run over the same sequence, and float addition and division are monotone, so 2 to 4 hold exactly, not within a tolerance.
5. The counts and times agree with the requests actually made. The test's oracle is the mock's own call history, and the spend ledger must record every one of those requests. When none of a group's requests failed, the row counts every returned response: its counts equal the replayed and live returns, its live time their live seconds, and the ledger's elapsed times give the same sums. When a request failed, a failed chunk or household keeps none of its time, so the row equals some subset of the returned responses, counts and times together, and never more.
6. Batch rows never record a cache hit and carry no live time.
7. Run-store and CSV round trips are byte-identical with the columns. Without them, the CSV round-trips byte-identical and stays without them.
8. Adding the columns leaves model_cost_latency and usage_summary_by_model identical.

Intended non-invariant: the counts are not additive across rows (each row of a group repeats them). live_elapsed_seconds is additive.

How the end-to-end property runs

test_rows_record_each_responses_provenance_and_agree_with_the_ledger drives run_no_tools_eval and run_no_tools_single_output_eval through the real request path: _request_predictions_once, escalation, repair rounds, explanation repair, retries, partial rounds, chunking, and failed chunks and households. Only litellm.completion, litellm.responses and the clock are mocked. The models cover chat completions with one output per request (Claude Sonnet 4.6) and unchunked (Claude Opus 4.6), and the Responses API unchunked (GPT-5.4) and three outputs per request (GPT-5.5). Hypothesis draws each response's cache flag, duration and shape: an answer to the requested tool (submit_outputs or submit_explanations), values only, nothing, a length cut, or a failure. Over 100 examples it produced partial replays (53%), full replays (35%), live households (21%) and households with no response (2%). A request failed in 43% of examples, 43% used the Responses API, and 13% answered an explanation repair. Each example checks invariants 1 to 5 and 7 on the rows, the written CSV and a run-store round trip. test_responses_api_requests_record_replays_and_live_calls pins a Responses API replay followed by a live repair.

Mutation check

Twenty-two mutants, applied one at a time, each break one rule: a replay keeping its time or counting as live, the aggregate counting only one response or ignoring an unknown one, the chunk path dropping provenance, a failed chunk turning provenance unknown, live time not being split, explanation repairs losing provenance or hiding a replay, answer replays going unseen, a failed household turning unknown, batch rows counting a hit, repair rows dropping counts, _append_rows ignoring the header, the legacy-repair merge keeping the source's values, the run store skipping the typed copies, the supervisor requiring the columns or accepting some of them, and the classifier misreading mixes and unknowns. All 22 are killed (~/reviews/policybench-row-cache-hit-calls/mutants-r2.log). Two first survived and led to tests: the supervisor requiring the columns (test_scenario_written_before_call_provenance_is_still_complete) and explanation replays going unseen (test_an_explanation_repair_records_its_own_provenance, which kills it on its own).

Review

Round 1 (GPT-6.1 Sol at ce67c35) asked for changes: an older repair file kept the source row's provenance (blocking, fixed above); invariant 5 was overstated and failures were untested; successful explanation repairs and the Responses API were uncovered. All three are addressed at 182decf.

Tests

• PYTHONHASHSEED=0 pytest -m "not slow" -n 4 at 182decf: 5169 passed, 58 skipped. Without the pinned seed, xdist workers collect tests/test_reference_audit_pins.py in different orders, because it parametrizes over frozensets; that predates this PR, CI runs without -n, and a separate task fixes it.
• ruff format --check and ruff check are clean on the changed files.

axiom: n/a: benchmark harness instrumentation, no policy change.

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
policybench-site	Ready	Preview	Oct 14, 2026 4:42pm UTC