feat: persist completed sample outcomes so interrupted runs resume instead of restarting

[jamespsterling]

TL;DR

Adds a SampleResultStore Effect service to the shared harness: completed per-sample outcomes are persisted (keyed by sampleId/epoch) and replayed on a re-run, so an interrupted run (e.g. a Temporal Activity retry after a worker kill) only re-executes samples that were actually in flight.

What changed?

• New src/harness/sample-result-store.ts: PersistedSampleOutcomeSchema (Zod), SampleResultStore Tag, SampleResultStoreService interface, sampleResultKey(sampleId, epoch), and NOOP_SAMPLE_RESULT_STORE.
• src/harness/run.ts: the sample/epoch stream now goes through evaluateOneResumable — read persisted outcome → return it if present, else evaluateOne then best-effort write. Read failure = cache miss; write failure never fails the run — but both are now logged via wLog (Failed to read persisted sample outcome… / Failed to persist sample outcome…) instead of being silently swallowed.
• src/runner/run-by-id.ts: RunBenchmarkInput gains optional sampleResultStore; the layer defaults to the noop store.
• package.json: exports ./sample-result-store.
• Test helpers/layers updated with noopSampleResultLayer; branch merged up to current main (post run-by-id refactor).

Why?

In the openrouter-web monorepo (which vendors this repo as a subtree), Temporal Cloud's Worker Controller scale-in kills Cloud Run workers mid-Activity; heartbeat-timeout retries currently restart every sample from scratch. Wiring a GCS-backed store happens downstream: OpenRouterTeam/openrouter-web#35767.

This write-through design was chosen over abort-time aggregate flushing (#52, now closed) after an empirical head-to-head: on hard SIGKILL, write-through preserved 8/13 completed samples vs 0 for the abort-flush approach; clean aborts and repeated interruptions were parity with exact-baseline aggregates. See the comparison summary on #52.

Scope guarding (epochs/range) is intentionally not needed here: outcomes are keyed per sampleId/epoch and the store instance is provided per run/session downstream, so a persisted outcome is only ever reused for the identical sample-epoch.

How to test

• bun test src/harness/run.test.ts src/harness/sample-result-resume.test.ts — fresh run, full resume (0 solver/scorer calls), partial resume (only missing keys evaluated), and throwing-store behavior (run completes, all evaluated, failures logged).
• Downstream: provide a SampleResultStoreService in runBenchmarkById, run once, interrupt, run again with the same store — completed samples return without invoking solver/scorer.

Benchmark impact

No score changes with the default noop store. With a persistent store, replayed samples reuse their recorded score/usage instead of re-evaluating, which is the intended behavior for Activity retries.

Reviewer focus

• PersistedSampleOutcomeSchema vs EvalOutcome compatibility (usage/generation-id accounting on replay).
• Best-effort semantics (catchAll + wLog) around read/write.

Checklist

• Tests cover changed behavior
• Public API or configuration changes are backward compatible, or the break is documented
• Benchmark changes document dataset provenance and licensing
• No credentials, private results, or restricted dataset contents are included
• Documentation is updated where needed

Link to Devin session: https://openrouter.devinenterprise.com/sessions/4899435489c14bba8242921472c31807
Requested by: @jamespsterling

[devin-ai-integration]

Original prompt from James

can you use temporal MCP & Datadog to debug the heartbeat failures on https://cloud.temporal.io/namespaces/benchmarking-gcp.kej4s/workflows/vgi-stepC2-provider-ignore-xiaomi-mimo-v2-5.xiaomi%2Fmimo-v2.5.auto-routing/01a02121-4362-7290-86a1-76569b236a1d/history

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR that start with 'DevinAI' or '@devin'.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment, CI, and merge conflict monitoring

[devin-ai-integration]

Devin Review found 1 potential issue.

View 2 additional findings in Devin Review. (Configure)

[devin-ai-integration]

E2E re-verification after main merge + logging fix (c55e2bc) ✅

Re-ran the deterministic resume harness (20 samples × ~200 ms, file-backed SampleResultStore) from the #48-vs-#52 comparison against this branch:

Scenario	Result
Uninterrupted baseline	exact expected aggregate (20 q / 10 correct / tokens 300 / cost 0.020)
Clean abort @8 → resume	8 preserved, 12 re-executed, aggregate exact
SIGKILL @8 → resume	8 preserved, 12 re-executed, aggregate exact
Failing store write	warning logged 20/20 (Failed to persist sample outcome; a resumed run will re-run it + sample_result_key) — previously silent

One nit found during verification (logged error was Effect's UnknownException wrapper instead of the underlying store error) is fixed in da40a06, which also addresses the review findings above (trajectory persistence, degraded-outcome skip).

Tested by Devin — session

[devin-ai-integration]

Fixed by f03ffb0 — ZodShape<T> now maps over keyof Required<T> so optional keys must appear in the schema shape too (ZodType<T[key]> still admits undefined for them, so .optional() continues to satisfy the pin). Confirmed your probe: with the fix, adding readonly probeField?: string to Score fails bun run typecheck with TS2741 at ScoreSchema, and the repo builds clean without the probe (typecheck + full suite 1370 pass). Thanks for measuring it — this closes the exact hole that let trajectory slip through.