feat(runner): persist and resume partial sample outcomes across aborted runs

[jamespsterling]

TL;DR

Adds an optional PartialOutcomeStoreService so an aborted run (e.g. Cloud Run SIGTERM) flushes completed sample outcomes durably, and the retry skips those sample/epochs and aggregates the combined result.

What changed?

• src/results/partial-outcome-store.ts (new): PartialOutcomeStoreService contract (read/write/remove) plus Zod schemas (SampleOutcomeSchema, PartialOutcomesSchema) to validate persisted JSON at runtime. Exported as ./partial-outcome-store.
• src/harness/run.ts:
  • RunConfig gains skipSampleEpochs?: ReadonlySet<string> and onOutcome?: (outcome: SampleOutcome) => void.
  • New helpers sampleEpochKey(sampleId, epoch) and aggregateOutcomes(outcomes) (loop-based fold into the existing FoldAccumulator/finalizeRun).
  • The sample-epoch stream filters out already-completed keys; each completed outcome invokes onOutcome before progress reporting.
• src/runner/run-by-id.ts: partialOutcomeStore? on the run input. Flow:
  1. read() prior outcomes (best-effort) and build the skip set.
  2. Run with skipSampleEpochs + collect new outcomes via onOutcome.
  3. On abort (abortSignal.aborted and new outcomes exist): write() prior + new outcomes, then rethrow/return the abort as before.
  4. On success: aggregate prior + new outcomes into the final RunResult and remove() the partial object.
  • Failures other than abort leave existing partial data untouched.

Why?

Cloud Run gives ~10s between SIGTERM and SIGKILL. Without a durable flush, every completed-but-unpersisted sample in an in-flight chunk is lost and re-run on retry. With this, the retry only reruns missing sample/epochs. Cancellation semantics are unchanged — an aborted run still surfaces as aborted; the flush is a side effect.

How to test

• bun test src/harness/run.test.ts src/results/partial-outcome-store.test.ts — covers skip-set filtering, onOutcome invocation, aggregation, and schema accept/reject.
• Simulate resume: run with a store whose read() returns prior outcomes; observe skipped sample/epochs in the run and a final result aggregating both sets, followed by remove().

Benchmark impact

No score changes for uninterrupted runs (store is optional and unused by default). Resumed runs aggregate persisted + new outcomes identically to a single full run over the same sample/epochs.

Reviewer focus

• Abort-flush condition in run-by-id.ts (only when abort signal fired and new outcomes exist; write errors are logged, never masked into a different failure).
• sampleEpochKey identity — stable across attempts so skip sets are safe.

Checklist

• Tests cover changed behavior
• Public API or configuration changes are backward compatible, or the break is documented
• Benchmark changes document dataset provenance and licensing
• No credentials, private results, or restricted dataset contents are included
• Documentation is updated where needed

Link to Devin session: https://openrouter.devinenterprise.com/sessions/4899435489c14bba8242921472c31807
Requested by: @jamespsterling

[devin-ai-integration]

Original prompt from James

I want to explore using SIGTERM/SIGKILL to alleviate the heartbeat timeouts on the temporal benchmarking workflows. What can we do to quickly stop & resume the benchmark within ~10s using SIGTERM/SIGKILL hooks? I want the most minimally invasive path if possible to test this. do we need a parallel activity with a short polling interval that can just listen for this event given it's scoped to like 10s? then this activity can signal to the long-running process to stop or even just collect what finished benchmarking chunks we have and terminate it if not stopped quickly enough?

Cloud Run sends a SIGTERM signal to your container to start a 10-second graceful shutdown window before forcing termination with SIGKILL. [1]
How Shutdown Signals Work

• SIGTERM (Signal 15): Sent to notify your container instance that it will shut down. Your application has 10 seconds to finish active requests, close database connections, and flush logs. [1, 2]
• SIGKILL (Signal 9): Sent immediately if your container is still running after the 10-second grace period ends. This signal cannot be caught or blocked, resulting in an abrupt stop. [1, 2]
• Environment behavior: If you use the first-generation environment and do not trap SIGTERM, the instance shuts down instantly. Second-generation environments strongly recommend installing a signal handler. [1]
  Best Practices for Handling Signals
• Catch SIGTERM: Write a signal handler in your application ... (734 chars truncated...)

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR that start with 'DevinAI' or '@devin'.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment, CI, and merge conflict monitoring

[devin-ai-integration]

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no potential bugs to report.

View in Devin Review to see 1 additional finding.

[devin-ai-integration]

Closing in favor of #48 (per-sample write-through persistence), based on an empirical head-to-head of both resume strategies on the same deterministic workload:

Findings

• Hard SIGKILL (no flush window): feat: persist completed sample outcomes so interrupted runs resume instead of restarting #48 preserved 8/13 completed samples across two runs; this PR's abort-time flush preserved 0 and re-executed all 20 samples — the flush never gets to run when the process dies without SIGTERM handling.
• Clean abort + resume: parity — both preserved outcomes at the abort point, resumed only the remainder, and produced final aggregates exactly matching the uninterrupted baseline.
• Repeated interruptions: both converged to correct results; feat: persist completed sample outcomes so interrupted runs resume instead of restarting #48 wasted 0 solver calls vs 5 here.
• Slow store write (~5s): this PR blocks shutdown waiting on the flush (risking the SIGKILL deadline); feat: persist completed sample outcomes so interrupted runs resume instead of restarting #48 exits fast but can drop the in-flight write.
• Store I/O: this PR is cheaper (≤~4 ops/cycle vs O(N) writes in feat: persist completed sample outcomes so interrupted runs resume instead of restarting #48), but write cost is negligible next to inference cost.

Worth porting into #48 (its write failures are currently silently swallowed):

• explicit wLog/eLog on partial-store write failures (this PR logs Failed to flush partial benchmark outcomes after abort);
• the scope guard (epochs/range) so stale persisted outcomes from a different run scope are never reused.