Problem
Combined runs of eval files intermittently hang far past the per-test timeout on provider retries: observed test durations of 362s and 582s against a 120s it(...) timeout (vitest reports the test failed at timeout but teardown blocks on the in-flight provider retry loop). Also seen: ProviderRetryError: Delay was aborted. Single-case runs are unaffected.
Impact: full-suite eval runs flake on infrastructure rather than behavior (~1 in 3 combined runs during the #773 verification campaign), which erodes trust in eval signal.
Direction
- Make provider retry/backoff abort promptly when the harness test context is cancelled (propagate AbortSignal through the retry loop).
- Consider a harness-level provider-call deadline below the per-test timeout so a stuck call fails the case with a clear infra-signature error instead of a generic timeout.
- policies/evals.md says to keep eval cases within 30s; the retry loop's worst case should respect that budget.
Problem
Combined runs of eval files intermittently hang far past the per-test timeout on provider retries: observed test durations of 362s and 582s against a 120s
it(...)timeout (vitest reports the test failed at timeout but teardown blocks on the in-flight provider retry loop). Also seen:ProviderRetryError: Delay was aborted. Single-case runs are unaffected.Impact: full-suite eval runs flake on infrastructure rather than behavior (~1 in 3 combined runs during the #773 verification campaign), which erodes trust in eval signal.
Direction