Skip to content

Gracefully handle load driver iteration failures - #461

Draft
THardy98 wants to merge 4 commits into
mainfrom
fix/graceful-driver-failure
Draft

Gracefully handle load driver iteration failures#461
THardy98 wants to merge 4 commits into
mainfrom
fix/graceful-driver-failure

Conversation

@THardy98

@THardy98 THardy98 commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

What changed

Omes currently stops a load run when one iteration fails. In Kubernetes this can make a transient workflow error terminate and immediately restart the load-driver container, instead of maintaining load for the configured duration.

This change makes CLI-driven runs continue after terminal iteration failures by default. It records each completed logical iteration, logs a final success/failure and throughput summary, and treats a run that completed with tolerated failures as a degraded but successful CLI run. Library callers still receive a typed IterationFailuresError, preserving their ability to apply a stricter policy.

Operators that require the existing assertion-style behavior can use --iteration-failure-policy=fail-fast. Retry behavior is unchanged and remains controlled independently by --max-iteration-attempts.

Observability

The load-driver Prometheus endpoint now exposes omes_iterations_total, labeled by scenario, outcome, and normalized status code. Full wrapped errors remain in structured per-iteration logs without introducing unbounded metric labels.

Validation

  • Focused tests pass for continuation, fail-fast behavior, cancellation, error classification, CLI policy, and Prometheus labels.
  • The focused suite passes under the race detector.
  • golangci-lint --new-from-rev=HEAD reports no new issues.
  • The broader affected-package run reaches the pre-existing TestKitchenSink external dev-server clone and times out locally; the remaining affected packages pass.

Fixes #453

@THardy98
THardy98 force-pushed the fix/graceful-driver-failure branch from 86955e9 to 7a4c0d5 Compare September 3, 2026 16:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature Request] Gracefully handle load driver failures

1 participant