Skip to content

Fail loudly when a workflow replays with a stale stored version - #30

Draft
sndre wants to merge 1 commit into
mainfrom
feat/version-gate-enforcement
Draft

sndre wants to merge 1 commit into
mainfrom
feat/version-gate-enforcement

Conversation

@sndre

@sndre sndre commented Sep 15, 2026

Copy link
Copy Markdown
Contributor

Goal

Workflow.version(changeId, minVersion, maxVersion) validated only its arguments; the version an
instance actually persisted was never checked against the range the current code declares. If an
author raised minVersion and pruned an old version's branch — or a deploy rolled maxVersion
back — while instances holding that version were still in flight, those instances replayed with the
stale value and silently took whatever the remaining code did with it. No error, no metric, no log.

This makes that case fail loudly instead.

What changed

  • New internal/VersionGateEnforcer, applied to the value version() hands back — the only place
    the persisted version is observable. On replay the named checkpoint is returned without
    re-invoking the backing action (ActionExecutor.executeAction), so a check inside
    VersionGateHelpers.getVersion would never fire.
  • When the stored version is outside [minVersion, maxVersion] the enforcer always logs at
    ERROR and emits the versionGate.storedVersionOutOfRange metric (tags: changeId, direction,
    enforced). When FeatureGate.Keys.ENFORCE_VERSION_GATE_MIN_VERSION is enabled for the app it
    additionally throws a NonRetryableError, moving the instance to a terminal ERROR state
    with no retry burn.
  • The enforcer is injected via SkipperInternalDeps, mirroring
    actionExecutor/skipperEngine/contextPropagator.

Key decisions

  • Error type = NonRetryableError → terminal ERROR without consuming the retry budget. A
    stale version is permanent, so a retryable error would burn budget for nothing.
  • Both bounds enforced — stored < minVersion (pruned branch) and stored > maxVersion
    (rollback, the more common case).
  • Rollout = opt-in gate + always-on metric/log. Default is detect-only, so upgrading is
    behavior-preserving; the metric makes a premature minVersion raise discoverable before the hard
    failure is switched on per app.

Testing

  • VersionGateEnforcerTest: boundaries, both directions, both gate states, error type, message,
    metric tags.
  • BaseWorkflowIntegTest version-gate suite: below-min and above-max replay → terminal ERROR;
    in-range replay → normal completion; gate-off → detect-only.
  • ./gradlew :test version-gate suites green (VersionGateEnforcerTest 6/6, WorkflowVersionGateTest
    3/3, SqliteWorkflowIntegTest TestVersionGate 7/7), stable across reruns.

Docs

versioning.md now documents that removing a version's path is detectable rather than silent.

🤖 Generated with Claude Code

Workflow.version(changeId, minVersion, maxVersion) validated only its
arguments; the version an instance actually persisted was never checked
against the range the current code declares. If an author raised minVersion
and pruned an old version's branch — or a deploy rolled maxVersion back —
while instances holding that version were still in flight, those instances
replayed with the stale value and silently took whatever the remaining code
did with it. No error, no metric, no log.

Add VersionGateEnforcer, applied to the value version() hands back — the only
place the persisted version is observable. On replay the named checkpoint is
returned without re-invoking the backing action (ActionExecutor.executeAction),
so a check inside VersionGateHelpers.getVersion would never fire. When the
stored version is outside [minVersion, maxVersion] the enforcer always logs at
ERROR and emits the versionGate.storedVersionOutOfRange metric; when the
enforce_version_gate_min_version feature gate is enabled it additionally fails
the instance with a NonRetryableError (terminal ERROR state, no retry burn).

- Both bounds are enforced: stored < minVersion (pruned branch) and
  stored > maxVersion (rollback).
- Enforcement is opt-in, so upgrading is behavior-preserving; the metric and
  log make a premature minVersion raise discoverable before the throw is on.

The enforcer is injected via SkipperInternalDeps, mirroring
actionExecutor/skipperEngine/contextPropagator.

Tests: VersionGateEnforcerTest covers the boundaries, both directions, both
gate states, error type, message, and metric tags; BaseWorkflowIntegTest's
version-gate suite adds below-min and above-max replay -> terminal ERROR,
in-range replay -> normal completion, and gate-off -> detect-only. Docs:
versioning.md now states that removing a version's path is detectable rather
than silent.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant