Repository navigation
Conversation
miyunari
force-pushed
the
fix/162-revoke-readiness-on-failure
branch
from
November 4, 2026 16:39
9596edd to
7765e42
Compare
In continuous mode the validation agent only ever set `ready = true` and never revoked it, so a model that started failing re-validation after the initial success kept serving traffic indefinitely. A tampered or swapped model was detected and logged, but the pod stayed in the Service endpoints. Add a `failureThreshold` field to `continuousValidation` that controls how many consecutive periodic failures are tolerated before the agent marks itself not-ready, which removes the pod from the Service until a later validation succeeds again. The webhook translates the field into an internal `--max-consecutive-failures` flag on the validation agent; users only ever interact with the CRD field. - `failureThreshold: 0` keeps the previous fail-open behavior (never revoke). - `failureThreshold: 1` (default) is fail-closed: a single failure removes the pod from rotation. - Higher values tolerate transient blips before revoking. The periodic loop is extracted from main() into `continuousValidationLoop` plus a pure `handleValidationResult` helper so the readiness logic is unit tested (the package had no tests before). BREAKING: the default of 1 means an existing continuous-validation deployment whose model starts failing re-validation will now be removed from rotation, where before it stayed ready. Set `failureThreshold: 0` to keep the old behavior. Fixes #162 Signed-off-by: Nina Bongartz <pninak@web.de>
miyunari
force-pushed
the
fix/162-revoke-readiness-on-failure
branch
from
November 6, 2026 19:48
7765e42 to
a4f00ec
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #162. In continuous mode the validation agent only ever set
ready = trueand had no path back to not-ready, so once a pod became ready it kept serving traffic regardless of later periodic validation results. A model that was tampered with or swapped on the volume after startup was detected and logged on the next periodic check — but the pod stayed in the Service endpoints. Startup is fail-closed; steady state was fail-open forever.What changed
continuousValidation.failureThreshold— the number of consecutive failed periodic re-validations tolerated before the pod is marked not-ready and removed from the Service endpoints. A later successful validation restores readiness.--max-consecutive-failuresflag on the validation agent (both native-sidecar and legacy-sidecar paths). Users only interact with the CRD field; the flag is an implementation detail.main()intocontinuousValidationLoop+ a purehandleValidationResulthelper so the readiness logic is unit-tested —cmd/validation-agent/had no tests before.GetConfigHashnow includesfailureThresholdso configuration-drift detection notices changes.examples/continuous-validation.yamlupdated.Semantics
failureThreshold01(default)N > 1The default of
1is fail-closed. An existing continuous-validation deployment whose model starts failing re-validation (including a transient failure, e.g. a brief volume/network hiccup) will now be removed from rotation, where previously it stayed ready. Operators who want the old behavior can setfailureThreshold: 0; those who want to tolerate blips can set a higher value.This default was chosen deliberately as security-first — the whole point of continuous validation is to stop serving a model that no longer validates.
Testing
make test— all packages pass (new unit tests for the readiness logic + loop, new webhook arg assertions, race-clean).make lint— no new findings beyond the repo's pre-existing baseline.Notes
The issue author (who proposed this direction) suggested two separate flags (
--fail-closed+--max-consecutive-failures). This PR collapses them into a singlefailureThresholdinteger where0= off, which covers both cases with one user-facing knob.🤖 Generated with Claude Code