Repository navigation
Edges serve stale WAF config for an unmeasured length of time #266
Description
Summary
TrafficProtectionPolicy spec changes made on the upstream control plane take an unbounded, unobservable amount of time to reach the edge data plane. During that window an edge serves the stale (previously-valid-or-invalid) WAF config, with no signal — on the cluster or in metrics — that a given edge is behind the control plane for a given policy.
This surfaced while investigating #242: a tenant corrected an inverted-paranoia TPP upstream, but the edge continued serving the old inverted config (which 500s 100% of the route via CRS 901500 fail-closed) for a window before converging.
Evidence
Observed on prod us-central-1-alice, policy ns-c5f1cf4e…/kev1n-proxy:
| Source | generation | paranoiaLevels |
|---|---|---|
| upstream (karmada control plane) | 8 | {detection: 2, blocking: 2} (valid — tenant-corrected) |
edge us-central-1-alice (first read) |
7 | {detection: 1, blocking: 2} (inverted — 500s all traffic) |
edge us-central-1-alice (later read) |
8 | {detection: 2, blocking: 2} (converged) |
So the projection is eventually consistent — it did converge — but the edge served the broken inverted config for the lag window, and nothing exposed that the edge was a generation behind. This is not a stuck/broken sync; it's propagation latency with zero visibility.
Why it matters
- WAF config is failure-sensitive: an inverted-paranoia policy 500s 100% of a route (TrafficProtectionPolicy Enforce mode: 500 with truncated/empty body on all traffic #242, Mechanism A). Any remediation applied upstream has a silent tail where the edge is still broken.
- With ~15+
*-aliceedges and 500+ project control planes, there is no per-edge/per-policy way to answer "is this edge serving the current generation of this TPP?" - The same latency applies to any TPP change (mode flips Observe↔Enforce, ruleSet edits), not just paranoia corrections — cf. fix: retrigger EG translation on TPP spec changes #255 (retrigger EG translation on TPP spec change).
Asks
- Observability: expose the applied TPP generation / resourceVersion per edge (metric or status), so control-plane vs edge skew is queryable. A
spec.observedGeneration-style signal on the edge-side attachment, or a metric labeled by policy + edge, would let us alert on lag. - Bound / quantify the latency: measure typical and worst-case upstream→edge propagation time for a TPP spec change; decide whether it needs an SLO.
- (stretch) Consider surfacing staleness at the control plane so a tenant/operator sees "not yet applied on N edges" after an edit.
Not this issue
- The upstream admission gate now rejects new inverted TPPs (
self.detection >= self.blocking, verified live on karmada) — recurrence prevention is tracked separately. - Already-persisted inverted TPPs / their neutralization via fix: flag inverted TrafficProtectionPolicy paranoia levels via status condition #252 — tracked in TrafficProtectionPolicy Enforce mode: 500 with truncated/empty body on all traffic #242.
Metadata
Metadata
Assignees
Labels
Type
Fields
Priority
Activity
ecv commentedon Jul 14, 2026
Proposed regression-coverage acceptance
Propagation-staleness is invisible today and the parity gate (#225) only reads a single point in time. With the prod-fidelity env live (#275/#276) and policy delivery landing in #227, attach explicit coverage:
test/e2e/scenario applies a TPP change upstream, then asserts the edge converges to the new generation and that convergence is observable — aspec.observedGeneration-style signal or per-edge/per-policy metric — not inferred from a request eventually succeeding.Depends on the observability signal from this issue's Ask #1 existing first — the test asserts against it.
mattdjenkinson commentedon Aug 4, 2026
Implementation in progress across NSO + infra + portal + datumctl for tenant-visible WAF readiness.
What lands
ProgrammedonTrafficProtectionPolicy(keepAccepted= attach-only). True only when all edges have applied the current generation; message carriesM/N. Reasons:Programmed/Pending/PartialFailure.nso_extension_tpp_{cache,applied}_generationgauges andgenerationin datum-gateway route metadata.apps/.../federated/tpp-aggregation, wired viaclusters/staging/infrastructure/network-services-operator-tpp-aggregation.yaml).status.ancestors[].conditions.Explicit follow-ups
Protecting/ synthetic 403) — deferred (cloud-portal#1336 family).waf-configuratione2e to gate onProgrammed.feat: Show WAF Protected/Monitoring from Programmed (#1407)
mattdjenkinson commentedon Aug 5, 2026
Ask #2 (measure → SLO): samples justify a 30s budget.
nowaf-testhelp-d9c5g5portfolio-lmpn0uwaf-test-3gsu2vAlert PR: #337 —
TrafficProtectionPolicyApplyLagSLOViolationwhennso_extension_tpp_cache_generationstays ahead ofnso_extension_tpp_applied_generationfor 30s (~10× measured lag).ecv commentedon Aug 10, 2026
Went through #331 and #335 against what this issue actually asked for, and checked prod rather than trusting the release notes. Leaving this open — a good chunk landed, but not the part the title is about.
What is genuinely done, and verified live: every TrafficProtectionPolicy now carries an ancestor
Programmedcondition that the edge extension server stamps only after a successful WAF apply, Karmada AND-aggregates it across members withM/N edges programmed generation Nin the message, and the upstream reconciler mirrors it back withPending/PartialFailure/Programmed. The aggregation ResourceInterpreterCustomization is applied and healthy on the prod Karmada hub, all seventeen edge clusters are on v0.25.2, andnso_extension_tpp_cache_generation/nso_extension_tpp_applied_generationare reporting per edge and per policy across the whole fleet. #335 makes idle proxies backfill the condition instead of waiting for someone to touch the spec. That covers Ask #1 in its status form, and it covers the stretch Ask #3 — a tenant can now see "not yet applied on N edges" after an edit.What is not done is Ask #2, and the gap is more than "the alert PR hasn't merged yet".
The measurement happened and 30s was chosen as the budget, but #337 is still open and there is no
TrafficProtectionPolicy*alert in prod today, so nothing would page. More importantly, the two gauges the alert watches are both edge-local: cache generation is the generation of the projected object sitting in the edge extension server's informer, and applied generation is what it last pushed into Envoy. The skew between them is the edge-internal translate leg. It is not the control-plane-to-edge leg, which is the leg this issue was opened about — in the original incident the hub was at generation 8 while the edge object was still at 7, and on that edge both gauges would have read 7 with no skew at all. Consistent with that,cache > appliedhas been flat zero across all seventeen edges for the entire five days since rollout. So the alert as drafted would not have fired for the event that opened this issue.The upstream
Programmedcondition does close that loop, because the controller compares the aggregated observedGeneration against the upstream generation and reports Pending while the edge is behind. But that is a boolean plus a count, not a duration — there is still no series that answers "how long has this policy been behind on this edge", which is what an SLO needs to be measurable rather than just declared. One thing I could not convince myself of and would like a second pair of eyes on: that comparison is between the aggregated observedGeneration reported off the downstream object and the upstream object's generation, and I do not think anything keeps those two generation counters in step, since each apiserver assigns its own. If they can drift the condition could read Pending or Programmed for the wrong reason. I have not reproduced that, so treat it as a question rather than a finding.And the half in the title — the edge serving stale config during the window — is untouched.
Programmedreports the window, it does not shorten or gate it. A tenant who corrects a broken policy still gets an unbounded tail of the old config being served, and infra#3740 shows the same shape on the create path where enforcement misses the 30s budget roughly one run in three. Nothing shipped changes that behavior.The regression-coverage boxes I proposed above are also still unticked. #331 added a
Programmed=Trueassertion to the enforce e2e, which is real progress, but it is a create-path assertion — there is no scenario that changes a TPP upstream and asserts the edge converges to the new generation, none that corrects an inverted-paranoia policy and asserts the 500s stop exactly when the corrected generation is reported, and nothing asserting the skew is queryable during the window.Suggested split, if that seems right: keep #266 for the lag half — land #337 or a successor that watches the control-plane-to-edge leg, add a duration signal, and attach the three coverage scenarios — and open a separate issue for the behavioral half, that an edge should not serve stale enforce config for an unbounded window, linked to infra#3740 as the create-path sibling. Happy to be talked out of the split if you would rather keep it as one thread.
#331
#335
#337
https://github.com/datum-cloud/infra/issues/3740
[-]Edge serves stale TrafficProtectionPolicy config during upstream→edge propagation with no lag visibility[/-][+]Edges serve stale WAF config for an unmeasured length of time[/+]vvoytovych commentedon Sep 30, 2026
I looked at why a TPP edit can wait for an unrelated change before an edge applies it. This is about the three edges that serve traffic today, on Envoy Gateway v1.7.4 and NSO v0.31.0.
What I found
networking.datumapis.com/tpp-generation-<name>on the Gateway (tpp.go).gateway.envoyproxy.io/annotations into its xDS model (listener.go, route.go).main. I can share it.networking.datumapis.com/connector-ready-generationon the Gateway (retrigger.go). So a Connector going online or offline also waits for an unrelated rebuild. The Programmed backfill uses the TPP annotation (tpp.go), so it waits too.cache > appliedstays at zero, and the fix: Alert when TPP edge apply lag exceeds 30s #337 alert would not fire for a missed rebuild.nso_extension_hook_calls_totalover both pods of each edge, scraped every 30 s.This corrects the report I sent in July, which said the defect "looks already fixed, just not deployed". The fix was deployed, and it cannot work.
What I plan
policyResources, and the two retriggers should not.mergeGateways, and so remove that trigger too.Let me know if you have concerns. Thanks.
mattdjenkinson commentedon Oct 1, 2026
Hey @vvoytovych this looks good to me, do you want to assign this issue to yourself?
vvoytovych commentedon Oct 5, 2026
Assigned. Thanks @mattdjenkinson
vvoytovych commentedon Oct 5, 2026
I tested this on a running controller: Envoy Gateway v1.9.2 and NSO v0.31.0, with the production config and the us-east-1 objects. For the status tests, Envoy Gateway had only the edge's rights on our API group. We are splitting this issue: the WAF stays here, and the Connector half moves to #451.
What the test showed
policyResourcessetting in infra (helm-release.yaml).extensionManager.resourcesand the two retrigger annotations, never reach the hook (README.md).What we fix here, for the WAF
Programmedcan report success before every edge has the new generation.Programmedstamp costs one more full rebuild per edit.Programmedis reported.The Connector half, for #451
Let me know if you have concerns. Thanks.
vvoytovych commentedon Oct 14, 2026
Closing: a WAF policy edit is now timed to every edge, and a policy that stays behind raises an alert.
Programmedis set per target. It is True only when every edge that holds the policy has built its current generation: #581, #593, datum-cloud/infra#6900, #616. The same changes answer the generation-counter question above (#545).waf-mode-flipswitches a live policy to Enforce in every scheduled run and waits forProgrammedon the new generation (the wait, datum-cloud/infra#6951). It ran in 7 scheduled production runs from 2026-10-13 16:20Z to 2026-10-14 04:21Z. The edit took 4.3 s at the least, 7.3 s at the median and 19.2 s at the most:chainsaw_test_step_duration_seconds{test="waf-mode-flip",step="flip-to-enforce",environment="prod"}.TrafficProtectionPolicyNotProgrammedSLOViolation(#620), atwarning: a target that is Accepted and not Programmed for 300 s. It replaces #337, whose two gauges stay equal on an edge that misses a rebuild.The three regression checks proposed above:
waf-mode-flipabove. A True left over from the previous generation never passes it.test: Assert TPP Programmed tracks generation bumps #336 asks the same in NSO's own tests.
waf-inverted-paranoia-rejectchecks it in every run. So no inverted policy can exist to correct.ProgrammedFalse for the new generation until every edge has built it. An edit that never gets there failswaf-mode-flipafter 12 minutes.An edge still serves the previous config until it builds the new one. That window is now measured and alerted, not removed.
Follow-ups: #547 removes the two edge gauges, and datum-cloud/infra#6768 alerts when Envoy Gateway fails to turn a change into edge config.