[polylane]

The end-to-end check that deletes a projected copy on the hub and expects it back failed on CI even though the copy was back. Its delete helper also waited for the copy to disappear, but the controller rebuilds it within the same second, so that wait never won and the step reported a deadline error. The check now deletes without waiting and confirms a fresh copy appears. Chainsaw's delete operation polls until the resource is gone, and rebuilds land in the same second, so the operation times out although the behavior under test is correct. The step now deletes without waiting and polls for a replacement with a new UID, the same pattern the repository already uses for this race.

flowchart LR
  A["step deletes hub copy"] --> B["controller rebuilds copy<br/>within the same second"]
  B --> C["old: blocking delete waits<br/>for the copy to be absent"]
  C --> D["wait times out<br/>step fails though copy is back"]
  B --> E["new: delete --wait=false,<br/>poll for a new UID"]
  E --> F["step passes when a fresh copy appears"]

Loading

Fix walkthrough

Watch the fix explained (1:01). Recorded for commit 98e4be29e98340d7e1ffd38302dca8ff3ffff785.

Production impact

Reachability e2e test edit touches no deployed resource

The diff only edits a Chainsaw test file under test/e2e/; it changes no operator source and reaches no deployed resource. The repository's only outbound edge is defines to the ghcr.io/datum-cloud/network-services-operator image, whose built artifact this test file does not affect, so no runtime behavior changes on merge.

Observability

• The E2E Tests GitHub Actions workflow is the signal: the vpcendpointslice-reachability-event scenario must stay green.
• The step now prints timed out waiting for the hub copy to come back: original UID <uid>, last observation <uid|deletionTimestamp> on failure, naming the UID and deletion state it saw.

Polylane assessed this pull request at 98e4be2.

Causal chain

• Signal (alert): CI failing: E2E Tests on datum-cloud/network-services-operator@main
• Surfacing site: datum-cloud/network-services-operator at test/e2e/vpcendpointslice-reachability-event/chainsaw-test.yaml#A copy deleted on the hub comes back within seconds
• Mechanism: Chainsaw's delete operation removes the hub EndpointSlice and then waits for it to be NotFound; the cell's vpcendpointslice writeback controller recreates the copy within the same second, so the wait never observes absence and the operation fails with context deadline exceeded even though the copy returned.
• Producer: datum-cloud/network-services-operator, at test/e2e/vpcendpointslice-reachability-event/chainsaw-test.yaml#A copy deleted on the hub comes back within seconds
• Trigger: push to main at a888a28 (E2E Tests run #2225)
• Cadence check: Fails only when the build observes the rebuilt copy before it observes absence: failed once at 09:35:26Z, passed in the prior run at 18:46Z.
• Last observed: 2026-11-11T09:35:26Z
• Blast radius: 0 other resource(s), 0 other tenant(s); data at risk: none named
• Producer evidence:
  • GitHub Actions run 42994595394, job 130435402129: only vpcendpointslice-reachability-event failed; step "A copy deleted on the hub comes back within seconds" logged DELETE OK at 09:35:11.410 then DELETE ERROR 'context deadline exceeded' at 09:35:26.389.
  • The step used chainsaw's blocking delete; kyverno/chainsaw v0.2.15 pkg/engine/operations/delete/operation.go deletes then waitForDeletion polls until NotFound.
  • Catch-block dump at 09:35:26 shows the EndpointSlice exists again with creationTimestamp 2026-11-11T09:35:11Z, uid 36a1bdd4-2059-4493-96a8-736cf9a470af and no deletionTimestamp, so it was rebuilt rather than stuck terminating.
  • test/e2e/networkinterfaceclaim-bound-interface-delete/chainsaw-test.yaml#L132-L148 already documents this class of race and fixes it with 'kubectl delete --wait=false' plus a poll for a replacement with a new UID.
  • Prior main run 42876388883 on 59dac55 passed the same scenario with DELETE OK/DONE in about 28 ms.

1 file changed (+23/-6)

• test/e2e/vpcendpointslice-reachability-event/chainsaw-test.yaml: modified, +23/-6

Repository lint: make lint (declared in CLAUDE.md) did not finish within its time budget, so its verdict is not known; run it before merging.

---

Generated by Polylane. You can ask follow-ups by mentioning @polylane in a comment. If you close this pull request without merging, add a comment that says why. Polylane reads it to make its next fixes better.