[polylane]

The e2e environment runs the cell controller manager on the upstream cluster but installed only the control-plane custom resources there, so a kind the manager watches was missing. The manager exited and restarted every two minutes, and while its watches were down a resource deletion went unseen, so an end-to-end check failed intermittently. The environment now installs the full cell custom resource set before the cell starts, so the manager keeps running and deleted resources are collected promptly.

flowchart LR
  A["test-infra:up applies config/e2e to the upstream cell"] --> B["cell CRD set config/crd/downstream is never applied there"]
  B --> C["cell-manager watch for NetworkLoadBalancerBackendHealth cannot sync"]
  C --> D["controller Start fails after the 2-minute cache-sync timeout"]
  D --> E["manager exits code 1 and restarts"]
  E --> F["write-back watch is down during the restart window"]
  G["claim delete removes the cell NetworkInterface"] --> H["delete event is lost while the watch is down"]
  F --> H
  H --> I["hub copy not collected before the test deadline"]
  K["fix: cell-crds applies config/crd/downstream to the upstream"] -.-> B

Loading

Detection evidence

Affected: int_17ad4b457001o98datdn4tkm

Why this fix

The write-back controller collects a published NetworkInterface only from the source delete event or a ten-minute sweep, and the test waits 120s. The controller manager was restarting every two minutes because the upstream test cluster never served NetworkLoadBalancerBackendHealth, a kind its controller watches: this harness installs the cell custom resource set on the hub and the downstream cluster but never on the cluster that plays the cell. Installing the same set on the upstream cell stops the restart cycle, so the delete is delivered and the copy collected promptly. The ten-minute sweep stays as the backstop for a genuinely missed event.

Production impact: E2E cell CRD install only adds manifests to a local test cluster

The only changed path, Taskfile.test-infra.yml, builds a local kind cluster and applies Kubernetes manifests to it during CI; it does not push an image or deploy to any connected production cluster. I checked the repository's one outbound deploy edge, the image repository, and this diff does not alter the image or its deployment.

Fix walkthrough

Watch the fix explained (1:27). Recorded for commit 48296a7cce313993dbde25a2f17928e44440e745.

Checked and not affected (1)

• datum-cloud/network-services-operator

Observability

• Existing signal: the cell-controller-manager pod restart count and the controller-runtime.source.Kind if kind is a CRD, it should be installed before calling Start error, which this change removes.
• Existing signal: the write-back controller's collecting a published interface with no interface behind it sweep log.
• No new instrumentation; the change is test-environment configuration.

Polylane assessed this pull request at 48296a7.

Causal chain

• Signal (alert): Intermittent ns-operator e2e: published NetworkInterface not collected within 120s after the claim is deleted
• Surfacing site: network-services-operator E2E Tests (GitHub Actions run 40994545976, job 123864506923) at test/e2e/networkinterface-project-visibility/chainsaw-test.yaml#Deleting the interface collects the published copy
• Mechanism: The claim delete at 03:15:54 removed the cell's NetworkInterface. The write-back controller collects the hub copy only from that delete event or a 10-minute sweep; the cell-controller-manager was in a restart cycle then, so the event was never delivered and the copy persisted past the test's 120s assertion (timed out 03:17:54).
• Producer: cell-controller-manager (network-services-operator cell-manager) in the e2e upstream cell, at datum-cloud/network-services-operator:internal/cmd/cell/cell.go#controller registrations and Taskfile.test-infra.yml#prepare-upstream
• Trigger: task test-infra:up deploys the upstream cell without applying config/crd/downstream, the set production's platform installs on every cell.
• Blast radius: unknown other resource(s), unknown other tenant(s); data at risk: No data loss. A published NetworkInterface copy outlives its deleted source for up to the 10-minute sweep whenever a delete event lands in a manager restart window.
• Producer evidence:
  • Run #2048 failure dump: cell-controller-manager Restart Count 5; last terminated instance 03:21:07-03:23:07 (the two-minute CacheSyncTimeout); manager error if kind is a CRD, it should be installed before calling Start kind NetworkLoadBalancerBackendHealth.networking.datumapis.com no matches for kind ... v1alpha.
  • internal/cmd/cell/cell.go registers networkloadbalancerbackendhealth unconditionally on mgr.GetLocalManager().
  • kustomize build config/e2e (applied to the upstream) emits no networkloadbalancerbackendhealths CRD; config/crd/downstream does; the apply log shows it created only for the downstream cluster and the Karmada hub.

1 file changed (+13/-0)

• Taskfile.test-infra.yml: modified, +13/-0

Repository lint: make lint (declared in CLAUDE.md) did not finish within its time budget, so its verdict is not known; run it before merging.

---

Generated by Polylane. You can ask follow-ups by mentioning @polylane in a comment.

[ecv]

Approve.

Test harness only: it applies the same config/crd/downstream set to the kind cluster that plays the cell, so the cell controller manager's NetworkLoadBalancerBackendHealth watch can sync and the manager stops exiting every two minutes. Nothing that ships changes.

• cell-crds runs before prepare-upstream, so any CRD present in both the cell set and the control-plane set is applied from the cell set first and then overwritten by the control-plane copy, which is the version the upstream manager expects. Both come from this repo at the same commit, so they should agree anyway.
• The root cause is the right one: the delete event was lost while the watch was down, and the ten-minute sweep is longer than the test's 120s wait. Fixing the restart rather than lengthening the wait keeps the test honest.

Follow-ups:

• The manager exiting on a kind its cluster does not serve would do the same on a real cell missing that CRD. A startup check that names the missing kind would make that obvious instead of a two-minute restart loop.