Skip to content

A FluxInstance kustomize patch that matches no target is silently discarded #2996

Closed

Description

@devantler

🤖 Generated by the Agentic Engineer

Evidence

A FluxInstance.spec.kustomize.patches entry whose target matches no resource is a silent
no-op
: the patch is simply not applied, with no error, no warning, and no change to the
FluxInstance's Ready condition.

Measured 2026-08-06 with kubectl kustomize over a minimal OCIRepository shaped like the live
flux-system one, changing only the patch target:

Patch target verify: present in rendered output exit
kind: OCIRepository, name: flux-system (correct) yes 0
kind: OCIRepository, name: NOT-THE-ROOT-SOURCE no 0
kind: GitRepository, name: flux-system (wrong kind) no 0

A true baseline with no patches block rendered no verify:, and a post-revert control rendered it
again — so the arms differ only in the target selector.

Affected audience and impact

Platform maintainers, and any agent doing reliability or supply-chain work through this file.
k8s/providers/hetzner/infrastructure/controllers/flux-instance/flux-instance.yaml currently carries
6 patches, and nothing in the repo or CI asserts that any of them still matches a resource:

  • three replicas: 2 patches that provide the HA active/standby for kustomize-, helm- and
    notification-controller,
  • a topologySpreadConstraints patch that exists specifically so one bad worker cannot decapitate
    reconciliation (the 2026-05-28 Cilium incident recorded in that file's own comments),
  • an ndots: 1 patch that suppresses thousands of NXDOMAIN queries per hour,
  • a --requeue-dependency=5s arg patch.

If any target stops matching — a controller renamed upstream, a component dropped from
spec.components, a label changed in a distribution bump, a typo during an edit — the corresponding
tuning silently disappears and the cluster keeps reporting healthy. The file's own history shows
this class is live rather than theoretical: an earlier label-selector approach to the replicas patch
was silently resolved the other way by flux-operator, leaving source-controller at 2 replicas, and
that was only caught by a forced reconcile and a live read.

This becomes security-critical once #2922 lands, because the cosign verify block on the root
OCIRepository will be delivered through this exact mechanism: a mistargeted patch would leave the
root GitOps source unverified while every signal — green CI, a clean FluxInstance reconcile, a
successful deploy — reads correct. That is precisely the dead control class #2627 exists to
eliminate, and the same "a present control is not a working control" problem #2946 is chartered on.

Expected behaviour

A patch in the FluxInstance that matches nothing fails loudly — at the latest in CI, ideally before
merge — rather than being silently discarded.

Acceptance criteria

  • A check enumerates every spec.kustomize.patches[].target in the FluxInstance manifests and
    fails when a target matches no resource that the operator will generate for that instance.
  • The check runs on the same CI path that already validates these manifests, and is registered on
    the path filter covering k8s/providers/*/infrastructure/controllers/flux-instance/**.
  • RED/GREEN proof: the check fails on a deliberately mistargeted patch and passes on the current
    6 patches unmodified. An ablation per target kind (name-mismatch and kind-mismatch) is part of
    the deliverable, since both were shown above to exit 0.
  • The failure message names the offending target and what it failed to match, so the fix is
    obvious from the log.

Notes on approach

Rendering the full operator output offline would be the strongest oracle (flux operator build
exists for this), but the flux CLI is not installed on the CI runners or the agent host today —
verified absent. A first increment that parses the manifests and checks each target against the
declared spec.components plus the known operator-generated resource set (the sync OCIRepository
and Kustomization, both named from spec.sync.name/namespace) needs no new tooling and catches the
realistic cases above. Adding the CLI can be a later, separate step.

Size

Small. One check plus its RED/GREEN ablations; no cluster access required.

Part of #2627.

Activity

  1. added theissue type on Aug 6, 2026
  2. coderabbitai commented on Aug 6, 2026

    @coderabbitai
    Contributor
    🔗 Related PRs

    #2319 - feat(security): cosign-verify the infrastructure Flux OCI artifact [merged]
    #2491 - ci: validate rendered talos patches on the PR event [merged]
    #2635 - fix(deploy): reboot stale nodes so containerd adopts refreshed GHCR auth [merged]
    #2919 - fix(security): move Flux cosign verify to the path KSail reads [merged]
    #2944 - fix(ci): assert Flux verification is in effect, not merely configured [merged]


    📝 Issue Planner

    Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.

    • Create Plan

    🧪 Issue enrichment is currently in open beta.

    You can configure auto-planning by selecting labels in the issue_enrichment configuration.

    To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

    issue_enrichment:
      auto_enrich:
        enabled: false

    💬 Have feedback or questions? Drop into our discord!

  3. moved this to 📥 Backlog in 🌊 Project Boardon Aug 6, 2026
  4. devantler commented on Aug 6, 2026

    @devantler
    ContributorAuthor

    🤖 Generated by the Agentic Engineer

    Scope update: one patch is now guarded, the general case is not

    #3001 adds a 7th kustomize patch to the same FluxInstance — the cosign spec.verify on the root
    OCIRepository — and guards that one target specifically in scripts/validate-flux-verify. So the
    count this issue reasons about has moved: 7 patches, 1 guarded, 6 unguarded.

    That check is deliberately narrow. It knows one target (kind: OCIRepository, name: flux-system) and
    asserts the verify patch points at it; it says nothing about the controller-args, three replicas: 2,
    topologySpread or ndots patches. This issue's general case stands.

    One measurement from building it, in case it saves a step here: the pipeline never renders these
    patches at all.
    kubectl kustomize copies the patch list into the FluxInstance verbatim and exits 0
    whatever the target says — flux-operator applies them later, in the cluster. So a general guard cannot
    work by rendering and diffing; it has to either (a) check each target against the resources the
    operator is known to generate, or (b) read the live FluxInstance's applied patch count back after a
    deploy. (a) is static and catches it pre-merge; (b) catches drift the repo cannot see. They answer
    different questions.

    Worth noting the two are not equally urgent: an unguarded replicas: 2 patch that silently misses
    degrades resilience and is visible in kubectl get deploy. An unguarded security patch that
    silently misses looks identical to one that works. That asymmetry is why the verify patch got its
    guard first rather than waiting for the general one.

  5. self-assigned this
    on Sep 19, 2026
  6. added a commit that references this issue on Sep 19, 2026
  7. moved this from 📥 Backlog to 👀 In Review in 🌊 Project Boardon Sep 19, 2026
  8. moved this from 👀 In Review to ✅ Done in 🌊 Project Boardon Sep 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

Fields

Priority

None yet

Effort

None yet

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions