Repository navigation
security(longhorn): clear the pre-opt-in C-0211 residual in longhorn-system without detaching volumes unintentionally #3918
Description
🤖 Generated by the Agentic Engineer
Evidence
longhorn-system is opted in to pod-security.devantler.tech/baseline-context, and the mutation
works — but 26 of its 41 pods still carry neither fsGroupChangePolicy nor seLinuxOptions.
Measured against live prod specs 2026-09-20, the split is temporal and completely clean:
- every clean pod was created on or after
2026-09-06T20:46:02Z - every residual pod was created on or before
2026-08-26T13:14:47Z
#3636 landed the opt-in on 2026-09-06. The rule matches CREATE only, so this is the documented
behaviour working as designed: pods created after the opt-in carry both fields, and pods created
before it keep their original spec until something recreates them.
Nothing in longhorn-system is going to recreate them on its own:
12x DaemonSet/engine-image-ei-* three engine versions, oldest pod 2026-06-16
4x DaemonSet/longhorn-csi-plugin
4x InstanceManager/instance-manager-*
6x ReplicaSet/csi-{attacher,provisioner,resizer}
Longhorn keeps an engine-image DaemonSet per engine version for as long as any volume references
that version, and an InstanceManager per node for as long as volumes are attached there.
Audience and problem
This is the last chunk of the C-0211 residual in an already opted-in namespace, so it will not
close by itself and it is invisible in the rollout state — the namespace looks done.
It is deliberately not folded into #3239's per-namespace opt-in work, because the risk class is
different. Opting a namespace in is a posture change with no runtime effect on running pods. Clearing
this residual means recreating storage-plane pods, and recreating an InstanceManager requires
detaching the volumes it serves. That is a storage availability decision, and it should not ride
along inside a security sweep.
Expected behaviour
Every pod in longhorn-system carries both fields, reached by a sequenced roll that never detaches a
volume without intent.
Acceptance criteria
- The three workload classes are sequenced by disruption, least first: the CSI sidecar
ReplicaSets andlonghorn-csi-pluginDaemonSet roll normally; theengine-imageDaemonSets
are checked for whether any volume still references each engine version before touching them;
theInstanceManagerpods are handled last and explicitly, against attachment state. - Any
engine-imageDaemonSet whose version no longer backs a volume is retired rather than
rolled — that removes the pods instead of recreating them. - Re-measured afterwards from live pod specs: 0 of the namespace's pods missing either field.
- No volume left detached or degraded, verified after the roll rather than assumed.
Not in scope
The kube-system opt-in (34 pods missing fsGroupChangePolicy, 26 missing seLinuxOptions, not yet
opted in) is the larger remaining population and belongs to #3239's ordinary per-namespace rollout.
The pod-versus-workload-spec distinction on #3522 is also separate and unaffected: clean pods do not
by themselves move the Kubescape C-0211 score, because Kubescape reads the stored workload spec.
Size
Small in diff, medium in care — the sequencing and the post-roll verification are the work.
Part of #3239
Metadata
Metadata
Assignees
Labels
Type
Fields
Projects
- Status
Activity
devantler commentedon Sep 20, 2026
Re-measured scope for this issue (2026-09-20)
While re-measuring C-0211's residual for #3938 / PR #3939, this namespace's share was measured directly against the live PROD stored controller specs — the surface Kubescape reads.
longhorn-system is 8 of 12 scanned workloads, and it is now the largest single contributor to a portfolio-wide residual of 14 of 39 (observability 6; flux-system 0 of 5 and velero 0 of 2 are fully closed). By kind, this namespace's residual is 4 Deployments and 4 DaemonSets — no CronJobs and no StatefulSets.
Two corrections from that measurement change how this work should be checked, both recorded in full on #3239:
seLinuxOptionsobject, not a populatedlevel. No workload in the population setsseLinuxOptions.levelat all, and every object present is empty{}. Alevel-based acceptance test reports every workload failing and would never register this issue's progress.add-baseline-context-optin-controllers*is the family that reaches this namespace's 8.Also worth carrying into the remediation: this namespace's residual is bounded by pod age, not by a policy defect — the mutation rules are
CREATE-only by design, so what remains is workloads not recreated since the 2026-09-06 opt-in. Separation was perfect on that boundary when last checked, so recreation is the mechanism rather than a rule fix.The per-workload detail is deliberately not reproduced here; it is derivable from the stored specs with the predicates described on #3239.
devantler commentedon Sep 20, 2026
Independently re-measured against live prod, and the split is confirmed — with two corrections
Measured 2026-09-20T21:20Z against the live PROD stored controller specs (
kubectl get deploy,daemonset -n longhorn-system), independently of the 18:24Z measurement. The residual is 8 of 11 controllers, and it lands exactly on the disruption classes this issue defines:csi-attacher,csi-provisioner,csi-resizer,csi-snapshotterengine-image-ei-493e04e7,engine-image-ei-75a03ec3,engine-image-ei-a4d05f02,longhorn-csi-pluginlonghorn-driver-deployer,longhorn-ui,longhorn-managerThat corroborates "4 Deployments and 4 DaemonSets — no CronJobs and no StatefulSets".
Correction 1 — the body's workload enumeration omits
csi-snapshotterThe evidence block lists the sidecar ReplicaSets as
6x ReplicaSet/csi-{attacher,provisioner,resizer}. There are four residual CSI sidecar Deployments, not three:csi-snapshotteris also missing both fields. Anything sequencing off the body's list would leave a workload behind and then fail acceptance criterion 3.Correction 2 — test
seLinuxOptionsper CONTAINER, not on the podThe 18:24Z comment already corrected
level-vs-object. There is a second axis, and it fails in the same direction:seLinuxOptionsis absent at pod level on all 11 controllers, including the 3 that are fully clean. The baseline supplies it per container. So a pod-level predicate reports 11 of 11 failing and would never register progress either.The predicate that discriminates:
Under it, 8 fail and 3 pass, matching the Kubescape-side count. Pod-level absence is by design (#3469) and is not a gap.
Decomposition
The three classes carry genuinely different risk, so I have split the least-disruptive one out rather than sequencing all three in one change — the CSI sidecar Deployments roll normally and need no volume-attachment reasoning. Filed as a sub-issue; the
engine-imageandInstanceManagerwork stays here.devantler commentedon Sep 21, 2026
Progress, measured in prod 2026-09-21 ~02:05Z
csi-attacher,csi-provisioner,csi-resizer,csi-snapshotter) now carryfsGroupChangePolicy: OnRootMismatchandseLinuxOptionson every container.longhorn-csi-plugin; the other three areengine-image-ei-*, one per engine version.0), so 10 of 14 volumes stay on v1.11.1 or v1.12.0 indefinitely and every engine image stays referenced. security(longhorn): enable automatic engine upgrades so stale engine-image DaemonSets can retire #3966 turns the upgrade on (draft fix(longhorn): enable automatic engine upgrades so stale engine images can retire #3967). Once it deploys, the two old images should lose their last reference and be removed, which leaveslonghorn-csi-pluginand the default engine-image DaemonSet for a scoped retrofit like security(longhorn): retrofit C-0211 onto the four CSI sidecar Deployments #3953's.chore: agent-claim #3918 nonce=8b9cf16ac691bd739677756738a142c7
chore: agent-claim #3918 nonce=3631be58b030ef0da7d6c9bb983d7a89
devantler commentedon Sep 21, 2026
Progress, measured in prod 2026-09-21 ~16:10Z
ei-493e04e7(longhorn-engine v1.12.1) is left. It is the default and 70 engines and replicas reference it.ei-75a03ec3andei-a4d05f02are gone, along with their DaemonSets, so the "retire, don't roll" criterion is met for both.longhorn-csi-pluginis clean. security(longhorn): retrofit C-0211 onto the longhorn-csi-plugin DaemonSet #3999 applied it: both fields are on the stored spec and 4/4 pods are ready.DaemonSet/engine-image-ei-493e04e7. It still has neither field. Because it is still referenced, it has to be rolled, not retired. That is security(longhorn): retrofit C-0211 onto the default engine-image DaemonSet #4025, with draft security(longhorn): retrofit C-0211 onto the default engine-image DaemonSet #4026, the same retrofit pattern as security(longhorn): retrofit C-0211 onto the longhorn-csi-plugin DaemonSet #3999.instance-manager-*pods (created 2026-08-26). They are the last class, and security(longhorn): clear the pre-opt-in C-0211 residual in longhorn-system without detaching volumes unintentionally #3918's final criterion still applies to them: handle them against attachment state.devantler commentedon Sep 22, 2026
Progress, measured in prod 2026-09-22 ~03:10Z
engine-image-ei-493e04e7is clean. security(longhorn): retrofit C-0211 onto the default engine-image DaemonSet #4026 merged; the DaemonSet's 4/4 pods now carry both fields. That closes the second workload class.instance-manager-*pods remain (one each onprod-worker-1/2/3, all created 2026-08-26). Every other pod inlonghorn-systemcarries both fields.So the last class is not a manifest change. It clears the next time each node restarts in a controlled way (a drain-and-reboot, such as the pending Talos upgrade in #3713). I'm leaving this open until then, and will re-measure from live pod specs afterwards against the last two criteria.
devantler commentedon Sep 25, 2026
Verified done, measured in prod 2026-09-25 ~18:20Z (read-only)
instance-manager-*pods were recreated on 2026-09-23 (19:01Z–19:30Z) in a controlled node restart. Each now carriesfsGroupChangePolicy: OnRootMismatchandseLinuxOptionson every container.longhorn-systemare missing either field. This covers init containers too.attached/healthy.longhorn-csi-plugin(security(longhorn): retrofit C-0211 onto the longhorn-csi-plugin DaemonSet #3999) were done first. The two unreferenced engine images were then retired, not rolled, and the referenced one was rolled (security(longhorn): retrofit C-0211 onto the default engine-image DaemonSet #4026). The instance managers were last, and only through a drain-and-restart.All four acceptance criteria are met, so I'm closing this. #3239 no longer waits on anything from this issue. As the issue already says, clean pods do not move Kubescape's stored-spec score; that stays with #3522.