Decide the four Talos machine-config fields that have no v1.14 document equivalent

🤖 Generated by the Agentic Engineer

Evidence

#3850 migrates our Talos machine-config patches onto the v1.14 configuration documents. Testing
every conflict class individually against talosctl v1.14.0 (measurement)
found that eight of the twelve classes migrate cleanly and four have no v1.14 home at all:

`v1alpha1` field	patch file	why it is stuck
`machine.kubelet.extraMounts`	`talos/workers/mount-longhorn-data.yaml`	`KubeletConfig` has five fields and this is not one; rejected `unknown keys found during decoding`. Known upstream gap (siderolabs/talos#14365).
`machine.install.extraKernelArgs`, `grubUseUKICmdline`	`talos/cluster/enable-apparmor.yaml`	`UnattendedInstallConfig` carries only `reboot`, `installer.image`, `provisioning.{diskSelector,wipe}`.
`cluster.apiServer.extraVolumes`	`talos/cluster/enable-audit-logging.yaml`	`KubeAPIServerConfig` has no volume surface.
`machine.files` → `/etc/kubernetes/...`	`talos/cluster/enable-audit-logging.yaml`	`EtcFileConfig` refuses the path: `user etc file ... is managed by Talos`.

Searched the complete v1.14 schema set (103 documents, 254 KB from talosctl docs --config):
extraKernelArgs, grubUseUKICmdline, extraVolumes and extraMounts appear in none of them. A
positive control on the same corpus (extraArgs, image, taints, nodeIP, installer) hits, so
those are real absences rather than a dead search.

The problem

Each of these four keeps a v1alpha1 section alive, and a live v1alpha1 section is exactly what
talosctl validate rejects once the corresponding document exists in the generated base — which in
v1.14 it always does. So they do not merely fail to migrate: they individually block #3850 from
going green, and #3850 blocks #3713, which blocks #3626 and Kubernetes v1.37 (#3534).

Each one also carries real production function: Longhorn replica storage, AppArmor enforcement, and
API-server audit logging.

The escape, and why it is not obviously right

$patch: delete on a base document does free the v1alpha1 field — verified for KubeletConfig,
after which extraMounts validates. But those documents are where the base's pinned images live
(ghcr.io/siderolabs/kubelet:v1.37.0; factory.talos.dev/metal-installer/<digest>:v1.14.0).
Deleting one either silently drops its pin to a Talos default or forces a generated,
version-specific digest to be hardcoded into a patch that provisions production nodes. That
trade-off is the decision this issue exists to make.

Expected behaviour

Each of the four has a decided, recorded answer — an upstream capability we wait for, a re-expression
that preserves the function, or an accepted $patch: delete with its pin cost understood — so that
#3850 can be implemented as a single settings-preserving change.

Acceptance criteria

Longhorn's /var/lib/longhorn bind mount has a decided answer, and it is verified to still mount.
security=apparmor has a decided answer, verified as still enforced on a node.
The API-server audit policy has a decided answer covering both the file placement and its
mount into the API server, verified by audit events still being written.
Where the answer is $patch: delete, the affected image pin is shown to be preserved or the
regression is explicitly accepted in writing.
Where the answer is an upstream fix, it is named with an identifier and a blocker line, per our
external-blocker convention.

Size and risk

Medium. Mostly decision and verification rather than code, but each answer changes how production
nodes are provisioned, so each wants evidence rather than assertion.

Blocks #3850. Part of #3626.

🤖 Generated by the Agentic Engineer

Two of the open questions are now answered by measurement

This issue's evidence was gathered against machinery v1.14.0-alpha.2. Two things have changed
since, and both narrow the decision.

1. The four absences are real in the shipped release — not an alpha gap

v1.14.1 was released 2026-09-15 and v1.15.0-alpha.0 now exists. I searched the complete
config/types/ tree of each (fetched from the Go module proxy, read-only) for the four fields,
with a positive control on the same corpus so a dead search would show:

field	`v1.14.1`	`v1.15.0-alpha.0`
`extraMounts`	0 files	0 files
`extraKernelArgs`	0 files	0 files
`grubUseUKICmdline`	0 files	0 files
`extraVolumes`	0 files	0 files
control: `extraArgs`	2 files ✅	2 files ✅
control: `taints`	1 file ✅	1 file ✅
control: `nodeIP`	2 files ✅	—

Consequence for acceptance criterion 5: "wait for an upstream capability" currently has no
target version to wait for. The gap is not closed in the shipped v1.14.1, and it is not closed
in the first alpha of the next minor either. Any answer of the form "wait for upstream" would
therefore have to be recorded with an unbounded blocker, which our own external-blocker convention
does not accept. That effectively reduces the live options to re-express or accepted
$patch: delete.

2. `$patch: delete` cannot silently drop the kubelet pin — the schema forbids it

The issue records the pin cost as "either silently drops its pin to a Talos default or forces a
hardcoded digest". The first half does not hold. In v1.14.1,
config/types/k8s/kubelet.go declares:

//   schemaRequired: true
KubeletImage string `yaml:"image"`

image is required on a KubeletConfig document — and note it is the only image field in that
package without omitempty (proxy.go and coredns.go both have image,omitempty). So deleting
the base document and supplying our own does not quietly inherit a default: validation forces the
pin to be restated where a reviewer can see it.

Consequence for acceptance criterion 4: the cost is not a silent regression but an explicit
version coupling — ghcr.io/siderolabs/kubelet:v<k8s> would have to move in lockstep with
spec.cluster.kubernetesVersion. That is a guardable cost rather than an unknown one: this repo
already enforces exactly this shape of lockstep in
scripts/tests/refresh-flux-ghcr-auth/contracts_test.go, which asserts setup-talosctl.sh's
TALOS_VERSION equals ksail.prod.yaml's Talos version. The same contract test extended to the
kubelet image would keep the pin honest.

Still open — deliberately not decided here

Criteria 1–3 each require the function to be verified still working (Longhorn mounting,
AppArmor enforced, audit events written), which needs the migration actually applied. Nothing above
substitutes for that, and I am not closing this Spike on a schema reading.

Why this matters more than it did yesterday

I diagnosed #3534 (Kubernetes v1.37.0) today and proved the chain empirically — the compatibility
gate delegates to upstream machinery/compatibility, and probing it directly gives Talos 1.13.9 +
k8s 1.37.0 ❌ against Talos 1.14.0 + k8s 1.37.0 ✅ (with 1.13.9 + 1.36.4 ✅ as the control). So this
Spike now gates a Kubernetes minor-version bump as well as the Talos migration:

#3851 → #3850 → #3713 / #3534

Blocker: siderolabs/talos — extraMounts/extraKernelArgs/grubUseUKICmdline/extraVolumes document support | upstream | last-verified 2026-09-16: absent in v1.14.1 (released 2026-09-15) and in v1.15.0-alpha.0; no target release

chore: agent-claim #3851 nonce=290534e585eaa4f5c26ee283fc3bc86a

🤖 Generated by the Agentic Engineer

Decision — all four fields now have an answer, and two of them are not what this issue assumed

I rebuilt the render+validate mirror of CI's validate-talos job and reproduced this issue's RED
baseline exactly (talos/cp 8, talos/workers 4, talos-local/cp 6, talos-local/workers
3), then tested each field in an isolated fixture — one patch against a fresh v1.14 base —
with a positive control (empty patch ⇒ PASS) and a negative control (v1alpha1 kubelet.extraMounts
⇒ FAIL) proving the harness can report both outcomes.

1. Audit logging — RESOLVED by re-expression. No `$patch: delete`, and nothing is lost.

The v1.14 base generates a KubeAuditPolicyConfig document. Its configuration field is
documented as "the literal Kubernetes audit policy configuration". Moving our policy inline there
means Talos places and mounts the policy itself — which retires both halves of criterion 3 at
once:

machine.files writing /var/etc/kubernetes/audit/audit-policy.yaml — no longer needed
cluster.apiServer.extraVolumes mounting it into the API server — no longer needed
audit-log-* flags move to KubeAPIServerConfig.extraArgs (which exists); audit-policy-file
is dropped because Talos owns that path now.

Verified: PASS on control-planes/cloud with no machine.files and no extraVolumes.

2. `machine.files` was never a blocker

Worth correcting explicitly, because it is listed as one of the four. In the RED baseline it appears
as WARNING: .machine.files is deprecated; use dedicated configuration documents instead — a
deprecation warning, not one of the 8 errors. It never blocked talosctl validate. Under (1) it
becomes unnecessary anyway, but it was not holding anything up.

3. Longhorn `machine.kubelet.extraMounts` — `$patch: delete` + restate the pin

No v1.14 home: 0 of 103 schema documents mention extraMounts, against five positive controls
(extraArgs 7, taints 1, nodeIP 1, installer 1, configuration 56) on the same corpus, so the
zero is a real absence rather than a dead search. Verified PASS, pin retained when restated.

One welcome side effect: once KubeletConfig is deleted, v1alpha1 machine.kubelet is live again,
so all five kubelet patches stay exactly as they are — evict-pods-before-oom,
gc-unused-images-before-pressure, rotate-server-certificates, label-nodes and
mount-longhorn-data need no migration at all. The migration gets smaller.

4. AppArmor — this is not a Talos decision. KSail already solved it.

This is the finding that reshapes the issue. KSail v7.184.0 — the version this repo actually pins
(KSAIL_VERSION: "7.184.0" in ci.yaml/cd.yaml) — already folds machine.install.extraKernelArgs
into the Image Factory schematic, then clears the field and pins grubUseUKICmdline=true. Its own
doc comment on reconcileFoldedKernelArgs describes our patch verbatim:

a patch may carry grubUseUKICmdline=false (e.g. a pre-fold workaround that let the host-built
cmdline include the args), but once the args live in the UKI cmdline, false would make GRUB
rebuild the cmdline host-side and silently drop them — so it must become true.

talos/cluster/enable-apparmor.yaml is that pre-fold workaround. So in production:

extraKernelArgs: [security=apparmor] is an input to the schematic, not a machine-config
setting — KSail consumes it and removes it from the rendered config;
grubUseUKICmdline: false is already inert, overridden to true by KSail on every render.

The $patch: delete escape is NOT available here, and that is worth stating plainly because it
is the obvious move. ksail.prod.yaml says the installer image is "derived during config
generation — there is no hand-maintained install-image patch", computed from a schematic carrying
siderolabs/iscsi-tools, siderolabs/util-linux-tools and siderolabs/qemu-guest-agent.
Hardcoding that digest into a patch would freeze the schematic and drop those extensions on the next
change — breaking Longhorn's data plane. Omitting it falls back to a bare installer with no
extensions, which breaks Longhorn immediately. Both variants are wrong.

5. Criterion 4 — correcting my own earlier conclusion

My previous comment
said $patch: delete "cannot silently drop the kubelet pin — the schema forbids it", reasoning
from schemaRequired: true on KubeletConfig.image. That is wrong, and I measured it wrong way
round. schemaRequired binds only while the document exists. Delete the document and the
requirement goes with it:

render	`siderolabs/kubelet` refs	`talosctl validate`
unpatched base (control)	1	PASS
`$patch: delete` on `KubeletConfig`	0	PASS
`$patch: delete` + restated `image`	1	PASS

The same holds for UnattendedInstallConfig and the metal-installer digest. Controls: the patched
render does contain our own extraMounts/longhorn content, so the zero is not a dead grep.

So the silent pin drop this issue originally feared is real, it passes validation, and the
mitigation is a guard, not the schema — the shape already used by
scripts/tests/refresh-flux-ghcr-auth/contracts_test.go for TALOS_VERSION.

The CI-fidelity gap this exposes

CI validates a raw talosctl gen config + machineconfig patch render. Production's config comes
from KSail, which folds kernel args, clears the field and pins UKI cmdline true. For
machine.install, CI validates a config production never produces — which is why AppArmor shows up
as a Talos schema blocker here when it is really a CI-fidelity issue.

Where this leaves the chain

Every field now has a decided answer, so #3850 is implementable. What criteria 1–3 still ask for
— Longhorn actually mounting, AppArmor actually enforced, audit events actually written — cannot be
produced by a spike: it needs the migration deployed. That evidence belongs on #3850's rollout, and I
am moving it there rather than holding the chain behind a verification that cannot happen first.

Follow-ups filed:

CI's Talos validation renders a config production never produces #3854 — CI's Talos validation renders a config production never produces (the fidelity gap above).
Guard the kubelet image pin against drifting from kubernetesVersion #3855 — guard the kubelet image pin against drifting from kubernetesVersion (criterion 4's mitigation).

Measured with talosctl v1.14.0 against a freshly generated base; KSail source read at tag
v7.184.0, the pinned version.

🤖 Generated by the Agentic Engineer

Closing: this Spike's definition of done — a recorded decision plus the follow-up issues it implies — is met.

All four fields are decided in the decision comment. Two needed no v1.14 home at all (the audit policy re-expresses as KubeAuditPolicyConfig, which retires the file write and the volume mount together), one takes $patch: delete plus a restated pin, and AppArmor turned out not to be a Talos question — KSail v7.184.0 already folds it into the Image Factory schematic.

Follow-ups: #3854 (CI renders a config production never produces) and #3855 (guard the kubelet pin).

Criteria 1–3 asked for live evidence that Longhorn, AppArmor and audit still work. That cannot precede the migration, so it now sits on #3850 rather than holding the chain behind a verification that cannot happen first. #3850 is unblocked.

Navigation Menu

Repository navigation

Decide the four Talos machine-config fields that have no v1.14 document equivalent #3851

Evidence

The problem

The escape, and why it is not obviously right

Expected behaviour

Acceptance criteria

Size and risk

Two of the open questions are now answered by measurement

1. The four absences are real in the shipped release — not an alpha gap

2. `$patch: delete` cannot silently drop the kubelet pin — the schema forbids it

Still open — deliberately not decided here

Why this matters more than it did yesterday

Decision — all four fields now have an answer, and two of them are not what this issue assumed

1. Audit logging — RESOLVED by re-expression. No `$patch: delete`, and nothing is lost.

2. `machine.files` was never a blocker

3. Longhorn `machine.kubelet.extraMounts` — `$patch: delete` + restate the pin

4. AppArmor — this is not a Talos decision. KSail already solved it.

5. Criterion 4 — correcting my own earlier conclusion

The CI-fidelity gap this exposes

Where this leaves the chain

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Decide the four Talos machine-config fields that have no v1.14 document equivalent #3851

Description

Evidence

The problem

The escape, and why it is not obviously right

Expected behaviour

Acceptance criteria

Size and risk

Activity

devantler commented on Sep 16, 2026

Two of the open questions are now answered by measurement

1. The four absences are real in the shipped release — not an alpha gap

2. $patch: delete cannot silently drop the kubelet pin — the schema forbids it

Still open — deliberately not decided here

Why this matters more than it did yesterday

devantler commented on Sep 16, 2026

Decision — all four fields now have an answer, and two of them are not what this issue assumed

1. Audit logging — RESOLVED by re-expression. No $patch: delete, and nothing is lost.

2. machine.files was never a blocker

3. Longhorn machine.kubelet.extraMounts — $patch: delete + restate the pin

4. AppArmor — this is not a Talos decision. KSail already solved it.

5. Criterion 4 — correcting my own earlier conclusion

The CI-fidelity gap this exposes

Where this leaves the chain

devantler commented on Sep 16, 2026

Metadata

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Issue actions

2. `$patch: delete` cannot silently drop the kubelet pin — the schema forbids it

1. Audit logging — RESOLVED by re-expression. No `$patch: delete`, and nothing is lost.

2. `machine.files` was never a blocker

3. Longhorn `machine.kubelet.extraMounts` — `$patch: delete` + restate the pin