Repository navigation
Migrate the Talos machine config onto the v1.14 configuration documents #3850
Description
🤖 Generated by the Agentic Engineer
Evidence
#3626 establishes that Kubernetes cannot advance past v1.36 until Talos moves to the v1.14 line, and states the prerequisite as "Talos moves to the v1.14 line first". That prerequisite has never been specified, and it is considerably more than a version bump.
#3713 (chore(deps): update dependency siderolabs/talos to v1.14.0) now demonstrates what it actually costs. With the stale talosctl digest fixed (commit a8203d4d, so setup-talosctl.sh succeeds and talosctl gen config runs), 🧪 Validate Talos Machine Config fails on the talos (cloud) overlay with 8 conflicts — job 104627163211:
8 errors occurred:
* UnattendedInstallConfig config is incompatible with v1alpha1 config (.machine.install)
* .machine.nodeAnnotations is already set in v1alpha1 config
* kubelet config is already set in v1alpha1 config (.machine.kubelet)
* cluster network config is already set in the v1alpha1 config (.machine.cluster.network). Please remove it and use only the new KubeNetworkConfig document to avoid conflicts
* kube-apiserver config is already set in v1alpha1 config (.cluster.apiServer)
* kube-controller-manager config is already set in v1alpha1 config (.cluster.controllerManager)
* cluster proxy config in v1alpha1 config (.machine.cluster.proxy) can't be used with KubeProxyConfig document, please remove it to avoid conflicts
* cluster network config in v1alpha1 config (.machine.cluster.network) can't be used with KubeFlannelCNIConfig document, please remove it to avoid conflicts
Preceded by WARNING: .machine.files is deprecated; use dedicated configuration documents instead.
Every other check on that head is green — 🔐 Validate Production Authorization, 🧪 Validate Manifests, kubescape, CodeQL and the rest. This one check, and CI - Required Checks as its consequence, are the only failures.
The problem
Talos v1.14 moves settings out of the monolithic v1alpha1 document into dedicated configuration documents, and refuses to validate when both describe the same thing. Our machine-config patches still set all eight in v1alpha1. So the Talos upgrade cannot land as a version bump, and while it cannot land, #3534 (Kubernetes v1.37) stays permanently unmergeable — that is the whole of #3626's cost, and this is its root.
The same upstream restructuring is breaking KSail independently (ksail#6776 — MachineConfig.Kubelet and seven sibling accessors removed), so this is one upstream change surfacing in two products.
Expected behaviour
The Talos machine-config patches are migrated onto the v1.14 documents, talosctl validate passes for both overlays, and #3713 (or its successor) can go green on the version bump alone.
Acceptance criteria
.machine.install→UnattendedInstallConfig.machine.kubelet→ the dedicated kubelet document.machine.cluster.network→KubeNetworkConfig(and theKubeFlannelCNIConfigconflict resolved with it).machine.cluster.proxy→KubeProxyConfig.cluster.apiServer→ the dedicated kube-apiserver document.cluster.controllerManager→ the dedicated kube-controller-manager document.machine.nodeAnnotationsmoved offv1alpha1.machine.filesmoved off the deprecated field🧪 Validate Talos Machine Configpasses for both thetalos (cloud)andtalos-local (container)overlays- The rendered config is diffed against the current one to show the migration is settings-preserving — no grant, no mount, no network setting silently dropped
Size and risk
Medium, and higher risk than its size suggests: this rewrites the machine config that provisions production nodes. It wants its own reviewed change with a rendered before/after diff — not an adaptation commit folded into a Renovate version bump.
Part of #3626.
Activity
devantler commentedon Sep 16, 2026
Three findings that change the shape of this work
Reproduced the failure locally against
talosctl v1.14.0with a faithful mirror of the CI step, then probed the current pin directly.1. The migration and the version bump are ATOMIC — this cannot land on
mainfirstThis is the load-bearing one, because the issue currently implies the opposite ("#3713 (or its successor) can go green on the version bump alone"), and acting on that reading would take
mainred.mainpinsTALOS_VERSION="1.13.9"in.github/scripts/setup-talosctl.sh, so CI onmainvalidates with talosctl 1.13.9 — and 1.13.9 does not register the v1.14 documents at all:Its
gen configemits exactly one non-v1alpha1document (HostnameConfig), against 27 for 1.14.0.So the migrated patches are unparseable on the current pin and the unmigrated patches are invalid on the next one. There is no ordering in which the two land separately and CI stays green in between. The Talos version bump, the digest, the
ksail.prod.yamlpin and all fourteen patch files have to move in one change.That also resolves the tension in the "Size and risk" note: this genuinely wants its own reviewed change rather than being folded into the Renovate bump — but that change has to carry the bump and supersede #3713, not land ahead of it.
2. The failure set is larger than the CI log showed —
talos-localfails tooCI reports only the first failing overlay/role. Locally, all four combinations fail, and
talos-localcontributes a conflict the acceptance criteria do not list:talostalostalos-localtalos-localThe extra one is
.cluster.allowSchedulingOnControlPlanes is already set in v1alpha1 config, fromtalos-local/cluster/allow-scheduling-on-control-planes.yaml. Criterion 9 already requires both overlays to pass, so this adds a mapping rather than changing the goal.3. Per-file attribution — the fourteen files that actually have to move
.machine.installtalos/cluster/enable-apparmor.yaml.machine.nodeAnnotationstalos/cluster/mark-ghcr-pull-revision.yaml.machine.kubelettalos/cluster/{gc-unused-images-before-pressure,evict-pods-before-oom,rotate-server-certificates}.yaml,talos/workers/{label-nodes,mount-longhorn-data}.yaml,talos-local/cluster/rotate-server-certificates.yaml.machine.cluster.proxy+.machine.cluster.networktalos/cluster/disable-default-cni-and-kube-proxy.yaml,talos-local/cluster/disable-default-cni-and-kube-proxy.yaml.cluster.apiServertalos/cluster/{enable-dex-oidc,enable-audit-logging}.yaml,talos-local/cluster/enable-dex-oidc.yaml.cluster.controllerManagertalos/cluster/gc-terminated-pods-sooner.yaml.cluster.allowSchedulingOnControlPlanestalos-local/cluster/allow-scheduling-on-control-planes.yaml.machine.files(deprecation warning only)talos/cluster/enable-audit-logging.yaml,talos-local/cluster/enable-dex-oidc.yamltalosctl docs --config <dir>generates the authoritative target schema for every one of these locally, which is what makes the mapping checkable rather than guessed.A caution on acceptance criterion 10
Criterion 10 asks for a rendered before/after diff proving the migration is settings-preserving. Note that the generated base also changes between 1.13.9 and 1.14.0 — 1 non-
v1alpha1document becomes 27 — so a naive diff of rendered output conflates upstream's restructuring with our patch migration, and the signal that matters (did one of our settings get dropped?) is buried. The diff needs to compare effective settings across the version boundary, not rendered documents.Not claimed
I am not holding this; the claim ref is retired. The reproduction is a faithful local mirror of the CI step and takes seconds to re-run, so whoever picks this up can iterate against
talosctl validatedirectly rather than through CI.chore: agent-claim #3850 nonce=b2a11cd402fea387755e50252f6fa9cd
devantler commentedon Sep 16, 2026
Feasibility matrix — four of the eight conflicts have no v1.14 document home
I built a local mirror of the CI step (
✅ Render + validate patched machine configs) that runs inseconds, reproduced the RED baseline exactly, and then tested every conflict class against
talosctlv1.14.0 individually. Every result below is an executedmachineconfig patch+validate, not a reading of the schema.RED baseline reproduced — and it is four failures, not one. CI reports only the first:
talos/control-planestalos/workerstalos-local/control-planestalos-local/workersThe structural fact this issue is missing
In v1.14 the generated base no longer carries these settings in
v1alpha1at all —gen configemits 28 documents and the
v1alpha1document has nomachine.kubelet,machine.install,machine.nodeAnnotations,cluster.network,cluster.proxy,cluster.apiServerorcluster.controllerManagersection. So "already set in v1alpha1" is our patches putting themback. That is good news for most of the migration: a patch document of the new kind merges cleanly
into the base document and validates.
Per-field results
v1alpha1fieldmachine.kubelet.extraArgsrotate-server-certificates(×2),workers/label-nodesKubeletConfig.extraArgsmachine.kubelet.extraConfigevict-pods-before-oom,gc-unused-images-before-pressureKubeletConfig.config(renamed)machine.nodeAnnotationsmark-ghcr-pull-revisionKubeNodeConfig.annotationscluster.controllerManager.extraArgsgc-terminated-pods-soonerKubeControllerManagerConfig.extraArgscluster.apiServer.extraArgsenable-dex-oidc(×2),enable-audit-loggingKubeAPIServerConfig.extraArgscluster.network.cni: nonedisable-default-cni-and-kube-proxy(×2)KubeFlannelCNIConfigcluster.proxy.disabled: truedisable-default-cni-and-kube-proxy(×2)KubeProxyConfig.enabled: falsecluster.allowSchedulingOnControlPlanestalos-local/allow-scheduling-on-control-planesKubeNodeConfig.taints: {}machine.kubelet.extraMountsworkers/mount-longhorn-datamachine.install.extraKernelArgs,grubUseUKICmdlineenable-apparmorcluster.apiServer.extraVolumesenable-audit-loggingmachine.files→/etc/kubernetes/...enable-audit-loggingEtcFileConfigrefuses the pathSearched exhaustively across the complete v1.14 schema set (103 documents, 254 KB from
talosctl docs --config);extraKernelArgs,grubUseUKICmdline,extraVolumes,extraMountsandallowSchedulingOnControlPlanesappear in none of them. Positive control run on the same corpus(
extraArgs,image,taints,nodeIP,installerall hit), so those zeros are real absences andnot a dead search.
The four blockers, concretely
KubeletConfighas exactly five fields —image,config,extraArgs,clusterDNS,defaultRuntimeSeccompProfileEnabled.extraMountsis rejected withunknown keys found during decoding. This is a known upstream gap (v1.14 KubeletConfig not support extraMounts. siderolabs/talos#14365), andthe upstream advice is to keep
machine.kubeletinv1alpha1— which is exactly what conflicts.UnattendedInstallConfigcarries onlyreboot,installer.imageandprovisioning.{diskSelector,wipe}.security=apparmorhas nowhere to go.KubeAPIServerConfighas noextraVolumes.EtcFileConfigrejectskubernetes/audit/audit-policy.yamlwithuser etc file ... is managed by Talos. The same document ataudit/audit-policy.yamlvalidates,so the mechanism works and only the path we need is refused — and relocating it outside
/etc/kubernetesneedsextraVolumes, which is blocker 3. These two compound.The escape, and its cost
$patch: deleteon a base document does work — deletingKubeletConfigfreesv1alpha1machine.kubelet(verified:extraMountsthen validates). But the deleted document is where thebase's pinned images live (
ghcr.io/siderolabs/kubelet:v1.37.0, andfactory.talos.dev/metal-installer/<digest>:v1.14.0for the installer). Deleting it either dropsthose pins to Talos defaults or forces us to hardcode a generated, version-specific digest into a
patch. Both are worse than the problem for a document that provisions production nodes.
Two corrections to the issue
talos-localadds.cluster.allowSchedulingOnControlPlanes, which the criteria do not mention.KubeProxyConfigis control-plane-only (the following document kinds are only allowed on control plane machines).disable-default-cni-and-kube-proxy.yamlcurrently lives incluster/and so applies to workers too. The kube-proxy half has to move to
control-planes/, splittinga file that today is one patch — a structural change the criteria do not anticipate.
Also worth recording
scripts/tests/refresh-flux-ghcr-auth/contracts_test.goassertsTALOS_VERSIONinsetup-talosctl.shequalsspec.cluster.talos.versioninksail.prod.yaml. CI pins 1.13.9,which cannot register the v1.14 kinds, so migrated patches go red until both pins move together.
There is no ordering where the migration and the bump land separately with CI green in between.
talos/pkg/machinery v1.14.0-alpha.2. ksail#6776 isabout accessors removed after alpha.2 and is a separate concern.
talosctl gen config, productionuses KSail. The patches have to satisfy both, and only the CI side is verified here.
Recommendation
Blockers 1–4 are upstream capability gaps, not patch bugs, so this issue cannot be closed by
migrating patches alone. I would split it: land the eight migratable classes, and track the four
blocked fields separately against an upstream fix, since each one of them keeps
v1alpha1in playand therefore keeps the conflict alive.
I have not opened a PR — a partial migration leaves
talosctl validatered, so there is nothingshippable until the blocked four have a decided answer. Claim retired; this is unclaimed again.
devantler commentedon Sep 16, 2026
#3851 is decided — this issue is unblocked, and it is smaller than it looked
The decision
resolved all four "no v1.14 home" fields. Two of them turned out not to need a home at all, so the
migration shrinks. Recording the concrete plan here, plus the verification this issue now owns.
The plan, file by file
Audit logging —
talos/cluster/enable-audit-logging.yamlis rewritten, not patched around. Thepolicy moves inline into a
KubeAuditPolicyConfigdocument, after which Talos places and mountsit itself: the
machine.fileswrite and thecluster.apiServer.extraVolumesmount both go away,and
audit-policy-filewith them. Theaudit-log-{path,maxage,maxbackup,maxsize}flags move toKubeAPIServerConfig.extraArgs. Verified PASS in isolation (control-planes/cloud).Kubelet — add a
$patch: deleteon the generatedKubeletConfigdocument and restateimage: ghcr.io/siderolabs/kubelet:v<k8s>. v1alpha1machine.kubeletis then live again, soevict-pods-before-oom,gc-unused-images-before-pressure,rotate-server-certificates,label-nodesandmount-longhorn-dataneed no edits at all. Verified PASS; pin retained.The restated pin is guarded by #3855 — restating it is not optional, because deleting the document
drops the pin silently and still validates.
AppArmor —
talos/cluster/enable-apparmor.yamlkeepsextraKernelArgs(it is KSail's schematicinput) and drops
grubUseUKICmdline: false, which KSail v7.184.0 already overrides totrueonevery render. The long comment justifying
falsedescribes pre-fold behaviour and should go with it.This file is not a Talos schema problem; the CI-side mismatch it exposes is #3854.
Carried from the earlier matrix
(tested there, not re-tested by me today):
nodeAnnotations→KubeNodeConfig.annotations;controllerManager.extraArgs→KubeControllerManagerConfig.extraArgs;apiServer.extraArgs(dex) →KubeAPIServerConfig.extraArgs; CNI none → deleteKubeFlannelCNIConfig;proxy.disabled→KubeProxyConfig.enabled: false;allowSchedulingOnControlPlanes→KubeNodeConfig.taints: {}.Two shape notes from that matrix still stand:
KubeProxyConfigis control-plane-only, sodisable-default-cni-and-kube-proxy.yamlhas to split out ofcluster/; and the Talos version bumpis atomic with these patches —
contracts_test.gopinsTALOS_VERSIONtoksail.prod.yaml, so there is no ordering where they land separately with CI green.Verification this issue now owns
#3851's criteria 1–3 asked for live evidence that the function survives. A spike cannot produce it —
it needs the migration deployed — so it moves here:
/var/lib/longhornstill bind-mounted on a worker, and a Longhorn volume stillattaches. The kubelet
extraMountspath is unchanged, so this is a regression check rather than anew risk.
security=apparmorstill on the node's kernel cmdline and still enforcing. This isthe one with a real behavioural change in the file, so it deserves the closest look: confirm the
arg arrives via the UKI cmdline now that
grubUseUKICmdlineis not forced false./var/log/audit/kube/audit.log. The policy now arrivesby a different mechanism, so confirm the content took effect (e.g. a secret read logged at
Metadata, an RBAC mutation atRequestResponse), not merely that the file exists.Ordering
#3854 (CI fidelity) is worth landing first if it is cheap, because until CI renders what
production renders, this migration's
machine.installresult cannot be trusted from a green check.It is not a hard blocker — I have not made it one — but it is the difference between CI attesting
this migration and CI attesting a config we do not deploy.
devantler commentedon Sep 16, 2026
Not implementable yet: production's config renderer cannot read the new documents
I picked this up to implement the plan above and stopped before opening a PR. Three findings change the plan, and the first one blocks it.
1. KSail's Talos library cannot decode the documents this migration adds (blocker)
Production does not render its machine config with
talosctl— KSail does, using the Talos config library it pins:pkg/machinery v1.14.0-alpha.2(unchanged in KSail v7.184.0, the version this repo pins). That pre-release predates part of the v1.14 split. Rendering a base with it (contract 1.14, Kubernetes 1.36.4) gives 16 documents on a control plane and 5 on a worker, against 28 fromtalosctl1.14.0 — with noKubeletConfig,KubeNodeConfig,UnattendedInstallConfigorKubeClusterConfig.Applying patches with that same library:
KubeNodeConfig(annotations; the control-plane taint for talos-local)"KubeNodeConfig" "v1alpha1": not registeredmachine.nodeAnnotations(control)KubeProxyConfig enabled: false,KubeFlannelCNIConfig $patch: delete,KubeControllerManagerConfig.extraArgs,KubeAuditPolicyConfigSo a migration that passes this repo's CI (which renders with
talosctl1.14.0) would fail at deploy time. TheKubeletConfig $patch: deletestep is also a no-op there, because that document does not exist in KSail's base.This makes #3850 depend on KSail moving to a Talos library that registers these kinds — ksail#6776, which removes the accessors blocking that bump — then a KSail release and a pin bump here. I have recorded that as a dependency on this issue.
2. The OIDC flags have to become an authentication config, not
KubeAPIServerConfig.extraArgsThe earlier matrix mapped the Dex flags to
KubeAPIServerConfig.extraArgs. On both bases, Talos rejects that:kube-apiserver extra argument "oidc-client-id" is not allowed: use KubeAuthenticationConfig. AKubeAuthenticationConfigwith ajwtissuer (URL, audiencepublic-client,email/groupsclaims with theoidc:prefixes) validates on both. Two cautions for whoever implements it:configurationobject on both bases. The base'sanonymousblock (health endpoints/livez,/readyz,/healthz) disappears unless the patch restates it.talosctl validatedoes not check the JWT content. An issuer ofnot a urlvalidates, so a mistake here would first surface as a kube-apiserver that will not start.The audit flags (
audit-log-pathand the three rotation flags) are accepted inKubeAPIServerConfig.extraArgs, as planned.3.
KubeNodeConfig.taints: {}does not remove the control-plane taintIt validates, but the rendered config still carries
node-role.kubernetes.io/control-plane: NoSchedule. Merging an empty map removes nothing, so talos-local would quietly stop scheduling workloads on its control planes. That item needs a key-level delete, and it is also affected by finding 1.What still holds
The file moves (control-plane-only settings out of
cluster/), the audit policy moving inline intoKubeAuditPolicyConfig, the proxy and CNI changes, and the AppArmor cleanup are unaffected. #3854 (CI rendering what production renders) is now more important: this result is exactly the kind of gap it describes, one layer belowmachine.install.Method: CI's render-and-validate step mirrored locally, with the baseline reproduced exactly (8/4/6/3 errors); a second base rendered with
pkg/machinery v1.14.0-alpha.2from the Go module proxy, and patches applied with that library's own patcher. Each probe patch was tested on its own, with the controls named above. KSail's binary itself was not run against a cluster.devantler commentedon Sep 27, 2026
Target Talos ≥ v1.14.1 for this migration, not v1.14.0. The
iscsi-toolsextension in v1.13.8–v1.14.0 ships open-iscsi 2.1.12, which has the regression behind the stale-iSCSI-record incidents (#3093, #3180; upstream longhorn/longhorn#13733). v1.14.1 is the first release with open-iscsi 2.1.13, which fixes it. The evidence is on #3180. Renovate's #4044 still proposes v1.14.0.