fix(kubernetes): preserve image volumes in job pod spec - #279
Merged
Conversation
The pod spec supplied through --kubernetes-pod-spec is decoded into a typed corev1.PodSpec. k8s.io/api was pinned at v0.26.2, which predates ImageVolumeSource (added in v0.31.0), so `volumes[].image` was silently dropped at decode time. The volume still reached the job pod, but with an empty VolumeSource, which the API server then defaults to emptyDir - so the job saw an empty directory instead of the mounted image. Image volumes were unusable on Kubernetes-based agents regardless of cluster version. Bump k8s.io/api, k8s.io/apimachinery and k8s.io/client-go to v0.35.4. No production code changes are needed - the field flows through once the typed PodSpec knows about it. Test_DeletePod and Test_DeleteSecret asserted a nil object from the fake clientset after deletion. Newer fake clientsets return a zero-valued object alongside the NotFound error, so assert on the error instead. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
skipi
force-pushed
the
mk/agent/k8s-api-bump
branch
from
August 31, 2026 11:49
d3d3d4e to
af61dee
Compare
dexyk
approved these changes
Aug 31, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
A Kubernetes image volume (KEP-4639, beta and on by default since k8s 1.33) declared in the pod spec passed via
--kubernetes-pod-specnever reaches the job pod:The volume arrives in the job pod with a completely empty
VolumeSource, which kube-apiserver'sSetDefaults_Volumethen turns into anemptyDir{}. The job sees an empty directory where the image contents should be, with no error anywhere. Creating the same pod directly withkubectlworks, so this is not a cluster or runtime limitation.Cause
LoadPodSpecdecodes the ConfigMap'spodkey into a typedcorev1.PodSpec:https://github.com/semaphoreci/agent/blob/master/pkg/kubernetes/client.go#L145-L149
go.modpinnedk8s.io/apiatv0.26.2.ImageVolumeSourcewas added inv0.31.0, so the typedVolumeSourcehad noImagefield and the value was discarded during deserialization.podSpecFromJobRequestthen carried the now-sourceless volume through to pod creation.This makes image volumes unusable on Kubernetes-based agents regardless of cluster version.
Fix
Bump
k8s.io/api,k8s.io/apimachineryandk8s.io/client-gofromv0.26.2tov0.35.4. No production code changes are required — the field flows through on its own once the typedPodSpecknows about it.v0.35.xbrackets current cluster versions within client-go's ±1 minor support window.v0.34.3andv0.37.0also build clean if a different target is preferred.The
godirective stays at1.25.0— nothing in the dependency graph requires more, so this does not raise the minimum Go needed to build the agent.Testing
Test__CreatePod/with image volume from pod specasserts the image volume survives into the created pod (reference, pull policy, and the corresponding volume mount). It fails onmasterand passes here.Test_DeletePod/Test_DeleteSecretassertedassert.Nil(t, pod)after deletion. Newer fake clientsets return a zero-valued object alongside theNotFounderror, so these now assertapierrors.IsNotFound(err). The delete code itself is unchanged.Test__DockerComposeExecutor,Test__Shell__SimpleHelloWorldUsingBase64Encoding,Test__StopJobWithExitCode) reproduce identically onmasterand are environment-dependent.Follow-up worth considering (not in this PR)
This bump fixes today's gap but not the underlying shape: unknown fields are dropped both at the typed decode and at
Pods().Create(ctx, *corev1.Pod), so any pod-spec field newer than the pinnedk8s.io/apiis lost the same silent way. Genuinely version-agnostic passthrough would needunstructuredplus the dynamic client.A cheaper mitigation in the meantime:
LoadPodSpeccurrently degrades silently. Logging a warning for keys present in the raw YAML but absent from the decoded struct would have surfaced this as "unknown fieldvolumes[0].image" in the agent log instead of a mysteryemptyDir. Worth a warning rather than a hard failure, so existing users with forward-looking specs are not broken.