Skip to content

fix(kubernetes): preserve image volumes in job pod spec - #279

Merged
skipi merged 1 commit into
masterfrom
mk/agent/k8s-api-bump
Aug 31, 2026
Merged

fix(kubernetes): preserve image volumes in job pod spec#279
skipi merged 1 commit into
masterfrom
mk/agent/k8s-api-bump

Conversation

@skipi

@skipi skipi commented Aug 31, 2026

Copy link
Copy Markdown
Member

Problem

A Kubernetes image volume (KEP-4639, beta and on by default since k8s 1.33) declared in the pod spec passed via --kubernetes-pod-spec never reaches the job pod:

pod: |
  volumes:
    - name: deps
      image:
        reference: registry.example.com/deps:abc123
        pullPolicy: IfNotPresent

The volume arrives in the job pod with a completely empty VolumeSource, which kube-apiserver's SetDefaults_Volume then turns into an emptyDir{}. The job sees an empty directory where the image contents should be, with no error anywhere. Creating the same pod directly with kubectl works, so this is not a cluster or runtime limitation.

Cause

LoadPodSpec decodes the ConfigMap's pod key into a typed corev1.PodSpec:

https://github.com/semaphoreci/agent/blob/master/pkg/kubernetes/client.go#L145-L149

go.mod pinned k8s.io/api at v0.26.2. ImageVolumeSource was added in v0.31.0, so the typed VolumeSource had no Image field and the value was discarded during deserialization. podSpecFromJobRequest then carried the now-sourceless volume through to pod creation.

This makes image volumes unusable on Kubernetes-based agents regardless of cluster version.

Fix

Bump k8s.io/api, k8s.io/apimachinery and k8s.io/client-go from v0.26.2 to v0.35.4. No production code changes are required — the field flows through on its own once the typed PodSpec knows about it.

v0.35.x brackets current cluster versions within client-go's ±1 minor support window. v0.34.3 and v0.37.0 also build clean if a different target is preferred.

The go directive stays at 1.25.0 — nothing in the dependency graph requires more, so this does not raise the minimum Go needed to build the agent.

Testing

  • New Test__CreatePod/with image volume from pod spec asserts the image volume survives into the created pod (reference, pull policy, and the corresponding volume mount). It fails on master and passes here.
  • Test_DeletePod / Test_DeleteSecret asserted assert.Nil(t, pod) after deletion. Newer fake clientsets return a zero-valued object alongside the NotFound error, so these now assert apierrors.IsNotFound(err). The delete code itself is unchanged.
  • Rest of the suite is unaffected; the remaining local failures (Test__DockerComposeExecutor, Test__Shell__SimpleHelloWorldUsingBase64Encoding, Test__StopJobWithExitCode) reproduce identically on master and are environment-dependent.

Follow-up worth considering (not in this PR)

This bump fixes today's gap but not the underlying shape: unknown fields are dropped both at the typed decode and at Pods().Create(ctx, *corev1.Pod), so any pod-spec field newer than the pinned k8s.io/api is lost the same silent way. Genuinely version-agnostic passthrough would need unstructured plus the dynamic client.

A cheaper mitigation in the meantime: LoadPodSpec currently degrades silently. Logging a warning for keys present in the raw YAML but absent from the decoded struct would have surfaced this as "unknown field volumes[0].image" in the agent log instead of a mystery emptyDir. Worth a warning rather than a hard failure, so existing users with forward-looking specs are not broken.

@skipi
skipi requested a review from dexyk August 31, 2026 11:16
The pod spec supplied through --kubernetes-pod-spec is decoded into a typed
corev1.PodSpec. k8s.io/api was pinned at v0.26.2, which predates
ImageVolumeSource (added in v0.31.0), so `volumes[].image` was silently
dropped at decode time. The volume still reached the job pod, but with an
empty VolumeSource, which the API server then defaults to emptyDir - so the
job saw an empty directory instead of the mounted image. Image volumes were
unusable on Kubernetes-based agents regardless of cluster version.

Bump k8s.io/api, k8s.io/apimachinery and k8s.io/client-go to v0.35.4. No
production code changes are needed - the field flows through once the typed
PodSpec knows about it.

Test_DeletePod and Test_DeleteSecret asserted a nil object from the fake
clientset after deletion. Newer fake clientsets return a zero-valued object
alongside the NotFound error, so assert on the error instead.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@skipi
skipi force-pushed the mk/agent/k8s-api-bump branch from d3d3d4e to af61dee Compare August 31, 2026 11:49
@skipi
skipi merged commit 03d1902 into master Aug 31, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants