Skip to content

fix: keep build-only NVIDIA prebakes unregistered with DKMS - #191

Draft
sulixu wants to merge 1 commit into
mainfrom
sulixu/gpu-prebake-no-dkms
Draft

sulixu wants to merge 1 commit into
mainfrom
sulixu/gpu-prebake-no-dkms

Conversation

@sulixu

@sulixu sulixu commented Sep 17, 2026

Copy link
Copy Markdown
Collaborator

Summary

Change build-only so it still compiles and installs the NVIDIA kernel modules and userspace libraries, but does not register NVIDIA with DKMS in the shared VHD.

  • Use --no-dkms only in build-only mode. Keep normal CUDA and GRID installation on --dkms.
  • Set the build-only module destination to /lib/modules/<target-kernel>/updates/dkms. This preserves the path used by AgentBaker's existing CPU/GRID cleanup when the installer no longer uses DKMS's Ubuntu destination.
  • Reject inherited NVIDIA DKMS directories or links before building. Reject any remaining NVIDIA DKMS registration or directory/link after building.
  • Propagate a failed build before writing a new success marker.
  • Add mocked-installer tests for the options, build checks, failure handling, and normal installation on a marked image.

This is the aks-gpu part of an alternative safety-only approach to Azure/AgentBaker#9545. It does not park or restore DKMS state.

Why

A shared VHD can become a CPU or opted-out GPU node. If that VHD contains active NVIDIA DKMS registration and node-time cleanup fails, a later kernel update can trigger an unwanted NVIDIA build.

Build-only must leave no active NVIDIA DKMS registration. The published image's safety must not depend on successful cleanup during node provisioning. Driver files still exist; this is not a driver-file removal change.

Compatibility and rollout — draft

  • This is not ready for standalone rollout. Companion AgentBaker changes must reject registered prebakes and require a complete current-kernel DKMS installation on managed CUDA nodes, including validation-only and PIS paths. Those changes are not included in this PR.
  • Deploy compatible CSE before selecting VHDs built with the new container behavior. Old callers can accept a loadable module without establishing DKMS registration.
  • Normal node installation remains unchanged. This does not enable AgentBaker to reuse prebaked .ko files to skip compilation. Validation-only CUDA nodes can require a full installation instead of restore-and-validate, so provisioning performance must be measured before rollout.
  • The existing install-skip-build implementation is unchanged and is not used by AgentBaker. It needs separate DKMS-state handling before it can safely consume these unregistered prebakes.
  • GRID still needs AgentBaker's CUDA-prebake cleanup before its own installation/validation path. This PR does not change that cleanup or fix its existing best-effort/validation-only limitations.
  • No image tags are changed, no images are published by this change itself, and existing VHDs or deployed nodes are not repaired.

Validation

  • shellcheck -S warning install.sh entrypoint.sh passes.
  • bats test/install.bats: 23 tests pass on host Bash and Linux Bash in a disposable container.
  • Tests use a fake NVIDIA installer and isolated directories. They do not compile or load a real NVIDIA driver.
  • Actual VHD build, managed CUDA/GRID provisioning, CPU security updates with failed cleanup, PIS, and GPU kernel-update/reboot E2E validation remain required before release.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants