Conversation
Contributor
|
Contributor
Windows Unit Test Results 3 files 14 suites 50s ⏱️ Results for commit 0f11354. |
Contributor
Contributor
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does / why we need it:
Draft companion to Azure/aks-gpu#191 and an alternative to #9545.
Some non-GPU nodes did not complete CUDA prebake cleanup. Later kernel security
updates could discover the remaining NVIDIA DKMS registration and try to rebuild
the driver. Shared-image safety must not depend on node-time cleanup succeeding.
The companion
aks-gpuchange compiles and installs the VHD's NVIDIA modules with--no-dkms. This PR checks that contract and prepares managed CUDA nodes to usethose images:
dkms statusreports NVIDIA state or/var/lib/dkms/nvidiaexists, including a dangling link. Check again in the finalVHD content test, and require the marker and compiled module.
installedrecord for the on-diskdriver version, running kernel, and architecture. A loadable module or an
added/builtrecord is not enough.installaction if installation is requestedor the required DKMS state is missing, including validation-only/PIS paths.
The normal NVIDIA installer still uses
--dkms. Check the installed stateafter installation or validation, and fail provisioning if it is incomplete.
paths. Do not add a duplicate GPU-node guard inside
ensureGPUDrivers.Unlike #9545, this design does not park or restore DKMS trees. It does not use
dkms addor enableinstall-skip-build.Trade-offs and draft blockers:
aks-gpubuild imageand update the builder image reference before using these build checks. No
image tags are changed here; an old container that registers NVIDIA will fail
the new check when
NVIDIA_CUDA_PREBAKEis enabled.Existing registered images and deployed nodes are not repaired by this change.
validation-only mode. This can lose the startup-time saving from validating a
loadable prebake. No startup-time improvement or measurement is claimed.
but return success. If the remaining driver passes validation, a validation-only
GRID node can finish without invoking the GRID installer. A mocked control-flow
check demonstrated this path; it has not been verified on a real GRID node.
cleanup plus kernel patching; CUDA install/validation/PIS; GRID and opt-out;
kernel/driver mismatches; and GPU kernel update plus reboot.
Validation:
existing GPU suite and 8 real-DKMS fixture tests.
repository's standard exclusions.
git diff --checkpassed.go test ./pkg/agent -count=1passed.The DKMS tests use fixture source/module trees. They test status parsing and
kernel-update discovery, not a real NVIDIA build or GPU installation.
Which issue(s) this PR fixes:
Addresses the failed-cleanup / unwanted NVIDIA DKMS rebuild scenario discussed
in #9545. No separate issue is linked. Requires Azure/aks-gpu#191.