You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Add a separate Ubuntu 24.04 Gen2 AMD GPU VHD (2404gen2amdgpucontainerd) for MI300X. The host contains the pinned AMDGPU driver, firmware, amd-smi, lspci, and numactl; ROCm compute libraries and AI frameworks stay in workload containers.
Build AMD through the manual .pipelines/.vsts-vhd-builder-amd.yaml, with no dependency from ordinary VHD pipelines and no production publication. AMD failures affect only that separate run.
Keep ordinary PR/release matrices, the base Packer template, component manifest/schema, Ubuntu tool installer, and common SKU settings unchanged. Small AMD guards load separate installers, metadata, content checks, and the baked bootstrap validator.
Keep default node/E2E behavior. Opt-in bootstrap validates the baked driver in nodePrep; the AMD scenario explicitly selects the dedicated image, checks eight GPUs and AMD SMI, and requires CPU-reference PyTorch training to complete.
Scope cleanup to this build's verified subscription, resource group and ownership tags; disable broad shared cleanup in the AMD pipeline.
Validation:
Repository validation passed: 1,340 ShellSpec examples; make test, make lint, make generate; E2E scenario/config tests and focused AMD tests. Tests also cover missing/broken AMD files without changing normal or NVIDIA paths, and all 45 existing Linux image names/outputs match the base.
Dedicated AMD build 181874913 succeeded, including capture/content checks and France Central replication. Validation commit 72e925a9 changes only the pipeline entrypoint; its image inputs match PR implementation 3ce6d173. Later changes affect the AMD test launcher and documentation, not the captured image.
Fresh image 2404gen2amdgpucontainerd/1.1789808022.4162 booted on MI300X, joined AKS and passed the AMD E2E scenario with no skips. Kernel 6.8.0-1067-azure, module 7.1.3.31500000, AMD SMI and eight GPUs were verified.
Before and after an independently verified reboot: 40 CPU-reference training steps per GPU; 100 BF16 distributed Transformer steps on eight GPUs with finite values, decreasing loss and identical full-model hashes; 42,631 total full-buffer transfer calls with 254.77 TB remote traffic and zero mismatches. Median remote-send bandwidth was 278.7–291.4 GB/s per GPU. All workloads exited successfully; host checks passed before and after each round.
Qualification / rollout:
These are bounded functional tests on a fresh dedicated image, not peak-performance, all-future-transfer or vendor security-fix guarantees. Runtime observations were healthy; full vendor security qualification remains outstanding. Native AKS image selection/driver policy, native ANC JSON, full embedded legacy provisioning, PIS, serviced-kernel/DKMS updates and Secure Boot require separate qualification. This test used scriptless NBC with baked ANC/scripts and explicitly selected the AMD SIG version.
See the AMD VHD documentation for provenance, bandwidth definitions and before/after values, package footprint, network prerequisites and reproducible E2E commands. Current PR checks should be read separately from the completed dedicated AMD bake and MI300X qualification.
The reason will be displayed to describe this comment to others. Learn more.
Copilot review overview
🔵 Needs a closer look
The NBC-to-ANC conversion currently discards the explicit GPU-driver opt-out.
Review effort: Balanced Findings: None
Previously missed (1)
In code that hasn't changed since last review
Preserve ConfigGPUDriverIfNeeded when converting NBC to ANC
e2e/scenario/node_config.go:286
🟡 Medium Risk — 🏗️ Architecture: This conversion enables AMD from the NBC but still hard-codes ConfigGpuDriver to true on the next line. Consequently, an NBC with EnableAMDGPU=true and ConfigGPUDriverIfNeeded=false loses the explicit driver opt-out when converted to ANC, contrary to the documented parity between the two provisioning contracts. Map both fields from the NBC (and exercise the false case in the existing conversion test).
Cleanup misses resource groups from earlier retry attempts
.pipelines/.vsts-vhd-builder-amd.yaml:78
🟡 Medium Risk — Operational implications: This cleanup records only $(PKR_RG_NAME) from the final Packer attempt. The shared build task has retryCountOnTaskFailure: 3, while packer.mk overwrites packer-output at the start of each attempt, so a failed earlier attempt can leave a different temporary resource group that this step can no longer discover. Persist and ownership-check every resource-group name emitted across retries, or disable automatic task retries for this isolated pipeline.
Privileged plugin container lacks Linux capability restrictions
e2e/scenario/scenario_gpu_amd.go:175
🔒 High Risk — Security: The manifest shipped with the pinned upstream v1.31.0.11 plugin keeps privileged: true but also drops all Linux capabilities. Omitting that restriction gives this third-party container the full capability set while it mounts /sys and the kubelet device-plugin directory, unnecessarily increasing host impact if the image is compromised. Mirror the upstream hardening here.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
componentsThis pull request updates cached components on Linux or Windows VHDs
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
github-actions
Bot
added
the
components
What this PR does / why we need it:
Add a separate Ubuntu 24.04 Gen2 AMD GPU VHD (
2404gen2amdgpucontainerd) for MI300X. The host contains the pinned AMDGPU driver, firmware,amd-smi,lspci, andnumactl; ROCm compute libraries and AI frameworks stay in workload containers..pipelines/.vsts-vhd-builder-amd.yaml, with no dependency from ordinary VHD pipelines and no production publication. AMD failures affect only that separate run.nodePrep; the AMD scenario explicitly selects the dedicated image, checks eight GPUs and AMD SMI, and requires CPU-reference PyTorch training to complete.Validation:
make test,make lint,make generate; E2E scenario/config tests and focused AMD tests. Tests also cover missing/broken AMD files without changing normal or NVIDIA paths, and all 45 existing Linux image names/outputs match the base.181874913succeeded, including capture/content checks and France Central replication. Validation commit72e925a9changes only the pipeline entrypoint; its image inputs match PR implementation3ce6d173. Later changes affect the AMD test launcher and documentation, not the captured image.2404gen2amdgpucontainerd/1.1789808022.4162booted on MI300X, joined AKS and passed the AMD E2E scenario with no skips. Kernel6.8.0-1067-azure, module7.1.3.31500000, AMD SMI and eight GPUs were verified.Qualification / rollout:
These are bounded functional tests on a fresh dedicated image, not peak-performance, all-future-transfer or vendor security-fix guarantees. Runtime observations were healthy; full vendor security qualification remains outstanding. Native AKS image selection/driver policy, native ANC JSON, full embedded legacy provisioning, PIS, serviced-kernel/DKMS updates and Secure Boot require separate qualification. This test used scriptless NBC with baked ANC/scripts and explicitly selected the AMD SIG version.
See the AMD VHD documentation for provenance, bandwidth definitions and before/after values, package footprint, network prerequisites and reproducible E2E commands. Current PR checks should be read separately from the completed dedicated AMD bake and MI300X qualification.
Which issue(s) this PR fixes:
None linked.