compute-domain: add managed ComputeDomain controller helm chart - #192
Open
Xu Xue (xuexu6666) wants to merge 12 commits into
Open
Xu Xue (xuexu6666) wants to merge 12 commits into
Xu Xue (xuexu6666) wants to merge 12 commits into
Conversation
Adds compute-domain/ holding the microsoft.managedcomputedomain helm chart: the cluster-scoped compute-domain-controller (Deployment + SA/ClusterRole/ ClusterRoleBinding) plus the compute-domain DeviceClasses. Delivered as an AKS core extension, mirroring managed DRANET (microsoft.manageddranet). driverManaged IMEX only: the controller spawns the per-ComputeDomain compute-domain-daemon DaemonSet from its own image at runtime. The kubelet-plugin (host systemd/deb) and nvidia-imex (aks-gpu) are delivered separately.
- values.yaml: point image at registry.k8s.io/dra-driver-nvidia/dra-driver-nvidia-gpu:v0.5.0 until the dalec/MCR onboarding lands (MCR path noted as the target). - chart/crds/: add computedomains + computedomaincliques CRDs so the controller can watch/reconcile them. Verified on a live 1.35 cluster: chart installs clean, controller 1/1 Running, all informers/RBAC OK (no forbidden), reconciles a ComputeDomain (creates the workload ResourceClaimTemplate + sets status).
Align with the managed-DRANET (microsoft.manageddranet) chart conventions: - standard Helm helpers (name/fullname/chart/labels/selectorLabels/serviceAccountName) with nameOverride/fullnameOverride - namespace = .Release.Namespace (extension-provided) instead of hardcoded kube-system - kubernetes.azure.com/managedby: aks gated on aks.managedByLabels.enabled - values.yaml restructured to dranet layout (image, rbac.create, serviceAccount, tolerations/nodeSelector/affinity, resources, podLabels/podAnnotations, deviceClasses.*.enabled, replicas) - Deployment fields driven from values Re-verified: helm lint clean, renders SA+ClusterRole+Binding+Deployment+2 DeviceClasses, controller 1/1 Running on a live 1.35 cluster.
A single-replica kube-system Deployment with system-cluster-critical priority and no PDB pins its node from cluster-autoscaler scale-down (CA won't evict kube-system pods lacking a PDB). The controller is stateless + leader-elected, so it's safe to drain/reschedule: add cluster-autoscaler.kubernetes.io/safe-to-evict: "true" (the same pattern the coredns-/konnectivity-agent-autoscaler helpers use). A PDB would be wrong here — minAvailable:1 on a single replica blocks all voluntary evictions.
…paced Role Match upstream's split: daemonsets + computedomaincliques are created only in the driver (release) namespace, so grant them via a namespaced Role/RoleBinding instead of the cluster-wide ClusterRole. The ClusterRole keeps only what's genuinely cluster-scoped (computedomains, resourceclaimtemplates, leases, nodes, pods reads). Verified: controller 1/1 Running, DaemonSet + ComputeDomainClique informers watch fine with the namespaced Role (no forbidden).
- ClusterRole nodes: add 'update' verb (was dropped). The controller labels/unlabels nodes with resource.nvidia.com/computeDomain=<UID> to place the per-CD daemon DaemonSet; without update this fails 'forbidden' and ComputeDomains never form. (Missed originally because the test cluster has no GPU nodes to label.) - Deployment: add CD_DAEMON_IMAGE_PULL_SECRET_NAMES env (gated on imagePullSecrets) so controller-spawned daemon pods can pull from a private registry. Verified against upstream: CRDs, DeviceClass selectors, daemonsets/cliques Role, and the rest of the controller env/command now match v0.5.0. Metrics env (HTTP_ENDPOINT/METRICS_PATH/PPROF_PATH) intentionally omitted (metrics disabled).
…ntion) Match the RP synth pattern (clusterhealthmonitor/.../resources.go): require a real linux AKS node (kubernetes.azure.com/cluster Exists, type != virtual-kubelet) with kubernetes.azure.com/mode=system as a HARD requirement (not the softer preferred used by coredns/metrics-server), tolerations node-role.kubernetes.io/master + CriticalAddonsOnly. Still no GPU-taint toleration, so it stays off GB nodes.
Actual usage is ~9-30Mi, but the controller watches all pods+nodes cluster-wide so its footprint scales with cluster size. Raise the request floor to 128Mi and the memory limit to 512Mi for headroom on large clusters. Still no CPU limit (avoids CFS throttling; matches dranet/upstream).
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds a dedicated
compute-domain/folder holding themicrosoft.managedcomputedomainhelm chart — the cluster-scoped half of the Grace-Blackwell (GB200/GB300) cross-node NVLink (MNNVL / IMEX) stack. Delivered as an AKS core extension, mirroring how managed DRANET (microsoft.manageddranet) is delivered.What the chart deploys
Deployment/compute-domain-controllerComputeDomainCRs, creates the workloadResourceClaimTemplate, and (driverManaged) spawns a per-ComputeDomaincompute-domain-daemonDaemonSetServiceAccount+ClusterRole+ClusterRoleBindingsystem:nodes)DeviceClass×2compute-domain-default-channel.nvidia.com(IMEX channel) +compute-domain-daemon.nvidia.com(daemon)What it deliberately does NOT deliver
compute-domain-daemon— created at runtime by the controller as a per-ComputeDomain DaemonSet, from the controller's own image (so the image must carrycompute-domain-controllerandcompute-domain-daemon).compute-domain-kubelet-plugin— host systemd (via thedra-driver-nvidia-gpudeb); its 2 read-onlyresource.nvidia.comreads bound tosystem:nodesare delivered separately.nvidia-imex— host binary installed by aks-gpu at boot; the daemon pod execs it via CDI. Hostnvidia-imex.servicestays off in driverManaged.IMEX mode: driverManaged only
Hard-defaults to
imex.mode: driverManaged.hostManagedis intentionally not offered — it would require AKS to own the host IMEX topology (nodes_config.cfg= rack peer set), which can't be built at node bootstrap (the peer set is cluster-scoped/converging) and just re-implements what the driverManaged DaemonSet does for free (via pod DNS names).Notes
kubernetes-sigs/dra-driver-nvidia-gpuv0.5.0 (deployments/helmtemplates).image.repository/image.taginvalues.yaml).helm lintclean;helm templaterenders SA + ClusterRole + Binding + Deployment + 2 DeviceClasses.