From 01c25d249dfab00e4dcdf3588a83a39de8c3b912 Mon Sep 17 00:00:00 2001 From: xuexu6666 Date: Sat, 19 Sep 2026 15:36:46 -0500 Subject: [PATCH 01/12] compute-domain: add managed ComputeDomain controller helm chart Adds compute-domain/ holding the microsoft.managedcomputedomain helm chart: the cluster-scoped compute-domain-controller (Deployment + SA/ClusterRole/ ClusterRoleBinding) plus the compute-domain DeviceClasses. Delivered as an AKS core extension, mirroring managed DRANET (microsoft.manageddranet). driverManaged IMEX only: the controller spawns the per-ComputeDomain compute-domain-daemon DaemonSet from its own image at runtime. The kubelet-plugin (host systemd/deb) and nvidia-imex (aks-gpu) are delivered separately. --- compute-domain/README.md | 44 ++++++++++ compute-domain/chart/Chart.yaml | 20 +++++ compute-domain/chart/templates/_helpers.tpl | 30 +++++++ .../chart/templates/deployment.yaml | 82 +++++++++++++++++++ .../chart/templates/deviceclass.yaml | 25 ++++++ compute-domain/chart/templates/rbac.yaml | 55 +++++++++++++ .../chart/templates/serviceaccount.yaml | 7 ++ compute-domain/chart/values.yaml | 44 ++++++++++ 8 files changed, 307 insertions(+) create mode 100644 compute-domain/README.md create mode 100644 compute-domain/chart/Chart.yaml create mode 100644 compute-domain/chart/templates/_helpers.tpl create mode 100644 compute-domain/chart/templates/deployment.yaml create mode 100644 compute-domain/chart/templates/deviceclass.yaml create mode 100644 compute-domain/chart/templates/rbac.yaml create mode 100644 compute-domain/chart/templates/serviceaccount.yaml create mode 100644 compute-domain/chart/values.yaml diff --git a/compute-domain/README.md b/compute-domain/README.md new file mode 100644 index 0000000..a14f18a --- /dev/null +++ b/compute-domain/README.md @@ -0,0 +1,44 @@ +# Managed ComputeDomain (GB200/GB300) + +Helm chart for the **AKS-managed NVIDIA ComputeDomain controller** — the cluster-scoped +piece of the Grace-Blackwell cross-node NVLink (MNNVL / IMEX) stack. It is delivered as an +AKS core extension (`microsoft.managedcomputedomain`), analogous to how managed DRANET +(`microsoft.manageddranet`) is delivered. + +## What this chart deploys + +| Object | Purpose | +|---|---| +| `Deployment/compute-domain-controller` | cluster-scoped controller: reconciles `ComputeDomain` CRs, creates the workload `ResourceClaimTemplate`, and (driverManaged) spawns a per-ComputeDomain `compute-domain-daemon` DaemonSet | +| `ServiceAccount` + `ClusterRole` + `ClusterRoleBinding` | controller identity + RBAC, bound to the SA (blast radius = the controller pod, **not** `system:nodes`) | +| `DeviceClass/compute-domain-default-channel.nvidia.com` | the IMEX channel device workloads claim | +| `DeviceClass/compute-domain-daemon.nvidia.com` | the daemon device (driverManaged) | + +## What this chart does NOT deliver + +- **`compute-domain-daemon`** — not a static object; the controller creates it as a + per-ComputeDomain DaemonSet at runtime, from the controller's **own image** (so the image + must carry both `compute-domain-controller` and `compute-domain-daemon`). +- **`compute-domain-kubelet-plugin`** — runs as a host **systemd** service (delivered via the + `dra-driver-nvidia-gpu` deb), not by this chart. Its node-identity RBAC (2 read-only + `resource.nvidia.com` reads bound to `system:nodes`) is delivered separately. +- **`nvidia-imex`** — the host binary, installed by **aks-gpu** at boot; the daemon pod execs + it via CDI. The host `nvidia-imex.service` stays **off** in driverManaged. + +## IMEX mode: driverManaged (only) + +This chart hard-defaults to `imex.mode: driverManaged`. `hostManaged` is intentionally not +offered: it would require AKS to own the host IMEX topology (`nodes_config.cfg` = the rack +peer set), which cannot be built at node bootstrap (the peer set is cluster-scoped and +converging) and would just re-implement what the driverManaged DaemonSet does automatically +(via pod DNS names). + +## Image + +The controller image (`compute-domain-controller` + `compute-domain-daemon`) is onboarded via +dalec/MCR; set `image.repository` / `image.tag` in `values.yaml`. + +## Upstream + +Chart wraps [`kubernetes-sigs/dra-driver-nvidia-gpu`](https://github.com/kubernetes-sigs/dra-driver-nvidia-gpu) +`v0.5.0` (controller + DeviceClasses + RBAC derived from its `deployments/helm` templates). diff --git a/compute-domain/chart/Chart.yaml b/compute-domain/chart/Chart.yaml new file mode 100644 index 0000000..39fd75f --- /dev/null +++ b/compute-domain/chart/Chart.yaml @@ -0,0 +1,20 @@ +apiVersion: v2 +name: microsoft.managedcomputedomain +description: | + AKS-managed NVIDIA ComputeDomain controller for Grace-Blackwell (GB200/GB300). + Deploys the cluster-scoped compute-domain-controller (and its RBAC + DeviceClasses) + that binds GB nodes into cross-node NVLink (MNNVL) IMEX domains. The controller + dynamically creates a per-ComputeDomain compute-domain-daemon DaemonSet from its + own image (driverManaged IMEX). The node-local compute-domain kubelet-plugin and + the host nvidia-imex binary are delivered separately (deb / aks-gpu), not by this chart. +type: application +version: 0.0.1 +appVersion: "0.5.0" +keywords: + - nvidia + - dra + - compute-domain + - imex + - mnnvl + - gb200 + - gb300 diff --git a/compute-domain/chart/templates/_helpers.tpl b/compute-domain/chart/templates/_helpers.tpl new file mode 100644 index 0000000..7c3d60d --- /dev/null +++ b/compute-domain/chart/templates/_helpers.tpl @@ -0,0 +1,30 @@ +{{/* Common name for the controller workload/identity. */}} +{{- define "computedomain.name" -}} +compute-domain-controller +{{- end -}} + +{{/* Namespace (defaults to kube-system, overridable). */}} +{{- define "computedomain.namespace" -}} +{{- default "kube-system" .Values.namespace -}} +{{- end -}} + +{{/* Full controller image reference. */}} +{{- define "computedomain.image" -}} +{{- printf "%s:%s" .Values.image.repository .Values.image.tag -}} +{{- end -}} + +{{/* Standard labels — mirrors the managed-dranet chart. */}} +{{- define "computedomain.labels" -}} +helm.sh/chart: {{ .Chart.Name }}-{{ .Chart.Version }} +app.kubernetes.io/name: {{ include "computedomain.name" . }} +app.kubernetes.io/instance: {{ .Release.Name }} +app.kubernetes.io/version: {{ .Chart.AppVersion | quote }} +app.kubernetes.io/managed-by: {{ .Release.Service }} +kubernetes.azure.com/managedby: aks +{{- end -}} + +{{/* Selector labels. */}} +{{- define "computedomain.selectorLabels" -}} +app.kubernetes.io/name: {{ include "computedomain.name" . }} +app.kubernetes.io/instance: {{ .Release.Name }} +{{- end -}} diff --git a/compute-domain/chart/templates/deployment.yaml b/compute-domain/chart/templates/deployment.yaml new file mode 100644 index 0000000..a35e296 --- /dev/null +++ b/compute-domain/chart/templates/deployment.yaml @@ -0,0 +1,82 @@ +apiVersion: apps/v1 +kind: Deployment +metadata: + name: {{ include "computedomain.name" . }} + namespace: {{ include "computedomain.namespace" . }} + labels: + {{- include "computedomain.labels" . | nindent 4 }} +spec: + replicas: {{ .Values.controller.replicas }} + selector: + matchLabels: + {{- include "computedomain.selectorLabels" . | nindent 6 }} + template: + metadata: + labels: + {{- include "computedomain.labels" . | nindent 8 }} + spec: + serviceAccountName: {{ include "computedomain.name" . }} + priorityClassName: {{ .Values.controller.priorityClassName }} + # Control-plane pod: run on a Linux system node, tolerate control-plane taints. + nodeSelector: + kubernetes.io/os: linux + tolerations: + - key: CriticalAddonsOnly + operator: Exists + - key: node-role.kubernetes.io/control-plane + operator: Exists + effect: NoSchedule + containers: + - name: compute-domain + image: {{ include "computedomain.image" . }} + imagePullPolicy: {{ .Values.image.pullPolicy }} + command: ["compute-domain-controller", "-v", "$(LOG_VERBOSITY)"] + securityContext: + allowPrivilegeEscalation: false + readOnlyRootFilesystem: true + capabilities: + drop: ["ALL"] + resources: + {{- toYaml .Values.controller.resources | nindent 12 }} + env: + - name: LOG_VERBOSITY + value: "{{ .Values.logVerbosity }}" + # Verbosity for the dynamically-launched compute-domain-daemon pods + # (their pod spec is rendered by the controller, not this chart). + - name: LOG_VERBOSITY_CD_DAEMON + value: "{{ .Values.logVerbosity }}" + - name: POD_NAME + valueFrom: + fieldRef: + fieldPath: metadata.name + - name: NAMESPACE + valueFrom: + fieldRef: + fieldPath: metadata.namespace + # The controller spawns the per-ComputeDomain compute-domain-daemon + # DaemonSet from THIS image, so it must contain compute-domain-daemon too. + - name: IMAGE_NAME + value: {{ include "computedomain.image" . }} + # runc: explicit "void" so the controller pod grabs no GPUs. + - name: NVIDIA_VISIBLE_DEVICES + value: void + - name: IMEX_MODE + value: "{{ .Values.imex.mode }}" + - name: IMEX_ISOLATION + value: "{{ .Values.imex.isolation }}" + {{- if .Values.featureGates }} + - name: FEATURE_GATES + value: "{{- range $k, $v := .Values.featureGates }}{{ $k }}={{ $v }},{{- end }}" + {{- end }} + - name: LEADER_ELECTION_ENABLED + value: "{{ .Values.controller.leaderElection.enabled }}" + - name: LEADER_ELECTION_LEASE_LOCK_NAME + value: "{{ include "computedomain.name" . }}" + - name: LEADER_ELECTION_LEASE_LOCK_NAMESPACE + value: "{{ include "computedomain.namespace" . }}" + - name: LEADER_ELECTION_LEASE_DURATION + value: "{{ .Values.controller.leaderElection.leaseDuration }}" + - name: LEADER_ELECTION_RENEW_DEADLINE + value: "{{ .Values.controller.leaderElection.renewDeadline }}" + - name: LEADER_ELECTION_RETRY_PERIOD + value: "{{ .Values.controller.leaderElection.retryPeriod }}" diff --git a/compute-domain/chart/templates/deviceclass.yaml b/compute-domain/chart/templates/deviceclass.yaml new file mode 100644 index 0000000..d05a81b --- /dev/null +++ b/compute-domain/chart/templates/deviceclass.yaml @@ -0,0 +1,25 @@ +# DeviceClasses the compute-domain kubelet-plugin publishes devices under. +# resource.k8s.io/v1 is GA as of Kubernetes 1.34; GB runs 1.35. +--- +apiVersion: resource.k8s.io/v1 +kind: DeviceClass +metadata: + name: compute-domain-default-channel.nvidia.com + labels: + {{- include "computedomain.labels" . | nindent 4 }} +spec: + selectors: + - cel: + expression: "device.driver == 'compute-domain.nvidia.com' && device.attributes['compute-domain.nvidia.com'].type == 'channel' && device.attributes['compute-domain.nvidia.com'].id == 0" +--- +# driverManaged only: the per-ComputeDomain daemon claims a 'daemon' device. +apiVersion: resource.k8s.io/v1 +kind: DeviceClass +metadata: + name: compute-domain-daemon.nvidia.com + labels: + {{- include "computedomain.labels" . | nindent 4 }} +spec: + selectors: + - cel: + expression: "device.driver == 'compute-domain.nvidia.com' && device.attributes['compute-domain.nvidia.com'].type == 'daemon'" diff --git a/compute-domain/chart/templates/rbac.yaml b/compute-domain/chart/templates/rbac.yaml new file mode 100644 index 0000000..f404dfa --- /dev/null +++ b/compute-domain/chart/templates/rbac.yaml @@ -0,0 +1,55 @@ +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + name: {{ include "computedomain.name" . }} + labels: + {{- include "computedomain.labels" . | nindent 4 }} +rules: + # Reconcile the ComputeDomain CRs and set their status. + - apiGroups: ["resource.nvidia.com"] + resources: ["computedomains"] + verbs: ["get", "list", "watch", "update"] + - apiGroups: ["resource.nvidia.com"] + resources: ["computedomains/status"] + verbs: ["update"] + # Workload ResourceClaimTemplate the controller manages for each ComputeDomain. + - apiGroups: ["resource.k8s.io"] + resources: ["resourceclaimtemplates"] + verbs: ["get", "list", "watch", "create", "update", "delete"] + # Leader election. + - apiGroups: ["coordination.k8s.io"] + resources: ["leases"] + verbs: ["get", "create", "update"] + - apiGroups: [""] + resources: ["nodes"] + verbs: ["get", "list", "watch"] + - apiGroups: [""] + resources: ["pods"] + verbs: ["get", "list", "watch"] + # driverManaged only: the controller creates a per-ComputeDomain + # compute-domain-daemon DaemonSet and the ComputeDomainClique objects. + # (Dropped under hostManaged IMEX, which this managed chart does not use.) + - apiGroups: ["apps"] + resources: ["daemonsets"] + verbs: ["get", "list", "watch", "create", "update", "patch", "delete"] + - apiGroups: ["resource.nvidia.com"] + resources: ["computedomaincliques"] + verbs: ["get", "list", "watch", "update"] +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRoleBinding +metadata: + name: {{ include "computedomain.name" . }} + labels: + {{- include "computedomain.labels" . | nindent 4 }} +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: {{ include "computedomain.name" . }} +subjects: + # Bound to the controller ServiceAccount (NOT system:nodes) — blast radius is the + # single controller pod. The node-identity kubelet-plugin binding (2 read-only + # resource.nvidia.com reads to system:nodes) is delivered separately. + - kind: ServiceAccount + name: {{ include "computedomain.name" . }} + namespace: {{ include "computedomain.namespace" . }} diff --git a/compute-domain/chart/templates/serviceaccount.yaml b/compute-domain/chart/templates/serviceaccount.yaml new file mode 100644 index 0000000..ba57b65 --- /dev/null +++ b/compute-domain/chart/templates/serviceaccount.yaml @@ -0,0 +1,7 @@ +apiVersion: v1 +kind: ServiceAccount +metadata: + name: {{ include "computedomain.name" . }} + namespace: {{ include "computedomain.namespace" . }} + labels: + {{- include "computedomain.labels" . | nindent 4 }} diff --git a/compute-domain/chart/values.yaml b/compute-domain/chart/values.yaml new file mode 100644 index 0000000..1644392 --- /dev/null +++ b/compute-domain/chart/values.yaml @@ -0,0 +1,44 @@ +# Values for microsoft.managedcomputedomain. +# The consumer is the AKS core-extension helm runner (not the public); the Azure.* +# block is injected by the extension system at install time (see dranet's chart). + +# Namespace the controller runs in. kube-system matches the managed-dranet model. +namespace: kube-system + +image: + # Compute-domain controller image (contains BOTH compute-domain-controller and + # compute-domain-daemon; the controller spawns the daemon DaemonSet from this same + # image via the IMAGE_NAME env). Onboarded via dalec/MCR. + repository: mcr.microsoft.com/oss/v2/nvidia/compute-domain-controller + tag: "v0.5.0" + pullPolicy: IfNotPresent + +# IMEX deployment mode. driverManaged = the controller creates a per-ComputeDomain +# compute-domain-daemon DaemonSet that runs nvidia-imex (host binary via CDI). +# hostManaged is intentionally NOT supported by this managed chart: it would require +# AKS to own host IMEX topology (nodes_config.cfg) which is impossible at node +# bootstrap and duplicates what the DaemonSet does for free. +imex: + mode: driverManaged + isolation: domain + +logVerbosity: 4 + +controller: + replicas: 1 + priorityClassName: system-cluster-critical + leaderElection: + enabled: true + leaseDuration: "15s" + renewDeadline: "10s" + retryPeriod: "2s" + resources: + requests: + cpu: 50m + memory: 64Mi + limits: + memory: 256Mi + +# Feature gates passed to the controller (FEATURE_GATES env). Left empty = upstream +# defaults (IMEXDaemonsWithDNSNames on, requires driver >= 570.158 which R580 satisfies). +featureGates: {} From 3bfc95d54ee0f83db1a4f75a9b4f7f0a228152c0 Mon Sep 17 00:00:00 2001 From: xuexu6666 Date: Sat, 19 Sep 2026 15:43:57 -0500 Subject: [PATCH 02/12] compute-domain: use upstream image for testing + ship ComputeDomain CRDs - values.yaml: point image at registry.k8s.io/dra-driver-nvidia/dra-driver-nvidia-gpu:v0.5.0 until the dalec/MCR onboarding lands (MCR path noted as the target). - chart/crds/: add computedomains + computedomaincliques CRDs so the controller can watch/reconcile them. Verified on a live 1.35 cluster: chart installs clean, controller 1/1 Running, all informers/RBAC OK (no forbidden), reconciles a ComputeDomain (creates the workload ResourceClaimTemplate + sets status). --- ...ource.nvidia.com_computedomaincliques.yaml | 84 +++++++++ .../resource.nvidia.com_computedomains.yaml | 163 ++++++++++++++++++ compute-domain/chart/values.yaml | 6 +- 3 files changed, 251 insertions(+), 2 deletions(-) create mode 100644 compute-domain/chart/crds/resource.nvidia.com_computedomaincliques.yaml create mode 100644 compute-domain/chart/crds/resource.nvidia.com_computedomains.yaml diff --git a/compute-domain/chart/crds/resource.nvidia.com_computedomaincliques.yaml b/compute-domain/chart/crds/resource.nvidia.com_computedomaincliques.yaml new file mode 100644 index 0000000..b3eddef --- /dev/null +++ b/compute-domain/chart/crds/resource.nvidia.com_computedomaincliques.yaml @@ -0,0 +1,84 @@ +--- +apiVersion: apiextensions.k8s.io/v1 +kind: CustomResourceDefinition +metadata: + annotations: + controller-gen.kubebuilder.io/version: v0.17.1 + name: computedomaincliques.resource.nvidia.com +spec: + group: resource.nvidia.com + names: + kind: ComputeDomainClique + listKind: ComputeDomainCliqueList + plural: computedomaincliques + singular: computedomainclique + scope: Namespaced + versions: + - name: v1beta1 + schema: + openAPIV3Schema: + description: |- + ComputeDomainClique holds information about a specific clique within a ComputeDomain. + It is created in the driver namespace and named as ".". + properties: + apiVersion: + description: |- + APIVersion defines the versioned schema of this representation of an object. + Servers should convert recognized schemas to the latest internal value, and + may reject unrecognized values. + More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources + type: string + daemons: + items: + description: ComputeDomainDaemonInfo provides information about each + daemon in a ComputeDomainClique. + properties: + cliqueID: + type: string + index: + description: |- + The Index field is used to ensure a consistent IP-to-DNS name + mapping across all machines within an IMEX domain. Each node's index + directly determines its DNS name within a given NVLink partition + (i.e. clique). In other words, the 2-tuple of (CliqueID, Index) will + always be unique. This field is marked as optional (but not + omitempty) in order to support downgrades and avoid an API bump. + type: integer + ipAddress: + type: string + nodeName: + type: string + status: + default: NotReady + description: |- + The Status field tracks the readiness of the IMEX daemon running on + this node. It gets switched to Ready whenever the IMEX daemon is + ready to broker GPU memory exchanges and switches to NotReady when + it is not. It is marked as optional in order to support downgrades + and avoid an API bump. + enum: + - Ready + - NotReady + type: string + required: + - cliqueID + - ipAddress + - nodeName + type: object + type: array + x-kubernetes-list-map-keys: + - nodeName + x-kubernetes-list-type: map + kind: + description: |- + Kind is a string value representing the REST resource this object represents. + Servers may infer this from the endpoint the client submits requests to. + Cannot be updated. + In CamelCase. + More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds + type: string + metadata: + type: object + type: object + served: true + storage: true diff --git a/compute-domain/chart/crds/resource.nvidia.com_computedomains.yaml b/compute-domain/chart/crds/resource.nvidia.com_computedomains.yaml new file mode 100644 index 0000000..0811b3c --- /dev/null +++ b/compute-domain/chart/crds/resource.nvidia.com_computedomains.yaml @@ -0,0 +1,163 @@ +--- +apiVersion: apiextensions.k8s.io/v1 +kind: CustomResourceDefinition +metadata: + annotations: + controller-gen.kubebuilder.io/version: v0.17.1 + name: computedomains.resource.nvidia.com +spec: + group: resource.nvidia.com + names: + kind: ComputeDomain + listKind: ComputeDomainList + plural: computedomains + singular: computedomain + scope: Namespaced + versions: + - name: v1beta1 + schema: + openAPIV3Schema: + description: ComputeDomain prepares a set of nodes to run a multi-node workload + in. + properties: + apiVersion: + description: |- + APIVersion defines the versioned schema of this representation of an object. + Servers should convert recognized schemas to the latest internal value, and + may reject unrecognized values. + More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources + type: string + kind: + description: |- + Kind is a string value representing the REST resource this object represents. + Servers may infer this from the endpoint the client submits requests to. + Cannot be updated. + In CamelCase. + More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds + type: string + metadata: + type: object + spec: + description: ComputeDomainSpec provides the spec for a ComputeDomain. + properties: + channel: + description: ComputeDomainChannelSpec provides the spec for a channel + used to run a workload inside a ComputeDomain. + properties: + allocationMode: + default: Single + description: |- + Allows for requesting all IMEX channels (the maximum per IMEX domain) or + precisely one. + enum: + - All + - Single + type: string + resourceClaimTemplate: + description: ComputeDomainResourceClaimTemplate provides the details + of the ResourceClaimTemplate to generate. + properties: + name: + type: string + required: + - name + type: object + required: + - resourceClaimTemplate + type: object + numNodes: + default: 0 + description: |- + Intended number of IMEX daemons (i.e., individual compute nodes) in the + ComputeDomain. Must be zero or greater. + + With `featureGates.IMEXDaemonsWithDNSNames=true` (the default), this is + recommended to be set to zero (default). Workload must implement and consult its + own source of truth for the number of workers online before trying to + share GPU memory (and hence triggering IMEX interaction). When non-zero, + `numNodes` is used only for automatically updating the global + ComputeDomain `Status` (indicating `Ready` when the number of ready IMEX + daemons equals `numNodes`). In this mode, a `numNodes` value greater than + zero in particular does not gate the startup of IMEX daemons: individual + IMEX daemons are started immediately without waiting for its peers, and + any workload pod gets released right after its local IMEX daemon has + started. + + With `featureGates.IMEXDaemonsWithDNSNames=false`, `numNodes` must be set + to the expected number of worker nodes joining the ComputeDomain. In that + mode, all workload pods are held back (with containers in state + `ContainerCreating`) until the underlying IMEX domain has been joined by + `numNodes` IMEX daemons. Pods from more than `numNodes` nodes trying to + join the ComputeDomain may lead to unexpected behavior. + + The `numNodes` parameter is deprecated and will be removed in the next + API version. + minimum: 0 + type: integer + required: + - channel + type: object + x-kubernetes-validations: + - message: A computeDomain.spec is immutable + rule: self == oldSelf + status: + description: |- + Global ComputeDomain status. Can be used to guide debugging efforts. + Workload however should not rely on inspecting this field at any point + during its lifecycle. + properties: + nodes: + items: + description: ComputeDomainNode provides information about each node + added to a ComputeDomain. + properties: + cliqueID: + type: string + index: + description: |- + The Index field is used to ensure a consistent IP-to-DNS name + mapping across all machines within an IMEX domain. Each node's index + directly determines its DNS name within a given NVLink partition + (i.e. clique). In other words, the 2-tuple of (CliqueID, Index) will + always be unique. This field is marked as optional (but not + omitempty) in order to support downgrades and avoid an API bump. + type: integer + ipAddress: + type: string + name: + type: string + status: + default: NotReady + description: |- + The Status field tracks the readiness of the IMEX daemon running on + this node. It gets switched to Ready whenever the IMEX daemon is + ready to broker GPU memory exchanges and switches to NotReady when + it is not. It is marked as optional in order to support downgrades + and avoid an API bump. + enum: + - Ready + - NotReady + type: string + required: + - cliqueID + - ipAddress + - name + type: object + type: array + x-kubernetes-list-map-keys: + - name + x-kubernetes-list-type: map + status: + default: NotReady + enum: + - Ready + - NotReady + type: string + required: + - status + type: object + type: object + served: true + storage: true + subresources: + status: {} diff --git a/compute-domain/chart/values.yaml b/compute-domain/chart/values.yaml index 1644392..d3858d3 100644 --- a/compute-domain/chart/values.yaml +++ b/compute-domain/chart/values.yaml @@ -8,8 +8,10 @@ namespace: kube-system image: # Compute-domain controller image (contains BOTH compute-domain-controller and # compute-domain-daemon; the controller spawns the daemon DaemonSet from this same - # image via the IMAGE_NAME env). Onboarded via dalec/MCR. - repository: mcr.microsoft.com/oss/v2/nvidia/compute-domain-controller + # image via the IMAGE_NAME env). + # TESTING: upstream image is used until the dalec/MCR onboarding lands + # (target: mcr.microsoft.com/oss/v2/nvidia/compute-domain-controller). + repository: registry.k8s.io/dra-driver-nvidia/dra-driver-nvidia-gpu tag: "v0.5.0" pullPolicy: IfNotPresent From 4aa1866961e184b8c443c70bd5c854dd7996b08c Mon Sep 17 00:00:00 2001 From: xuexu6666 Date: Mon, 21 Sep 2026 10:50:22 -0500 Subject: [PATCH 03/12] compute-domain: refactor chart to standard managed-extension form Align with the managed-DRANET (microsoft.manageddranet) chart conventions: - standard Helm helpers (name/fullname/chart/labels/selectorLabels/serviceAccountName) with nameOverride/fullnameOverride - namespace = .Release.Namespace (extension-provided) instead of hardcoded kube-system - kubernetes.azure.com/managedby: aks gated on aks.managedByLabels.enabled - values.yaml restructured to dranet layout (image, rbac.create, serviceAccount, tolerations/nodeSelector/affinity, resources, podLabels/podAnnotations, deviceClasses.*.enabled, replicas) - Deployment fields driven from values Re-verified: helm lint clean, renders SA+ClusterRole+Binding+Deployment+2 DeviceClasses, controller 1/1 Running on a live 1.35 cluster. --- compute-domain/chart/templates/_helpers.tpl | 38 +++++---- .../chart/templates/deployment.yaml | 61 ++++++++++----- .../chart/templates/deviceclass.yaml | 14 +++- compute-domain/chart/templates/rbac.yaml | 18 +++-- .../chart/templates/serviceaccount.yaml | 11 ++- compute-domain/chart/values.yaml | 77 +++++++++++++------ 6 files changed, 153 insertions(+), 66 deletions(-) diff --git a/compute-domain/chart/templates/_helpers.tpl b/compute-domain/chart/templates/_helpers.tpl index 7c3d60d..c481aaa 100644 --- a/compute-domain/chart/templates/_helpers.tpl +++ b/compute-domain/chart/templates/_helpers.tpl @@ -1,30 +1,40 @@ -{{/* Common name for the controller workload/identity. */}} {{- define "computedomain.name" -}} -compute-domain-controller +{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" -}} {{- end -}} -{{/* Namespace (defaults to kube-system, overridable). */}} -{{- define "computedomain.namespace" -}} -{{- default "kube-system" .Values.namespace -}} +{{- define "computedomain.fullname" -}} +{{- if .Values.fullnameOverride -}} +{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" -}} +{{- else -}} +{{- $name := default .Chart.Name .Values.nameOverride -}} +{{- if contains $name .Release.Name -}} +{{- .Release.Name | trunc 63 | trimSuffix "-" -}} +{{- else -}} +{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" -}} +{{- end -}} +{{- end -}} {{- end -}} -{{/* Full controller image reference. */}} -{{- define "computedomain.image" -}} -{{- printf "%s:%s" .Values.image.repository .Values.image.tag -}} +{{- define "computedomain.chart" -}} +{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" -}} {{- end -}} -{{/* Standard labels — mirrors the managed-dranet chart. */}} {{- define "computedomain.labels" -}} -helm.sh/chart: {{ .Chart.Name }}-{{ .Chart.Version }} +helm.sh/chart: {{ include "computedomain.chart" . }} app.kubernetes.io/name: {{ include "computedomain.name" . }} app.kubernetes.io/instance: {{ .Release.Name }} app.kubernetes.io/version: {{ .Chart.AppVersion | quote }} app.kubernetes.io/managed-by: {{ .Release.Service }} -kubernetes.azure.com/managedby: aks {{- end -}} -{{/* Selector labels. */}} {{- define "computedomain.selectorLabels" -}} -app.kubernetes.io/name: {{ include "computedomain.name" . }} -app.kubernetes.io/instance: {{ .Release.Name }} +app: {{ include "computedomain.name" . }} +{{- end -}} + +{{- define "computedomain.serviceAccountName" -}} +{{- include "computedomain.fullname" . -}} +{{- end -}} + +{{- define "computedomain.image" -}} +{{- printf "%s:%s" .Values.image.repository .Values.image.tag -}} {{- end -}} diff --git a/compute-domain/chart/templates/deployment.yaml b/compute-domain/chart/templates/deployment.yaml index a35e296..c642a86 100644 --- a/compute-domain/chart/templates/deployment.yaml +++ b/compute-domain/chart/templates/deployment.yaml @@ -1,12 +1,15 @@ apiVersion: apps/v1 kind: Deployment metadata: - name: {{ include "computedomain.name" . }} - namespace: {{ include "computedomain.namespace" . }} + name: {{ include "computedomain.fullname" . }} + namespace: {{ .Release.Namespace }} labels: {{- include "computedomain.labels" . | nindent 4 }} + {{- if .Values.aks.managedByLabels.enabled }} + kubernetes.azure.com/managedby: aks + {{- end }} spec: - replicas: {{ .Values.controller.replicas }} + replicas: {{ .Values.replicas }} selector: matchLabels: {{- include "computedomain.selectorLabels" . | nindent 6 }} @@ -14,18 +17,38 @@ spec: metadata: labels: {{- include "computedomain.labels" . | nindent 8 }} + {{- include "computedomain.selectorLabels" . | nindent 8 }} + {{- if .Values.aks.managedByLabels.enabled }} + kubernetes.azure.com/managedby: aks + {{- end }} + {{- with .Values.podLabels }} + {{- toYaml . | nindent 8 }} + {{- end }} + {{- with .Values.podAnnotations }} + annotations: + {{- toYaml . | nindent 8 }} + {{- end }} spec: - serviceAccountName: {{ include "computedomain.name" . }} - priorityClassName: {{ .Values.controller.priorityClassName }} - # Control-plane pod: run on a Linux system node, tolerate control-plane taints. + serviceAccountName: {{ include "computedomain.serviceAccountName" . }} + {{- with .Values.priorityClassName }} + priorityClassName: {{ . }} + {{- end }} + {{- with .Values.imagePullSecrets }} + imagePullSecrets: + {{- toYaml . | nindent 8 }} + {{- end }} + {{- with .Values.nodeSelector }} nodeSelector: - kubernetes.io/os: linux + {{- toYaml . | nindent 8 }} + {{- end }} + {{- with .Values.tolerations }} tolerations: - - key: CriticalAddonsOnly - operator: Exists - - key: node-role.kubernetes.io/control-plane - operator: Exists - effect: NoSchedule + {{- toYaml . | nindent 8 }} + {{- end }} + {{- with .Values.affinity }} + affinity: + {{- toYaml . | nindent 8 }} + {{- end }} containers: - name: compute-domain image: {{ include "computedomain.image" . }} @@ -37,7 +60,7 @@ spec: capabilities: drop: ["ALL"] resources: - {{- toYaml .Values.controller.resources | nindent 12 }} + {{- toYaml .Values.resources | nindent 12 }} env: - name: LOG_VERBOSITY value: "{{ .Values.logVerbosity }}" @@ -69,14 +92,14 @@ spec: value: "{{- range $k, $v := .Values.featureGates }}{{ $k }}={{ $v }},{{- end }}" {{- end }} - name: LEADER_ELECTION_ENABLED - value: "{{ .Values.controller.leaderElection.enabled }}" + value: "{{ .Values.leaderElection.enabled }}" - name: LEADER_ELECTION_LEASE_LOCK_NAME - value: "{{ include "computedomain.name" . }}" + value: "{{ include "computedomain.fullname" . }}" - name: LEADER_ELECTION_LEASE_LOCK_NAMESPACE - value: "{{ include "computedomain.namespace" . }}" + value: "{{ .Release.Namespace }}" - name: LEADER_ELECTION_LEASE_DURATION - value: "{{ .Values.controller.leaderElection.leaseDuration }}" + value: "{{ .Values.leaderElection.leaseDuration }}" - name: LEADER_ELECTION_RENEW_DEADLINE - value: "{{ .Values.controller.leaderElection.renewDeadline }}" + value: "{{ .Values.leaderElection.renewDeadline }}" - name: LEADER_ELECTION_RETRY_PERIOD - value: "{{ .Values.controller.leaderElection.retryPeriod }}" + value: "{{ .Values.leaderElection.retryPeriod }}" diff --git a/compute-domain/chart/templates/deviceclass.yaml b/compute-domain/chart/templates/deviceclass.yaml index d05a81b..5dbdc30 100644 --- a/compute-domain/chart/templates/deviceclass.yaml +++ b/compute-domain/chart/templates/deviceclass.yaml @@ -1,25 +1,35 @@ # DeviceClasses the compute-domain kubelet-plugin publishes devices under. # resource.k8s.io/v1 is GA as of Kubernetes 1.34; GB runs 1.35. +{{- if .Values.deviceClasses.computeDomainDefaultChannel.enabled }} --- apiVersion: resource.k8s.io/v1 kind: DeviceClass metadata: - name: compute-domain-default-channel.nvidia.com + name: {{ .Values.deviceClasses.computeDomainDefaultChannel.name | default "compute-domain-default-channel.nvidia.com" }} labels: {{- include "computedomain.labels" . | nindent 4 }} + {{- if .Values.aks.managedByLabels.enabled }} + kubernetes.azure.com/managedby: aks + {{- end }} spec: selectors: - cel: expression: "device.driver == 'compute-domain.nvidia.com' && device.attributes['compute-domain.nvidia.com'].type == 'channel' && device.attributes['compute-domain.nvidia.com'].id == 0" +{{- end }} +{{- if .Values.deviceClasses.computeDomainDaemon.enabled }} --- # driverManaged only: the per-ComputeDomain daemon claims a 'daemon' device. apiVersion: resource.k8s.io/v1 kind: DeviceClass metadata: - name: compute-domain-daemon.nvidia.com + name: {{ .Values.deviceClasses.computeDomainDaemon.name | default "compute-domain-daemon.nvidia.com" }} labels: {{- include "computedomain.labels" . | nindent 4 }} + {{- if .Values.aks.managedByLabels.enabled }} + kubernetes.azure.com/managedby: aks + {{- end }} spec: selectors: - cel: expression: "device.driver == 'compute-domain.nvidia.com' && device.attributes['compute-domain.nvidia.com'].type == 'daemon'" +{{- end }} diff --git a/compute-domain/chart/templates/rbac.yaml b/compute-domain/chart/templates/rbac.yaml index f404dfa..d111370 100644 --- a/compute-domain/chart/templates/rbac.yaml +++ b/compute-domain/chart/templates/rbac.yaml @@ -1,9 +1,13 @@ +{{- if .Values.rbac.create }} apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: - name: {{ include "computedomain.name" . }} + name: {{ include "computedomain.fullname" . }} labels: {{- include "computedomain.labels" . | nindent 4 }} + {{- if .Values.aks.managedByLabels.enabled }} + kubernetes.azure.com/managedby: aks + {{- end }} rules: # Reconcile the ComputeDomain CRs and set their status. - apiGroups: ["resource.nvidia.com"] @@ -39,17 +43,21 @@ rules: apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: - name: {{ include "computedomain.name" . }} + name: {{ include "computedomain.fullname" . }} labels: {{- include "computedomain.labels" . | nindent 4 }} + {{- if .Values.aks.managedByLabels.enabled }} + kubernetes.azure.com/managedby: aks + {{- end }} roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole - name: {{ include "computedomain.name" . }} + name: {{ include "computedomain.fullname" . }} subjects: # Bound to the controller ServiceAccount (NOT system:nodes) — blast radius is the # single controller pod. The node-identity kubelet-plugin binding (2 read-only # resource.nvidia.com reads to system:nodes) is delivered separately. - kind: ServiceAccount - name: {{ include "computedomain.name" . }} - namespace: {{ include "computedomain.namespace" . }} + name: {{ include "computedomain.serviceAccountName" . }} + namespace: {{ .Release.Namespace }} +{{- end }} diff --git a/compute-domain/chart/templates/serviceaccount.yaml b/compute-domain/chart/templates/serviceaccount.yaml index ba57b65..c40c87e 100644 --- a/compute-domain/chart/templates/serviceaccount.yaml +++ b/compute-domain/chart/templates/serviceaccount.yaml @@ -1,7 +1,14 @@ apiVersion: v1 kind: ServiceAccount metadata: - name: {{ include "computedomain.name" . }} - namespace: {{ include "computedomain.namespace" . }} + name: {{ include "computedomain.serviceAccountName" . }} + namespace: {{ .Release.Namespace }} labels: {{- include "computedomain.labels" . | nindent 4 }} + {{- if .Values.aks.managedByLabels.enabled }} + kubernetes.azure.com/managedby: aks + {{- end }} + {{- with .Values.serviceAccount.annotations }} + annotations: + {{- toYaml . | nindent 4 }} + {{- end }} diff --git a/compute-domain/chart/values.yaml b/compute-domain/chart/values.yaml index d3858d3..c471a2a 100644 --- a/compute-domain/chart/values.yaml +++ b/compute-domain/chart/values.yaml @@ -1,46 +1,75 @@ # Values for microsoft.managedcomputedomain. # The consumer is the AKS core-extension helm runner (not the public); the Azure.* -# block is injected by the extension system at install time (see dranet's chart). +# block is injected by the extension system at install time (same model as managed DRANET). -# Namespace the controller runs in. kube-system matches the managed-dranet model. -namespace: kube-system +nameOverride: compute-domain-controller +fullnameOverride: compute-domain-controller image: # Compute-domain controller image (contains BOTH compute-domain-controller and # compute-domain-daemon; the controller spawns the daemon DaemonSet from this same # image via the IMAGE_NAME env). - # TESTING: upstream image is used until the dalec/MCR onboarding lands + # TESTING: upstream image until the dalec/MCR onboarding lands # (target: mcr.microsoft.com/oss/v2/nvidia/compute-domain-controller). repository: registry.k8s.io/dra-driver-nvidia/dra-driver-nvidia-gpu tag: "v0.5.0" pullPolicy: IfNotPresent +imagePullSecrets: [] + +aks: + managedByLabels: + enabled: false + +rbac: + create: true + +serviceAccount: + annotations: {} # IMEX deployment mode. driverManaged = the controller creates a per-ComputeDomain # compute-domain-daemon DaemonSet that runs nvidia-imex (host binary via CDI). -# hostManaged is intentionally NOT supported by this managed chart: it would require -# AKS to own host IMEX topology (nodes_config.cfg) which is impossible at node -# bootstrap and duplicates what the DaemonSet does for free. +# hostManaged is intentionally NOT supported by this managed chart (see README). imex: mode: driverManaged isolation: domain logVerbosity: 4 -controller: - replicas: 1 - priorityClassName: system-cluster-critical - leaderElection: - enabled: true - leaseDuration: "15s" - renewDeadline: "10s" - retryPeriod: "2s" - resources: - requests: - cpu: 50m - memory: 64Mi - limits: - memory: 256Mi - -# Feature gates passed to the controller (FEATURE_GATES env). Left empty = upstream -# defaults (IMEXDaemonsWithDNSNames on, requires driver >= 570.158 which R580 satisfies). +replicas: 1 + +leaderElection: + enabled: true + leaseDuration: "15s" + renewDeadline: "10s" + retryPeriod: "2s" + +# Feature gates passed to the controller (FEATURE_GATES env). Empty = upstream defaults +# (IMEXDaemonsWithDNSNames on, requires driver >= 570.158 which R580 satisfies). featureGates: {} + +deviceClasses: + computeDomainDefaultChannel: + enabled: true + name: compute-domain-default-channel.nvidia.com + computeDomainDaemon: + enabled: true + name: compute-domain-daemon.nvidia.com + +priorityClassName: system-cluster-critical +nodeSelector: + kubernetes.io/os: linux +tolerations: + - key: CriticalAddonsOnly + operator: Exists + - key: node-role.kubernetes.io/control-plane + operator: Exists + effect: NoSchedule +affinity: {} +podAnnotations: {} +podLabels: {} +resources: + requests: + cpu: 50m + memory: 64Mi + limits: + memory: 256Mi From 6199fec1fb217f0ea93e278d9282c046e8f684c2 Mon Sep 17 00:00:00 2001 From: xuexu6666 Date: Mon, 21 Sep 2026 13:07:47 -0500 Subject: [PATCH 04/12] compute-domain: mark controller safe-to-evict for cluster autoscaler MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A single-replica kube-system Deployment with system-cluster-critical priority and no PDB pins its node from cluster-autoscaler scale-down (CA won't evict kube-system pods lacking a PDB). The controller is stateless + leader-elected, so it's safe to drain/reschedule: add cluster-autoscaler.kubernetes.io/safe-to-evict: "true" (the same pattern the coredns-/konnectivity-agent-autoscaler helpers use). A PDB would be wrong here — minAvailable:1 on a single replica blocks all voluntary evictions. --- compute-domain/chart/templates/deployment.yaml | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/compute-domain/chart/templates/deployment.yaml b/compute-domain/chart/templates/deployment.yaml index c642a86..4b93f09 100644 --- a/compute-domain/chart/templates/deployment.yaml +++ b/compute-domain/chart/templates/deployment.yaml @@ -24,10 +24,14 @@ spec: {{- with .Values.podLabels }} {{- toYaml . | nindent 8 }} {{- end }} - {{- with .Values.podAnnotations }} annotations: + # Stateless, leader-elected controller — safe for the cluster autoscaler to + # drain/reschedule. Without this a single-replica kube-system Deployment pins + # its node from scale-down (CA won't evict kube-system pods lacking a PDB). + cluster-autoscaler.kubernetes.io/safe-to-evict: "true" + {{- with .Values.podAnnotations }} {{- toYaml . | nindent 8 }} - {{- end }} + {{- end }} spec: serviceAccountName: {{ include "computedomain.serviceAccountName" . }} {{- with .Values.priorityClassName }} From c09a4a5ff2c6cac446ec45a046bfff6a269c2eb9 Mon Sep 17 00:00:00 2001 From: xuexu6666 Date: Mon, 21 Sep 2026 14:25:56 -0500 Subject: [PATCH 05/12] compute-domain: drop unnecessary resource.k8s.io/v1 GA comment in deviceclass --- compute-domain/chart/templates/deviceclass.yaml | 1 - 1 file changed, 1 deletion(-) diff --git a/compute-domain/chart/templates/deviceclass.yaml b/compute-domain/chart/templates/deviceclass.yaml index 5dbdc30..04c1036 100644 --- a/compute-domain/chart/templates/deviceclass.yaml +++ b/compute-domain/chart/templates/deviceclass.yaml @@ -1,5 +1,4 @@ # DeviceClasses the compute-domain kubelet-plugin publishes devices under. -# resource.k8s.io/v1 is GA as of Kubernetes 1.34; GB runs 1.35. {{- if .Values.deviceClasses.computeDomainDefaultChannel.enabled }} --- apiVersion: resource.k8s.io/v1 From 63e3e526bbb923b59017774fb2682d935385452d Mon Sep 17 00:00:00 2001 From: xuexu6666 Date: Mon, 21 Sep 2026 14:31:02 -0500 Subject: [PATCH 06/12] compute-domain: drop hostManaged note from rbac comment --- compute-domain/chart/templates/rbac.yaml | 1 - 1 file changed, 1 deletion(-) diff --git a/compute-domain/chart/templates/rbac.yaml b/compute-domain/chart/templates/rbac.yaml index d111370..76078ef 100644 --- a/compute-domain/chart/templates/rbac.yaml +++ b/compute-domain/chart/templates/rbac.yaml @@ -32,7 +32,6 @@ rules: verbs: ["get", "list", "watch"] # driverManaged only: the controller creates a per-ComputeDomain # compute-domain-daemon DaemonSet and the ComputeDomainClique objects. - # (Dropped under hostManaged IMEX, which this managed chart does not use.) - apiGroups: ["apps"] resources: ["daemonsets"] verbs: ["get", "list", "watch", "create", "update", "patch", "delete"] From 83a62bf947063cc34cb1bfb43567e4f841a753ee Mon Sep 17 00:00:00 2001 From: xuexu6666 Date: Mon, 21 Sep 2026 14:35:33 -0500 Subject: [PATCH 07/12] compute-domain: scope daemonsets/computedomaincliques RBAC to a namespaced Role Match upstream's split: daemonsets + computedomaincliques are created only in the driver (release) namespace, so grant them via a namespaced Role/RoleBinding instead of the cluster-wide ClusterRole. The ClusterRole keeps only what's genuinely cluster-scoped (computedomains, resourceclaimtemplates, leases, nodes, pods reads). Verified: controller 1/1 Running, DaemonSet + ComputeDomainClique informers watch fine with the namespaced Role (no forbidden). --- compute-domain/chart/templates/rbac.yaml | 48 +++++++++++++++++++----- 1 file changed, 38 insertions(+), 10 deletions(-) diff --git a/compute-domain/chart/templates/rbac.yaml b/compute-domain/chart/templates/rbac.yaml index 76078ef..461d5d2 100644 --- a/compute-domain/chart/templates/rbac.yaml +++ b/compute-domain/chart/templates/rbac.yaml @@ -1,4 +1,6 @@ {{- if .Values.rbac.create }} +# Cluster-scoped: reconcile ComputeDomain CRs, manage the workload ResourceClaimTemplate, +# leader election, and read nodes/pods cluster-wide. apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: @@ -9,18 +11,15 @@ metadata: kubernetes.azure.com/managedby: aks {{- end }} rules: - # Reconcile the ComputeDomain CRs and set their status. - apiGroups: ["resource.nvidia.com"] resources: ["computedomains"] verbs: ["get", "list", "watch", "update"] - apiGroups: ["resource.nvidia.com"] resources: ["computedomains/status"] verbs: ["update"] - # Workload ResourceClaimTemplate the controller manages for each ComputeDomain. - apiGroups: ["resource.k8s.io"] resources: ["resourceclaimtemplates"] verbs: ["get", "list", "watch", "create", "update", "delete"] - # Leader election. - apiGroups: ["coordination.k8s.io"] resources: ["leases"] verbs: ["get", "create", "update"] @@ -30,8 +29,39 @@ rules: - apiGroups: [""] resources: ["pods"] verbs: ["get", "list", "watch"] - # driverManaged only: the controller creates a per-ComputeDomain - # compute-domain-daemon DaemonSet and the ComputeDomainClique objects. +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRoleBinding +metadata: + name: {{ include "computedomain.fullname" . }} + labels: + {{- include "computedomain.labels" . | nindent 4 }} + {{- if .Values.aks.managedByLabels.enabled }} + kubernetes.azure.com/managedby: aks + {{- end }} +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: {{ include "computedomain.fullname" . }} +subjects: + - kind: ServiceAccount + name: {{ include "computedomain.serviceAccountName" . }} + namespace: {{ .Release.Namespace }} +--- +# Namespaced: the per-ComputeDomain compute-domain-daemon DaemonSets and the +# ComputeDomainClique objects the controller creates live in the driver namespace, +# so these verbs are scoped there rather than cluster-wide (driverManaged only). +apiVersion: rbac.authorization.k8s.io/v1 +kind: Role +metadata: + name: {{ include "computedomain.fullname" . }} + namespace: {{ .Release.Namespace }} + labels: + {{- include "computedomain.labels" . | nindent 4 }} + {{- if .Values.aks.managedByLabels.enabled }} + kubernetes.azure.com/managedby: aks + {{- end }} +rules: - apiGroups: ["apps"] resources: ["daemonsets"] verbs: ["get", "list", "watch", "create", "update", "patch", "delete"] @@ -40,9 +70,10 @@ rules: verbs: ["get", "list", "watch", "update"] --- apiVersion: rbac.authorization.k8s.io/v1 -kind: ClusterRoleBinding +kind: RoleBinding metadata: name: {{ include "computedomain.fullname" . }} + namespace: {{ .Release.Namespace }} labels: {{- include "computedomain.labels" . | nindent 4 }} {{- if .Values.aks.managedByLabels.enabled }} @@ -50,12 +81,9 @@ metadata: {{- end }} roleRef: apiGroup: rbac.authorization.k8s.io - kind: ClusterRole + kind: Role name: {{ include "computedomain.fullname" . }} subjects: - # Bound to the controller ServiceAccount (NOT system:nodes) — blast radius is the - # single controller pod. The node-identity kubelet-plugin binding (2 read-only - # resource.nvidia.com reads to system:nodes) is delivered separately. - kind: ServiceAccount name: {{ include "computedomain.serviceAccountName" . }} namespace: {{ .Release.Namespace }} From b038ec174b774637eaeb3698bbe85290015e4019 Mon Sep 17 00:00:00 2001 From: xuexu6666 Date: Mon, 21 Sep 2026 14:39:09 -0500 Subject: [PATCH 08/12] compute-domain: reconverge RBAC + env with upstream v0.5.0 - ClusterRole nodes: add 'update' verb (was dropped). The controller labels/unlabels nodes with resource.nvidia.com/computeDomain= to place the per-CD daemon DaemonSet; without update this fails 'forbidden' and ComputeDomains never form. (Missed originally because the test cluster has no GPU nodes to label.) - Deployment: add CD_DAEMON_IMAGE_PULL_SECRET_NAMES env (gated on imagePullSecrets) so controller-spawned daemon pods can pull from a private registry. Verified against upstream: CRDs, DeviceClass selectors, daemonsets/cliques Role, and the rest of the controller env/command now match v0.5.0. Metrics env (HTTP_ENDPOINT/METRICS_PATH/PPROF_PATH) intentionally omitted (metrics disabled). --- compute-domain/chart/templates/deployment.yaml | 6 ++++++ compute-domain/chart/templates/rbac.yaml | 4 +++- 2 files changed, 9 insertions(+), 1 deletion(-) diff --git a/compute-domain/chart/templates/deployment.yaml b/compute-domain/chart/templates/deployment.yaml index 4b93f09..f818969 100644 --- a/compute-domain/chart/templates/deployment.yaml +++ b/compute-domain/chart/templates/deployment.yaml @@ -84,6 +84,12 @@ spec: # DaemonSet from THIS image, so it must contain compute-domain-daemon too. - name: IMAGE_NAME value: {{ include "computedomain.image" . }} + {{- if .Values.imagePullSecrets }} + # Pull-secret names propagated to the controller-spawned compute-domain-daemon + # DaemonSet pods (they're created from this same image). + - name: CD_DAEMON_IMAGE_PULL_SECRET_NAMES + value: "{{- range $i, $s := .Values.imagePullSecrets }}{{ if $i }},{{ end }}{{ $s.name }}{{- end }}" + {{- end }} # runc: explicit "void" so the controller pod grabs no GPUs. - name: NVIDIA_VISIBLE_DEVICES value: void diff --git a/compute-domain/chart/templates/rbac.yaml b/compute-domain/chart/templates/rbac.yaml index 461d5d2..45bd18e 100644 --- a/compute-domain/chart/templates/rbac.yaml +++ b/compute-domain/chart/templates/rbac.yaml @@ -24,8 +24,10 @@ rules: resources: ["leases"] verbs: ["get", "create", "update"] - apiGroups: [""] + # update: the controller labels/unlabels nodes with + # resource.nvidia.com/computeDomain= to place the per-CD daemon DaemonSet. resources: ["nodes"] - verbs: ["get", "list", "watch"] + verbs: ["get", "list", "watch", "update"] - apiGroups: [""] resources: ["pods"] verbs: ["get", "list", "watch"] From 48b3467e25e3949abdb1bdd0faf9a17d8189c476 Mon Sep 17 00:00:00 2001 From: xuexu6666 Date: Mon, 21 Sep 2026 14:47:19 -0500 Subject: [PATCH 09/12] compute-domain: require controller on the system pool (RP synth convention) Match the RP synth pattern (clusterhealthmonitor/.../resources.go): require a real linux AKS node (kubernetes.azure.com/cluster Exists, type != virtual-kubelet) with kubernetes.azure.com/mode=system as a HARD requirement (not the softer preferred used by coredns/metrics-server), tolerations node-role.kubernetes.io/master + CriticalAddonsOnly. Still no GPU-taint toleration, so it stays off GB nodes. --- compute-domain/chart/values.yaml | 31 +++++++++++++++++++++++++------ 1 file changed, 25 insertions(+), 6 deletions(-) diff --git a/compute-domain/chart/values.yaml b/compute-domain/chart/values.yaml index c471a2a..a674e32 100644 --- a/compute-domain/chart/values.yaml +++ b/compute-domain/chart/values.yaml @@ -56,15 +56,34 @@ deviceClasses: name: compute-domain-daemon.nvidia.com priorityClassName: system-cluster-critical -nodeSelector: - kubernetes.io/os: linux +nodeSelector: {} tolerations: - - key: CriticalAddonsOnly - operator: Exists - - key: node-role.kubernetes.io/control-plane + # Matches the RP synth convention (cluster-health-monitor). No GPU-taint toleration + # on purpose, so the controller stays off GB nodes. + - key: node-role.kubernetes.io/master operator: Exists effect: NoSchedule -affinity: {} + - key: CriticalAddonsOnly + operator: Exists +# Require a real linux AKS node (not an ACI virtual-kubelet) on the system pool — +# mirrors the RP synth convention (cluster-health-monitor-synth): mode=system is a +# hard requirement, not a soft preference. +affinity: + nodeAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + nodeSelectorTerms: + - matchExpressions: + - key: kubernetes.azure.com/cluster + operator: Exists + - key: type + operator: NotIn + values: ["virtual-kubelet"] + - key: kubernetes.io/os + operator: In + values: ["linux"] + - key: kubernetes.azure.com/mode + operator: In + values: ["system"] podAnnotations: {} podLabels: {} resources: From 611356715809edc1b63d07642d5b70663608a1e9 Mon Sep 17 00:00:00 2001 From: xuexu6666 Date: Mon, 21 Sep 2026 14:51:09 -0500 Subject: [PATCH 10/12] compute-domain: drop tolerations comment in values --- compute-domain/chart/values.yaml | 2 -- 1 file changed, 2 deletions(-) diff --git a/compute-domain/chart/values.yaml b/compute-domain/chart/values.yaml index a674e32..1e2d0a6 100644 --- a/compute-domain/chart/values.yaml +++ b/compute-domain/chart/values.yaml @@ -58,8 +58,6 @@ deviceClasses: priorityClassName: system-cluster-critical nodeSelector: {} tolerations: - # Matches the RP synth convention (cluster-health-monitor). No GPU-taint toleration - # on purpose, so the controller stays off GB nodes. - key: node-role.kubernetes.io/master operator: Exists effect: NoSchedule From d33f159b7fc5a91ab68817ec4a8084d59b3e0a3d Mon Sep 17 00:00:00 2001 From: xuexu6666 Date: Mon, 21 Sep 2026 14:57:27 -0500 Subject: [PATCH 11/12] compute-domain: drop nodes:update comment in rbac --- compute-domain/chart/templates/rbac.yaml | 2 -- 1 file changed, 2 deletions(-) diff --git a/compute-domain/chart/templates/rbac.yaml b/compute-domain/chart/templates/rbac.yaml index 45bd18e..09a3bb1 100644 --- a/compute-domain/chart/templates/rbac.yaml +++ b/compute-domain/chart/templates/rbac.yaml @@ -24,8 +24,6 @@ rules: resources: ["leases"] verbs: ["get", "create", "update"] - apiGroups: [""] - # update: the controller labels/unlabels nodes with - # resource.nvidia.com/computeDomain= to place the per-CD daemon DaemonSet. resources: ["nodes"] verbs: ["get", "list", "watch", "update"] - apiGroups: [""] From 9eaad9dacf55d5adbaa4a590b60abd68914cbec0 Mon Sep 17 00:00:00 2001 From: xuexu6666 Date: Mon, 21 Sep 2026 22:16:56 -0500 Subject: [PATCH 12/12] compute-domain: bump controller memory req/limit to 128Mi/512Mi Actual usage is ~9-30Mi, but the controller watches all pods+nodes cluster-wide so its footprint scales with cluster size. Raise the request floor to 128Mi and the memory limit to 512Mi for headroom on large clusters. Still no CPU limit (avoids CFS throttling; matches dranet/upstream). --- compute-domain/chart/values.yaml | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/compute-domain/chart/values.yaml b/compute-domain/chart/values.yaml index 1e2d0a6..0a575d3 100644 --- a/compute-domain/chart/values.yaml +++ b/compute-domain/chart/values.yaml @@ -87,6 +87,9 @@ podLabels: {} resources: requests: cpu: 50m - memory: 64Mi + memory: 128Mi limits: - memory: 256Mi + # No CPU limit on purpose (avoids CFS throttling; matches dranet/upstream). + # Memory limit headroom: the controller's footprint scales with cluster size + # (it watches all pods/nodes), so 512Mi rather than a tight bound. + memory: 512Mi