diff --git a/compute-domain/README.md b/compute-domain/README.md new file mode 100644 index 0000000..a14f18a --- /dev/null +++ b/compute-domain/README.md @@ -0,0 +1,44 @@ +# Managed ComputeDomain (GB200/GB300) + +Helm chart for the **AKS-managed NVIDIA ComputeDomain controller** — the cluster-scoped +piece of the Grace-Blackwell cross-node NVLink (MNNVL / IMEX) stack. It is delivered as an +AKS core extension (`microsoft.managedcomputedomain`), analogous to how managed DRANET +(`microsoft.manageddranet`) is delivered. + +## What this chart deploys + +| Object | Purpose | +|---|---| +| `Deployment/compute-domain-controller` | cluster-scoped controller: reconciles `ComputeDomain` CRs, creates the workload `ResourceClaimTemplate`, and (driverManaged) spawns a per-ComputeDomain `compute-domain-daemon` DaemonSet | +| `ServiceAccount` + `ClusterRole` + `ClusterRoleBinding` | controller identity + RBAC, bound to the SA (blast radius = the controller pod, **not** `system:nodes`) | +| `DeviceClass/compute-domain-default-channel.nvidia.com` | the IMEX channel device workloads claim | +| `DeviceClass/compute-domain-daemon.nvidia.com` | the daemon device (driverManaged) | + +## What this chart does NOT deliver + +- **`compute-domain-daemon`** — not a static object; the controller creates it as a + per-ComputeDomain DaemonSet at runtime, from the controller's **own image** (so the image + must carry both `compute-domain-controller` and `compute-domain-daemon`). +- **`compute-domain-kubelet-plugin`** — runs as a host **systemd** service (delivered via the + `dra-driver-nvidia-gpu` deb), not by this chart. Its node-identity RBAC (2 read-only + `resource.nvidia.com` reads bound to `system:nodes`) is delivered separately. +- **`nvidia-imex`** — the host binary, installed by **aks-gpu** at boot; the daemon pod execs + it via CDI. The host `nvidia-imex.service` stays **off** in driverManaged. + +## IMEX mode: driverManaged (only) + +This chart hard-defaults to `imex.mode: driverManaged`. `hostManaged` is intentionally not +offered: it would require AKS to own the host IMEX topology (`nodes_config.cfg` = the rack +peer set), which cannot be built at node bootstrap (the peer set is cluster-scoped and +converging) and would just re-implement what the driverManaged DaemonSet does automatically +(via pod DNS names). + +## Image + +The controller image (`compute-domain-controller` + `compute-domain-daemon`) is onboarded via +dalec/MCR; set `image.repository` / `image.tag` in `values.yaml`. + +## Upstream + +Chart wraps [`kubernetes-sigs/dra-driver-nvidia-gpu`](https://github.com/kubernetes-sigs/dra-driver-nvidia-gpu) +`v0.5.0` (controller + DeviceClasses + RBAC derived from its `deployments/helm` templates). diff --git a/compute-domain/chart/Chart.yaml b/compute-domain/chart/Chart.yaml new file mode 100644 index 0000000..39fd75f --- /dev/null +++ b/compute-domain/chart/Chart.yaml @@ -0,0 +1,20 @@ +apiVersion: v2 +name: microsoft.managedcomputedomain +description: | + AKS-managed NVIDIA ComputeDomain controller for Grace-Blackwell (GB200/GB300). + Deploys the cluster-scoped compute-domain-controller (and its RBAC + DeviceClasses) + that binds GB nodes into cross-node NVLink (MNNVL) IMEX domains. The controller + dynamically creates a per-ComputeDomain compute-domain-daemon DaemonSet from its + own image (driverManaged IMEX). The node-local compute-domain kubelet-plugin and + the host nvidia-imex binary are delivered separately (deb / aks-gpu), not by this chart. +type: application +version: 0.0.1 +appVersion: "0.5.0" +keywords: + - nvidia + - dra + - compute-domain + - imex + - mnnvl + - gb200 + - gb300 diff --git a/compute-domain/chart/crds/resource.nvidia.com_computedomaincliques.yaml b/compute-domain/chart/crds/resource.nvidia.com_computedomaincliques.yaml new file mode 100644 index 0000000..b3eddef --- /dev/null +++ b/compute-domain/chart/crds/resource.nvidia.com_computedomaincliques.yaml @@ -0,0 +1,84 @@ +--- +apiVersion: apiextensions.k8s.io/v1 +kind: CustomResourceDefinition +metadata: + annotations: + controller-gen.kubebuilder.io/version: v0.17.1 + name: computedomaincliques.resource.nvidia.com +spec: + group: resource.nvidia.com + names: + kind: ComputeDomainClique + listKind: ComputeDomainCliqueList + plural: computedomaincliques + singular: computedomainclique + scope: Namespaced + versions: + - name: v1beta1 + schema: + openAPIV3Schema: + description: |- + ComputeDomainClique holds information about a specific clique within a ComputeDomain. + It is created in the driver namespace and named as ".". + properties: + apiVersion: + description: |- + APIVersion defines the versioned schema of this representation of an object. + Servers should convert recognized schemas to the latest internal value, and + may reject unrecognized values. + More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources + type: string + daemons: + items: + description: ComputeDomainDaemonInfo provides information about each + daemon in a ComputeDomainClique. + properties: + cliqueID: + type: string + index: + description: |- + The Index field is used to ensure a consistent IP-to-DNS name + mapping across all machines within an IMEX domain. Each node's index + directly determines its DNS name within a given NVLink partition + (i.e. clique). In other words, the 2-tuple of (CliqueID, Index) will + always be unique. This field is marked as optional (but not + omitempty) in order to support downgrades and avoid an API bump. + type: integer + ipAddress: + type: string + nodeName: + type: string + status: + default: NotReady + description: |- + The Status field tracks the readiness of the IMEX daemon running on + this node. It gets switched to Ready whenever the IMEX daemon is + ready to broker GPU memory exchanges and switches to NotReady when + it is not. It is marked as optional in order to support downgrades + and avoid an API bump. + enum: + - Ready + - NotReady + type: string + required: + - cliqueID + - ipAddress + - nodeName + type: object + type: array + x-kubernetes-list-map-keys: + - nodeName + x-kubernetes-list-type: map + kind: + description: |- + Kind is a string value representing the REST resource this object represents. + Servers may infer this from the endpoint the client submits requests to. + Cannot be updated. + In CamelCase. + More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds + type: string + metadata: + type: object + type: object + served: true + storage: true diff --git a/compute-domain/chart/crds/resource.nvidia.com_computedomains.yaml b/compute-domain/chart/crds/resource.nvidia.com_computedomains.yaml new file mode 100644 index 0000000..0811b3c --- /dev/null +++ b/compute-domain/chart/crds/resource.nvidia.com_computedomains.yaml @@ -0,0 +1,163 @@ +--- +apiVersion: apiextensions.k8s.io/v1 +kind: CustomResourceDefinition +metadata: + annotations: + controller-gen.kubebuilder.io/version: v0.17.1 + name: computedomains.resource.nvidia.com +spec: + group: resource.nvidia.com + names: + kind: ComputeDomain + listKind: ComputeDomainList + plural: computedomains + singular: computedomain + scope: Namespaced + versions: + - name: v1beta1 + schema: + openAPIV3Schema: + description: ComputeDomain prepares a set of nodes to run a multi-node workload + in. + properties: + apiVersion: + description: |- + APIVersion defines the versioned schema of this representation of an object. + Servers should convert recognized schemas to the latest internal value, and + may reject unrecognized values. + More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources + type: string + kind: + description: |- + Kind is a string value representing the REST resource this object represents. + Servers may infer this from the endpoint the client submits requests to. + Cannot be updated. + In CamelCase. + More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds + type: string + metadata: + type: object + spec: + description: ComputeDomainSpec provides the spec for a ComputeDomain. + properties: + channel: + description: ComputeDomainChannelSpec provides the spec for a channel + used to run a workload inside a ComputeDomain. + properties: + allocationMode: + default: Single + description: |- + Allows for requesting all IMEX channels (the maximum per IMEX domain) or + precisely one. + enum: + - All + - Single + type: string + resourceClaimTemplate: + description: ComputeDomainResourceClaimTemplate provides the details + of the ResourceClaimTemplate to generate. + properties: + name: + type: string + required: + - name + type: object + required: + - resourceClaimTemplate + type: object + numNodes: + default: 0 + description: |- + Intended number of IMEX daemons (i.e., individual compute nodes) in the + ComputeDomain. Must be zero or greater. + + With `featureGates.IMEXDaemonsWithDNSNames=true` (the default), this is + recommended to be set to zero (default). Workload must implement and consult its + own source of truth for the number of workers online before trying to + share GPU memory (and hence triggering IMEX interaction). When non-zero, + `numNodes` is used only for automatically updating the global + ComputeDomain `Status` (indicating `Ready` when the number of ready IMEX + daemons equals `numNodes`). In this mode, a `numNodes` value greater than + zero in particular does not gate the startup of IMEX daemons: individual + IMEX daemons are started immediately without waiting for its peers, and + any workload pod gets released right after its local IMEX daemon has + started. + + With `featureGates.IMEXDaemonsWithDNSNames=false`, `numNodes` must be set + to the expected number of worker nodes joining the ComputeDomain. In that + mode, all workload pods are held back (with containers in state + `ContainerCreating`) until the underlying IMEX domain has been joined by + `numNodes` IMEX daemons. Pods from more than `numNodes` nodes trying to + join the ComputeDomain may lead to unexpected behavior. + + The `numNodes` parameter is deprecated and will be removed in the next + API version. + minimum: 0 + type: integer + required: + - channel + type: object + x-kubernetes-validations: + - message: A computeDomain.spec is immutable + rule: self == oldSelf + status: + description: |- + Global ComputeDomain status. Can be used to guide debugging efforts. + Workload however should not rely on inspecting this field at any point + during its lifecycle. + properties: + nodes: + items: + description: ComputeDomainNode provides information about each node + added to a ComputeDomain. + properties: + cliqueID: + type: string + index: + description: |- + The Index field is used to ensure a consistent IP-to-DNS name + mapping across all machines within an IMEX domain. Each node's index + directly determines its DNS name within a given NVLink partition + (i.e. clique). In other words, the 2-tuple of (CliqueID, Index) will + always be unique. This field is marked as optional (but not + omitempty) in order to support downgrades and avoid an API bump. + type: integer + ipAddress: + type: string + name: + type: string + status: + default: NotReady + description: |- + The Status field tracks the readiness of the IMEX daemon running on + this node. It gets switched to Ready whenever the IMEX daemon is + ready to broker GPU memory exchanges and switches to NotReady when + it is not. It is marked as optional in order to support downgrades + and avoid an API bump. + enum: + - Ready + - NotReady + type: string + required: + - cliqueID + - ipAddress + - name + type: object + type: array + x-kubernetes-list-map-keys: + - name + x-kubernetes-list-type: map + status: + default: NotReady + enum: + - Ready + - NotReady + type: string + required: + - status + type: object + type: object + served: true + storage: true + subresources: + status: {} diff --git a/compute-domain/chart/templates/_helpers.tpl b/compute-domain/chart/templates/_helpers.tpl new file mode 100644 index 0000000..c481aaa --- /dev/null +++ b/compute-domain/chart/templates/_helpers.tpl @@ -0,0 +1,40 @@ +{{- define "computedomain.name" -}} +{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" -}} +{{- end -}} + +{{- define "computedomain.fullname" -}} +{{- if .Values.fullnameOverride -}} +{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" -}} +{{- else -}} +{{- $name := default .Chart.Name .Values.nameOverride -}} +{{- if contains $name .Release.Name -}} +{{- .Release.Name | trunc 63 | trimSuffix "-" -}} +{{- else -}} +{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" -}} +{{- end -}} +{{- end -}} +{{- end -}} + +{{- define "computedomain.chart" -}} +{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" -}} +{{- end -}} + +{{- define "computedomain.labels" -}} +helm.sh/chart: {{ include "computedomain.chart" . }} +app.kubernetes.io/name: {{ include "computedomain.name" . }} +app.kubernetes.io/instance: {{ .Release.Name }} +app.kubernetes.io/version: {{ .Chart.AppVersion | quote }} +app.kubernetes.io/managed-by: {{ .Release.Service }} +{{- end -}} + +{{- define "computedomain.selectorLabels" -}} +app: {{ include "computedomain.name" . }} +{{- end -}} + +{{- define "computedomain.serviceAccountName" -}} +{{- include "computedomain.fullname" . -}} +{{- end -}} + +{{- define "computedomain.image" -}} +{{- printf "%s:%s" .Values.image.repository .Values.image.tag -}} +{{- end -}} diff --git a/compute-domain/chart/templates/deployment.yaml b/compute-domain/chart/templates/deployment.yaml new file mode 100644 index 0000000..f818969 --- /dev/null +++ b/compute-domain/chart/templates/deployment.yaml @@ -0,0 +1,115 @@ +apiVersion: apps/v1 +kind: Deployment +metadata: + name: {{ include "computedomain.fullname" . }} + namespace: {{ .Release.Namespace }} + labels: + {{- include "computedomain.labels" . | nindent 4 }} + {{- if .Values.aks.managedByLabels.enabled }} + kubernetes.azure.com/managedby: aks + {{- end }} +spec: + replicas: {{ .Values.replicas }} + selector: + matchLabels: + {{- include "computedomain.selectorLabels" . | nindent 6 }} + template: + metadata: + labels: + {{- include "computedomain.labels" . | nindent 8 }} + {{- include "computedomain.selectorLabels" . | nindent 8 }} + {{- if .Values.aks.managedByLabels.enabled }} + kubernetes.azure.com/managedby: aks + {{- end }} + {{- with .Values.podLabels }} + {{- toYaml . | nindent 8 }} + {{- end }} + annotations: + # Stateless, leader-elected controller — safe for the cluster autoscaler to + # drain/reschedule. Without this a single-replica kube-system Deployment pins + # its node from scale-down (CA won't evict kube-system pods lacking a PDB). + cluster-autoscaler.kubernetes.io/safe-to-evict: "true" + {{- with .Values.podAnnotations }} + {{- toYaml . | nindent 8 }} + {{- end }} + spec: + serviceAccountName: {{ include "computedomain.serviceAccountName" . }} + {{- with .Values.priorityClassName }} + priorityClassName: {{ . }} + {{- end }} + {{- with .Values.imagePullSecrets }} + imagePullSecrets: + {{- toYaml . | nindent 8 }} + {{- end }} + {{- with .Values.nodeSelector }} + nodeSelector: + {{- toYaml . | nindent 8 }} + {{- end }} + {{- with .Values.tolerations }} + tolerations: + {{- toYaml . | nindent 8 }} + {{- end }} + {{- with .Values.affinity }} + affinity: + {{- toYaml . | nindent 8 }} + {{- end }} + containers: + - name: compute-domain + image: {{ include "computedomain.image" . }} + imagePullPolicy: {{ .Values.image.pullPolicy }} + command: ["compute-domain-controller", "-v", "$(LOG_VERBOSITY)"] + securityContext: + allowPrivilegeEscalation: false + readOnlyRootFilesystem: true + capabilities: + drop: ["ALL"] + resources: + {{- toYaml .Values.resources | nindent 12 }} + env: + - name: LOG_VERBOSITY + value: "{{ .Values.logVerbosity }}" + # Verbosity for the dynamically-launched compute-domain-daemon pods + # (their pod spec is rendered by the controller, not this chart). + - name: LOG_VERBOSITY_CD_DAEMON + value: "{{ .Values.logVerbosity }}" + - name: POD_NAME + valueFrom: + fieldRef: + fieldPath: metadata.name + - name: NAMESPACE + valueFrom: + fieldRef: + fieldPath: metadata.namespace + # The controller spawns the per-ComputeDomain compute-domain-daemon + # DaemonSet from THIS image, so it must contain compute-domain-daemon too. + - name: IMAGE_NAME + value: {{ include "computedomain.image" . }} + {{- if .Values.imagePullSecrets }} + # Pull-secret names propagated to the controller-spawned compute-domain-daemon + # DaemonSet pods (they're created from this same image). + - name: CD_DAEMON_IMAGE_PULL_SECRET_NAMES + value: "{{- range $i, $s := .Values.imagePullSecrets }}{{ if $i }},{{ end }}{{ $s.name }}{{- end }}" + {{- end }} + # runc: explicit "void" so the controller pod grabs no GPUs. + - name: NVIDIA_VISIBLE_DEVICES + value: void + - name: IMEX_MODE + value: "{{ .Values.imex.mode }}" + - name: IMEX_ISOLATION + value: "{{ .Values.imex.isolation }}" + {{- if .Values.featureGates }} + - name: FEATURE_GATES + value: "{{- range $k, $v := .Values.featureGates }}{{ $k }}={{ $v }},{{- end }}" + {{- end }} + - name: LEADER_ELECTION_ENABLED + value: "{{ .Values.leaderElection.enabled }}" + - name: LEADER_ELECTION_LEASE_LOCK_NAME + value: "{{ include "computedomain.fullname" . }}" + - name: LEADER_ELECTION_LEASE_LOCK_NAMESPACE + value: "{{ .Release.Namespace }}" + - name: LEADER_ELECTION_LEASE_DURATION + value: "{{ .Values.leaderElection.leaseDuration }}" + - name: LEADER_ELECTION_RENEW_DEADLINE + value: "{{ .Values.leaderElection.renewDeadline }}" + - name: LEADER_ELECTION_RETRY_PERIOD + value: "{{ .Values.leaderElection.retryPeriod }}" diff --git a/compute-domain/chart/templates/deviceclass.yaml b/compute-domain/chart/templates/deviceclass.yaml new file mode 100644 index 0000000..04c1036 --- /dev/null +++ b/compute-domain/chart/templates/deviceclass.yaml @@ -0,0 +1,34 @@ +# DeviceClasses the compute-domain kubelet-plugin publishes devices under. +{{- if .Values.deviceClasses.computeDomainDefaultChannel.enabled }} +--- +apiVersion: resource.k8s.io/v1 +kind: DeviceClass +metadata: + name: {{ .Values.deviceClasses.computeDomainDefaultChannel.name | default "compute-domain-default-channel.nvidia.com" }} + labels: + {{- include "computedomain.labels" . | nindent 4 }} + {{- if .Values.aks.managedByLabels.enabled }} + kubernetes.azure.com/managedby: aks + {{- end }} +spec: + selectors: + - cel: + expression: "device.driver == 'compute-domain.nvidia.com' && device.attributes['compute-domain.nvidia.com'].type == 'channel' && device.attributes['compute-domain.nvidia.com'].id == 0" +{{- end }} +{{- if .Values.deviceClasses.computeDomainDaemon.enabled }} +--- +# driverManaged only: the per-ComputeDomain daemon claims a 'daemon' device. +apiVersion: resource.k8s.io/v1 +kind: DeviceClass +metadata: + name: {{ .Values.deviceClasses.computeDomainDaemon.name | default "compute-domain-daemon.nvidia.com" }} + labels: + {{- include "computedomain.labels" . | nindent 4 }} + {{- if .Values.aks.managedByLabels.enabled }} + kubernetes.azure.com/managedby: aks + {{- end }} +spec: + selectors: + - cel: + expression: "device.driver == 'compute-domain.nvidia.com' && device.attributes['compute-domain.nvidia.com'].type == 'daemon'" +{{- end }} diff --git a/compute-domain/chart/templates/rbac.yaml b/compute-domain/chart/templates/rbac.yaml new file mode 100644 index 0000000..09a3bb1 --- /dev/null +++ b/compute-domain/chart/templates/rbac.yaml @@ -0,0 +1,90 @@ +{{- if .Values.rbac.create }} +# Cluster-scoped: reconcile ComputeDomain CRs, manage the workload ResourceClaimTemplate, +# leader election, and read nodes/pods cluster-wide. +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + name: {{ include "computedomain.fullname" . }} + labels: + {{- include "computedomain.labels" . | nindent 4 }} + {{- if .Values.aks.managedByLabels.enabled }} + kubernetes.azure.com/managedby: aks + {{- end }} +rules: + - apiGroups: ["resource.nvidia.com"] + resources: ["computedomains"] + verbs: ["get", "list", "watch", "update"] + - apiGroups: ["resource.nvidia.com"] + resources: ["computedomains/status"] + verbs: ["update"] + - apiGroups: ["resource.k8s.io"] + resources: ["resourceclaimtemplates"] + verbs: ["get", "list", "watch", "create", "update", "delete"] + - apiGroups: ["coordination.k8s.io"] + resources: ["leases"] + verbs: ["get", "create", "update"] + - apiGroups: [""] + resources: ["nodes"] + verbs: ["get", "list", "watch", "update"] + - apiGroups: [""] + resources: ["pods"] + verbs: ["get", "list", "watch"] +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRoleBinding +metadata: + name: {{ include "computedomain.fullname" . }} + labels: + {{- include "computedomain.labels" . | nindent 4 }} + {{- if .Values.aks.managedByLabels.enabled }} + kubernetes.azure.com/managedby: aks + {{- end }} +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: {{ include "computedomain.fullname" . }} +subjects: + - kind: ServiceAccount + name: {{ include "computedomain.serviceAccountName" . }} + namespace: {{ .Release.Namespace }} +--- +# Namespaced: the per-ComputeDomain compute-domain-daemon DaemonSets and the +# ComputeDomainClique objects the controller creates live in the driver namespace, +# so these verbs are scoped there rather than cluster-wide (driverManaged only). +apiVersion: rbac.authorization.k8s.io/v1 +kind: Role +metadata: + name: {{ include "computedomain.fullname" . }} + namespace: {{ .Release.Namespace }} + labels: + {{- include "computedomain.labels" . | nindent 4 }} + {{- if .Values.aks.managedByLabels.enabled }} + kubernetes.azure.com/managedby: aks + {{- end }} +rules: + - apiGroups: ["apps"] + resources: ["daemonsets"] + verbs: ["get", "list", "watch", "create", "update", "patch", "delete"] + - apiGroups: ["resource.nvidia.com"] + resources: ["computedomaincliques"] + verbs: ["get", "list", "watch", "update"] +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: RoleBinding +metadata: + name: {{ include "computedomain.fullname" . }} + namespace: {{ .Release.Namespace }} + labels: + {{- include "computedomain.labels" . | nindent 4 }} + {{- if .Values.aks.managedByLabels.enabled }} + kubernetes.azure.com/managedby: aks + {{- end }} +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: Role + name: {{ include "computedomain.fullname" . }} +subjects: + - kind: ServiceAccount + name: {{ include "computedomain.serviceAccountName" . }} + namespace: {{ .Release.Namespace }} +{{- end }} diff --git a/compute-domain/chart/templates/serviceaccount.yaml b/compute-domain/chart/templates/serviceaccount.yaml new file mode 100644 index 0000000..c40c87e --- /dev/null +++ b/compute-domain/chart/templates/serviceaccount.yaml @@ -0,0 +1,14 @@ +apiVersion: v1 +kind: ServiceAccount +metadata: + name: {{ include "computedomain.serviceAccountName" . }} + namespace: {{ .Release.Namespace }} + labels: + {{- include "computedomain.labels" . | nindent 4 }} + {{- if .Values.aks.managedByLabels.enabled }} + kubernetes.azure.com/managedby: aks + {{- end }} + {{- with .Values.serviceAccount.annotations }} + annotations: + {{- toYaml . | nindent 4 }} + {{- end }} diff --git a/compute-domain/chart/values.yaml b/compute-domain/chart/values.yaml new file mode 100644 index 0000000..0a575d3 --- /dev/null +++ b/compute-domain/chart/values.yaml @@ -0,0 +1,95 @@ +# Values for microsoft.managedcomputedomain. +# The consumer is the AKS core-extension helm runner (not the public); the Azure.* +# block is injected by the extension system at install time (same model as managed DRANET). + +nameOverride: compute-domain-controller +fullnameOverride: compute-domain-controller + +image: + # Compute-domain controller image (contains BOTH compute-domain-controller and + # compute-domain-daemon; the controller spawns the daemon DaemonSet from this same + # image via the IMAGE_NAME env). + # TESTING: upstream image until the dalec/MCR onboarding lands + # (target: mcr.microsoft.com/oss/v2/nvidia/compute-domain-controller). + repository: registry.k8s.io/dra-driver-nvidia/dra-driver-nvidia-gpu + tag: "v0.5.0" + pullPolicy: IfNotPresent +imagePullSecrets: [] + +aks: + managedByLabels: + enabled: false + +rbac: + create: true + +serviceAccount: + annotations: {} + +# IMEX deployment mode. driverManaged = the controller creates a per-ComputeDomain +# compute-domain-daemon DaemonSet that runs nvidia-imex (host binary via CDI). +# hostManaged is intentionally NOT supported by this managed chart (see README). +imex: + mode: driverManaged + isolation: domain + +logVerbosity: 4 + +replicas: 1 + +leaderElection: + enabled: true + leaseDuration: "15s" + renewDeadline: "10s" + retryPeriod: "2s" + +# Feature gates passed to the controller (FEATURE_GATES env). Empty = upstream defaults +# (IMEXDaemonsWithDNSNames on, requires driver >= 570.158 which R580 satisfies). +featureGates: {} + +deviceClasses: + computeDomainDefaultChannel: + enabled: true + name: compute-domain-default-channel.nvidia.com + computeDomainDaemon: + enabled: true + name: compute-domain-daemon.nvidia.com + +priorityClassName: system-cluster-critical +nodeSelector: {} +tolerations: + - key: node-role.kubernetes.io/master + operator: Exists + effect: NoSchedule + - key: CriticalAddonsOnly + operator: Exists +# Require a real linux AKS node (not an ACI virtual-kubelet) on the system pool — +# mirrors the RP synth convention (cluster-health-monitor-synth): mode=system is a +# hard requirement, not a soft preference. +affinity: + nodeAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + nodeSelectorTerms: + - matchExpressions: + - key: kubernetes.azure.com/cluster + operator: Exists + - key: type + operator: NotIn + values: ["virtual-kubelet"] + - key: kubernetes.io/os + operator: In + values: ["linux"] + - key: kubernetes.azure.com/mode + operator: In + values: ["system"] +podAnnotations: {} +podLabels: {} +resources: + requests: + cpu: 50m + memory: 128Mi + limits: + # No CPU limit on purpose (avoids CFS throttling; matches dranet/upstream). + # Memory limit headroom: the controller's footprint scales with cluster size + # (it watches all pods/nodes), so 512Mi rather than a tight bound. + memory: 512Mi