Skip to content

feat(helm): make cluster-scoped RBAC optional - #3459

Open
ansjindal wants to merge 1 commit into
NVIDIA:mainfrom
ansjindal:feat/optional-cluster-scoped-rbac
Open

ansjindal wants to merge 1 commit into
NVIDIA:mainfrom
ansjindal:feat/optional-cluster-scoped-rbac

Conversation

@ansjindal

Copy link
Copy Markdown

Summary

Makes the gateway chart's cluster-scoped RBAC optional so a cluster-admin can apply the ClusterRole and ClusterRoleBinding once and a namespace-admin can install and upgrade the OpenShell release without cluster-scoped permissions. Both new flags default to true, so existing installs are unchanged.

Related Issue

Fixes #3043

Changes

  • Add an rbac values block following the shape proposed on the issue: rbac.create, rbac.clusterScoped.create, rbac.clusterScoped.clusterRoleName, rbac.clusterScoped.clusterRoleBindingName.
  • Gate clusterrole.yaml and clusterrolebinding.yaml on rbac.clusterScoped.create, independent of server.drivers.kubernetes.workspaceMode.
  • Gate the namespaced sandbox role.yaml / rolebinding.yaml on rbac.create. This is not merely a parent switch: in shared mode, Kubernetes escalation prevention stops an installer holding only the built-in admin role from creating a Role that grants agents.x-k8s.io verbs it does not itself hold, so such an installer needs rbac.create=false with all RBAC applied out of band. The certgen hook and credential driver RBAC keep their existing flags.
  • Add helpers that treat a missing rbac block as enabled, so helm upgrade --reuse-values from a pre-change release does not drop RBAC — the same pattern as openshell.workspaceResourcesEnabled.
  • Drive the ClusterRoleBinding roleRef from clusterRoleName so a separately applied ClusterRole can carry a cluster-admin-chosen name.
  • Add ci/values-namespace-admin.yaml, picked up automatically by the helm:lint overlay loop.
  • Document the cluster-admin / namespace-admin split, the cluster-scoped vs namespaced object inventory, and the migration path for an existing release in the chart README and docs/kubernetes/setup.mdx. This also corrects the documented ClusterRole name, which is openshell-node-reader-<namespace>.
  • Extend deploy/helm/test-split-ownership.sh to assert a rbac.clusterScoped.create=false render emits no cluster-scoped objects while keeping the gateway ServiceAccount.

Migration note for reviewers: Helm deletes objects that leave a release manifest, so setting the flag on a release that already owns the cluster-scoped objects deletes them, and the gateway loses TokenReview until a cluster-admin re-applies them. The docs prescribe annotating them with helm.sh/resource-policy=keep first; this was tested and hands ownership over with no gap.

Testing

  • mise run pre-commit passes
  • Unit tests added/updated
  • E2E tests added/updated (if applicable) — no existing E2E lane covers chart RBAC topology; verified on a live cluster instead, described below

mise run pre-commit (fmt + lint) exits 0 and fmt produces no diff. The lint half covers all 15 subtasks, including helm:lint across all 19 CI overlays, helm:docs:check, helm:test, license:check over 1045 files, clippy, and markdown/mermaid.

Unit tests — 14 new cases across clusterrole_test.yaml (5), clusterrolebinding_test.yaml (5), and a new rbac_test.yaml (4): default still renders both objects; each flag omits the right objects; omission is independent of workspace mode; name overrides land in metadata.name and roleRef; serviceAccount.create=false still yields a correct binding subject; and legacy values with no rbac key still create RBAC. Gateway suite is 163 tests / 15 suites, workspace 4 / 1, plus test-split-ownership.sh, whose new assertion was negative-controlled by flipping the flag to confirm it fails.

Live cluster (k3d, agent-sandbox v1.0.2, --kube-as-user impersonation of an identity holding only the built-in admin role in one namespace, asserted unable to get or create ClusterRoles/ClusterRoleBindings):

  • Negative control reproduces the issue exactly: clusterroles ... is forbidden ... at the cluster scope.
  • rbac.clusterScoped.create=false — cluster-admin applies the two cluster-scoped objects; namespace-admin installs and upgrades; StatefulSet rolls out; release still owns the namespaced Role/RoleBinding; the pre-created binding matches the release's ServiceAccount, which can create TokenReviews.
  • rbac.create=false — installer needs no RBAC write permission; helm get manifest confirms the release claims no RBAC; admin-owned objects survive install and upgrade.
  • serviceAccount.create=false + serviceAccount.name=my-existing-sa with RBAC applied separately: ClusterRoleBinding subject, RoleBinding subject, and pod serviceAccountName all resolve correctly; pod Running 1/1.
  • Gateway logs are clean — no forbidden, and it lists sandboxes through the namespaced Role.

Workspace modes — 23 assertions, all pass. managed and operator both install and upgrade as a namespace-admin with rbac.clusterScoped.create=false, roll out, and log no permission errors. Mode-specific ClusterRole grants were verified to land: managed gets namespace create/delete, cluster-wide sandboxes, and serviceaccount create; operator gets cluster-wide sandboxes but not namespace create; live kubectl auth can-i on the gateway SA matches per mode. In managed/operator the chart renders no namespaced sandbox Role, so an admin-only installer suffices; only shared needs one of the two documented remedies, both verified.

Backward compatibility — 15 assertions, each starting from a release installed with the pre-change chart from main and upgraded with this branch: default upgrade, --reuse-values, a pre-change values file, --set-json rbac=null, helm rollback to the pre-change revision, and uninstall leaving no cluster-scoped residue. All pass; the cluster-scoped objects survive and stay release-owned in every case.

Cluster-scope audit — all 19 CI overlays plus defaults were scanned against 16 cluster-scoped kinds (SCC, ClusterIssuer, GatewayClass, CRD, PV, StorageClass, webhooks, and others). Every overlay renders exactly ClusterRole + ClusterRoleBinding and nothing else, and zero with the flag off, confirming the object inventory the docs now publish.

Not covered: OpenShift SCC admission and Route behavior on a real OpenShift cluster. The three OpenShift overlays pass render-level checks and lint, but the live path needs an OpenShift cluster; test:e2e-kubernetes may be worth applying.

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)
  • Architecture docs updated (if applicable) — packaging change; no architecture/ doc covers chart RBAC topology

🤖 Generated with Claude Code

@github-actions

github-actions Bot commented Sep 18, 2026

Copy link
Copy Markdown

All contributors have signed the DCO ✍️ ✅
Posted by the DCO Assistant Lite bot.

The gateway chart always rendered the ClusterRole and ClusterRoleBinding, so
every install and upgrade required cluster-admin even when only namespaced
objects were needed. Installers that are namespace-admin GitOps or platform
controllers could not run the release at all, and clusters where cluster-scoped
RBAC is owned by a separate team had no supported way to split the install.

Add an rbac values block so a cluster-admin can apply the cluster-scoped objects
once and a namespace-admin can install and upgrade the release without
cluster-scoped permissions:

  rbac:
    create: true
    clusterScoped:
      create: true
      clusterRoleName: ""
      clusterRoleBindingName: ""

rbac.clusterScoped.create gates the ClusterRole and ClusterRoleBinding, and is
independent of the workspace mode. rbac.create additionally gates the namespaced
sandbox Role and RoleBinding, which matters because Kubernetes escalation
prevention stops an installer holding only the built-in admin role from creating
a Role that grants agents.x-k8s.io verbs it does not itself hold. The certgen
hook and credential driver RBAC keep their existing flags.

Both flags default to true, so current installs are unchanged. The helpers treat
a missing rbac block as enabled so upgrades with --reuse-values do not drop RBAC,
matching the existing workspaceResources pattern. The ClusterRoleBinding roleRef
follows clusterRoleName so a separately applied ClusterRole can carry a name the
cluster-admin chooses.

Document the migration for a release that already owns the cluster-scoped
objects: Helm deletes objects that leave the manifest, so annotate them with
helm.sh/resource-policy=keep before setting the flag, otherwise the gateway
loses TokenReview until a cluster-admin re-applies them.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: ansjindal <ansjindal@nvidia.com>
@ansjindal
ansjindal force-pushed the feat/optional-cluster-scoped-rbac branch from c5c9173 to e876b74 Compare September 18, 2026 12:23
@ansjindal

Copy link
Copy Markdown
Author

I have read the DCO document and I hereby sign the DCO.

@ansjindal

Copy link
Copy Markdown
Author

recheck

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(helm): gate cluster-scoped RBAC so the gateway chart can be installed without cluster-admin

1 participant