Skip to content

feat(helm): publish the CRDs as a separate cluster-readiness-engine-crds chart (ADR-107) - #560

Open
kaynetu wants to merge 3 commits into
mainfrom
claude/cluster-readiness-engine-impl-2aacca
Open

kaynetu wants to merge 3 commits into
mainfrom
claude/cluster-readiness-engine-impl-2aacca

Conversation

@kaynetu

@kaynetu kaynetu commented Oct 23, 2026

Copy link
Copy Markdown
Collaborator

Summary

Implements ADR-107 (#551). A plain helm upgrade never updated the NVCRE CRDs, so the new controller ran against the old schema and the API server silently pruned the fields it lacked.

Charts

  • The main chart's CRDs move into a local crds subchart (charts/crds/crds/), enabled by crds.enabled, default true. The default renders exactly as before. crds.enabled=false turns the CRDs off wherever Helm renders the chart.
  • New helm/cluster-readiness-engine-crds/ renders the same CRDs from templates/, so helm upgrade keeps them current, including removing fields. Each CRD carries helm.sh/resource-policy: keep (crds.keep). The template fails when files/ is empty, so an empty chart cannot publish.
  • make manifests writes both copies and both are committed (Ebot's note 1 on docs(designs): add ADR-107 for a separate cluster-readiness-engine-crds chart #551): verify-codegen catches drift, and test/helm renders the real charts from the source tree. helm-lint, helm-package and helm-push cover both charts; helm-push adds CRDS_CHART_DIGEST_FILE.

nvcrectl setup init (decision 7), first matching rule wins:

  1. Any release that owns an NVCRE CRD is failed, pending-* or unreadable: stop before the deps phase changes anything. This includes partial ownership, such as a Helm 4 adoption conflict leaving six CRDs owned by a failed release and one unowned.
  2. One deployed release owns every CRD the chart being installed carries: skip the apply and pass --skip-crds. The set comes from helm show crds, not a fixed seven, so a missing CRD or one a newer chart adds falls through.
  3. The release's values set crds.enabled=false: leave the CRDs alone.
  4. Otherwise apply as before, with a warning for split ownership.

Release: helm-publish pushes both charts; a new attest-crds-chart job signs the CRDs chart through attest.yml; chart-attested and the release gate check both. The release notes gain a CRDs-chart section and the CRD path change.

Docs: deployment.md covers installing, upgrading and adopting with the CRDs chart (Helm 4 --take-ownership / --force-conflicts), Helmfile and Flux ordering, and moving an Argo CD install. ADR-107 is accepted, with Ebot's notes 2 and 3 fixed; ADR-064 notes the amendment.

Review focus

  • The Argo CD handoff is simpler than the ADR proposed. Argo CD's repo-server never writes tracking metadata on CRDs (!kube.IsCRD(target)), and cascading deletion skips CRDs and keep resources (shouldBeDeleted), from v2.10.0 through v3.5.6. So the main Application never owns the CRDs, and "disable pruning around the switch" is unnecessary. ADR-107's Notes record the runs.
  • pkg/setup/crd_ownership.go: the two-stage decision. Stops happen before the deps phase; the full-set check runs after the read-only helm show crds.
  • release.yml: the gate's chart checks are now a verify_chart function run once per chart, with the main chart's messages unchanged.

Related Issue

Closes #357. AICR's adoption is tracked in NVIDIA/aicr#2525.

Type of Change

  • 🐛 Bug fix
  • ✨ New feature
  • 💥 Breaking change
  • 📚 Documentation
  • 🔧 Refactoring
  • 🔨 Build/CI

Component(s) Affected

  • API / CRDs
  • Controller / Reconcilers
  • Catalog / Workloads
  • CLI (nvcrectl)
  • Helm / Deployment
  • Documentation / CI
  • Other: ____________

Testing

  • Tests pass locally: make lint, make build, make test (unit and integration), go test ./test/helm/ ./test/releasepolicy/ ./test/docspolicy/, make verify-doc-links
  • Manual testing completed, on throwaway Kind clusters with Helm 4.3.1:
    • CRDs chart: install, a field added and then removed on upgrade, kept on helm uninstall, adoption with --take-ownership.
    • Real nvcrectl setup init with a controller image from this branch, through every rule: stale owner (apply), deployed owner (skip; the CRDs' helm managed-fields timestamp did not move across a full run), missing CRD (apply, recreated), failed owner after a Helm 4 nvcrectl-setup conflict (stop; main release revision unchanged), six CRDs owned by a failed release plus one unowned, with and without crds.enabled=false (stop), the documented --force-conflicts recovery, and crds.enabled=false set and kept (CRDs untouched).
    • Argo CD v3.5.6 handoff with automated pruning and existing NVCRE objects: the six-step sequence, the two-step sequence with pruning on under annotation and label tracking, and cascading deletes of the CRDs Application. All seven CRDs kept their UIDs and every object survived.
  • No breaking changes (or documented): the default install is unchanged. The CRD path inside the packaged main chart moves from crds/ to charts/crds/crds/, noted in the release notes; helm show crds returns the CRDs as before.

New golden files were generated for new cases only and reviewed; no existing golden changed. Release-policy tests were extended for the CRDs chart and mutation-checked: dropping it from the gate turns them red.

Checklist

  • Self-review completed
  • Commits are signed off for the DCO (git commit -s)
  • make manifests generate run (if *_types.go was modified)
  • Golden files updated (if integration test output changed)
  • Documentation updated (if needed)
  • Ready for review

…rds chart

Implements ADR-107 (#357). A plain helm upgrade never updated the NVCRE
CRDs, so the new controller ran against the old schema and the API server
silently pruned the fields it lacked.

Charts:
- Move the main chart's CRDs into a local crds subchart, enabled by
  crds.enabled (default true). The default renders exactly as before;
  crds.enabled=false turns the CRDs off wherever Helm renders the chart.
- Add helm/cluster-readiness-engine-crds, which renders the same CRDs from
  templates/ so helm upgrade keeps them current, each with
  helm.sh/resource-policy: keep (crds.keep). It fails to render when
  files/ is empty.
- make manifests writes both copies; both are committed, so verify-codegen
  catches drift. helm-lint, helm-package and helm-push cover both charts.

nvcrectl setup init (decision 7), in order:
- stop before the deps phase when any release that owns an NVCRE CRD is
  failed, pending or unreadable, including partial ownership;
- skip the apply and pass --skip-crds only when one deployed release owns
  every CRD the chart being installed carries;
- leave the CRDs alone when the release's values set crds.enabled=false;
- otherwise apply them as before.

Release: push, sign and attest the CRDs chart beside the main chart, and
verify both at the release gate.

Docs: deployment.md covers installing, upgrading and adopting with the
CRDs chart, and moving an Argo CD install to it. ADR-107 is accepted, with
the Argo CD handoff tested on Kind; ADR-064 notes the amendment.

Signed-off-by: Kayne Tu <kaynet@nvidia.com>
@github-actions

Copy link
Copy Markdown
Contributor

@coderabbitai

coderabbitai Bot commented Oct 23, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

Important

Review skipped

Review was skipped as selected files did not have any reviewable changes.

⛔ Files ignored due to path filters: 77
  • helm/cluster-readiness-engine-crds/files/nvcre.nvidia.com_bandwidthmeasurements.yaml is excluded by !helm/cluster-readiness-engine-crds/files/**
  • helm/cluster-readiness-engine-crds/files/nvcre.nvidia.com_certifications.yaml is excluded by !helm/cluster-readiness-engine-crds/files/**
  • helm/cluster-readiness-engine-crds/files/nvcre.nvidia.com_goodputmeasurements.yaml is excluded by !helm/cluster-readiness-engine-crds/files/**
  • helm/cluster-readiness-engine-crds/files/nvcre.nvidia.com_jobs.yaml is excluded by !helm/cluster-readiness-engine-crds/files/**
  • helm/cluster-readiness-engine-crds/files/nvcre.nvidia.com_logprofiles.yaml is excluded by !helm/cluster-readiness-engine-crds/files/**
  • helm/cluster-readiness-engine-crds/files/nvcre.nvidia.com_workflows.yaml is excluded by !helm/cluster-readiness-engine-crds/files/**
  • helm/cluster-readiness-engine-crds/files/nvcre.nvidia.com_workloadruns.yaml is excluded by !helm/cluster-readiness-engine-crds/files/**
  • helm/cluster-readiness-engine/Chart.lock is excluded by !**/*.lock
  • helm/cluster-readiness-engine/charts/crds/crds/nvcre.nvidia.com_bandwidthmeasurements.yaml is excluded by !helm/cluster-readiness-engine/charts/crds/crds/**
  • helm/cluster-readiness-engine/charts/crds/crds/nvcre.nvidia.com_certifications.yaml is excluded by !helm/cluster-readiness-engine/charts/crds/crds/**
  • helm/cluster-readiness-engine/charts/crds/crds/nvcre.nvidia.com_goodputmeasurements.yaml is excluded by !helm/cluster-readiness-engine/charts/crds/crds/**
  • helm/cluster-readiness-engine/charts/crds/crds/nvcre.nvidia.com_jobs.yaml is excluded by !helm/cluster-readiness-engine/charts/crds/crds/**
  • helm/cluster-readiness-engine/charts/crds/crds/nvcre.nvidia.com_logprofiles.yaml is excluded by !helm/cluster-readiness-engine/charts/crds/crds/**
  • helm/cluster-readiness-engine/charts/crds/crds/nvcre.nvidia.com_workflows.yaml is excluded by !helm/cluster-readiness-engine/charts/crds/crds/**
  • helm/cluster-readiness-engine/charts/crds/crds/nvcre.nvidia.com_workloadruns.yaml is excluded by !helm/cluster-readiness-engine/charts/crds/crds/**
  • pkg/setup/testdata/chart-values-probe/crds-disabled-kept/expected.json is excluded by !**/testdata/**
  • pkg/setup/testdata/chart-values-probe/crds-disabled-kept/input.yaml is excluded by !**/testdata/**
  • pkg/setup/testdata/chart-values-probe/crds-disabled-this-run/expected.json is excluded by !**/testdata/**
  • pkg/setup/testdata/chart-values-probe/crds-disabled-this-run/input.yaml is excluded by !**/testdata/**
  • pkg/setup/testdata/crd-ownership/disabled-owned-crd-missing/expected.json is excluded by !**/testdata/**
  • pkg/setup/testdata/crd-ownership/disabled-owned-crd-missing/input.yaml is excluded by !**/testdata/**
  • pkg/setup/testdata/crd-ownership/disabled-owned-deployed/expected.json is excluded by !**/testdata/**
  • pkg/setup/testdata/crd-ownership/disabled-owned-deployed/input.yaml is excluded by !**/testdata/**
  • pkg/setup/testdata/crd-ownership/disabled-owned-failed/expected.json is excluded by !**/testdata/**
  • pkg/setup/testdata/crd-ownership/disabled-owned-failed/input.yaml is excluded by !**/testdata/**
  • pkg/setup/testdata/crd-ownership/disabled-unowned/expected.json is excluded by !**/testdata/**
  • pkg/setup/testdata/crd-ownership/disabled-unowned/input.yaml is excluded by !**/testdata/**
  • pkg/setup/testdata/crd-ownership/fresh-cluster/expected.json is excluded by !**/testdata/**
  • pkg/setup/testdata/crd-ownership/fresh-cluster/input.yaml is excluded by !**/testdata/**
  • pkg/setup/testdata/crd-ownership/other-group-ignored/expected.json is excluded by !**/testdata/**
  • pkg/setup/testdata/crd-ownership/other-group-ignored/input.yaml is excluded by !**/testdata/**
  • pkg/setup/testdata/crd-ownership/owned-deployed-chart-adds-crd/expected.json is excluded by !**/testdata/**
  • pkg/setup/testdata/crd-ownership/owned-deployed-chart-adds-crd/input.yaml is excluded by !**/testdata/**
  • pkg/setup/testdata/crd-ownership/owned-deployed-chart-drops-crd/expected.json is excluded by !**/testdata/**
  • pkg/setup/testdata/crd-ownership/owned-deployed-chart-drops-crd/input.yaml is excluded by !**/testdata/**
  • pkg/setup/testdata/crd-ownership/owned-deployed-crd-missing/expected.json is excluded by !**/testdata/**
  • pkg/setup/testdata/crd-ownership/owned-deployed-crd-missing/input.yaml is excluded by !**/testdata/**
  • pkg/setup/testdata/crd-ownership/owned-deployed-one-of-seven/expected.json is excluded by !**/testdata/**
  • pkg/setup/testdata/crd-ownership/owned-deployed-one-of-seven/input.yaml is excluded by !**/testdata/**
  • pkg/setup/testdata/crd-ownership/owned-deployed/expected.json is excluded by !**/testdata/**
  • pkg/setup/testdata/crd-ownership/owned-deployed/input.yaml is excluded by !**/testdata/**
  • pkg/setup/testdata/crd-ownership/owned-failed/expected.json is excluded by !**/testdata/**
  • pkg/setup/testdata/crd-ownership/owned-failed/input.yaml is excluded by !**/testdata/**
  • pkg/setup/testdata/crd-ownership/owned-pending-upgrade/expected.json is excluded by !**/testdata/**
  • pkg/setup/testdata/crd-ownership/owned-pending-upgrade/input.yaml is excluded by !**/testdata/**
  • pkg/setup/testdata/crd-ownership/owned-release-gone/expected.json is excluded by !**/testdata/**
  • pkg/setup/testdata/crd-ownership/owned-release-gone/input.yaml is excluded by !**/testdata/**
  • pkg/setup/testdata/crd-ownership/owned-uninstalled/expected.json is excluded by !**/testdata/**
  • pkg/setup/testdata/crd-ownership/owned-uninstalled/input.yaml is excluded by !**/testdata/**
  • pkg/setup/testdata/crd-ownership/partly-owned-failed-disabled/expected.json is excluded by !**/testdata/**
  • pkg/setup/testdata/crd-ownership/partly-owned-failed-disabled/input.yaml is excluded by !**/testdata/**
  • pkg/setup/testdata/crd-ownership/partly-owned-failed/expected.json is excluded by !**/testdata/**
  • pkg/setup/testdata/crd-ownership/partly-owned-failed/input.yaml is excluded by !**/testdata/**
  • pkg/setup/testdata/crd-ownership/partly-owned-unreadable/expected.json is excluded by !**/testdata/**
  • pkg/setup/testdata/crd-ownership/partly-owned-unreadable/input.yaml is excluded by !**/testdata/**
  • pkg/setup/testdata/crd-ownership/partly-owned/expected.json is excluded by !**/testdata/**
  • pkg/setup/testdata/crd-ownership/partly-owned/input.yaml is excluded by !**/testdata/**
  • pkg/setup/testdata/crd-ownership/status-unreadable/expected.json is excluded by !**/testdata/**
  • pkg/setup/testdata/crd-ownership/status-unreadable/input.yaml is excluded by !**/testdata/**
  • pkg/setup/testdata/crd-ownership/two-releases-one-pending/expected.json is excluded by !**/testdata/**
  • pkg/setup/testdata/crd-ownership/two-releases-one-pending/input.yaml is excluded by !**/testdata/**
  • pkg/setup/testdata/crd-ownership/two-releases/expected.json is excluded by !**/testdata/**
  • pkg/setup/testdata/crd-ownership/two-releases/input.yaml is excluded by !**/testdata/**
  • pkg/setup/testdata/crd-ownership/unowned/expected.json is excluded by !**/testdata/**
  • pkg/setup/testdata/crd-ownership/unowned/input.yaml is excluded by !**/testdata/**
  • pkg/setup/testdata/helm-chart-args/release-owned-crds/expected.json is excluded by !**/testdata/**
  • pkg/setup/testdata/helm-chart-args/release-owned-crds/input.yaml is excluded by !**/testdata/**
  • pkg/setup/testdata/init-values-runs/crds-disabled-kept/expected.json is excluded by !**/testdata/**
  • pkg/setup/testdata/init-values-runs/crds-disabled-kept/input.yaml is excluded by !**/testdata/**
  • test/helm/testdata/render-crds-chart/keep-default/expected.json is excluded by !**/testdata/**
  • test/helm/testdata/render-crds-chart/keep-default/input.yaml is excluded by !**/testdata/**
  • test/helm/testdata/render-crds-chart/keep-false/expected.json is excluded by !**/testdata/**
  • test/helm/testdata/render-crds-chart/keep-false/input.yaml is excluded by !**/testdata/**
  • test/helm/testdata/render-crds-subchart/enabled-default/expected.json is excluded by !**/testdata/**
  • test/helm/testdata/render-crds-subchart/enabled-default/input.yaml is excluded by !**/testdata/**
  • test/helm/testdata/render-crds-subchart/enabled-false/expected.json is excluded by !**/testdata/**
  • test/helm/testdata/render-crds-subchart/enabled-false/input.yaml is excluded by !**/testdata/**
⚙️ Run configuration
📥 Commits

Reviewing files that changed from the base of the PR and between bf7e4d8 and 47093b1.

⛔ Files ignored due to path filters: 77
  • helm/cluster-readiness-engine-crds/files/nvcre.nvidia.com_bandwidthmeasurements.yaml is excluded by !helm/cluster-readiness-engine-crds/files/**
  • helm/cluster-readiness-engine-crds/files/nvcre.nvidia.com_certifications.yaml is excluded by !helm/cluster-readiness-engine-crds/files/**
  • helm/cluster-readiness-engine-crds/files/nvcre.nvidia.com_goodputmeasurements.yaml is excluded by !helm/cluster-readiness-engine-crds/files/**
  • helm/cluster-readiness-engine-crds/files/nvcre.nvidia.com_jobs.yaml is excluded by !helm/cluster-readiness-engine-crds/files/**
  • helm/cluster-readiness-engine-crds/files/nvcre.nvidia.com_logprofiles.yaml is excluded by !helm/cluster-readiness-engine-crds/files/**
  • helm/cluster-readiness-engine-crds/files/nvcre.nvidia.com_workflows.yaml is excluded by !helm/cluster-readiness-engine-crds/files/**
  • helm/cluster-readiness-engine-crds/files/nvcre.nvidia.com_workloadruns.yaml is excluded by !helm/cluster-readiness-engine-crds/files/**
  • helm/cluster-readiness-engine/Chart.lock is excluded by !**/*.lock
  • helm/cluster-readiness-engine/charts/crds/crds/nvcre.nvidia.com_bandwidthmeasurements.yaml is excluded by !helm/cluster-readiness-engine/charts/crds/crds/**
  • helm/cluster-readiness-engine/charts/crds/crds/nvcre.nvidia.com_certifications.yaml is excluded by !helm/cluster-readiness-engine/charts/crds/crds/**
  • helm/cluster-readiness-engine/charts/crds/crds/nvcre.nvidia.com_goodputmeasurements.yaml is excluded by !helm/cluster-readiness-engine/charts/crds/crds/**
  • helm/cluster-readiness-engine/charts/crds/crds/nvcre.nvidia.com_jobs.yaml is excluded by !helm/cluster-readiness-engine/charts/crds/crds/**
  • helm/cluster-readiness-engine/charts/crds/crds/nvcre.nvidia.com_logprofiles.yaml is excluded by !helm/cluster-readiness-engine/charts/crds/crds/**
  • helm/cluster-readiness-engine/charts/crds/crds/nvcre.nvidia.com_workflows.yaml is excluded by !helm/cluster-readiness-engine/charts/crds/crds/**
  • helm/cluster-readiness-engine/charts/crds/crds/nvcre.nvidia.com_workloadruns.yaml is excluded by !helm/cluster-readiness-engine/charts/crds/crds/**
  • pkg/setup/testdata/chart-values-probe/crds-disabled-kept/expected.json is excluded by !**/testdata/**
  • pkg/setup/testdata/chart-values-probe/crds-disabled-kept/input.yaml is excluded by !**/testdata/**
  • pkg/setup/testdata/chart-values-probe/crds-disabled-this-run/expected.json is excluded by !**/testdata/**
  • pkg/setup/testdata/chart-values-probe/crds-disabled-this-run/input.yaml is excluded by !**/testdata/**
  • pkg/setup/testdata/crd-ownership/disabled-owned-crd-missing/expected.json is excluded by !**/testdata/**
  • pkg/setup/testdata/crd-ownership/disabled-owned-crd-missing/input.yaml is excluded by !**/testdata/**
  • pkg/setup/testdata/crd-ownership/disabled-owned-deployed/expected.json is excluded by !**/testdata/**
  • pkg/setup/testdata/crd-ownership/disabled-owned-deployed/input.yaml is excluded by !**/testdata/**
  • pkg/setup/testdata/crd-ownership/disabled-owned-failed/expected.json is excluded by !**/testdata/**
  • pkg/setup/testdata/crd-ownership/disabled-owned-failed/input.yaml is excluded by !**/testdata/**
  • pkg/setup/testdata/crd-ownership/disabled-unowned/expected.json is excluded by !**/testdata/**
  • pkg/setup/testdata/crd-ownership/disabled-unowned/input.yaml is excluded by !**/testdata/**
  • pkg/setup/testdata/crd-ownership/fresh-cluster/expected.json is excluded by !**/testdata/**
  • pkg/setup/testdata/crd-ownership/fresh-cluster/input.yaml is excluded by !**/testdata/**
  • pkg/setup/testdata/crd-ownership/other-group-ignored/expected.json is excluded by !**/testdata/**
  • pkg/setup/testdata/crd-ownership/other-group-ignored/input.yaml is excluded by !**/testdata/**
  • pkg/setup/testdata/crd-ownership/owned-deployed-chart-adds-crd/expected.json is excluded by !**/testdata/**
  • pkg/setup/testdata/crd-ownership/owned-deployed-chart-adds-crd/input.yaml is excluded by !**/testdata/**
  • pkg/setup/testdata/crd-ownership/owned-deployed-chart-drops-crd/expected.json is excluded by !**/testdata/**
  • pkg/setup/testdata/crd-ownership/owned-deployed-chart-drops-crd/input.yaml is excluded by !**/testdata/**
  • pkg/setup/testdata/crd-ownership/owned-deployed-crd-missing/expected.json is excluded by !**/testdata/**
  • pkg/setup/testdata/crd-ownership/owned-deployed-crd-missing/input.yaml is excluded by !**/testdata/**
  • pkg/setup/testdata/crd-ownership/owned-deployed-one-of-seven/expected.json is excluded by !**/testdata/**
  • pkg/setup/testdata/crd-ownership/owned-deployed-one-of-seven/input.yaml is excluded by !**/testdata/**
  • pkg/setup/testdata/crd-ownership/owned-deployed/expected.json is excluded by !**/testdata/**
  • pkg/setup/testdata/crd-ownership/owned-deployed/input.yaml is excluded by !**/testdata/**
  • pkg/setup/testdata/crd-ownership/owned-failed/expected.json is excluded by !**/testdata/**
  • pkg/setup/testdata/crd-ownership/owned-failed/input.yaml is excluded by !**/testdata/**
  • pkg/setup/testdata/crd-ownership/owned-pending-upgrade/expected.json is excluded by !**/testdata/**
  • pkg/setup/testdata/crd-ownership/owned-pending-upgrade/input.yaml is excluded by !**/testdata/**
  • pkg/setup/testdata/crd-ownership/owned-release-gone/expected.json is excluded by !**/testdata/**
  • pkg/setup/testdata/crd-ownership/owned-release-gone/input.yaml is excluded by !**/testdata/**
  • pkg/setup/testdata/crd-ownership/owned-uninstalled/expected.json is excluded by !**/testdata/**
  • pkg/setup/testdata/crd-ownership/owned-uninstalled/input.yaml is excluded by !**/testdata/**
  • pkg/setup/testdata/crd-ownership/partly-owned-failed-disabled/expected.json is excluded by !**/testdata/**
  • pkg/setup/testdata/crd-ownership/partly-owned-failed-disabled/input.yaml is excluded by !**/testdata/**
  • pkg/setup/testdata/crd-ownership/partly-owned-failed/expected.json is excluded by !**/testdata/**
  • pkg/setup/testdata/crd-ownership/partly-owned-failed/input.yaml is excluded by !**/testdata/**
  • pkg/setup/testdata/crd-ownership/partly-owned-unreadable/expected.json is excluded by !**/testdata/**
  • pkg/setup/testdata/crd-ownership/partly-owned-unreadable/input.yaml is excluded by !**/testdata/**
  • pkg/setup/testdata/crd-ownership/partly-owned/expected.json is excluded by !**/testdata/**
  • pkg/setup/testdata/crd-ownership/partly-owned/input.yaml is excluded by !**/testdata/**
  • pkg/setup/testdata/crd-ownership/status-unreadable/expected.json is excluded by !**/testdata/**
  • pkg/setup/testdata/crd-ownership/status-unreadable/input.yaml is excluded by !**/testdata/**
  • pkg/setup/testdata/crd-ownership/two-releases-one-pending/expected.json is excluded by !**/testdata/**
  • pkg/setup/testdata/crd-ownership/two-releases-one-pending/input.yaml is excluded by !**/testdata/**
  • pkg/setup/testdata/crd-ownership/two-releases/expected.json is excluded by !**/testdata/**
  • pkg/setup/testdata/crd-ownership/two-releases/input.yaml is excluded by !**/testdata/**
  • pkg/setup/testdata/crd-ownership/unowned/expected.json is excluded by !**/testdata/**
  • pkg/setup/testdata/crd-ownership/unowned/input.yaml is excluded by !**/testdata/**
  • pkg/setup/testdata/helm-chart-args/release-owned-crds/expected.json is excluded by !**/testdata/**
  • pkg/setup/testdata/helm-chart-args/release-owned-crds/input.yaml is excluded by !**/testdata/**
  • pkg/setup/testdata/init-values-runs/crds-disabled-kept/expected.json is excluded by !**/testdata/**
  • pkg/setup/testdata/init-values-runs/crds-disabled-kept/input.yaml is excluded by !**/testdata/**
  • test/helm/testdata/render-crds-chart/keep-default/expected.json is excluded by !**/testdata/**
  • test/helm/testdata/render-crds-chart/keep-default/input.yaml is excluded by !**/testdata/**
  • test/helm/testdata/render-crds-chart/keep-false/expected.json is excluded by !**/testdata/**
  • test/helm/testdata/render-crds-chart/keep-false/input.yaml is excluded by !**/testdata/**
  • test/helm/testdata/render-crds-subchart/enabled-default/expected.json is excluded by !**/testdata/**
  • test/helm/testdata/render-crds-subchart/enabled-default/input.yaml is excluded by !**/testdata/**
  • test/helm/testdata/render-crds-subchart/enabled-false/expected.json is excluded by !**/testdata/**
  • test/helm/testdata/render-crds-subchart/enabled-false/input.yaml is excluded by !**/testdata/**

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration
📥 Commits

Reviewing files that changed from the base of the PR and between 79735fc and bf7e4d8.

📒 Files selected for processing: 5
  • docs/designs/107-crds-chart.md
  • pkg/setup/crd_ownership.go
  • pkg/setup/jobset_ownership.go
  • pkg/setup/recovery_inventory.go
  • test/helm/release_size_test.go
🚧 Files skipped from review as they are similar to previous changes: 1
  • docs/designs/107-crds-chart.md

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 10 remain after this review.

📝 Walkthrough

The PR adds a standalone CRDs Helm chart and moves the main chart’s CRDs into a conditional subchart. Manifest generation keeps both CRD copies aligned. Setup checks Helm ownership and release status before applying CRDs, leaving them unmanaged, or passing --skip-crds. The release process packages, publishes, attests, and verifies both charts. Tests and documentation cover chart rendering, setup rules, upgrades, and release checks.

Priority: ⬆️ High

Estimated code review effort: 4 (Complex) | ~60 minutes

Change: Feature · Severity of issue fixed: Medium

Possibly related issues

Possibly related PRs

Merge Risk: ⚪ Minimal · up to bf7e4

No actionable issue is established for these changes; they are mergeable after normal checks.

Pre-merge checks: Passed 4 · Failed 1

Failed Failed checks: 1 warning

Check name Status Explanation Resolution
Docstring Coverage Warning Docstring coverage is 69.23% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 52 functions across 16 files. (1 skipped:… Write docstrings for the functions missing them to satisfy the coverage threshold.
Passed Passed checks: 4 passed
Check name Status Explanation
Description check Passed The description clearly explains the separate CRDs chart, setup ownership behavior, release changes, documentation updates, and testing. It is directly related to the changeset.
Title check Passed The title clearly and concisely identifies the primary change: publishing CRDs as a separate Helm chart under ADR-107.
Linked Issues check Passed Directly linked issue #357 requires publishing the CRDs as a separate cluster-readiness-engine-crds Helm chart so consumers can upgrade CRDs as an ordinary, orderable release. The reviewed head cont…
Out of Scope Changes check Passed The changes support the CRDs-chart work requested by #357. The new chart, the main chart's crds subchart, make manifests, the Makefile packaging and push targets, and the release workflow publish …
Full details: Linked Issues check

Explanation

Directly linked issue #357 requires publishing the CRDs as a separate cluster-readiness-engine-crds Helm chart so consumers can upgrade CRDs as an ordinary, orderable release. The reviewed head contains helm/cluster-readiness-engine-crds/ with Chart.yaml, values.yaml, templates/crds.yaml, templates/_helpers.tpl, and seven CRD files under files/ (one per NVCRE CRD). templates/crds.yaml renders those files, and the summary states that rendering fails when files/ is empty. The crds.keep value defaults to true, which adds helm.sh/resource-policy: keep. The release workflow publishes and attests the chart (helm-publish, attest-crds-chart, release gate). test/helm/crds_test.go checks that the standalone and subchart CRD copies match. The main chart keeps the CRDs in a crds subchart enabled by crds.enabled, so the default install still delivers the CRDs. Helm, Helmfile, Flux, and Argo CD ordering are covered in docs/operations/deployment.md. No concrete coding requirement from #357 is unmet. Limit: this assessment relies on the whole-PR summary and the file listing at head. The incremental changes since the previous reviewed head were not re-read hunk by hunk. A crds.go diff from the previous head returned no content, so that file did not change in the incremental range.

Full details: Out of Scope Changes check

Explanation

The changes support the CRDs-chart work requested by #357. The new chart, the main chart's crds subchart, make manifests, the Makefile packaging and push targets, and the release workflow publish and attest both charts. The nvcrectl setup init ownership logic (pkg/setup/crd_ownership.go, helm.go, helm_values.go, setup.go) prevents the main chart and the CRDs chart from fighting over the same CRDs during setup. That is a direct consequence of publishing a second chart that owns the same CRDs. The documentation and ADR-107 changes describe the same install, upgrade, and adoption paths. The apiextensionsGroup and Helm managed-by constant refactors in jobset_ownership.go and recovery_inventory.go are small supporting changes. The release-size test in test/helm/release_size_test.go checks that both charts stay within the Helm Secret size budget that the chart split depends on. The .coderabbit.yaml, AGENTS.md, and CLAUDE.md path updates follow the moved CRD locations. No demonstrated unrelated behavior change is present.

Full details: Docstring Coverage

Explanation

Docstring coverage is 69.23% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 52 functions across 16 files. (1 skipped: 1 unsupported.)

Resolution

Write docstrings for the functions missing them to satisfy the coverage threshold.

✨ Finishing Touches: 💡 2
📝 Generate docstrings: 💡
  • Commit to this branch
  • Create a new PR
⚔️ Resolve merge conflicts: 💡
  • Resolve merge conflict in branch claude/cluster-readiness-engine-impl-2aacca
🧪 Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

ndipebot
ndipebot previously approved these changes Oct 23, 2026

@ndipebot ndipebot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I ran this with Helm 3.18.4 from a worktree at 79735fc9, and every mechanic ADR-107 claims holds when I actually run it.

helm template t helm/cluster-readiness-engine --include-crds gives 7 CRDs, 0 with --set crds.enabled=false. helm show crds gives 7 either way, which is exactly why pkg/setup has to read crds.enabled itself rather than infer it from the stream, and helm_values.go does. files/ is committed as a rename of the old top-level crds/ and not staged at package time, so test/helm's identity assertion renders real content instead of an empty tree. The fromYaml/toYaml round trip is lossless: all seven rendered spec blocks match charts/crds/crds/*.yaml after normalization, and maxLength: 65536, maxItems: 128 and minimum: -1 come back intact, no scientific notation. TestCRDsSubchartFollowsEnabled does pass --include-crds, so its goldens are 7 names vs [], not both empty.

helm dependency build --skip-refresh resolves the local crds dependency and leaves git status clean, helm lint passes on both charts, and helm package keeps charts/crds/crds/* in the main tgz and files/* in the CRDs one. go test ./pkg/setup/ ./test/helm/ passes. The only surviving references to helm/cluster-readiness-engine/crds/ are in ADRs 076, 077, 088, 094 and 097, which are records and correctly left alone.

The four rules in planCRDReconcile and confirmReleaseOwned line up with decision 7, and the stop sits after the confirmation prompt and before setupPhaseParams is built (setup.go:327-337), so nothing has touched the cluster by then. confirmReleaseOwned reads the chart's actual CRD set rather than a hardcoded seven, and owned-deployed-chart-adds-crd pins that. The 22 crd-ownership fixtures assert action, owner, skipCRDs, the status-read order and the full printed output, and gate_step_legacy_test.go gives the CRDs chart a digest distinct from the main chart's, so a check that only ever reached the main chart cannot pass on the wrong subject.

I modelled Helm's encodeRelease independently over the 1,889,347 bytes in files/ (JSON carrying chart.File.Data as base64 plus the rendered manifest, then gzip, then base64) and got 677,060 bytes. The ADR's 686 KB is honest and its "about 1.5x headroom" is right.

The Argo CD handoff you cut from six steps to two is grounded in named Argo CD source (!kube.IsCRD(target), shouldBeDeleted) rather than asserted, which is better than what the merged ADR proposed.

Two optional items inline: an unpaged full-CRD LIST in listGroupCRDs, and no CI guard on the release-Secret size the ADR itself calls close. Neither needs to block this.

/lgtm
/approve

Comment thread pkg/setup/crd_ownership.go
Comment thread docs/designs/107-crds-chart.md Outdated
planCRDReconcile needs only CRD names, labels and annotations, but listed
every CRD in the cluster with its full schema in one unpaged request. It
now lists PartialObjectMetadata and selects the NVCRE group by name
suffix, which the API server's <plural>.<group> naming makes equivalent.

TestReleaseSecretFitsBudget estimates both charts' release Secrets the way
Helm's Secrets driver encodes them and fails above 850 KiB, so the
headroom ADR-107 records breaks CI before it breaks an install.

Signed-off-by: Kayne Tu <kaynet@nvidia.com>
Brings in #510 (networkAttachments). Its CRD changes landed only in the
crds subchart's copy, so make manifests regenerated the CRDs chart's
files/ to match.

Signed-off-by: Kayne Tu <kaynet@nvidia.com>
@kaynetu
kaynetu requested a review from ndipebot October 23, 2026 21:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Publish the chart's CRDs as a separate cluster-readiness-engine-crds chart

2 participants