Repository navigation
Conversation
…rds chart Implements ADR-107 (#357). A plain helm upgrade never updated the NVCRE CRDs, so the new controller ran against the old schema and the API server silently pruned the fields it lacked. Charts: - Move the main chart's CRDs into a local crds subchart, enabled by crds.enabled (default true). The default renders exactly as before; crds.enabled=false turns the CRDs off wherever Helm renders the chart. - Add helm/cluster-readiness-engine-crds, which renders the same CRDs from templates/ so helm upgrade keeps them current, each with helm.sh/resource-policy: keep (crds.keep). It fails to render when files/ is empty. - make manifests writes both copies; both are committed, so verify-codegen catches drift. helm-lint, helm-package and helm-push cover both charts. nvcrectl setup init (decision 7), in order: - stop before the deps phase when any release that owns an NVCRE CRD is failed, pending or unreadable, including partial ownership; - skip the apply and pass --skip-crds only when one deployed release owns every CRD the chart being installed carries; - leave the CRDs alone when the release's values set crds.enabled=false; - otherwise apply them as before. Release: push, sign and attest the CRDs chart beside the main chart, and verify both at the release gate. Docs: deployment.md covers installing, upgrading and adopting with the CRDs chart, and moving an Argo CD install to it. ADR-107 is accepted, with the Argo CD handoff tested on Kind; ADR-064 notes the amendment. Signed-off-by: Kayne Tu <kaynet@nvidia.com>
|
Important Review skippedReview was skipped as selected files did not have any reviewable changes. ⛔ Files ignored due to path filters: 77
⚙️ Run configuration
⛔ Files ignored due to path filters: 77
You can disable this status message by setting the Use the checkbox below for a quick retry:
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info
📝 Walkthrough
Merge Risk: ⚪ Minimal · up to
|
ndipebot
left a comment
There was a problem hiding this comment.
I ran this with Helm 3.18.4 from a worktree at 79735fc9, and every mechanic ADR-107 claims holds when I actually run it.
helm template t helm/cluster-readiness-engine --include-crds gives 7 CRDs, 0 with --set crds.enabled=false. helm show crds gives 7 either way, which is exactly why pkg/setup has to read crds.enabled itself rather than infer it from the stream, and helm_values.go does. files/ is committed as a rename of the old top-level crds/ and not staged at package time, so test/helm's identity assertion renders real content instead of an empty tree. The fromYaml/toYaml round trip is lossless: all seven rendered spec blocks match charts/crds/crds/*.yaml after normalization, and maxLength: 65536, maxItems: 128 and minimum: -1 come back intact, no scientific notation. TestCRDsSubchartFollowsEnabled does pass --include-crds, so its goldens are 7 names vs [], not both empty.
helm dependency build --skip-refresh resolves the local crds dependency and leaves git status clean, helm lint passes on both charts, and helm package keeps charts/crds/crds/* in the main tgz and files/* in the CRDs one. go test ./pkg/setup/ ./test/helm/ passes. The only surviving references to helm/cluster-readiness-engine/crds/ are in ADRs 076, 077, 088, 094 and 097, which are records and correctly left alone.
The four rules in planCRDReconcile and confirmReleaseOwned line up with decision 7, and the stop sits after the confirmation prompt and before setupPhaseParams is built (setup.go:327-337), so nothing has touched the cluster by then. confirmReleaseOwned reads the chart's actual CRD set rather than a hardcoded seven, and owned-deployed-chart-adds-crd pins that. The 22 crd-ownership fixtures assert action, owner, skipCRDs, the status-read order and the full printed output, and gate_step_legacy_test.go gives the CRDs chart a digest distinct from the main chart's, so a check that only ever reached the main chart cannot pass on the wrong subject.
I modelled Helm's encodeRelease independently over the 1,889,347 bytes in files/ (JSON carrying chart.File.Data as base64 plus the rendered manifest, then gzip, then base64) and got 677,060 bytes. The ADR's 686 KB is honest and its "about 1.5x headroom" is right.
The Argo CD handoff you cut from six steps to two is grounded in named Argo CD source (!kube.IsCRD(target), shouldBeDeleted) rather than asserted, which is better than what the merged ADR proposed.
Two optional items inline: an unpaged full-CRD LIST in listGroupCRDs, and no CI guard on the release-Secret size the ADR itself calls close. Neither needs to block this.
/lgtm
/approve
planCRDReconcile needs only CRD names, labels and annotations, but listed every CRD in the cluster with its full schema in one unpaged request. It now lists PartialObjectMetadata and selects the NVCRE group by name suffix, which the API server's <plural>.<group> naming makes equivalent. TestReleaseSecretFitsBudget estimates both charts' release Secrets the way Helm's Secrets driver encodes them and fails above 850 KiB, so the headroom ADR-107 records breaks CI before it breaks an install. Signed-off-by: Kayne Tu <kaynet@nvidia.com>
Brings in #510 (networkAttachments). Its CRD changes landed only in the crds subchart's copy, so make manifests regenerated the CRDs chart's files/ to match. Signed-off-by: Kayne Tu <kaynet@nvidia.com>
Summary
Implements ADR-107 (#551). A plain
helm upgradenever updated the NVCRE CRDs, so the new controller ran against the old schema and the API server silently pruned the fields it lacked.Charts
crdssubchart (charts/crds/crds/), enabled bycrds.enabled, defaulttrue. The default renders exactly as before.crds.enabled=falseturns the CRDs off wherever Helm renders the chart.helm/cluster-readiness-engine-crds/renders the same CRDs fromtemplates/, sohelm upgradekeeps them current, including removing fields. Each CRD carrieshelm.sh/resource-policy: keep(crds.keep). The template fails whenfiles/is empty, so an empty chart cannot publish.make manifestswrites both copies and both are committed (Ebot's note 1 on docs(designs): add ADR-107 for a separate cluster-readiness-engine-crds chart #551):verify-codegencatches drift, andtest/helmrenders the real charts from the source tree.helm-lint,helm-packageandhelm-pushcover both charts;helm-pushaddsCRDS_CHART_DIGEST_FILE.nvcrectl
setup init(decision 7), first matching rule wins:failed,pending-*or unreadable: stop before thedepsphase changes anything. This includes partial ownership, such as a Helm 4 adoption conflict leaving six CRDs owned by a failed release and one unowned.deployedrelease owns every CRD the chart being installed carries: skip the apply and pass--skip-crds. The set comes fromhelm show crds, not a fixed seven, so a missing CRD or one a newer chart adds falls through.crds.enabled=false: leave the CRDs alone.Release:
helm-publishpushes both charts; a newattest-crds-chartjob signs the CRDs chart throughattest.yml;chart-attestedand the release gate check both. The release notes gain a CRDs-chart section and the CRD path change.Docs:
deployment.mdcovers installing, upgrading and adopting with the CRDs chart (Helm 4--take-ownership/--force-conflicts), Helmfile and Flux ordering, and moving an Argo CD install. ADR-107 is accepted, with Ebot's notes 2 and 3 fixed; ADR-064 notes the amendment.Review focus
!kube.IsCRD(target)), and cascading deletion skips CRDs andkeepresources (shouldBeDeleted), from v2.10.0 through v3.5.6. So the main Application never owns the CRDs, and "disable pruning around the switch" is unnecessary. ADR-107's Notes record the runs.pkg/setup/crd_ownership.go: the two-stage decision. Stops happen before thedepsphase; the full-set check runs after the read-onlyhelm show crds.release.yml: the gate's chart checks are now averify_chartfunction run once per chart, with the main chart's messages unchanged.Related Issue
Closes #357. AICR's adoption is tracked in NVIDIA/aicr#2525.
Type of Change
Component(s) Affected
Testing
make lint,make build,make test(unit and integration),go test ./test/helm/ ./test/releasepolicy/ ./test/docspolicy/,make verify-doc-linkshelm uninstall, adoption with--take-ownership.nvcrectl setup initwith a controller image from this branch, through every rule: stale owner (apply), deployed owner (skip; the CRDs'helmmanaged-fields timestamp did not move across a full run), missing CRD (apply, recreated), failed owner after a Helm 4nvcrectl-setupconflict (stop; main release revision unchanged), six CRDs owned by a failed release plus one unowned, with and withoutcrds.enabled=false(stop), the documented--force-conflictsrecovery, andcrds.enabled=falseset and kept (CRDs untouched).crds/tocharts/crds/crds/, noted in the release notes;helm show crdsreturns the CRDs as before.New golden files were generated for new cases only and reviewed; no existing golden changed. Release-policy tests were extended for the CRDs chart and mutation-checked: dropping it from the gate turns them red.
Checklist
git commit -s)make manifests generaterun (if*_types.gowas modified)