Skip to content

fix(helm): apply manager.image.pullPolicy to the controller container - #531

Merged
ndipebot merged 8 commits into
NVIDIA:mainfrom
kang0225:fix/511-manager-image-pull-policy
Oct 23, 2026
Merged

ndipebot merged 8 commits into
NVIDIA:mainfrom
kang0225:fix/511-manager-image-pull-policy

Conversation

@kang0225

@kang0225 kang0225 commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

Summary

manager.image.pullPolicy is declared in values.yaml (default IfNotPresent), but templates/deployment.yaml never renders imagePullPolicy, so setting it has no effect.

The value now defaults to "", and imagePullPolicy is rendered only when it is set. With the value unset the chart renders exactly as before, so the API server still picks the policy from the tag, as the air-gap guidance expects. The value is also added to the key chart values table, and the air-gap step now mentions it.

Related Issue

Closes #511

Type of Change

  • 🐛 Bug fix
  • ✨ New feature
  • 💥 Breaking change
  • 📚 Documentation
  • 🔧 Refactoring
  • 🔨 Build/CI

Component(s) Affected

  • API / CRDs
  • Controller / Reconcilers
  • Catalog / Workloads
  • CLI (nvcrectl)
  • Helm / Deployment
  • Documentation / CI
  • Other: ____________

Testing

  • Tests pass locally (commit ef2eb2d): go test ./test/helm/render_test.go -v -count=1 (22 cases, none skipped), helm lint helm/cluster-readiness-engine, and go vet ./test/helm/.
  • Full Helm test execution completed: the package compiles, but go test ./test/helm/ -v -count=1 fails during catalog initialization on Windows due to embedded-file path handling. The same initialization failure reproduces on upstream main (765ff4a).
  • Manual testing completed: on kind (Kubernetes v1.37.0), main's chart with pullPolicy=Always left the Deployment at IfNotPresent. After upgrading to this branch it was Always, and IfNotPresent again once the override was removed.
  • No breaking changes (or documented): with the value unset, renders are byte-identical to main. helm upgrade --reuse-values from an older release carries the old IfNotPresent default, which now renders; only a latest tag behaves differently.

Checklist

  • Self-review completed
  • Commits are signed off for the DCO (git commit -s)
  • make manifests generate run (if *_types.go was modified): not modified
  • Golden files updated (if integration test output changed): no golden file changed
  • Documentation updated (if needed)
  • Ready for review

@copy-pr-bot

copy-pr-bot Bot commented Oct 20, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@coderabbitai

coderabbitai Bot commented Oct 20, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration
📥 Commits

Reviewing files that changed from the base of the PR and between 7a2a438 and a4643f2.

📒 Files selected for processing: 1
  • docs/operations/deployment.md

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

📝 Walkthrough

The manager image pull policy defaults to empty. The Helm template validates supported values and renders imagePullPolicy only when a policy is configured. Documentation describes manager image selection, preloaded images, workload defaults, and Helm upgrade handling. Render tests cover omitted, empty, null, explicit, tag-based, digest-pinned, and invalid policies.

Priority: ➖ Normal

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Bug fix · Severity of issue fixed: Medium

Merge Risk: ⚪ Minimal · up to a4643

No actionable merge risk remains for the pull-policy change.

Pre-merge checks: Passed 4 · Failed 1

Failed Failed checks: 1 warning

Check name Status Explanation Resolution
Docstring Coverage Warning Docstring coverage is 75.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 8 functions across 1 files. (1 skipped: 1… Write docstrings for the functions missing them to satisfy the coverage threshold.
Passed Passed checks: 4 passed
Check name Status Explanation
Description check Passed The description clearly explains the Helm pull-policy change, its default behavior, documentation updates, testing, and compatibility considerations.
Title check Passed The title clearly identifies the main change: applying manager.image.pullPolicy to the controller container.
Linked Issues check Passed Issue #511 requirements are met. values.yaml uses an empty default. deployment.yaml omits imagePullPolicy for an unset value and renders supported explicit values, including Always. Helm tests…
Out of Scope Changes check Passed The changes stay within issue #511. Template validation, render-test helpers, Helm cases, and deployment and setup documentation support the pull-policy behavior. No unrelated behavior change is shown…
Full details: Linked Issues check

Explanation

Issue #511 requirements are met. values.yaml uses an empty default. deployment.yaml omits imagePullPolicy for an unset value and renders supported explicit values, including Always. Helm tests cover omission, explicit policies, digest-pinned images, and invalid values. The deployment documentation describes the setting and the tag-based default behavior.

Full details: Out of Scope Changes check

Explanation

The changes stay within issue #511. Template validation, render-test helpers, Helm cases, and deployment and setup documentation support the pull-policy behavior. No unrelated behavior change is shown.

Full details: Docstring Coverage

Explanation

Docstring coverage is 75.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 8 functions across 1 files. (1 skipped: 1 unsupported.)

Resolution

Write docstrings for the functions missing them to satisfy the coverage threshold.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

@kaynetu kaynetu left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This fixes #511 as proposed. The value now defaults to "", imagePullPolicy is rendered only when set, and the docs table and air-gap step are updated. With the value unset, the chart renders the same as main, which the air-gap guidance depends on. Defaulting to omission, with an opt-in override, is the right design: wiring up the old IfNotPresent default would have silently changed every :latest install. The test helper checks that the field is actually absent rather than empty, which is the right thing to check here.

A few follow-ups, most important first:

  1. Upgrade note for --reuse-values. You noted this in the description, but it should also be in the deployment guide. Upgrading with --reuse-values from an older release carries the old IfNotPresent default forward, and it now renders. Only a :latest or untagged controller image behaves differently. --set manager.image.pullPolicy= once, or --reset-then-reuse-values (Helm 3.14+), avoids it. The paragraph near deployment.md:311 already covers both flags for rbac.*, so the note could go there.

  2. setup init keeps pullPolicy but works out the image afresh each run (ADR-098). Before this change a kept value did nothing. Now a kept Never applies to whatever image the next run picks. If that image isn't preloaded, the rollout stalls on ErrImageNeverPull. A sentence in docs/cli-reference/setup.md would cover it.

  3. Validate the value. --set manager.image.pullPolicy=always passes the chart and only fails when the API server rejects the Deployment. setup init also keeps the bad value, so every later run fails the same way. The chart already uses fail for a bad rbac.scope; a check against Always/IfNotPresent/Never would match that.

  4. Air-gap step wording. "override it with manager.image.pullPolicy when needed" could name the case it's for: IfNotPresent or Never for a preloaded controller image.

  5. Tests (nits):

    • managerImagePullPolicy repeats the decode-and-find loop from managerImage and managerArgs. One helper that returns the unstructured manager container would serve all three and still tell a missing field from an empty one.
    • The "latest tag leaves pull policy to Kubernetes" case only checks that the field is omitted, which the default case already covers.
  6. Rebase. The branch predates #528, so on GitHub the diff appears to remove the "MPI workers stuck installing sshd" troubleshooting section and revert the FAQ. It merges cleanly, but a rebase would make the diff readable.

@kang0225

Copy link
Copy Markdown
Contributor Author

Thanks for the detailed review. I'll address the follow-ups and rebase onto the latest main.

Default the value to empty and render imagePullPolicy only when set,
so the chart value takes effect and an unset value keeps the API
server's tag-based default. Document the value and add test/helm cases.

Closes NVIDIA#511

Signed-off-by: kang0225 <kjskkdkmj@gmail.com>
Reject unsupported pull policies during chart rendering so configuration
errors surface before deployment. Keep imagePullPolicy omitted for
empty or null values.

Document upgrade behavior and the preloading requirement for a retained
Never policy. Share manager container lookup across render test helpers
and remove the redundant latest-tag omission case.

Signed-off-by: Kang-minjong <kjskkdkmj@gmail.com>
@kang0225
kang0225 force-pushed the fix/511-manager-image-pull-policy branch from 2668e04 to ef2eb2d Compare October 21, 2026 09:37
@kang0225

Copy link
Copy Markdown
Contributor Author

I've pushed the fixes and rebased onto main. I also reran the Helm render tests and helm lint locally, and both passed. Thanks for the review!

kaynetu
kaynetu previously approved these changes Oct 21, 2026

@kaynetu kaynetu left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the quick turnaround. All six follow-ups are addressed:

  1. The --reuse-values note is now under the upgrade step in deployment.md and covers both --set-string manager.image.pullPolicy= and --reset-then-reuse-values.
  2. setup.md now says to preload each new controller image when Never is kept.
  3. The chart rejects any value other than empty, Always, IfNotPresent or Never, and names the bad value in the error.
  4. The air-gap step names the preloaded-image case.
  5. managerContainerFromRendered is now shared by the args, image and pull-policy helpers, and the redundant latest case is gone.
  6. After the rebase, the diff shows only the intended files.

I merged this onto current main and ran test/helm, which passes. With the value unset, helm template output is byte-identical to main. A null value, from a values file or from --set, omits the field, and false, " Always" and [] are all rejected with the new message. setup init reads helm get values without --all, so it keeps only values someone actually set and can't bring back the old default; the new note correctly scopes that to helm upgrade --reuse-values.

Two optional nits, neither blocking:

  • The upgrade note says the old default "is now rendered". Once this ships, "now" will be ambiguous. Naming the release ("from vX.Y on…") would read better later. The note's manual line breaks also don't match the wrapping of the surrounding text.
  • The two commits use different author names (kang0225 and Kang-minjong) with the same email. Each sign-off matches its own commit, so DCO is fine, but squashing on merge would tidy it.

@kaynetu

kaynetu commented Oct 21, 2026

Copy link
Copy Markdown
Collaborator

/ok to test eabab67

@github-actions

Copy link
Copy Markdown
Contributor

ndipebot
ndipebot previously approved these changes Oct 22, 2026

@ndipebot ndipebot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The fix is right and the shape of it is right. imagePullPolicy only renders when the value is set, the default moves to "" so an unset release renders byte-identical to before, and the fail guard sits before the render guard so false and 0 get rejected instead of quietly falling through the truthiness check. The upgrade note about --reuse-values carrying the old IfNotPresent default is the part I would have forgotten, and it is already there.

Two small things, both optional.

  1. The manager.image.pullPolicy row in the chart values table explains the empty case but does not name Always, IfNotPresent or Never. The rbac.scope row a few lines down spells its allowed set out inline, so naming the values here would match what the table already does. I would also add a sentence to the upgrade note at deployment.md:469 saying a kept value outside the enum now fails the render, since that is the one way an existing release can break on upgrade, and point at --set-string manager.image.pullPolicy= (which clears it) or nvcrectl setup init --reset-values.

  2. TestHelmTemplateRejectsInvalidImagePullPolicy pins the fail text with a Go table and a fmt.Sprintf of the expected string, while the chart's only other fail site is pinned with a testutil.TestCaseParser negative golden at test/helm/testdata/render-rbac/invalid-scope/error. The two guards in templates/ are symmetric now but their tests are not, so whoever adds a third has two precedents to pick from. Worth moving the four rejected values to render-pull-policy/<case>/{input.yaml,error}? The positive cases can stay a table.

Comment thread docs/operations/deployment.md Outdated
Comment thread test/helm/render_test.go
List Always, IfNotPresent and Never in the chart values table, since the
chart now rejects anything else. In the --reuse-values upgrade note, name
v0.8.2 instead of "now", say how to clear a kept value the chart rejects,
and unwrap the paragraph to match the surrounding list items.

Signed-off-by: Kang-minjong <kjskkdkmj@gmail.com>
@kang0225
kang0225 dismissed stale reviews from ndipebot and kaynetu via a4643f2 October 22, 2026 15:28
@kang0225

Copy link
Copy Markdown
Contributor Author

Thanks! Updated the docs in a4643f2. Could you take another look and run /ok to test again?

@kang0225
kang0225 requested review from kaynetu and ndipebot October 22, 2026 15:58

@kaynetu kaynetu left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. The values table now names the accepted policies, and the upgrade note is anchored to v0.8.2 and says how to clear a kept value the chart now rejects. If another release is cut before this merges, bump the version in that note.

@NVIDIA NVIDIA deleted a comment from copy-pr-bot Bot Oct 22, 2026
@kaynetu

kaynetu commented Oct 22, 2026

Copy link
Copy Markdown
Collaborator

/ok to test aa97c84

@kaynetu

kaynetu commented Oct 22, 2026

Copy link
Copy Markdown
Collaborator

/ok to test 57277da

@ndipebot ndipebot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still good at this head. The only thing that landed since I last looked is the README line from #550 coming in with main, so nothing in the chart or the tests moved.

@ndipebot

Copy link
Copy Markdown
Collaborator

/ok-to-test ef9117b

@ndipebot
ndipebot merged commit b7531f2 into NVIDIA:main Oct 23, 2026
20 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: Chart value manager.image.pullPolicy is ignored; the Deployment never sets imagePullPolicy

3 participants