docs(kubernetes-support): name what older clusters silently lose

[ayuskauskas]

Refs #411 (item 8). Does not close it. Part of #223, based on feature/package-as-jobs.

The problem

docs/kubernetes-support.md claimed the operator "relies only on core, long-stable Kubernetes APIs" and "gates on no version-specific features", and marked ~1.23 – 1.32 as expected to work.

That was true before package execution moved to batch/v1 Jobs. It isn't now. The Jobs path depends on podFailurePolicy, the DisruptionTarget pod condition, the FailureTarget Job condition, podReplacementPolicy, the batch.kubernetes.io/* pod labels, suspend, and ttlSecondsAfterFinished.

The failure mode is what makes this worth documenting rather than just fixing a range. An apiserver that doesn't know a field drops it and returns success — it doesn't reject the Job. So on an older cluster the operator creates a healthy-looking Job with the field simply absent, the property it guaranteed is gone, and there is no error, event, or log line. Nothing surfaces until the situation that field existed to handle actually occurs.

What this adds

A cumulative table — each row lists what newly stops working below that version, so reading top-down accumulates:

Going below	What stops working
1.33	— (support floor; where CI stops, not where features stop)
1.29	podReplacementPolicy → two executors can share one host copyDir
1.27	batch.kubernetes.io/* pod labels → pruning, last-logs, container-name lookup all no-op
1.26	podFailurePolicy + DisruptionTarget + FailureTarget → disruptions count as failures
1.23	job tracking with finalizers → attempt accounting unreliable
1.22	ttlSecondsAfterFinished → finished Jobs never GC'd
1.21	spec.suspend → pause can't stop a running stage

The old 1.23–1.32 band splits into 1.29–1.32 (every field at least beta-on-by-default; losses theoretical) and 1.23–1.28 (real degradation).

The row that matters most, and why it got worse

Below 1.26 the Ignore-on-DisruptionTarget rule disappears, so evictions, preemptions and taint-manager kills count toward backoffLimit like genuine failures.

At the old unbounded backoffLimit that was harmless. With #402 making it finite (default 3), a couple of unrelated disruptions can exhaust the budget and park a package that never failed. So this degradation is materially sharper than when #411 was filed, which is worth a reviewer's attention.

Honesty about the numbers

The versions track upstream feature-gate graduation, not measured NodeWright behaviour — none of these clusters are in CI, and I did not stand any up. #411 explicitly said "please confirm the floor before editing the table", so treat these as needing a maintainer's check against upstream before merge. The page now says as much in a note rather than implying a tested promise.

I also removed the same "core APIs only" claim where it was repeated in the Version Selection Guide, so the two sections can't drift apart.

Testing

Docs only. markdownlint-cli2 --config ci/.markdownlint-cli2.yaml — 0 issues.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: 0eacb6a5-6726-47a3-9801-e2152474366d

📥 Commits

Reviewing files that changed from the base of the PR and between 6db86ef and 50f3be6.

📒 Files selected for processing (1)

• docs/kubernetes-support.md

---

📝 Walkthrough

Walkthrough

The Kubernetes support documentation now includes a version-specific support matrix for Job feature degradation below Kubernetes 1.33. It documents omitted fields, disabled feature gates, and effects on replacement safety, pod discovery, failure handling, disruption handling, deadlines, attempt accounting, cleanup, and suspension. It also updates version guidance for Kubernetes 1.33–1.36 and older clusters.

Estimated code review effort: 1 (Trivial) | ~5 minutes

Suggested reviewers: lockwobr, rice-riley

🚥 Pre-merge checks | ✅ 4

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly and concisely identifies the documentation change about behavior that older Kubernetes clusters silently lose.
Description check	✅ Passed	The description directly explains the Kubernetes compatibility documentation changes, their rationale, version thresholds, and validation performed.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch jobs-migration/411-k8s-support-matrix

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 3

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@docs/kubernetes-support.md`:
- Line 33: Update the Kubernetes 1.26 entry to remove the incorrect claim that
JOB_BACKOFF_LIMIT defaults to 3 and causes a finite backoff budget. Document
that JobSpec.BackoffLimit is configured to math.MaxInt32, resulting in
effectively unlimited retries, while preserving the surrounding version-specific
behavior and deadline-log statement.
- Line 31: Update the Kubernetes 1.29 compatibility entry describing
podReplacementPolicy so it no longer guarantees that concurrent executor writes
are only a race without corruption. Either implement synchronization for shared
generation-based copyDir and package writes and add a concurrent replacement
test, or revise the documentation to explicitly state that concurrent
replacements may corrupt data.
- Around line 14-15: Update the Kubernetes compatibility documentation to use
separate version thresholds for podFailurePolicy/DisruptionTarget support and
the deadline FailureTarget signal: retain 1.26 as the beta availability point,
but identify 1.31 as the start of the delayed-terminal FailureTarget behavior.
Revise the support table and the claims near the Kubernetes support overview and
older-cluster limitations so versions 1.29–1.30 are not described as having the
FailureTarget deadline signal.

🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: 8bdcdb6c-7c18-4793-8eb1-14eebbebb0bc

📥 Commits

Reviewing files that changed from the base of the PR and between 15a317f and 1493459.

📒 Files selected for processing (1)

• docs/kubernetes-support.md

[ayuskauskas]

Corrected the 1.27 row in 6db86ef — it overstated the cost, in two directions.

Completion is not affected. It is read from the Job's Complete condition (job_controller.go:164), never from pods. The succeeded-container-name lookup feeds one variable whose only downstream use is containerName == InterruptContainerName (:342) — and interrupt Jobs source that from the Job's own label (:312), not from a pod. For every other stage it is cosmetic, which the comment above the call already said:

For non-interrupt stages the exact name is cosmetic; read it from the succeeded pod when present, else leave it empty.

Listing it as a cost was wrong.

"Failure evidence stops being retained" was backwards. Losing pruneFailedAttempts means archive pods accumulate instead of being trimmed to two per stage. That is a scale problem, not an evidence problem.

The row now names the two real costs — unbounded archive accumulation, and the last-logs snapshot never firing for a container that never started — and says explicitly that completion is unaffected.

---

One thing this dug up that is worth a reviewer's eye, and that I am not putting in the doc because it is speculative:

After #402, jobFailureIsGenuine becomes a fourth caller of childPods. On a sub-1.27 cluster it would find no archives and return false for BackoffLimitExceeded, so the Job-side erroring write never happens.

That does not break parking, because the Pod watch is the other writer and it is unaffected — child pods inherit nodewright.nvidia.com/name from the Job's pod template, not from the batch.kubernetes.io/* labels, so ownedPod() still matches and pod evidence still writes erroring. The park predicate is (terminal Failed ∧ entry erroring), so it still holds.

What it does mean is that on those clusters the park loses its backstop and rests entirely on the Pod watch — which is precisely the two-writer invariant documented in #413. Worth knowing, not worth a table row.

[ayuskauskas]

All three addressed in 50f3be6. All three were right.

FailureTarget is a later threshold than podFailurePolicy — correct, and it's worse than framed. podFailurePolicy and DisruptionTarget are beta-on from 1.26, but 1.29–1.30 report a deadline expiry directly as JobFailed; the delayed-terminal behaviour that raises FailureTarget first arrives in 1.31. So everything hanging off that condition breaks five minors earlier than the table claimed — and it isn't only the log snapshot: failureTargetStale, the path that surfaces erroring for a Job wedged on an unreachable node, keys on it too (job_controller.go:509, 761, 773). Now its own 1.31 row, removed from the 1.26 row, with the intro band split to match.

"Race, not corruption" — a guarantee I had no business making. Flag files make re-execution idempotent; they are not a lock and do not order two concurrent cp -r runs into the same copyDir, so a step script can read a file another attempt is mid-way through overwriting. Now described as a possible-corruption configuration. Not taking the heavy-lift half (add synchronization + a concurrent-replacement test) — this is a docs PR, and the configuration is both non-default and below the support floor. Documenting it accurately is the fix; engineering around it would be a separate change with its own justification.

The JOB_BACKOFF_LIMIT default of 3 — correct, and a good catch on branch scope. That knob doesn't exist here; it arrives with #402, and on this branch BackoffLimit is math.MaxInt32. I was describing the post-#402 world on a PR that doesn't contain it. Rephrased to hold in both regimes rather than forward-referencing an unmerged PR.

One thing I fixed that none of the three named: the 1.27 row still listed the last-logs snapshot as a loss, but the table is cumulative and it's already gone at 1.31. De-duplicated.

markdownlint-cli2 0 issues.