docs(operator): correct where a timed-out stage's logs actually live - #458
Conversation
The Jobs release notes said a stage killed by its own deadline gets a last-logs snapshot. It does not, and manual validation on 2026-08-12 and again on 2026-08-13 found the annotation empty in every package-stage case: a per-attempt deadline fails the pod in place, so the archive holds the logs and snapshotFailureLogs deliberately skips. The annotation is still written for an interrupt Job's whole-stage deadline, where the Job controller deletes the pod, so the note now says that instead of promising it for every timeout. Refs #449, which tracks the fact that the fallback is currently unreachable for package Jobs at all. Signed-off-by: Alex Yuskauskas <ayuskauskas@nvidia.com>
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: ASSERTIVE Plan: Enterprise Run ID: 📒 Files selected for processing (1)
📝 WalkthroughWalkthroughFailure-log handling now retains full logs for stage-timeout failures. When interrupt-job pod deletion occurs, the Job annotation records a best-effort log tail of approximately 16 KiB. The previous stage-timeout annotation snapshot was removed. Estimated code review effort: 1 (Trivial) | ~3 minutes Mergeability Score: ⚪ Minimal · up to This documentation-only change corrects the release-note description of where timed-out stage logs are stored without changing runtime behavior. No actionable merge-blocking risk remains after normal checks and review. Suggested reviewers: 🚥 Pre-merge checks | ✅ 4✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
Important
Base this work on
feature/package-as-jobs, notmain.The Jobs migration (#223) is integrating on that branch.
maindoes not yet executepackages as Jobs, so a PR opened against
mainwill be missing the code this dependson.
feature/package-as-jobsmerges tomainonce #305 lands.That branch squash-merges, so if your branch is stacked on another PR in this epic,
replay only your own commits when restacking:
git rebase --onto origin/feature/package-as-jobs <last-inherited-commit>Part of #223. Refs #449 — this corrects the claim; the underlying gap is tracked there.
The claim that was wrong
The Jobs release notes promised:
It does not. A per-attempt
stageTimeoutfails the pod in place rather than deleting it, so the timed-out attempt survives as an ordinary full-log archive andsnapshotFailureLogsdeliberately skips ("a genuine failed archive already holds full logs"). I wrote that sentence in #402 from the pre-#402 design, where a Job-level deadline could delete the running pod.Manual validation found the annotation empty in every package-stage case, twice:
last-logsstageTimeout: 20s,SLEEP_LEN: 600)93caf2acSIGTERM93caf2acstageTimeout: 60s)93caf2ac3e3886ea3e3886eaWhat it says now
That the archive holds the logs for a timed-out attempt, and that the
last-logsfallback belongs to the interrupt Job's whole-stage deadline — the one case where the pod really is deleted.Deliberately not claiming the fallback works for package stages, because per #449 it currently cannot fire for them at all: the snapshot is gated on
FailureTarget, which a package Job only reaches once the final attempt has already failed, at which point either a genuine archive exists (skip) or no live pod remains to read. Fixing that is #449's job; this PR just stops the release notes promising something users will not find.Docs only — no code, no test changes.