Skip to content

fix(sc): regenerate buffered prompts on resume - #4067

Draft
mehraakash wants to merge 1 commit into
NVIDIA-NeMo:mainfrom
mehraakash:akamehra/sc-replay-free-resume
Draft

fix(sc): regenerate buffered prompts on resume#4067
mehraakash wants to merge 1 commit into
NVIDIA-NeMo:mainfrom
mehraakash:akamehra/sc-replay-free-resume

Conversation

@mehraakash

Copy link
Copy Markdown
Contributor

Summary

  • honor checkpointing.load_replay_buffer=false in SingleController native TQ restores
  • validate the native replay snapshot before removing canonical and staging rows
  • reconstruct the exact buffered prompts from stable prompt_idx tags and queue full fresh generations on the restored policy
  • keep this change separate from rollout telemetry and Slurm/Pyxis changes

Validation

  • git diff --check
  • Python compile check for changed implementation and test files
  • Added a focused replay-free restore test. Local pytest collection is currently blocked because this checkout requires Python 3.13.14 and the available environment is Python 3.13.5 without the complete project dependencies; CI should run the full supported environment.

Context

This is the current-main forward port of the replay-free resume behavior used by the SWE-E2E convergence work. Current main uses native TQ snapshots plus rollout-recovery ledgers, so this intentionally replaces the historical pending_rollouts.pt implementation instead of mechanically rebasing it.

Signed-off-by: Akash Mehra <akamehra@nvidia.com>
@copy-pr-bot

copy-pr-bot Bot commented Sep 9, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@mehraakash

Copy link
Copy Markdown
Contributor Author

/ok to test 9f1bc3a

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant