Skip to content

Python: fix: unique temporary filenames for concurrent FileCheckpointStorage saves (#8182) - #8220

Closed
Atik (ptimizeroracle) wants to merge 2 commits into
microsoft:mainfrom
ptimizeroracle:fix/concurrent-checkpoint-save-8182
Closed

Python: fix: unique temporary filenames for concurrent FileCheckpointStorage saves (#8182)#8220
Atik (ptimizeroracle) wants to merge 2 commits into
microsoft:mainfrom
ptimizeroracle:fix/concurrent-checkpoint-save-8182

Conversation

@ptimizeroracle

@ptimizeroracle Atik (ptimizeroracle) commented Sep 10, 2026

Copy link
Copy Markdown

description

fixes #8182

every writer used the same temp name, {id}.json.tmp. so when two saves of one checkpoint id run at the same time, one os.replace eats the temp file the other writer is about to replace, and that save dies with a raw FileNotFoundError

now each writer gets its own temp name, {id}.json.{pid}.{uuid}.tmp. still ends in .tmp so the *.json globs in list_checkpoints and list_checkpoint_ids never pick these up

validation

before: 20 concurrent saves of one id, several of them crash in os.replace
after: same probe, 0 errors

added test_file_checkpoint_storage_concurrent_saves_same_id (10 concurrent saves, no exceptions, final state loads)

test_checkpoint.py 47/47 local, ruff clean

i only changed the temp name. no exception type changes, that question stays in the issue thread

ai assistance: i led this contribution and used a coding agent to help write the change and its test. validation was run locally. happy to use a different suffix format if you have one

…saves (microsoft#8182)

Signed-off-by: ptimizeroracle <contact@binblok.com>
Copilot AI balanced review requested due to automatic review settings September 10, 2026 09:01
@agent-framework-automation agent-framework-automation Bot added the python Usage: [Issues, PRs], Target: Python label Sep 10, 2026
@github-actions github-actions Bot changed the title fix: unique temporary filenames for concurrent FileCheckpointStorage saves (#8182) Python: fix: unique temporary filenames for concurrent FileCheckpointStorage saves (#8182) Sep 10, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Failed writes can leave an unbounded number of uniquely named temporary files.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Prevents concurrent checkpoint saves from racing on a shared temporary file.

Changes:

  • Generates a unique temporary filename per save.
  • Adds a concurrent same-ID save regression test.
File summaries
File Description
_checkpoint.py Uses PID/UUID-based temporary filenames.
test_checkpoint.py Tests concurrent saves and final-state loading.
Review details
  • Files reviewed: 2/2 changed files
  • Comments generated: 1
  • Review effort level: Balanced

💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.

Comment on lines 335 to 338
tmp_path = file_path.with_suffix(f".json.{os.getpid()}.{uuid.uuid4().hex}.tmp")
with open(tmp_path, "w") as f:
json.dump(encoded_checkpoint, f, indent=2, ensure_ascii=False)
os.replace(tmp_path, file_path)
@ptimizeroracle

Copy link
Copy Markdown
Author

@microsoft-github-policy-service agree

Signed-off-by: ptimizeroracle <contact@binblok.com>
@ptimizeroracle

Copy link
Copy Markdown
Author

addressed in 6b7311d: failed writes now unlink their own temp file before re-raising, so failures don't accumulate. added a test that simulates a replace failure and asserts no .tmp leftovers. 48/48 local.

@ptimizeroracle

Copy link
Copy Markdown
Author

closing this duplicate in favor of #7757, which already covers the same concurrent-save temp-file race and is under active review

the reproduction in #8182 and the regression tests here remain available as additional evidence

thanks for the ongoing work on the fix

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

python Usage: [Issues, PRs], Target: Python

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Python: FileCheckpointStorage.save crashes on concurrent saves of the same checkpoint id: fixed .tmp filename race

2 participants