[cpruijsen]

Description

RunAnywayTarget deletes stale PID directories under its shared temp dir on every construct. If two pipelines do that at once, os.stat or shutil.rmtree can see a directory the other process already removed and raise FileNotFoundError.

This treats that cleanup as best-effort: shutil.rmtree(..., ignore_errors=True), and OSError from the isdir/stat check is ignored, matching done() in the same class which already swallows OSError from concurrent makedirs.

Motivation and Context

Fixes #3233.

The reporter asked for shutil.rmtree(path, ignore_errors=True) on the rmtree line. That is the right fix for the rmtree race. The same loop also calls os.stat after os.path.isdir, which can raise the same FileNotFoundError if the directory disappears in between.

Decision: use the reporter's ignore_errors=True and also wrap the per-path check in except OSError.
Alternative: only add ignore_errors=True.
Why: ignore_errors does not cover the os.stat window; test_cleanup_ignores_concurrent_stat fails without the try/except.
Happy to drop the try/except and ship the one-argument change if that is preferred.

Have you tested this? If so, how?

I have included unit tests.

test_cleanup_ignores_concurrent_rmtree and test_cleanup_ignores_concurrent_stat fail with FileNotFoundError without the change and pass with it. test_cleanup_removes_stale_pid_directory checks that expired directories are still removed.

[cpruijsen]

Closing this one so it does not sit in the queue.