fix(episode): stage frames in unique tempdir and publish atomically (#634) - #640
Conversation
…ebbian-Robotics#634) Signed-off-by: shobhitagnihotri69 <shobhitagnihotri69@users.noreply.github.com>
|
| capture_output=True, | ||
| text=True, | ||
| check=False, | ||
| if not output_directory.exists(): |
There was a problem hiding this comment.
Incomplete cache returns missing frames
If a JPEG is missing from an existing cache directory in a reused workdir, this check skips extraction and returns a path to the missing file. A caller that reads the frame then gets FileNotFoundError. The previous check verified every expected JPEG and rebuilt an incomplete cache.
Knowledge Base Used: Episode storage and identity
| with suppress(OSError): | ||
| staging_dir.rename(output_dir) | ||
| finally: | ||
| shutil.rmtree(staging_dir, ignore_errors=True) |
There was a problem hiding this comment.
Rename errors discard extracted frames
If the rename fails for a reason other than another caller publishing first, suppress(OSError) hides the failure and finally deletes the successfully extracted frames. frames() can then return an empty list; the same pattern in frames_at_indices() can return paths to files that do not exist. Check that a complete destination was published before treating a rename error as a cache hit.
| with ThreadPoolExecutor(max_workers=2) as pool: | ||
| futures = [pool.submit(extract) for _ in range(2)] | ||
| results = [future.result() for future in futures] |
There was a problem hiding this comment.
Tests may miss publication race
These workers are not synchronized, so the first extraction may finish before the second checks the cache. In that case the test passes through a cache hit without exercising concurrent staging or publication. The indexed-frame test has the same gap; synchronize both workers after they pass the cache-miss check so the tests reliably cover the race.
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
Summary
Prevents concurrency collisions and orphaned staging directories in
Episode.frames()andEpisode.frames_at_indices()when multiple callers or threads share an explicitworkdir.Why
Fixes #634.
When multiple
Episodecallers share oneworkdir, concurrent extractions previously collided on the fixed.tmpstaging directory name or unlinked each other's in-progress files. Furthermore,frames_at_indicesinvokedshutil.rmtree(output_directory)on cache misses, which could blow away an in-progress or completed cache directory created by a concurrent caller.Following maintainer guidance on #635:
tempfile.mkdtemp(dir=self.workdir, prefix=...)) rather than a fixed.tmpname.rename. If the output directory already exists (another caller finished first), the local staging directory is removed infinallyand the winner's cache is used.shutil.rmtree(output_directory)inframes_at_indices()since with atomic publishing the output directory is either completely published or absent.Validation
Ran linting and type checks: