Skip to content

fix(benchmark): replace a kept cell holding read-only git files on retry - #24

Merged
ryanduguid merged 1 commit into
mainfrom
ryan/harness-retry-cleanup
Oct 31, 2026
Merged

ryanduguid merged 1 commit into
mainfrom
ryan/harness-retry-cleanup

Conversation

@ryanduguid

Copy link
Copy Markdown
Collaborator

Summary

Fixes a Windows failure in the benchmark's retry: a retried cell raised Destination path ... already exists, skipped the retry and stranded its session in the scratch directory, which the run deletes at the end. Only benchmarks/agentic/run.py changes. No rule text, scoring or retry policy changes, and no paid session has run.

Cause: _one removed the kept workspace of the failed attempt with shutil.rmtree(..., ignore_errors=True). That workspace holds a git repository from _git_snapshot, whose objects are read-only, and Windows will not delete a read-only file. The delete failed silently, so shutil.move nested the new workspace inside the old directory.

_keep_cell now removes the old directory, clearing the read-only flag when needed, raises if anything remains, then moves. Seen in the first live pilot attempt on 31 October, where 2 cells failed this way.

Evidence

  • Before: with the old delete-and-move behaviour, the new selftest case keep cell prints XX on Windows (Python 3.15).
    After: it prints ok.
  • run.py --selftest passes at this head, with the new case.
  • node --test tests/*.test.js gives 154 passed, 2 failed, 4 skipped, the same as on main (the 2 are correctness.test.js: email bare unfenced code, csv timeout; cause not established).

Merge danger

Door: two-way. Blast radius: benchmarks

Only the harness's cell storage changes.

Unverified

  • The new case passes on Linux with or without the fix, because a read-only file does not block deletion there. It runs in CI, so it guards the contract but only fails on Windows without the fix.
  • No live cell has run with the fix: the account is at its usage limit until 10pm.

On Windows a retried cell's earlier workspace holds read-only git objects.
rmtree(ignore_errors=True) left it behind, so the move nested the new workspace
and raised 'Destination path already exists', which skipped the retry and
stranded the paid session. Remove read-only files, and raise if a leftover stays.
@sonarqubecloud

Copy link
Copy Markdown

@ryanduguid
ryanduguid merged commit 940859f into main Oct 31, 2026
9 checks passed
@ryanduguid
ryanduguid deleted the ryan/harness-retry-cleanup branch October 31, 2026 04:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant