Skip to content

Fix Windows rewatch CI flakes from files opened without delete sharing - #9034

Merged
cknitt merged 4 commits into
masterfrom
windows-share-delete
Oct 25, 2026
Merged

cknitt merged 4 commits into
masterfrom
windows-share-delete

Conversation

@cknitt

@cknitt cknitt commented Oct 25, 2026 •

Copy link
Copy Markdown
Member

Windows-only CI failures (~18 jobs, 2026-10-20 to 10-24) were all in the OCaml rewatch test steps:

  • Canonical suite (5×): rm: cannot remove 'lib/watch.lock': Device or resource busy, then "Watcher process N did not exit". The watcher re-reads the lock every 100 ms through Unix.openfile, which on Windows doesn't share delete unless O_SHARE_DELETE is passed. An rm that lands during a read fails, and the watcher is left orphaned. Rust's File::open shares delete by default, so this started with the OCaml port.
  • Parity (1×): mv of a watched directory failed with Permission denied. Same mechanism: the watcher hashes a file inside it.
  • Focused, retained-parse (6×): after B is edited, the log never shows A being parsed again within 20 s. The log dump from change 3 caught this on this PR's first Windows run: A was parsed, but its line was lost from REWATCH_COMPILER_CALL_LOG. Parallel compiler requests each reopen that log in append mode, and on Windows appending is seek-then-write, so one line can overwrite another. The initial build's compile line for A was missing from the same log. So the duplicate-event explanation in Fix watch tests that flake on duplicate change events #8902 was wrong.
  • Parity (4×): "The hosted runner lost communication with the server", about 45 min into the step despite timeout-minutes: 20, and no log survived.

Changes:

  1. File_util.open_for_reading passes O_SHARE_DELETE (a no-op on Unix). All read-only opens and the lock publisher use it.
  2. exit_watcher in rewatch/tests/utils.sh retries the rm and kills a watcher that still doesn't exit, so it can't run into later tests.
  3. The retained-parse test prints watch.log and calls.log when its wait fails.
  4. A watchdog in the parity step (Windows only) reports the largest processes with their command lines and stops rescript/bsc processes when one exceeds 3 GB, more than 64 are running, or free memory falls below 1.5 GB. The step then fails with that report instead of the runner dying silently.
  5. Writes to the compiler call log go through a mutex.

New tests: a unit test in file_util_tests.ml renames and deletes a file held open by open_for_reading. It only proves anything on the Windows unit-test step. I checked the watchdog wrapper locally with a stub powershell.exe. The PowerShell script itself first runs on Windows CI.

🤖 Generated with Claude Code

https://claude.ai/code/session_01GtJW7McE7BZqNZGt5VULiK

cknitt and others added 3 commits October 25, 2026 05:42
Unix.openfile opens files on Windows without FILE_SHARE_DELETE unless
asked, so while the watcher read lib/watch.lock (every 100 ms) or hashed
a source, other processes could not delete or rename it. Removing the
lock then failed with "Device or resource busy" and the watcher kept
running; Rust's File::open shares delete by default.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GtJW7McE7BZqNZGt5VULiK
Signed-off-by: Christoph Knittel <ck@cca.io>
A single failed rm left the watcher running into later tests, where the
runner later killed it as an orphan. Retry for five seconds, and kill the
watcher when it still does not exit so the failure stays in its test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GtJW7McE7BZqNZGt5VULiK
Signed-off-by: Christoph Knittel <ck@cca.io>
The retained-parse test now prints watch.log and calls.log when the edit
of B does not re-parse A. A watchdog in the parity step reports and stops
runaway rescript/bsc processes: four Windows runners lost contact with
GitHub during that step, past its timeout, without any log.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GtJW7McE7BZqNZGt5VULiK
Signed-off-by: Christoph Knittel <ck@cca.io>
@github-actions

github-actions Bot commented Oct 25, 2026 •

Copy link
Copy Markdown

Changelog

🐛 Bug fix

  • On Windows, let other programs delete or rename files while the build system reads them, so stopping a watcher by removing lib/watch.lock no longer fails intermittently.

Semantic Diff

4 commits · Full page

Open files with delete sharing on Windows

314e53f · cknitt + Claude · Oct 25, 2026, 05:42 UTC

Unix.openfile opens files on Windows without FILE_SHARE_DELETE unless asked, so while the watcher read lib/watch.lock (every 100 ms) or hashed a source, other processes could not delete or rename it. Removing the lock then failed with "Device or resource busy" and the watcher kept running; Rust's File::open shares delete by default.

Retry removing watch.lock and stop a watcher that ignores it

a1da0c0 · cknitt + Claude · Oct 25, 2026, 05:43 UTC

A single failed rm left the watcher running into later tests, where the runner later killed it as an orphan. Retry for five seconds, and kill the watcher when it still does not exit so the failure stays in its test.

Diagnose Windows rewatch CI failures that leave no evidence

94ef5ea · cknitt + Claude · Oct 25, 2026, 05:43 UTC

The retained-parse test now prints watch.log and calls.log when the edit of B does not re-parse A. A watchdog in the parity step reports and stops runaway rescript/bsc processes: four Windows runners lost contact with GitHub during that step, past its timeout, without any log.

✨ Serialize writes to the compiler call log

c5694c7 · cknitt + Claude · Oct 25, 2026, 06:18 UTC · new

Parallel domains appended to REWATCH_COMPILER_CALL_LOG through separate handles, and Windows emulates append as seek-then-write, so one domain's line could overwrite another's. The retained-parse test then missed A's parse line (and the initial build's implementation line for A) although A was parsed: the Windows "flake" in that test was a lost log line.

Parallel domains appended to REWATCH_COMPILER_CALL_LOG through separate
handles, and Windows emulates append as seek-then-write, so one domain's
line could overwrite another's. The retained-parse test then missed A's
parse line (and the initial build's implementation line for A) although A
was parsed: the Windows "flake" in that test was a lost log line.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GtJW7McE7BZqNZGt5VULiK
Signed-off-by: Christoph Knittel <ck@cca.io>
@cknitt

cknitt commented Oct 25, 2026

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 25, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-25T06:29:13.016866Z c5694c7 Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c5694c7e44

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +21 to +22
rm -f "$WATCHDOG_RUNNING"
if [ -s "$WATCHDOG_REPORT" ]; then

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Wait for the watchdog before reading and deleting its report

On Windows, if the parity step finishes while the watchdog is sleeping or collecting process details, removing the sentinel does not stop its current iteration: windows_process_watchdog.ps1 checks the sentinel only before its five-second sleep. This cleanup immediately reads the report and deletes its directory without waiting for PowerShell, so an in-flight watchdog action can be missed and the step can return success despite stopping processes. The watchdog can also continue into subsequent CI steps. Save the background PID and wait for it to finish after removing the sentinel, before inspecting the report or deleting its directory. This also satisfies the scoped process-ownership requirement.

AGENTS.md reference: rewatch-ocaml/AGENTS.md:L29-L31

Useful? React with 👍 / 👎.

@pkg-pr-new

pkg-pr-new Bot commented Oct 25, 2026

Copy link
Copy Markdown

Open in StackBlitz

rescript

npm i https://pkg.pr.new/rescript@9034

@rescript/belt

npm i https://pkg.pr.new/@rescript/belt@9034

@rescript/darwin-arm64

npm i https://pkg.pr.new/@rescript/darwin-arm64@9034

@rescript/linux-arm64

npm i https://pkg.pr.new/@rescript/linux-arm64@9034

@rescript/linux-x64

npm i https://pkg.pr.new/@rescript/linux-x64@9034

@rescript/runtime

npm i https://pkg.pr.new/@rescript/runtime@9034

@rescript/win32-x64

npm i https://pkg.pr.new/@rescript/win32-x64@9034

commit: c5694c7

@github-actions

Copy link
Copy Markdown

@cknitt
cknitt requested a review from fhammerschmidt October 25, 2026 06:38
@cknitt
cknitt marked this pull request as ready for review October 25, 2026 06:38
@cknitt
cknitt enabled auto-merge (squash) October 25, 2026 06:38
@cknitt
cknitt merged commit 5689a30 into master Oct 25, 2026
26 checks passed
@cknitt
cknitt deleted the windows-share-delete branch October 25, 2026 06:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants