vm: inotify: sync file events in batches

[fredericmorin-flare]

Related: #1543, #1643.

Problem

With --mount-inotify, every host file event is synced by running two limactl shell commands in sequence: a stat, then the sync. Each command takes about 100ms, and the watcher waits while they run. Events are handled at roughly 4 per second. Changing a few hundred files at once, for example switching branches or formatting a project, takes close to a minute to reach the containers. Watchers such as pants --loop or pantsd keep running on stale files until then. The rate limit also drops events above 50 unique files per 500ms.

Making this faster exposes a feedback loop. On macOS, a chmod run in the VM on a file that was previously opened for writing from the VM emits a write event on the host. Since #1643, every sync does both (chmod, then : >>). So syncing a file emits an event for that same file, which is synced again, endlessly. The rate limit only slowed this loop down. It shows in the daemon log as the same files being synced every second or so, long after they were last changed.

Change

Batching (sync file events in batches):

• Events are collected for 100ms, deduplicated by path, and synced with a single command. The command is split when its arguments would exceed 64KiB, well below Linux's 128KiB limit for a single argument (the command reaches the VM's shell as one string).
• The existence check moves into the sync script ([ -e "$f" ]), which removes the separate stat command.
• The sync runs in the background, so events keep being received while it runs. Events received meanwhile are synced right after it returns.
• The rate limit is removed, since it dropped events. Batching bounds the number of commands run in the VM.

Loop fix (skip events for files unchanged since their last sync):

• When a file is synced, its state on the host (inode, size, mode, modification time) is remembered for a minute. Events for a file whose state hasn't changed since are skipped: either the sync caused them, or the VM has already been notified of that state. The syncs only change the file's ctime, so real edits are never skipped (their mtime changes, or their inode for atomic saves).

Event buffer (buffer the file events channel):

• notify drops events when the channel it sends to is full, and that channel had a buffer of one. Most events of a burst were lost before reaching the handler. The buffer is now 1024.

Large bursts (keep up with large bursts of file events):

• The watcher only forwards the paths of the events, so it keeps up with notify. The checks on the host (stat, directory, unchanged since the last sync) move to the sync, once per path and batch.
• chmod runs once per batch and file mode instead of once per file. Starting a process for each file is what took most of the time in the VM (about 680µs per file, down to 260µs).
• Up to 4 sync commands run at the same time (about 105µs per file).

Results

macOS 26 (Apple M3 Pro), vz + virtiofs, docker runtime. The test rewrites N existing files from the host in one burst, then measures how long until every file's IN_CLOSE_WRITE is seen in a container watching the directory:

	200 files	2000 files	27,469 files (a project checkout, 4,855 directories)
master (49dfe87)	3/200 delivered; the rest were dropped, nothing more within 3 min	5/2000	not measured
this PR	200/200 after 0.3s	2000/2000 after 0.7s	27,469/27,469 after 3.0s; another run: 27,375/27,469 after 3.4s

The time is measured from the end of the burst on the host. Writing the 27,469 files took 7–9s.

Before the loop fix, one host write to a file made the daemon sync it ~25 times over 3 seconds, and it kept going. With the fix, it syncs once; the echo event is received and skipped.

Known limitation

In bursts of tens of thousands of files, macOS can still drop events before notify sees them. FSEvents then reports UserDropped and MustScanSubDirs on the root of the watched directory, flags that notify discards unless a watcher subscribes to them. This is how the second 27k run lost 94 files. Fixing it needs changes in how events are received from FSEvents, which I'd like to address in a follow-up.

Tests

• events_test.go: generated commands (one per file mode, split when too large), batching (deduplication, events received during a sync form the next batch), and the syncer (directories and missing files skipped, unchanged files skipped, changed files synced again).
• events_linux_test.go (Linux only, runs in CI): runs one batch command over several files, one of them missing, and checks that each existing file gets IN_ATTRIB and IN_CLOSE_WRITE, no IN_MODIFY, and keeps its content and modification time.

All tests pass on macOS and on Linux, with -race, and golangci-lint reports no issues.

LLM usage disclosure

This change was written with the help of an LLM (Claude): the investigation, the fix and the tests. I reviewed every line and ran the tests and measurements above myself.

🤖 Generated with Claude Code