Repository navigation
fix: index the files a directory brings when it is moved into the watched root - #1110
Merged
Merged
Conversation
Integration (watch_scan_correctness) and CLI e2e (cli_e2e) tests that stage a populated directory outside the root, rename it in, and wait for its files to become rows. They fail today: inotify reports the move as one event naming the directory, and the watch path never walks beneath it.
inotify reports a directory rename-in as one Modify(Name(To)) event naming the directory and nothing for the files already inside it. translate_event turned that into Modified(dir); process_file_event ran match_all on the directory path, which no file glob matches, and returned. notify did install the recursive watch, so later writes were seen, but nothing backfilled the contents until a cold rescan. - watcher: Name(To | Any | Other) -> Created; Name(Both), inotify's paired summary after a From/To, is dropped so a rename is not indexed twice. - lib: a Created path that is a directory is walked with the startup scan (scanner::scan_subtree) and every matched file beneath it is upserted. A Modified directory is not walked. is_dir and the walk sit behind the FileSystem trait so unit tests stay isolated. - scanner: walk takes a separate start directory; depth for the reserved .dirsql check and gitignore frames comes from the root-relative path. The same walk closes the mkdir-then-immediate-write race, where a file landed before notify was watching the new directory.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1096
Problem
A directory moved into the watched root (the atomic-publish pattern: stage elsewhere,
mvinto place) never produced rows. The files inside only showed up after a cold rescan, or after each one was individually touched.Pre-fix reproduction (
/tmp/dirsql-1096-repro.sh, worktree binary atmain, 3/3 runs):One of the three runs also showed step 2 at
n=1: themkdir-then-write control raced, with the file landing beforenotifyhad a watch on the new directory. Same root-cause class.Root cause
inotify reports a directory rename-in as a single
Modify(Name(To))event naming the directory and emits nothing for the files already inside it.translate_eventmapped that toModified(<dir>);process_file_eventthen ranmatch_allon the directory path, which no file glob matches, and returned early.notifydoes install the recursive watch on the new directory (which is why a latertouchwas seen), but nothing ever backfilled the existing contents.Fix
watcher.rs:Modify(Name(To | Any | Other))maps toFileEvent::Created;Modify(Name(Both))(inotify's cookie-paired summary after a From/To) is dropped so a rename is not indexed twice.Fromstill maps toDeleted.lib.rs: aCreatedevent whose path is a directory walks it (index_subtree) and upserts every matched file beneath, relative to the root the watch uses. AModifieddirectory is not walked (achmodon a tree should not re-run everyon-file). The walk andis_dirsit behind theFileSystemtrait so the unit tests stay isolated (FakeFs).scanner.rs:walktakes a separate start directory;scan_subtree(root, dir, matcher)reuses the startup scan so watch and startup share one definition of "which files are rows". Depth for the reserved.dirsqlcheck and gitignore frames is computed from the root-relative path rather thanWalkDir's depth.This also covers a fresh
mkdirfollowed by an immediate write: the directory'sCreateevent triggers the same walk, closing the race above.Red/green: 286a9d0 pushed the tests alone; rust CI run 34391474344 "Integration tests" (job 102600547465) failed on exactly
server_indexes_a_directory_moved_into_the_rootwitha directory moved into the root must index the files it brought; rows: ["Hello-World.json", "Second-Post.json"](45 passed; 1 failed).watch_scan_correctnessred confirmed locally (cargo fail-fast stopped CI at the first binary).Manual run
Fixed binary, same script:
Nested move (
dirsql server, globposts/**/*.json,mv stage/carol root/posts/carolwithfirst.jsonanddrafts/second.jsoninside):The mirror case (directory moved out of the root leaves its rows behind) is unchanged and tracked as #1109.
E2E Verification
packages/python/e2e-attestations/<branch>.jsonreceipt written ifpackages/pythonchanged (just e2e-attest-python)packages/ts/e2e-attestations/<branch>.jsonreceipt written ifpackages/tschanged (just e2e-attest-ts)cargo test --workspace --features cli;just preflight;just e2e-attest-python(runsjust test-e2e);just e2e-attest-ts(runspnpm test:e2e);npx -y testing-conventions unit coverage --language rust packages/rust/srccargo test --workspace --features cliexit 0 (full suite incl. the 3 new integration/e2e tests);just preflight0 failing pairs; Python e2e 37 passed, TS e2e 16 passed (receipts committed in 753d068 / dc9e3b8); rust unit coverage lines 94.02 / functions 92.56 / regions 93.31 / branch 79.74 against floors 92 / 91 / 91 / 72.Changelog / Migrations
<root>/<pkg>/changelog.d/for each changed package (or:skip-changelogtrailer on a commit with reason) —packages/rust/changelog.d/2026-09-09-watch-moved-directory.md<root>/<pkg>/migrations.d/(or: not required -- additive/bugfix only) — not required. Considered under "Behavior changes without code changes": the watch now emits theInsertevents a restart would already have produced for the same files, so no consumer could have depended on their absence without also breaking on every restart. Bugfix only.Parity
Parity maintained, no drift. The fix is entirely in the shared Rust core; the
FileSystemtrait is crate-private and no SDK surface changed.PARITY.mdnotes it.