Skip to content

fix: index the files a directory brings when it is moved into the watched root - #1110

Merged
thekevinscott merged 4 commits into
mainfrom
claude/1096-watch-moved-dir
Sep 16, 2026
Merged

thekevinscott merged 4 commits into
mainfrom
claude/1096-watch-moved-dir

Conversation

@thekevinbot

@thekevinbot thekevinbot commented Sep 9, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes #1096

Problem

A directory moved into the watched root (the atomic-publish pattern: stage elsewhere, mv into place) never produced rows. The files inside only showed up after a cold rescan, or after each one was individually touched.

Pre-fix reproduction (/tmp/dirsql-1096-repro.sh, worktree binary at main, 3/3 runs):

1. initial scan            -> [{"n":1}]   (want n=1)
2. control: mkdir + write  -> [{"n":2}]   (want n=2)
3. mv a dir into the root  -> [{"n":2}]   (want n=3)
4. ...then touch the file  -> [{"n":3}]   (want n=3)
5. cold rescan             -> [{"n":3}]   (want n=3)

One of the three runs also showed step 2 at n=1: the mkdir-then-write control raced, with the file landing before notify had a watch on the new directory. Same root-cause class.

Root cause

inotify reports a directory rename-in as a single Modify(Name(To)) event naming the directory and emits nothing for the files already inside it. translate_event mapped that to Modified(<dir>); process_file_event then ran match_all on the directory path, which no file glob matches, and returned early. notify does install the recursive watch on the new directory (which is why a later touch was seen), but nothing ever backfilled the existing contents.

Fix

  • watcher.rs: Modify(Name(To | Any | Other)) maps to FileEvent::Created; Modify(Name(Both)) (inotify's cookie-paired summary after a From/To) is dropped so a rename is not indexed twice. From still maps to Deleted.
  • lib.rs: a Created event whose path is a directory walks it (index_subtree) and upserts every matched file beneath, relative to the root the watch uses. A Modified directory is not walked (a chmod on a tree should not re-run every on-file). The walk and is_dir sit behind the FileSystem trait so the unit tests stay isolated (FakeFs).
  • scanner.rs: walk takes a separate start directory; scan_subtree(root, dir, matcher) reuses the startup scan so watch and startup share one definition of "which files are rows". Depth for the reserved .dirsql check and gitignore frames is computed from the root-relative path rather than WalkDir's depth.

This also covers a fresh mkdir followed by an immediate write: the directory's Create event triggers the same walk, closing the race above.

Red/green: 286a9d0 pushed the tests alone; rust CI run 34391474344 "Integration tests" (job 102600547465) failed on exactly server_indexes_a_directory_moved_into_the_root with a directory moved into the root must index the files it brought; rows: ["Hello-World.json", "Second-Post.json"] (45 passed; 1 failed). watch_scan_correctness red confirmed locally (cargo fail-fast stopped CI at the first binary).

Manual run

Fixed binary, same script:

1. initial scan            -> [{"n":1}]   (want n=1)
2. control: mkdir + write  -> [{"n":2}]   (want n=2)
3. mv a dir into the root  -> [{"n":3}]   (want n=3)
4. ...then touch the file  -> [{"n":3}]   (want n=3)
5. cold rescan             -> [{"n":3}]   (want n=3)

Nested move (dirsql server, glob posts/**/*.json, mv stage/carol root/posts/carol with first.json and drafts/second.json inside):

before mv : [{"path":"posts/alice/hello.json"}]
after  mv : [{"path":"posts/alice/hello.json"},{"path":"posts/carol/drafts/second.json"},{"path":"posts/carol/first.json"}]

The mirror case (directory moved out of the root leaves its rows behind) is unchanged and tracked as #1109.

E2E Verification

  • Ran e2e suites locally for every affected SDK
  • Python SDK e2e: pass
  • TypeScript SDK e2e: pass
  • Rust core e2e (if applicable): pass
  • packages/python/e2e-attestations/<branch>.json receipt written if packages/python changed (just e2e-attest-python)
  • packages/ts/e2e-attestations/<branch>.json receipt written if packages/ts changed (just e2e-attest-ts)
  • Command(s) run: cargo test --workspace --features cli; just preflight; just e2e-attest-python (runs just test-e2e); just e2e-attest-ts (runs pnpm test:e2e); npx -y testing-conventions unit coverage --language rust packages/rust/src
  • Result summary: Rust cargo test --workspace --features cli exit 0 (full suite incl. the 3 new integration/e2e tests); just preflight 0 failing pairs; Python e2e 37 passed, TS e2e 16 passed (receipts committed in 753d068 / dc9e3b8); rust unit coverage lines 94.02 / functions 92.56 / regions 93.31 / branch 79.74 against floors 92 / 91 / 91 / 72.

Changelog / Migrations

  • Changelog fragment added under <root>/<pkg>/changelog.d/ for each changed package (or: skip-changelog trailer on a commit with reason) — packages/rust/changelog.d/2026-09-09-watch-moved-directory.md
  • Migration fragment added under <root>/<pkg>/migrations.d/ (or: not required -- additive/bugfix only) — not required. Considered under "Behavior changes without code changes": the watch now emits the Insert events a restart would already have produced for the same files, so no consumer could have depended on their absence without also breaking on every restart. Bugfix only.

Parity

Parity maintained, no drift. The fix is entirely in the shared Rust core; the FileSystem trait is crate-private and no SDK surface changed. PARITY.md notes it.

Integration (watch_scan_correctness) and CLI e2e (cli_e2e) tests that stage
a populated directory outside the root, rename it in, and wait for its files
to become rows. They fail today: inotify reports the move as one event naming
the directory, and the watch path never walks beneath it.
inotify reports a directory rename-in as one Modify(Name(To)) event naming
the directory and nothing for the files already inside it. translate_event
turned that into Modified(dir); process_file_event ran match_all on the
directory path, which no file glob matches, and returned. notify did install
the recursive watch, so later writes were seen, but nothing backfilled the
contents until a cold rescan.

- watcher: Name(To | Any | Other) -> Created; Name(Both), inotify's paired
  summary after a From/To, is dropped so a rename is not indexed twice.
- lib: a Created path that is a directory is walked with the startup scan
  (scanner::scan_subtree) and every matched file beneath it is upserted.
  A Modified directory is not walked. is_dir and the walk sit behind the
  FileSystem trait so unit tests stay isolated.
- scanner: walk takes a separate start directory; depth for the reserved
  .dirsql check and gitignore frames comes from the root-relative path.

The same walk closes the mkdir-then-immediate-write race, where a file
landed before notify was watching the new directory.
@thekevinbot
thekevinbot marked this pull request as ready for review September 9, 2026 19:09
@thekevinscott
thekevinscott merged commit 1b53954 into main Sep 16, 2026
77 checks passed
@thekevinscott
thekevinscott deleted the claude/1096-watch-moved-dir branch September 16, 2026 21:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Watch: a directory moved into the root is never indexed

2 participants