[thekevinbot]

Fixes #1572

Guarantee

"A read must guarantee that it is accurate to the filesystem." A query on a long-lived Rust DirSQL (SDK, server, REPL; Python through the core) now reads what is on disk when it starts. Every freshness test asserts at t+0, the moment the write returns, against a DirSQL built fresh over the same tree. No sleep, retry, poll or timeout precedes an assertion.

Scope after the split: Rust tests + core fix, Linux and Windows. macOS CI is #1574. The Python binding tests ship in the next PR; the TypeScript binding tests and the JS-thread apply step in the one after (until then a TS query reads what the watcher applied, as before).

Mechanism

Before a query on a declared table, under one lock and one transaction:

1. Walk each table anchor with the build's own walker (.gitignore re-read on every walk).
2. Fingerprint each match, following symlinks: device, inode, size, mtime, ctime.
3. Re-read a file if it is new, its fingerprint moved, or its last read was unsettled. Delete the rows of a known file the walk no longer finds. A per-table hook reruns once over its table when any of its files changed or went.
4. Commit, then record the fingerprints. A failed commit returns Err and records nothing.

Read-your-writes without a marker file. A read is settled iff the file's ctime is more than 2s before the read started. utime cannot set ctime, and any write after the read began bumps ctime to at least that time, so a settled fingerprint that is unchanged proves the content is what was read. An unsettled file is re-read on every query until it settles. No content hash: a hash cannot see an A→B→A change made mid-parse. Files the build indexed are adopted without a re-read iff their ctime is more than 2s before the build started.

Windows. ctime is NTFS ChangeTime (GetFileInformationByHandleEx(FileBasicInfo)); device/inode are volume serial and file index. The file is opened with FILE_READ_ATTRIBUTES only. A ChangeTime of 0 counts as never settled.

Events. Row events a query's re-reads produce go to an outbox once a watcher is wanted (start_watching, poll_events, watch). The next watch batch or poll drains the outbox first, under the same lock, so events stay totally ordered. When the watcher later reports the same change, re-applying identical content yields no event, so each change arrives exactly once.

Unchanged surfaces. The CLI one-shot (dirsql query) queries straight after its own build, so fresh reads are off there. The TS binding is off too: its on_file runs only on the JS thread.

Foreign keys. If a file on disk breaks a foreign key, the query fails with the ForeignKeyViolation a build over that tree returns. Serving the pre-violation rows would be a stale read.

Each way a change could be missed, and the test that closes it

Way to miss a change	Test
Create / edit / delete / rename in and out of the glob, queried right after	fresh_declared_hammer.rs every_*_is_read_by_the_next_query (2000 rounds each)
Atomic save (write temp, rename over)	every_atomic_save_…, an_atomic_save_over_the_original_is_read
Same-size edit inside one mtime tick	every_same_size_edit_…, a_same_size_edit_within_one_mtime_tick_is_read
Edit with mtime/atime put back by utime	every_same_size_edit_with_its_times_restored_…, an_edit_whose_times_are_put_back_is_read
Delete and recreate under the same name	a_file_deleted_and_recreated_under_the_same_name_is_read
New / deleted subdirectory	a_new_subdirectory_is_read, a_deleted_subdirectory_is_dropped
.gitignore edited	a_gitignore_change_changes_which_files_are_read
Symlink retargeted, or its target edited	a_retargeted_symlink_is_read, an_edit_to_a_symlink_target_is_read
Write racing a query's read (torn or stale read)	a_file_rewritten_while_queries_run_never_reads_torn_and_ends_exact, a_file_written_in_place_while_queries_run_ends_exact; unit a_settled_file_is_read_once_then_not_again
Write landing during the build's read	unit a_built_file_unchanged_since_before_the_build_is_not_read_again (adopt only if settled); oracle long-lived run
Per-table hook file changed or deleted	every_edit_to_a_table_hook_file_…; unit a_per_table_hook_reruns_when_one_of_its_files_changes_and_when_one_goes
Hook fails on the new content	unit a_file_whose_hook_fails_at_query_time_keeps_no_rows_and_reports_the_error
query_ordered / query_each skipping the check	unit query_ordered_and_query_each_read_fresh_rows_too
Random interleavings, idle gaps, bursts	fresh_declared_oracle.rs (seeds 1–4, 150 ops each; long-lived instance)
Event dropped or duplicated between query and watcher	fresh_declared_events.rs; units a_query_hands_its_row_events_to_the_next_poll_once, a_query_after_start_watching_hands_its_row_events_to_the_consumer, a_change_a_query_read_reaches_the_next_poll_once_when_the_watcher_reports_it_too
FK orphan on disk	a_query_over_an_orphaned_child_fails_as_the_build_would

The four freshness suites ran 20 times locally with no flake.

Residual gaps (stat cannot see these)

• Writes through a shared mmap: the kernel updates ctime lazily.
• NFS with clock skew or attribute caching.
• The system clock stepped backwards across the 2s settle window.
• Windows: ChangeTime is settable via SetFileInformationByHandle.

The shared-mmap, NFS and clock-step limits are shared by every stat-based tool and are documented in docs/reference/sdk.md.

Windows open-handle writes: not a gap

tests/fresh_declared_open_handle.rs tests open-handle writes. Each of its four tests opens 300 files for writing, waits past the 2s settle window, and queries once so every file's read is trusted on its stat alone. Then, with the handles still open, it writes each file and queries at t+0, asserting the new rows. The four tests cover a same-size rewrite and a resize, each with and without sync_all (FlushFileBuffers). That is 1,200 rounds with no sleep before any assert.

• Windows CI passed: log. Those runs are with no flush, a same-size rewrite, and the handle still open. Under them, NTFS ChangeTime still moved before write returned.
• The test is load-bearing. When must_read stops comparing fingerprints for settled files, all four tests fail locally.
• An earlier version re-wrote one file in a loop: log. It passed trivially, because the settle window forced a re-read on every round. It was replaced.

Cost of the settle rule: a file changed within the last 2s is re-parsed on every query until it settles.

Changes to existing tests

• Fixtures that test the watch layer over a fake filesystem whose walk lists nothing (populated_db, fk_db), the FK hold test, and the two move-in watch tests turn fresh reads off, so they still test what the watcher applies.
• Three watch move-out tests looped until a query showed the row gone, then asserted the Delete event. A query now drops the row first and queues the event for the next poll, so they loop until the event arrives. Assertions unchanged.
• persist_fans_out_and_composite_key_round_trips checks "extracted once" right after the build; the just-written file is unsettled, so the queries after it re-parse it by design.

Mutation exemption

testing-conventions.toml exempts the cfg(windows) and non-unix fingerprint bodies in src/freshness.rs from mutation. The Linux mutation runner compiles them out, so no Linux test can reach them. The Windows job runs tests/fresh_declared_*.rs, whose t+0 asserts fail if the Windows fingerprint misses a change.

Red run

Commit test: declared tables read exactly what is on disk at t+0 went red in CI (run); every failure was a staleness assertion.

• Linux (log): stopped at fresh_declared_edges, all 13 failing.
• Windows (log): exactly the 25 new tests failed.

Perf

Release builds, a 20,000-file JSON tree, per-table jq hook, measured with hyperfine against the branch's merge-base.

Measure	Base	This PR
CLI one-shot dirsql query 'SELECT count(*) FROM records' -c … (60 runs)	26.5 ms ± 1.8	26.5 ms ± 1.5
Long-lived query on the declared table (REPL, 100 queries, per query)	~0 (stale)	32.1 ms
The same count on the equivalent path-table './**/*.json' (per query)	—	34.1 ms

The one-shot is unchanged (fresh reads are off there). A declared-table query now costs a walk and a stat per match, slightly below the equivalent path-table query. cargo bench -p dirsql (db, differ, matcher, scanner) covers modules this PR does not touch and ran clean.

Downstream acceptance repros (Python, by hand)

I built the worktree with uv run maturin develop --release at 97eff27, then ran both scripts with uv run python.

• dq-watch.py. Two cases:
  • With no watch() running, a file written after ready() appears in the next query: no watch [turns/0, turns/1].
  • With watch() consumed, turns/2 appears in the query and arrives as one insert event. turns/1 sends no event, because it was written and queried before watch() started.
• dq-race.py. It consumes watch(), writes a file, then queries straight away. Over 6 runs of 20 writes: immediate misses 0 of 20 in every run, and max lag 0.001 to 0.002 s, which is the query's own cost.

E2E Verification

• Ran e2e suites locally for every affected SDK
• Python SDK e2e: pass (31 passed)
• TypeScript SDK e2e: pass (18 passed)
• Rust core e2e: pass (cargo test --workspace --features cli, all 84 test binaries green)
• packages/python/e2e-attestations/bot-1572-fresh-declared-1103.json receipt written
• packages/ts/e2e-attestations/bot-1572-fresh-declared-1103.json receipt written
• Command(s) run: testing-conventions e2e attest 'uv run python -m pytest tests/e2e/{declared_table_name,fanout,on_file_query,query_contract,watch_speed,server_speed,repl_interactive,sigint_interrupt,use_case_dossier,use_case_join,use_case_notebook,use_case_transcripts,frontmatter_parse_speed,jsonl_parse_speed}_test.py -q' (the e2e tests that touch declared tables, watch, server or REPL; the path-table glob speed tests are unaffected); just e2e-attest-ts; cargo test --workspace --features cli
• Result summary: all pass. Hand check: the /tmp repros (fresh, fresh2 with and without watch, fresh_path) all read the new state at t+0.

Changelog / Migrations

• Changelog fragment added under packages/rust/changelog.d/ and packages/ts/changelog.d/
• Migration fragment with the same slug added under packages/rust/migrations.d/ and packages/ts/migrations.d/

[thekevinbot]

Restarted red-first as #1577