A declared table reads what is on disk when each query runs

docs/howto/parse-files-into-columns.md

@@ -83,7 +83,7 @@ the fastest way to see whether a parser produces the rows you expect.
 
 The inline flag re-scans and re-parses every file on every query, defines one
 parser for the whole query, and forgets everything when the process exits. When
-you want a **watcher** that keeps the rows fresh, **persistence** across
+you want **row events** as files change, **persistence** across
 restarts, or **different parsers for different file sets**, move the parser into
 a `.dirsql.toml` [`[[table]]`](../reference/config.md#table) — and paste the
 command in verbatim:
@@ -110,7 +110,8 @@ dirsql query "SELECT title, author FROM posts ORDER BY title" -c ./.dirsql.toml
 ```
 
 Same command, same rows. The difference is what a declared table brings: it is
-indexed on build, kept fresh by the watcher, survives restarts with
+indexed on build, re-reads only the files that changed on each query, streams
+row events as files change, survives restarts with
 [`--persist`](./persist.md), and a config can declare
 [many tables](./define-tables.md) — each with its own `on-file` — where the flag
 gives every path-table one parser. In both spellings the table's columns are

docs/howto/react-to-changes.md

@@ -72,6 +72,9 @@ works — `EventSource` in a browser, an SSE client library, or plain
 
 ## Notes
 
+- Querying never waits on this stream: a query reads what is on disk when it
+  starts ([`query`](../reference/sdk.md#query)). A change a query reads first
+  still arrives here, once.
 - Events reflect *row* changes, not raw filesystem events: a file edit
   that leaves a table's rows identical emits nothing, and one file change
   can emit several events. The diffing model is part of

docs/reference/sdk.md

@@ -289,6 +289,24 @@ db.query_with(
 - Without `params`, a statement with placeholders fails the same way: a
   placeholder is never left unbound.
 
+- **Accurate to the filesystem.** A query reads what is on disk when it
+  starts: a file written, edited, renamed or deleted before the call is
+  reflected in its rows, with no watcher, wait or retry. Before reading, the
+  query walks each named table's glob and stats every match; a file whose stat
+  moved, or that changed within the last two seconds, is parsed again, and a
+  file that is gone loses its rows. The row events those re-reads produce reach
+  [`watch`](#watch) streams and the next poll, once each. If a file on disk
+  breaks a foreign key, the query fails with the error a build over that tree
+  returns. In TypeScript the re-reads run on the JavaScript thread, where
+  `onFile` runs, and the SQL then runs on the threadpool.
+
+  The check trusts the file's stat, so it shares the limits of every
+  stat-based tool (`make`, `rsync`, `git status`): a write through a shared
+  memory map can land without the file's times moving until the writer
+  unmaps or syncs it; a network filesystem that caches attributes can report
+  a stale stat; and a clock stepped backwards can stamp a new write with a
+  time older than the last read.
+
 - **Read-only.** Each statement is classified by SQLite itself
   (`sqlite3_stmt_readonly`); anything classified as a write — `INSERT`,
   `UPDATE`, `DELETE`, `DROP`, `CREATE`, `ALTER`, `VACUUM`, … — is rejected
@@ -369,7 +387,9 @@ to stop.
 The watcher is live once both `watch()` and [`ready`](#ready) have returned,
 in either order (a Rust `DirSQL` is ready when `build()` returns); a file
 written after that point produces an event and reaches the index. A write
-made while the initial scan is still running may be missed.
+made while the initial scan is still running may be missed by the stream, but
+never by a query (see [`query`](#query)). A change a query read first still
+arrives on the stream, once.
 
 An instance with no named tables cannot be watched: path-tables emit no
 watch events, so `watch()` (and the polling primitives) raise an error naming

packages/python/changelog.d/2026-11-03-fresh-declared-reads.md

@@ -0,0 +1,3 @@
+**Fixed**
+
+A query on a long-lived `DirSQL` now reads what is on disk when it starts (core change, surfaced through the Python SDK). Before reading, it walks each named table's glob, re-runs `on_file` on every file whose stat moved or that changed within the last two seconds, and drops the rows of files that are gone, so a write made just before a query is never missed. The row events those re-reads produce reach `watch()` streams once each, including a change read before the first `watch()` call.

packages/python/e2e-attestations/bot-1572-fresh-red-1104.json

@@ -0,0 +1,7 @@
+{
+  "command": "uv run python -m pytest tests/e2e/declared_table_name_test.py tests/e2e/fresh_declared_repros_test.py tests/e2e/on_file_query_test.py tests/e2e/use_case_dossier_test.py tests/e2e/use_case_join_test.py tests/e2e/use_case_notebook_test.py tests/e2e/use_case_transcripts_test.py tests/e2e/frontmatter_parse_speed_test.py tests/e2e/jsonl_parse_speed_test.py tests/e2e/fanout_test.py tests/e2e/plugin_discovery_test.py tests/e2e/plugin_extension_discovery_test.py tests/e2e/extension_package_test.py tests/e2e/server_speed_test.py tests/e2e/query_contract_test.py tests/e2e/repl_interactive_test.py tests/e2e/watch_speed_test.py tests/e2e/sigint_interrupt_test.py -q",
+  "ran_at": 1793815824,
+  "exit_code": 0,
+  "commit": "501e509a06e56f8cdd91445dae06bb55180f2787",
+  "branch": "bot/1572-fresh-red-1104"
+}

packages/python/migrations.d/2026-11-03-fresh-declared-reads.md

@@ -0,0 +1,23 @@
+### Queries read what is on disk
+
+#### Summary
+
+A query on a long-lived `DirSQL` brings its named tables in line with the files on disk before reading. No API or configuration changes are required.
+
+#### Required changes
+
+_None._
+
+#### Deprecations removed
+
+_None._
+
+#### Behavior changes without code changes
+
+A query re-runs `on_file` on the files that changed since the index last read them, so code that waited on a watch event or slept before querying no longer needs to. A change a query reads first still arrives on `watch()`, once.
+
+Each query walks its tables' globs and stats every match, about what the same query on a path-table costs. When a file on disk breaks a foreign key, the query raises the error a build over that tree raises, until the file is fixed.
+
+#### Verification
+
+Build a `DirSQL` over a directory with a declared table, write a matching file, then `await db.query(...)` the table with no wait; the row for the new file is in the result.

packages/python/tests/e2e/fresh_declared_repros_test.py

@@ -0,0 +1,169 @@
+"""E2E: a long-lived ``DirSQL`` reads what is on disk when each query runs.
+
+Regressions for the reported repros, against the installed build: a declared
+table read with no watch running, a change written between ``ready()`` and
+``watch()`` starting, and a query racing a consumed ``watch()``. Every query
+runs the moment the write before it returns. No mocks: real binding, real
+filesystem, real SQLite.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import json
+from pathlib import Path
+
+import pytest
+
+from dirsql import DirSQL, Table
+
+RACE_ROUNDS = 1000
+
+
+def _turns(root: Path) -> Table:
+    def on_file(path):
+        return [{"id": Path(path).parent.relative_to(root).as_posix()}]
+
+    return Table(
+        name="turns",
+        ddl="CREATE TABLE turns (id TEXT PRIMARY KEY)",
+        glob="turns/*/a.txt",
+        on_file=on_file,
+    )
+
+
+def _records() -> Table:
+    def on_file(path):
+        p = Path(path)
+        return [{"name": p.name, "v": json.loads(p.read_text())["v"]}]
+
+    return Table(
+        name="t",
+        ddl="CREATE TABLE t (name TEXT, v INTEGER)",
+        glob="*.json",
+        on_file=on_file,
+    )
+
+
+def _add_turn(root: Path, n: int) -> None:
+    (root / f"turns/{n}").mkdir()
+    (root / f"turns/{n}/a.txt").write_text("x")
+
+
+async def _turn_ids(db: DirSQL) -> list[str]:
+    return [row["id"] for row in await db.query("SELECT id FROM turns ORDER BY id")]
+
+
+async def _records_of(db: DirSQL) -> list[tuple[str, int]]:
+    rows = await db.query("SELECT name, v FROM t ORDER BY name")
+    return [(row["name"], row["v"]) for row in rows]
+
+
+async def _open_turns(root: Path) -> DirSQL:
+    db = DirSQL(str(root), no_ignore=True, tables=[_turns(root)])
+    await db.ready()
+    return db
+
+
+async def _open_records(root: Path) -> DirSQL:
+    db = DirSQL(str(root), tables=[_records()])
+    await db.ready()
+    return db
+
+
+def describe_a_declared_table_with_no_watch_running():
+    @pytest.mark.asyncio
+    async def it_reads_a_file_written_after_ready(tmp_path):
+        (tmp_path / "turns/0").mkdir(parents=True)
+        (tmp_path / "turns/0/a.txt").write_text("x")
+        db = await _open_turns(tmp_path)
+        _add_turn(tmp_path, 1)
+        assert await _turn_ids(db) == ["turns/0", "turns/1"]
+
+    @pytest.mark.asyncio
+    async def it_reads_a_create_an_edit_and_a_delete(tmp_path):
+        (tmp_path / "a.json").write_text(json.dumps({"v": 1}))
+        db = await _open_records(tmp_path)
+        assert await _records_of(db) == [("a.json", 1)]
+        (tmp_path / "b.json").write_text(json.dumps({"v": 2}))
+        (tmp_path / "a.json").write_text(json.dumps({"v": 10}))
+        assert await _records_of(db) == [("a.json", 10), ("b.json", 2)]
+        (tmp_path / "b.json").unlink()
+        assert await _records_of(db) == [("a.json", 10)]
+
+
+def describe_a_change_written_before_watch_starts():
+    @pytest.mark.asyncio
+    async def it_is_read_by_the_next_query_and_streamed_once(tmp_path):
+        (tmp_path / "turns/0").mkdir(parents=True)
+        (tmp_path / "turns/0/a.txt").write_text("x")
+        db = await _open_turns(tmp_path)
+        _add_turn(tmp_path, 1)
+        assert await _turn_ids(db) == ["turns/0", "turns/1"]
+        stream = db.watch()
+        # Unread by any query, so only the watcher reports it, after every
+        # change before it.
+        _add_turn(tmp_path, 2)
+        inserted = []
+        async for event in stream:
+            assert event.action == "insert", event
+            inserted.append(event.row["id"])
+            if event.row["id"] == "turns/2":
+                break
+        assert inserted == ["turns/1", "turns/2"]
+        assert await _turn_ids(db) == ["turns/0", "turns/1", "turns/2"]
+
+
+def describe_a_query_racing_a_consumed_watch():
+    @pytest.mark.asyncio
+    async def it_reads_every_write_before_it(tmp_path):
+        (tmp_path / "turns").mkdir()
+        db = await _open_turns(tmp_path)
+        stream = db.watch()
+
+        async def consume():
+            async for _ in stream:
+                pass
+
+        task = asyncio.create_task(consume())
+        misses = []
+        for n in range(RACE_ROUNDS):
+            _add_turn(tmp_path, n)
+            if len(await db.query("SELECT id FROM turns")) != n + 1:
+                misses.append(n)
+        task.cancel()
+        assert misses == []
+
+    @pytest.mark.asyncio
+    async def it_reads_a_create_with_a_watch_consumed(tmp_path):
+        (tmp_path / "a.json").write_text(json.dumps({"v": 1}))
+        db = await _open_records(tmp_path)
+        stream = db.watch()
+
+        async def consume():
+            async for _ in stream:
+                pass
+
+        task = asyncio.create_task(consume())
+        (tmp_path / "b.json").write_text(json.dumps({"v": 2}))
+        assert await _records_of(db) == [("a.json", 1), ("b.json", 2)]
+        task.cancel()
+
+
+def describe_a_path_table():
+    @pytest.mark.asyncio
+    async def it_reads_a_create_an_edit_and_a_delete(tmp_path):
+        (tmp_path / "a.json").write_text(json.dumps({"v": 1}))
+        db = DirSQL(str(tmp_path), no_ignore=True)
+        await db.ready()
+        sql = "SELECT path, content FROM './*.json' ORDER BY path"
+
+        async def read():
+            return [(row["path"], row["content"]) for row in await db.query(sql)]
+
+        assert await read() == [("a.json", '{"v": 1}')]
+        (tmp_path / "b.json").write_text(json.dumps({"v": 2}))
+        (tmp_path / "a.json").write_text(json.dumps({"v": 10}))
+        assert await read() == [("a.json", '{"v": 10}'), ("b.json", '{"v": 2}')]
+        (tmp_path / "b.json").unlink()
+        assert await read() == [("a.json", '{"v": 10}')]