Progress reporting on the startup scan

PARITY.md

@@ -43,6 +43,19 @@ on any SDK: the root-joining Rust `DirSQL::from_config(root)` /
 TypeScript never had a root-joiner — only the explicit `config=` — so nothing
 was removed there. This **restores parity** with the CLI's no-`-c` behavior (#602).
 
+**Progress on construction — at parity across all three SDKs (#957), no drift.**
+Building an index reports the walk and the ingest pass on stderr while they
+run, then erases the live line and leaves one summary of what each cost. The
+reporter lives in the shared core (`progress`, driven from
+`DirSQL::prepare_resolved` and `finish_build`), so all three SDKs got it in the
+same commit and no SDK carries its own copy. It is terminal-only and
+slow-work-only by default (nothing on a pipe, nothing under half a second);
+`DIRSQL_PROGRESS=always` / `=never` overrides it, and the variable is read by
+the core rather than by any launcher, so it behaves identically from the CLI,
+Python, Rust and TypeScript. No SDK exposes a knob for it: an SDK-level
+`progress=` argument would be exactly the kind of per-SDK surface this table
+exists to prevent.
+
 | API                        | Python                                         | Rust                                                 | TypeScript                                              |
 |----------------------------|------------------------------------------------|------------------------------------------------------|---------------------------------------------------------|
 | Constructor                | `DirSQL(root=None, *, tables=None, ignore=None, no_ignore=False, config=None, persist=False, persist_path=None, extensions=None)` | `DirSQL::builder().root(..).tables(..).ignore(..).config(..).persist(Option<path>).extensions(..).build()` (also `DirSQL::new`/`with_ignore` shortcuts) | `new DirSQL(configPath)` or `new DirSQL({ root?, tables?, ignore?, config?, persist?, persistPath?, extensions?, noIgnore? })` + `await db.ready` |

docs/reference/cli.md

@@ -460,3 +460,47 @@ Turn discovery off with either:
 
 A plugin that declares itself but is missing its module or its `dirsql.toml`
 fragment is a launcher error naming the package — never a silent skip.
+
+## Progress reporting
+
+Building the index over a large tree is not instant: the walk visits every
+file, then each matched file costs one `on-file` round trip plus whatever the
+table's `ddl` fires on insert. On a big corpus that is minutes. dirsql reports
+the two phases on **stderr** while they run:
+
+```
+dirsql: scanning 128413 files
+dirsql: indexing 9204/41231 files (22%)
+```
+
+Each line is rewritten in place. When a phase ends its line is erased and
+replaced by one summary of what it cost:
+
+```
+dirsql: scanned 128413 files in 4.2s
+dirsql: indexed 41231 files in 3m12s
+```
+
+stdout is untouched — it carries the query result and nothing else.
+
+By default this is **terminal-only, and only for work slow enough to wonder
+about**: a phase that finishes in under half a second prints nothing at all,
+and a run whose stderr is a pipe or a file prints nothing regardless of how
+long it takes. `dirsql "…" 2>run.log` and `dirsql "…" | jq` are byte-for-byte
+what they were before.
+
+Override with `DIRSQL_PROGRESS`:
+
+| Value | Effect |
+|---|---|
+| unset, or `auto` | Report only on a terminal, and only once a phase has run for half a second. The default. |
+| `always`, `1`, `true` | Report from the first update, terminal or not. Use it to watch a scan whose stderr is redirected. |
+| `never`, `0`, `false` | Report nothing, ever. |
+
+Values are case-insensitive and surrounding whitespace is ignored; anything
+unrecognized reads as `auto`, so a typo cannot stop a scan from running.
+
+The setting is read by the **core**, not the CLI, so it governs an index built
+from any SDK as well — a Python or TypeScript program that builds a `DirSQL`
+with a terminal attached gets the same two phases on stderr, and the same
+silence when piped.

docs/reference/sdk.md

@@ -396,3 +396,22 @@ not fit that range is a hard error, not a lossy conversion: Python raises
 A TypeScript `bigint` **within** `i64` range maps to `INTEGER`. Only a real
 `bytes`/`bytearray` (Python) or `Buffer`/`Uint8Array` (TypeScript) maps to
 `BLOB` — a list/array of integers does not.
+
+## Progress on construction
+
+Constructing a `DirSQL` walks the tree and ingests every matched file, which
+on a large corpus is the slowest thing your program does. The core reports
+both phases on **stderr** while they run, then erases the live line and leaves
+one summary of what each cost:
+
+```
+dirsql: indexed 41231 files in 3m12s
+```
+
+This is terminal-only by default, and only for a phase that runs longer than
+half a second — a program whose stderr is a pipe, a file, or a log collector
+sees nothing, whatever the scan costs. Set `DIRSQL_PROGRESS=never` to
+guarantee silence even on a terminal, or `DIRSQL_PROGRESS=always` to report
+regardless. The full table is in the
+[CLI reference](cli.md#progress-reporting); the setting lives in the shared
+core, so it behaves identically from all three SDKs.

packages/python/e2e-attestations/claude-tackle-957-lrm0z6.json

@@ -0,0 +1,7 @@
+{
+  "command": "uv run python -m pytest tests/e2e/ -x -q",
+  "ran_at": 1787261032,
+  "exit_code": 0,
+  "commit": "eed954811817292c5c3dad0577eb90ab551e2e73",
+  "branch": "claude/tackle-957-lrm0z6"
+}

packages/rust/changelog.d/2026-08-20-scan-progress-reporting.md

@@ -0,0 +1,3 @@
+**Added**
+
+- **The startup scan reports its progress on stderr.** A cold scan over a large corpus was silent: the walk visits every file, then each matched file costs an `on-file` round trip plus whatever the table's `ddl` fires on insert, and on a big tree that is minutes with nothing on the terminal — no way to tell a slow scan from a wedged one. Both phases now draw a counter that is rewritten in place (`dirsql: indexing 9204/41231 files (22%)`), erased when the phase ends and replaced by one summary of what it cost (`dirsql: indexed 41231 files in 3m12s`). It is **terminal-only and slow-work-only** by default: a phase finishing in under half a second prints nothing, and a run whose stderr is a pipe or a file prints nothing at any duration, so `| jq` pipelines, `2>` redirects and CI logs are byte-for-byte unchanged. `DIRSQL_PROGRESS=always` reports regardless of terminal and from the first update; `DIRSQL_PROGRESS=never` guarantees silence. The reporter lives in the core (`progress`), so it governs an index built from any SDK, not just the CLI, and it adds no dependency — the line is a `\r`-rewritten counter rather than a drawn bar, which needs the terminal width and therefore a crate. (#957)

packages/rust/migrations.d/2026-08-20-scan-progress-reporting.md

@@ -0,0 +1,62 @@
+### The startup scan reports progress on stderr (#957)
+
+#### Summary
+
+Building an index — from the CLI or from any SDK — now writes a progress
+counter to stderr while the directory walk and the ingest pass run, and one
+summary line when each finishes. Nothing about the API changes; what changes is
+that a process that used to write nothing to stderr on a successful build may
+now write to it. The behavior is gated on stderr being a terminal *and* the
+phase running longer than half a second, so a piped, redirected or fast run is
+unaffected.
+
+#### Required changes
+
+_None._ No signature changed and no configuration is required. The
+`DIRSQL_PROGRESS` environment variable is the opt-out for a program that
+insists on an empty stderr.
+
+#### Deprecations removed
+
+_None._
+
+#### Behavior changes without code changes
+
+- **stderr may carry progress lines on a terminal.** A build whose walk or
+  ingest phase runs longer than 500 ms and whose stderr is a terminal draws
+  `dirsql: scanning <n> files` / `dirsql: indexing <n>/<total> files (<p>%)`,
+  rewritten in place, then erases the line and prints
+  `dirsql: scanned <n> files in <t>` / `dirsql: indexed <n> files in <t>`.
+  Set `DIRSQL_PROGRESS=never` to restore an unconditionally empty stderr.
+- **Non-terminal stderr is unchanged.** A pipe, a file, a log collector or a
+  CI runner gets nothing, at any duration — the default mode consults
+  `stderr.is_terminal()` before writing a byte. Anything asserting on captured
+  stderr keeps passing.
+- **stdout is unchanged**, on every path. Progress never shares the stream the
+  query result is printed on.
+- **`dirsql::scanner::scan_directory` is unchanged.** The reporting walk is a
+  new sibling, `scan_directory_reporting(root, matcher, &mut on_file)`, which
+  the existing two-argument function now delegates to with a no-op callback.
+  Direct callers (benches, in-workspace tools) compile as before.
+
+#### Verification
+
+```bash
+# A slow scan on a terminal: the counter is drawn, erased, and summarized.
+dirsql query "SELECT count(*) FROM items" -c ./.dirsql.toml
+# dirsql: indexing 40/41 files (97%)      (rewritten in place, then erased)
+# dirsql: indexed 41 files in 2.2s
+# [{"count(*)":41}]
+
+# Piped: stderr is empty, exactly as before.
+dirsql query "SELECT count(*) FROM items" -c ./.dirsql.toml 2>err.txt | jq .
+wc -c err.txt   # -> 0
+
+# Forced on without a terminal, for watching a redirected run.
+DIRSQL_PROGRESS=always dirsql query "SELECT 1" -c ./.dirsql.toml 2>&1 >/dev/null | tail -1
+# -> dirsql: indexed 41 files in 2.2s
+
+# Opted out, even on a terminal.
+DIRSQL_PROGRESS=never dirsql query "SELECT count(*) FROM items" -c ./.dirsql.toml
+# -> [{"count(*)":41}]      (nothing on stderr)
+```

packages/rust/src/lib.rs

@@ -30,6 +30,8 @@ pub mod path_table;
 #[doc(hidden)]
 pub mod persist;
 #[doc(hidden)]
+pub mod progress;
+#[doc(hidden)]
 pub mod scanner;
 #[doc(hidden)]
 pub mod sql_literal;
@@ -52,7 +54,8 @@ use crate::persist::{
     hash_file, is_trusted, meta_is_compatible, now_ns, read_cached_files, read_meta,
     resolve_persist_path, upsert_file, write_meta,
 };
-use crate::scanner::scan_directory;
+use crate::progress::Progress;
+use crate::scanner::scan_directory_reporting;
 use crate::watcher::{FileEvent, Watcher};
 use futures_channel::mpsc::{UnboundedReceiver, UnboundedSender, unbounded};
 use std::collections::HashMap;
@@ -328,8 +331,9 @@ pub struct DirSQL {
 /// Internal scan seam: the startup directory walk, injectable so unit tests
 /// can observe whether it ran. The walk has no result a caller can tell apart
 /// from not walking, so *whether* it ran takes a seam. Production always
-/// passes [`scan_directory`].
-type ScanFn<'a> = &'a dyn Fn(&Path, &TableMatcher) -> Vec<(PathBuf, String)>;
+/// passes [`scan_directory_reporting`]; the third argument is the walk's
+/// running file count, which drives the progress line.
+type ScanFn<'a> = &'a dyn Fn(&Path, &TableMatcher, &mut dyn FnMut(u64)) -> Vec<(PathBuf, String)>;
 
 impl DirSQL {
     /// Start building a `DirSQL`. See [`DirSQLBuilder`] for the available
@@ -711,7 +715,7 @@ impl DirSQL {
     /// JS main thread for the napi-rs TypeScript binding).
     #[doc(hidden)]
     pub fn prepare_resolved(resolved: ResolvedBuild) -> Result<PreparedBuild> {
-        Self::prepare_resolved_with(resolved, &scan_directory)
+        Self::prepare_resolved_with(resolved, &scan_directory_reporting)
     }
 
     fn prepare_resolved_with(resolved: ResolvedBuild, scan: ScanFn<'_>) -> Result<PreparedBuild> {
@@ -751,7 +755,16 @@ impl DirSQL {
         let scanned = if table_names.is_empty() {
             Vec::new()
         } else {
-            scan(&root, &matcher)
+            // The walk reports a running count rather than a fraction: it does
+            // not know how many files there are until it has found them all.
+            let mut progress = Progress::scanning();
+            let mut seen = 0;
+            let scanned = scan(&root, &matcher, &mut |count| {
+                seen = count;
+                progress.update(count, None);
+            });
+            progress.finish(seen);
+            scanned
         };
 
         // When persist is enabled, files whose stat tuple matches the cache
@@ -953,12 +966,21 @@ impl DirSQL {
 
         let snapshot_ns = now_ns();
         let mut on_file_failures: Vec<OnFileFailure> = Vec::new();
-        for ScannedFile {
-            rel_path,
-            table_name,
-            stat,
-        } in scanned_files
+        // The phase a user actually waits on: one `on_file` round trip per
+        // file, plus whatever the table's DDL triggers on insert. The count is
+        // known up front, so this one reports a fraction.
+        let total_files = scanned_files.len() as u64;
+        let mut progress = Progress::indexing();
+        for (
+            done,
+            ScannedFile {
+                rel_path,
+                table_name,
+                stat,
+            },
+        ) in scanned_files.into_iter().enumerate()
         {
+            progress.update(done as u64, Some(total_files));
             let on_file = on_file_map.get(&table_name).ok_or_else(|| {
                 DirSqlError::Ddl(format!("missing on-file function for table {table_name}"))
             })?;
@@ -1028,6 +1050,8 @@ impl DirSQL {
             }
         }
 
+        progress.finish(total_files);
+
         // Write the meta block last; ingest and meta commit atomically in the
         // single transaction opened above. A crash mid-build leaves the cache
         // exactly as it was before the build started (detected via meta on
@@ -2268,8 +2292,18 @@ mod internal_tests {
             }
         }
 
-        fn scan(&self, _root: &Path, _matcher: &TableMatcher) -> Vec<(PathBuf, String)> {
+        /// Mirrors the real walk closely enough for the seam: it reports a
+        /// running file count as it goes, then returns what it found.
+        fn scan(
+            &self,
+            _root: &Path,
+            _matcher: &TableMatcher,
+            on_file: &mut dyn FnMut(u64),
+        ) -> Vec<(PathBuf, String)> {
             self.calls.set(self.calls.get() + 1);
+            for (index, _) in self.found.iter().enumerate() {
+                on_file(index as u64 + 1);
+            }
             self.found.clone()
         }
     }
@@ -2308,7 +2342,7 @@ mod internal_tests {
 
         let prepared = DirSQL::prepare_resolved_with(
             resolved_over(dir.path(), Vec::new(), false),
-            &|root, matcher| scan.scan(root, matcher),
+            &|root, matcher, on_file| scan.scan(root, matcher, on_file),
         )
         .unwrap();
 
@@ -2329,7 +2363,7 @@ mod internal_tests {
 
         let prepared = DirSQL::prepare_resolved_with(
             resolved_over(dir.path(), vec![txt_table()], false),
-            &|root, matcher| scan.scan(root, matcher),
+            &|root, matcher, on_file| scan.scan(root, matcher, on_file),
         )
         .unwrap();
 
@@ -2351,7 +2385,7 @@ mod internal_tests {
 
         let prepared = DirSQL::prepare_resolved_with(
             resolved_over(dir.path(), Vec::new(), true),
-            &|root, matcher| scan.scan(root, matcher),
+            &|root, matcher, on_file| scan.scan(root, matcher, on_file),
         )
         .unwrap();