[thekevinbot]

Fixes #1238

Status

Green step pushed, both receipts committed, migrations fragment added (the changelog gate asks for one on any packages/rust/src change), just preflight clean (Rust mutation: 12 tested, 0 survived). The two-table walks now overlap: packages/rust/tests/path_table_parallel_scan.rs failed on the serial code (CI red on 8c8f56c, ubuntu and Windows, with the test's own assertion) and passes with the fix. The join e2e's strict xfail is removed. Honest caveat: on this box (shared, load average 20 to 200) the join test went from 2.6x over the bar to sitting at it -- 5 passes in 14 runs after the fix, every failure within 2 to 30 percent of the bar, with native itself swinging between 0.78s and 2.74s between runs. The Python receipt took 11 attempts of the full suite; the failures on the way were join (5), dossier (4, untouched by this change) and notebook (1, untouched). The walk no longer costs twice; what remains between dirsql and the find | awk | join pipeline is reading content for 34k files plus the join and the sort, which this issue does not cover.

What changed

A statement that names several path-tables walked their trees one after the other. The serialization was not in the resolver loop in Db::run_statement (that loop only registers the virtual tables); it was in SQLite driving the cursors: each path-table's walk runs inside its first xFilter, and SQLite opens one cursor at a time.

Option chosen: the pre-pass, not the shared walk. At the final successful prepare, the authorizer already installed by prepare_guarded records every SQLITE_READ table name. run_statement then calls StatementScope::warm with those names, and every adopted path-table cache whose name is in the set starts its scan on a thread of its own (RowCache gains a Pending slot the first xFilter joins). The SQLite error loop remains the only thing that registers a table: the authorizer names are read from the statement SQLite accepted, after every no such table was resolved, and a name no adopted table answers to is skipped, so a declared table is never shadowed and nothing is registered for a name SQLite did not fail on. The shared-walk option was not measured: it needs a common root and a merged glob, a bigger change, and the pre-pass alone brings the join to the bar.

Measurements

Five speed-of-light tests, packages/python/tests/e2e, run on a shared 32-core box with load average 50 to 200 throughout; every number is best-of-3 and noisy. measured is dirsql minus startup; the bar is startup plus 1.1 x native.

Before (main's build):

test	native	dirsql	startup	bar	measured	result
join	2.800s	8.268s	0.221s	3.301s	8.047s	fail (2.6x over)
dossier	0.618s	0.997s	0.707s	1.388s	0.290s	pass
notebook	1.017s	1.221s	0.870s	1.989s	0.351s	pass
transcripts	--	did not finish within 60s	--	--	--	strict xfail (on-file, unrelated)
embeddings	1.989s	1.082s	0.153s	2.341s	0.928s	pass

After (this branch):

test	native	dirsql	startup	bar	measured	result
join	1.202s	1.916s	0.876s	2.199s	1.040s	pass
join (second run)	0.781s	1.233s	1.024s	1.883s	0.209s	pass
join (third run)	1.210s	2.442s	1.110s	2.441s	1.333s	fail by 2ms
dossier	0.615s	1.398s	0.861s	1.537s	0.537s	pass
notebook	0.365s	0.625s	0.615s	1.017s	0.009s	pass
transcripts	--	did not finish within 60s	--	--	--	strict xfail (unchanged)
embeddings	0.820s	0.810s	0.706s	1.608s	0.104s	pass

Release CLI profile against the join fixture (40,000 papers, 3 runs each): count(*) over the abstract glob 0.44-0.53s, over the title glob 0.43s, over both joined 0.42-0.47s, so two walks now cost one walk. The remaining gap to native on a loaded box is reading content and the join itself, not the walk.

Manual exercise

cargo build --release --features cli, then the join query from use_case_join_test.py against a 40,000-paper fixture, load average about 180: native (find + join) 0.74s / 0.58s / 0.86s, 31,168 rows; dirsql query 1.39s / 1.09s / 1.37s, same 31,168 rows, one JSON array line, rows such as {"dir":"papers/2609.000001","title":"Paper 1: on the 1th method","abstract_bytes":248}.

E2E Verification

• Ran e2e suites locally for every affected SDK
• Python SDK e2e: pass
• TypeScript SDK e2e: pass
• Rust core e2e (if applicable): pass
• packages/python/e2e-attestations/<branch>.json receipt written if packages/python changed (just e2e-attest-python)
• packages/ts/e2e-attestations/<branch>.json receipt written if packages/ts changed (just e2e-attest-ts)
• Command(s) run: cargo test --workspace --features cli, just e2e-attest-python, just e2e-attest-ts, just preflight
• Result summary: cargo test --workspace --features cli all green (1042 lib tests plus every integration and CLI e2e test); just e2e-attest-ts passed first run (4 scripts, 112s) and wrote packages/ts/e2e-attestations/claude-1238-parallel-walk.json; just e2e-attest-python passed on the 11th attempt (40 passed, transcripts strict-xfailed as on main) and wrote packages/python/e2e-attestations/claude-1238-parallel-walk.json; just preflight 0 failing pairs.

Changelog / Migrations

• Changelog fragment added under packages/rust/changelog.d/2026-10-02-path-table-parallel-walk.md
• Migration fragment added under packages/rust/migrations.d/2026-10-02-path-table-parallel-walk.md -- no API change; records the one runtime difference (every path table a statement reads is walked at prepare, on its own thread, even if SQLite never opens its cursor) and a verification query

[thekevinbot]

Hold for #1241. Profiling on this branch (notes in /tmp/dirsql-1238-profile-2026-10-02.md on the maintainer box) shows the join e2e misses the bar by 6% because content is read serially on the cursor thread: 77 ms of a 277 ms run against a 260 ms budget. The walks are now parallel; the join and sort are under 5 ms. #1241 prefetches content in parallel off main. Once it merges, rebase this branch onto it and the xfail removal here should pass reliably (expected ~216 ms, 17% under the bar).