[thekevinbot]

Fixes #1285
Fixes #1244

Batched on-file ingest no longer trails the shell pipeline. The transcripts speed-of-light test now compares against the fair native (find projects -name '*.jsonl' -exec python3 messages.py {} + | python3 count.py), and with this change it passes, so its xfail is removed.

What the profile showed

Phase timing on the transcripts fixture (4,096 files, three parser invocations, 273,067 rows / 37 MB of JSON), release build:

Phase	Before
Parser invocations (run back to back)	1.63-1.75 s
JSON parse	0.107 s
Shape rows into DDL order	0.080 s
SQLite insert	0.205 s
Drop + commit	~0.09 s
Query	0.063 s

The parser takes about as long as it does in the native pipeline. dirsql's extra cost was that every row got parsed, shaped and inserted only after all invocations had exited. The shell pipe consumes each batch while the next one runs.

Changes

• on_file::run_streaming hands each invocation's output to a parser thread while the next invocation runs. finish_build shapes and inserts each chunk inside the same outer transaction as it arrives, so ingest overlaps the parser like a shell pipe does. Invocations stay sequential and the error precedence stays the same: an earlier invocation's bad output still outranks a later one's failure.
• impl ToSql for Value borrows text and blobs instead of cloning them. Before, every string was copied on bind.

SQLite insert cost on its own

For 273,067 rows / 37 MB:

Insert path	Time
dirsql before this PR	0.205 s
dirsql after this PR (copy-free bind)	0.163 s
Python sqlite3.executemany replay of the same rows	0.104 s

The Python replay used a temp DB. Pragma variants (journal off, 64 MB cache, 64 KiB pages, :memory:) moved it only to 0.093-0.104 s.

Timings

Release build, 1-minute load under 16, minimum of at least 6 runs. The ratio is (dirsql - startup) / native.

Files	Native fair	dirsql before	dirsql after	Ratio before	Ratio after
4096	1.709-1.727 s	2.251 s	1.812-1.828 s	1.280x	1.031-1.035x
2048	0.860 s	-	0.993 s	-	1.108x

At 2048 files dirsql passes absolute paths, so it needs two invocations where find with relative paths needs one. That leaves an extra interpreter start plus the last chunk's ingest exposed. Changing what argv the hook receives would change behavior, so it is out of scope. The test grows the fixture until native takes at least 1 s, which is 4096 files on this box.

Red / green

• Red: CI run 37341334790. In the Unit tests job (111868877244), db::tests::text_and_blob_values_bind_without_copying failed with left Owned(Text("abc")) and right Borrowed(Text([97, 98, 99])).
• Red, locally: the fair-native transcripts test under --runxfail reported "dirsql 2.250s minus startup 0.038s = 2.211s, over 1.1x native 1.735s = 1.908s".
• Green: the transcripts test passes without xfail.

E2E Verification

• Ran e2e suites locally for every affected SDK
• Python SDK e2e: pass (40 passed)
• TypeScript SDK e2e: pass
• Rust core e2e (if applicable): pass (cargo test --workspace --features cli)
• packages/python/e2e-attestations/<branch>.json receipt written if packages/python changed (just e2e-attest-python)
• packages/ts/e2e-attestations/<branch>.json receipt written if packages/ts changed (just e2e-attest-ts)
• Command(s) run: cargo test --workspace --features cli, just e2e-attest-python, just e2e-attest-ts
• Result summary: all pass. A manual dirsql query on the 2048-file fixture gives the same per-role counts as the native pipeline (assistant 68268, user 68267).

Changelog / Migrations

• Changelog fragment added under <root>/<pkg>/changelog.d/ for each changed package (or: skip-changelog trailer on a commit with reason)
• Migration fragment with the same slug added under <root>/<pkg>/migrations.d/ for each changed package (every subsection but Summary _None._ when nothing breaks)