Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -202,7 +202,7 @@ The `dirsql` Rust crate handles all heavy lifting:

### `db` -- Ephemeral SQLite

Wraps `rusqlite` with an anonymous disk-backed temp database (#402) — ephemeral like `:memory:`, but index pages spill to disk so resident memory does not scale with the corpus. Handles DDL execution (run verbatim -- no injected columns, epic #358), row insertion with per-file ownership recorded in the internal `_dirsql_internal_rows` table, querying, and row deletion by file path. The internal bookkeeping tables (`_dirsql_internal_rows`, `_dirsql_files`, `_dirsql_meta`) are a private surface: a SQLite authorizer installed on the `query()` path denies any read (or schema `PRAGMA`) targeting the reserved `_dirsql_*` namespace, so they are unreachable through the public query surface (issue #378) while the engine still writes them in the same transaction as the user rows.
Wraps `rusqlite` with an anonymous disk-backed temp database (#402) — ephemeral like `:memory:`, but index pages spill to disk so resident memory does not scale with the corpus. Handles DDL execution (run verbatim -- no injected columns, epic #358), row insertion with per-file ownership recorded in the internal `_dirsql_internal_rows` table (a row at a time) or `_dirsql_internal_ranges` (a per-table hook's batch, as one rowid range), querying, and row deletion by file path. The internal bookkeeping tables (`_dirsql_internal_rows`, `_dirsql_internal_ranges`, `_dirsql_files`, `_dirsql_meta`) are a private surface: a SQLite authorizer installed on the `query()` path denies any read (or schema `PRAGMA`) targeting the reserved `_dirsql_*` namespace, so they are unreachable through the public query surface (issue #378) while the engine still writes them in the same transaction as the user rows.

**A table's name is declared, never derived.** Both `[[table]]` and the SDK
`Table` take a required `name`; nothing tokenizes the `CREATE TABLE` head of
Expand Down
5 changes: 3 additions & 2 deletions PARITY.md
Original file line number Diff line number Diff line change
Expand Up @@ -216,7 +216,8 @@ at once with no binding changes and no public-surface change.

**Verbatim table columns (#361, epic #358) — parity by construction, no drift.**
User tables carry exactly the columns declared in their DDL (row ownership is
tracked in the internal `_dirsql_internal_rows` table, not injected columns).
tracked in the internal `_dirsql_internal_rows` and `_dirsql_internal_ranges`
tables, not injected columns).
This lives entirely in the shared Rust core (`create_table` runs DDL verbatim;
`query` returns vanilla rows), so `PRAGMA table_info` and `SELECT *` report the
same user-only columns across all three SDKs.
Expand All @@ -232,7 +233,7 @@ together.

**Internal tables unreachable through `query()` (#378, epic #358) — parity by
construction, no drift.** The internal bookkeeping tables (`_dirsql_internal_rows`,
`_dirsql_files`, `_dirsql_meta`) are denied on the `query()` path by a SQLite
`_dirsql_internal_ranges`, `_dirsql_files`, `_dirsql_meta`) are denied on the `query()` path by a SQLite
authorizer in the shared Rust core (`db::query`), so a read of any `_dirsql_*`
table fails identically across all three SDKs (and the CLI's `POST /query`) with
a "not authorized" error — no per-binding surface.
Expand Down
7 changes: 7 additions & 0 deletions packages/python/e2e-attestations/claude-1243-ingest-perf.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
{
"command": "just test-e2e",
"ran_at": 1790978388,
"exit_code": 0,
"commit": "cb8a9e14d99d059576c313c41cac87b9d865f020",
"branch": "claude/1243-ingest-perf"
}
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
**Changed** A per-table `on-file` hook's rows are owned as one rowid range in the new internal `_dirsql_internal_ranges` table instead of one `_dirsql_internal_rows` entry per row, and are laid out in DDL column order once rather than rebuilt per row. Ingesting the transcripts fixture's 273k rows drops from 0.99 s to 0.58 s; the full run from 2.65 s to 2.17 s. Refresh and watch keep their delete-before-reingest semantics; rows, order and query results are unchanged.
45 changes: 45 additions & 0 deletions packages/rust/migrations.d/2026-10-02-batch-rows-owned-by-range.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
### Core: a per-table hook's rows are owned as one rowid range

#### Summary

A table fed by an `on-file` command no longer records one
`_dirsql_internal_rows` entry per ingested row. The batch is owned as a single
`(table, first_rowid, last_rowid)` range in a new internal table,
`_dirsql_internal_ranges`, created alongside `_dirsql_internal_rows` on every
connection, including an existing persisted cache. No signature, config key
or CLI flag changes.

#### Required changes

_None._

#### Deprecations removed

_None._

#### Behavior changes without code changes

- `_dirsql_internal_ranges` joins the reserved `_dirsql_*` namespace: a
`query()` that reads it is denied like the other internal tables.
- A persisted cache gains the table on next open; a cold rebuild clears it
with the rest of the bookkeeping.
- Rows, their order, `rowid`s and query results are unchanged.

#### Verification

```bash
cd "$(mktemp -d)"
printf 'a\n' > a.txt
printf 'b\n' > b.txt
cat > .dirsql.toml <<'TOML'
[[table]]
name = "items"
ddl = "CREATE TABLE items (name TEXT)"
glob = "*.txt"
on-file = "sh -c 'printf \"[\"; sep=\"\"; for f in \"$@\"; do printf \"%s{\\\"name\\\":\\\"%s\\\"}\" \"$sep\" \"$(basename \"$f\")\"; sep=\",\"; done; printf \"]\\n\"' sh"
TOML
dirsql query "SELECT name FROM items ORDER BY name" -c .dirsql.toml
# expected: [{"name":"a.txt"},{"name":"b.txt"}]
dirsql query "SELECT * FROM _dirsql_internal_ranges" -c .dirsql.toml; echo "exit: $?"
# expected: a "not authorized" error and a non-zero exit
```
Loading
Loading