From 3e442c3c29a099a0b717ec66825488a3832a7128 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 21 Aug 2026 02:40:28 +0000 Subject: [PATCH] docs: state the path-table/declared-table trade-off, ship index templates Two gaps, one page's worth of work. The trade-off between the two table kinds was true but scattered across the path-tables reference, the vtab's hidden `content` column and `db.rs`'s use of `temp`. Explanation now states it as one deliberate choice: a path-table stores nothing, which is exactly why it is always current, needs no setup, and can carry no index; a declared table costs a config entry and buys the watcher, persistence and a `ddl` batch to hang indexes off. The asymmetry is structural -- a table cannot both hold no state and index its state -- so the pages that used to gesture at it (path-tables, query-without-config, define-tables) now link there instead. The templates are the error-prone half. `ddl` has been a wide-open batch since #956, and the triggers are where it goes wrong: a malformed FTS5 'delete' trigger leaves stale rows in results and never errors. The new how-to ships both templates whole rather than abstracting them, plus the diagnostics -- a LEFT JOIN that exposes stale FTS entries as a null base row, and a `json_array_length(embed(...))` probe for the vec0 column width, since guessing it is the other easy way to paste something that does not work. Both were run against real dirsql and real SQLite while writing: FTS5 including delete propagation over a warm cache (and a no-delete-trigger control that stays stale, confirming the check discriminates), and vec0 against sqlite-vec 0.1.9 for trigger-time insert, KNN, and delete. Uncovered #1008 doing so, noted inline on the vector template. Links added inside ARCHITECTURE.md use absolute doc-site URLs. The `../docs/` form already there rendered as `./../docs/...` once included into the Explanation page, dropping the site's `/dirsql/` base -- that one existing occurrence is fixed the same way. Fixes #960 Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01SFEXWZPYdkXVkRjjy8tvdZ --- ARCHITECTURE.md | 51 ++++++- docs/.vitepress/config.ts | 1 + docs/howto/define-tables.md | 4 +- docs/howto/query-without-config.md | 4 +- docs/howto/search-indexes.md | 220 +++++++++++++++++++++++++++++ docs/reference/path-tables.md | 8 +- 6 files changed, 284 insertions(+), 4 deletions(-) create mode 100644 docs/howto/search-indexes.md diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index f8a099548..8b72364a3 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -31,6 +31,55 @@ prior `[table.columns]` source-dispatch and the `format` / `each` config keys were ripped out; the per-format parser zoo (`Format::Json`, `Csv`, `Yaml`, `Toml`, `Frontmatter`, …) is gone. +## Two table kinds, opposite trade-offs + +The two kinds above are not a general mechanism and a limited one. They are a +choice the user makes per table, and each buys what the other gives up. + +A **path-table** costs nothing to reach for. `FROM './**/*.json'` needs no +config file, no DDL and no hook; the table is minted in `temp` when the +statement runs, so it lives and dies with the connection. Nothing is stored: +the filesystem is walked during the query and `content` is read only if a +column names it. What you get for that is freshness with no machinery — a file +written a second ago is in the next statement's results, because there is no +copy of the tree to go stale. + +A **declared table** costs a `[[table]]` entry and an `on-file` hook. In +exchange it is an ordinary SQLite table: rows are inserted during the build, +the watcher keeps them current as files change, `--persist` carries them +across restarts, and its `ddl` batch can create anything SQLite can — a +`CREATE INDEX`, an FTS5 vtab, a `vec0` vtab, and the triggers that keep them +current. + +The asymmetry is structural, not an unfinished path-table. Indexes, +persistence and the watcher all need rows that outlive the statement that +asked for them, and a path-table deliberately has none — it stores nothing, +which is exactly why it is always current and needs no setup. A table cannot +both hold no state and index its state. + +So the trade reads: + +| | Path-table | Declared table | +| --- | --- | --- | +| Setup | none | `[[table]]` entry + `on-file` hook | +| Cost per query | walks the matched tree | reads rows from SQLite | +| Freshness | always current | watcher-maintained | +| Survives restart | no | with `--persist` | +| Indexes (B-tree, FTS5, vector) | no | in the `ddl` batch | + +Reach for a path-table when the question is ad hoc, or the tree is small +enough that walking it is not worth avoiding. Declare a table when the same +tree is queried repeatedly, is large, or wants keyword or vector search. The +two are ordinary SQLite tables once resolved, so a query is free to join one +against the other. + +The user-facing versions of both sides are +[Query files without a config](https://thekevinscott.github.io/dirsql/howto/query-without-config) +and +[Define tables for your files](https://thekevinscott.github.io/dirsql/howto/define-tables); +the index templates are in +[Add a search index to a table](https://thekevinscott.github.io/dirsql/howto/search-indexes). + ## Read-only by design **dirsql never modifies the directory it indexes.** It opens files for reading @@ -165,7 +214,7 @@ reinvent what SQLite expresses natively*. `validate_identifier` remains: `name` is spliced into `format!()`-built INSERT/DELETE SQL, so it is still the injection guard. -**Named tables are real; path-tables are virtual.** A declared `[[table]]` (or programmatic `Table`) is a real SQLite table whose rows are inserted on build and maintained by the watcher — the `db` module above. A [path-table](../docs/reference/path-tables.md) (`SELECT * FROM './'`, epic path-as-table) is a `dirsql_path` **virtual table** (`vtab.rs` / `path_table.rs`, rusqlite's `vtab` feature): no rows are stored, no reconcile or watcher runs, and the filesystem is walked live at query time (`xFilter`/`xNext` enumerate matched files; `xColumn` supplies stat values and lazily reads `content`). SQLite stays the entire query engine either way; the vtab only enumerates rows and supplies column values. Path-tables are registered on demand — see *Query execution* below. +**Named tables are real; path-tables are virtual.** A declared `[[table]]` (or programmatic `Table`) is a real SQLite table whose rows are inserted on build and maintained by the watcher — the `db` module above. A [path-table](https://thekevinscott.github.io/dirsql/reference/path-tables) (`SELECT * FROM './'`, epic path-as-table) is a `dirsql_path` **virtual table** (`vtab.rs` / `path_table.rs`, rusqlite's `vtab` feature): no rows are stored, no reconcile or watcher runs, and the filesystem is walked live at query time (`xFilter`/`xNext` enumerate matched files; `xColumn` supplies stat values and lazily reads `content`). SQLite stays the entire query engine either way; the vtab only enumerates rows and supplies column values. Path-tables are registered on demand — see *Query execution* below. ### `scanner` -- Directory traversal diff --git a/docs/.vitepress/config.ts b/docs/.vitepress/config.ts index 074fb437a..a03c00821 100644 --- a/docs/.vitepress/config.ts +++ b/docs/.vitepress/config.ts @@ -56,6 +56,7 @@ export default defineConfig({ { text: 'Parse your files into columns', link: '/howto/parse-files-into-columns' }, { text: 'Query JSON file contents', link: '/howto/query-json' }, { text: 'Search documents by meaning', link: '/howto/search-by-meaning' }, + { text: 'Add a search index to a table', link: '/howto/search-indexes' }, { text: "Skip files you don't want indexed", link: '/howto/skip-files' }, { text: 'Load a SQLite extension', link: '/howto/load-extension' }, { text: 'Keep the index across restarts', link: '/howto/persist' }, diff --git a/docs/howto/define-tables.md b/docs/howto/define-tables.md index ae0294e9c..a956b981d 100644 --- a/docs/howto/define-tables.md +++ b/docs/howto/define-tables.md @@ -107,7 +107,9 @@ maintain nothing. `name` still has to be the row table (`posts`); the virtual table lives beside it under its own name. The batch runs once, when the table is created, and editing any part of it rebuilds a [persistent cache](./persist.md) from scratch. Full rules: -[Batch `ddl`](../reference/config.md#batch-ddl). +[Batch `ddl`](../reference/config.md#batch-ddl); pasteable FTS5 and vector +templates, with the trigger mistakes that fail silently, are in +[Add a search index to a table](./search-indexes.md). ## Going further diff --git a/docs/howto/query-without-config.md b/docs/howto/query-without-config.md index e2981c6c1..23d34653e 100644 --- a/docs/howto/query-without-config.md +++ b/docs/howto/query-without-config.md @@ -55,7 +55,9 @@ question over a few hundred files, wasteful for a large tree you query repeatedly. When that day comes, [declare a table](./define-tables.md): it is indexed once, kept fresh by the watcher, and can persist across restarts. The zero-config query is the floor; a named table is the escalation path — nothing -you write here has to be thrown away to get there. +you write here has to be thrown away to get there. The two sides in full, and +why indexes belong to only one of them, are in +[how `dirsql` thinks](../explanation.md#two-table-kinds-opposite-trade-offs). ## Notes diff --git a/docs/howto/search-indexes.md b/docs/howto/search-indexes.md new file mode 100644 index 000000000..f3c137bac --- /dev/null +++ b/docs/howto/search-indexes.md @@ -0,0 +1,220 @@ +# Add a search index to a table + +Make a declared table answer search queries instead of scanning: a B-tree for +lookups, an FTS5 index for keywords, a `vec0` index for meaning. All three are +statements in the table's [`ddl` batch](../reference/config.md#batch-ddl), so +SQLite does the work and `dirsql` maintains nothing. + +The templates below are meant to be pasted and edited. Both are shaped around +one worked example — notes in `notes/*.md`, parsed into a `notes` table by an +`extract.py` that prints `{"slug": …, "title": …, "body": …}` per file, exactly +as in [Define tables for your files](./define-tables.md). + +## Why triggers, and why only two + +`dirsql` writes file rows with plain `INSERT` and `DELETE`; an update is a +delete and an insert in one transaction, and there is no `UPDATE` path on user +rows. So an insert trigger and a delete trigger cover every way a row can +change — during the initial build, on each [watcher](./react-to-changes.md) +event, and when a [persistent cache](./persist.md) reconciles a tree that +moved on. There is no third trigger to write and no maintenance command to run. + +The triggers are also the part that goes wrong, which is why the templates +spell them out rather than hiding them behind an abstraction. + +## Keyword search (FTS5) + +FTS5 ships inside SQLite — no extension, no install. Declare an +external-content index beside the row table and let the two triggers feed it: + +```toml +[[table]] +name = "notes" +glob = "notes/**/*.md" +on-file = "python3 extract.py {path}" +ddl = ''' +CREATE TABLE notes (slug TEXT, title TEXT, body TEXT); + +CREATE VIRTUAL TABLE notes_fts USING fts5( + body, + content='notes', + content_rowid='rowid', + tokenize='porter unicode61' +); +CREATE TRIGGER notes_ai AFTER INSERT ON notes BEGIN + INSERT INTO notes_fts(rowid, body) VALUES (new.rowid, new.body); +END; +CREATE TRIGGER notes_ad AFTER DELETE ON notes BEGIN + INSERT INTO notes_fts(notes_fts, rowid, body) + VALUES ('delete', old.rowid, old.body); +END; +''' +``` + +`content='notes'` makes the index store only its own search structures and +read column values back from `notes`, so the text is not duplicated. +`tokenize='porter unicode61'` stems English on top of the default +case-folding, so *deploying* matches *deploy*; drop the `porter` half to match +whole words only. + +Query the index and join back to the row table on `rowid`: + +```bash +dirsql query " +SELECT n.slug, + bm25(notes_fts) AS score, + snippet(notes_fts, 0, '[', ']', '…', 8) AS excerpt +FROM notes_fts JOIN notes AS n ON n.rowid = notes_fts.rowid +WHERE notes_fts MATCH 'deploy' +ORDER BY score" -c ./.dirsql.toml +``` + +```json +[{"excerpt":"Rebase before you [deploy]. Keep pull requests small.","score":-1.0476190476190478e-6,"slug":"branches"}, + {"excerpt":"…the spaghetti goes in. [Deploy] the garlic late.","score":-8.461538461538463e-7,"slug":"pasta"}] +``` + +**`bm25()` returns negative scores, and better matches are more negative** — +so `ORDER BY score` ascending is best-first, with no `DESC`. `snippet()`'s +arguments are the table, the column index, the open and close markers, the +ellipsis, and the token budget. + +### Check the delete trigger + +A wrong or missing delete trigger fails **silently**: the index keeps rows for +files that are gone, and they keep matching. Nothing errors. + +The symptom is a hit that has no row behind it, which a `LEFT JOIN` exposes: + +```bash +dirsql query " +SELECT n.slug +FROM notes_fts LEFT JOIN notes AS n ON n.rowid = notes_fts.rowid +WHERE notes_fts MATCH 'frost'" -c ./.dirsql.toml --persist +``` + +```json +[{"slug":null}] +``` + +A `null` slug is a stale index entry — the base row is gone and the index did +not hear about it. With the `notes_ad` trigger above, the deleted file's entry +is gone too and the query returns nothing. + +Note the `--persist` flag: without it every run rebuilds the index from +scratch, so a broken delete trigger cannot show itself. Deletes are visible +against a [warm cache](./persist.md) and through the watcher, which is exactly +where the staleness would have bitten. + +## Vector search (`vec0`) + +Two pieces beyond SQLite: the +[`sqlite-vec`](https://github.com/asg017/sqlite-vec) extension supplies the +`vec0` virtual table, and +[`dirsql-plugin-embeddings`](../plugins.md#dirsql-plugin-embeddings) supplies +`embed()`. Installing the plugin brings both — its config fragment declares +the extension too, and the launcher +[discovers it](../reference/cli.md#plugins): + +```bash +uvx --with dirsql-plugin-embeddings dirsql query "…" -c ./.dirsql.toml +``` + +Loading `sqlite-vec` by hand instead (another runtime, a pinned build) is +[Load a SQLite extension](./load-extension.md). + +### Find your model's dimension + +A `vec0` column declares a fixed width, so the template needs the number of +components your model emits. `embed()` returns the vector as JSON text, so ask +it: + +```bash +uvx --with dirsql-plugin-embeddings dirsql query \ + "SELECT json_array_length(embed('probe', 'minishlab/potion-retrieval-32M')) AS dims" +``` + +Run it once per model id you intend to use, and substitute the answer for the +`512` in the template below. (The first call for a model downloads it — see +[model](../plugins.md#model).) Getting it wrong is at least loud — the build fails naming both +numbers: + +``` +dirsql query: failed to load config: SQLite error: Dimension mismatch for +inserted vector for the "embedding" column. Expected 8 dimensions but received 4. +``` + +### The template + +```toml +[[table]] +name = "notes" +glob = "notes/**/*.md" +on-file = "python3 extract.py {path}" +ddl = ''' +CREATE TABLE notes (slug TEXT, title TEXT, body TEXT); + +-- Width must equal what the probe printed for the model id named below. +CREATE VIRTUAL TABLE notes_vec USING vec0(embedding float[512]); +CREATE TRIGGER notes_vi AFTER INSERT ON notes BEGIN + INSERT INTO notes_vec(rowid, embedding) + VALUES (new.rowid, embed(new.body, 'minishlab/potion-retrieval-32M')); +END; +CREATE TRIGGER notes_vd AFTER DELETE ON notes BEGIN + DELETE FROM notes_vec WHERE rowid = old.rowid; +END; +''' +``` + +The delete side is an ordinary `DELETE`, not FTS5's `'delete'` command row — +`vec0` is a normal-looking table that way. + +Query it with `sqlite-vec`'s KNN form, joined back on `rowid`: + +```bash +uvx --with dirsql-plugin-embeddings dirsql query " +SELECT n.slug, v.distance +FROM notes_vec AS v JOIN notes AS n ON n.rowid = v.rowid +WHERE v.embedding MATCH embed('how do I cook pasta?', 'minishlab/potion-retrieval-32M') + AND k = 3 +ORDER BY v.distance" -c ./.dirsql.toml +``` + +`MATCH` plus `k = 3` is what makes this a top-k scan rather than a full one — +`vec0` uses `k`, not `LIMIT`. `distance` is supplied by the virtual table, +closest first. + +**Name the same model id in the trigger and the query.** Nothing checks that +the two agree, and vectors from different models are not comparable. If the +models happen to share a width the mismatch is entirely silent — rankings just +get worse. Spelling the id out in both places, rather than leaning on the +default in either, is the cheap defence; it also puts the model in the +[config hash](../reference/config.md#batch-ddl), so changing it rebuilds the +index instead of mixing old vectors with new. + +Rows are embedded once, at ingest, and cached on disk by content and model +([vector cache](../plugins.md#vector-cache)). A search then costs one +`embed()` call for the query text plus an in-process scan. + +::: warning Editing `ddl` wedges a persisted `vec0` cache +Under `--persist`, editing any part of a `ddl` batch whose cache holds a +`vec0` table currently fails with `SQLite error: no such module: vec0`, and +keeps failing until the cache is deleted (`rm -rf /.dirsql`). Tracked in +[#1008](https://github.com/thekevinscott/dirsql/issues/1008); FTS5 is +unaffected. +::: + +## What a rebuild does + +`ddl` runs once, when the table is created. Editing any character of it +changes the config hash, which drops a persisted cache and re-ingests every +file — so a new index, a different tokenizer or a changed model id all rebuild +from scratch rather than leaving a half-migrated index behind. The full rules +are in [Batch `ddl`](../reference/config.md#batch-ddl). + +## Going further + +- Ranked semantic search over files with no config at all — + [Search documents by meaning](./search-by-meaning.md). +- Why indexes belong to declared tables and not to path-tables — + [how `dirsql` thinks](../explanation.md). diff --git a/docs/reference/path-tables.md b/docs/reference/path-tables.md index fd76dd138..1f155fb22 100644 --- a/docs/reference/path-tables.md +++ b/docs/reference/path-tables.md @@ -190,7 +190,13 @@ That is the right trade for a hundreds-of-files, run-it-once question. When the same tree is queried repeatedly, or is large, declare a [table](/reference/config) for it instead — a declared table is indexed on build, kept fresh by the watcher, and (with `--persist`) survives restarts, so -its rows are read from SQLite rather than re-walked each time. +its rows are read from SQLite rather than re-walked each time. It is also the +only place a keyword or vector index can live +([templates](/howto/search-indexes)). + +Neither kind is the better one: each buys what the other gives up, and the +choice is stated as one trade in +[how `dirsql` thinks](/explanation#two-table-kinds-opposite-trade-offs). ## Skip rules