diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index f8a099548..8b72364a3 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -31,6 +31,55 @@ prior `[table.columns]` source-dispatch and the `format` / `each` config keys were ripped out; the per-format parser zoo (`Format::Json`, `Csv`, `Yaml`, `Toml`, `Frontmatter`, …) is gone. +## Two table kinds, opposite trade-offs + +The two kinds above are not a general mechanism and a limited one. They are a +choice the user makes per table, and each buys what the other gives up. + +A **path-table** costs nothing to reach for. `FROM './**/*.json'` needs no +config file, no DDL and no hook; the table is minted in `temp` when the +statement runs, so it lives and dies with the connection. Nothing is stored: +the filesystem is walked during the query and `content` is read only if a +column names it. What you get for that is freshness with no machinery — a file +written a second ago is in the next statement's results, because there is no +copy of the tree to go stale. + +A **declared table** costs a `[[table]]` entry and an `on-file` hook. In +exchange it is an ordinary SQLite table: rows are inserted during the build, +the watcher keeps them current as files change, `--persist` carries them +across restarts, and its `ddl` batch can create anything SQLite can — a +`CREATE INDEX`, an FTS5 vtab, a `vec0` vtab, and the triggers that keep them +current. + +The asymmetry is structural, not an unfinished path-table. Indexes, +persistence and the watcher all need rows that outlive the statement that +asked for them, and a path-table deliberately has none — it stores nothing, +which is exactly why it is always current and needs no setup. A table cannot +both hold no state and index its state. + +So the trade reads: + +| | Path-table | Declared table | +| --- | --- | --- | +| Setup | none | `[[table]]` entry + `on-file` hook | +| Cost per query | walks the matched tree | reads rows from SQLite | +| Freshness | always current | watcher-maintained | +| Survives restart | no | with `--persist` | +| Indexes (B-tree, FTS5, vector) | no | in the `ddl` batch | + +Reach for a path-table when the question is ad hoc, or the tree is small +enough that walking it is not worth avoiding. Declare a table when the same +tree is queried repeatedly, is large, or wants keyword or vector search. The +two are ordinary SQLite tables once resolved, so a query is free to join one +against the other. + +The user-facing versions of both sides are +[Query files without a config](https://thekevinscott.github.io/dirsql/howto/query-without-config) +and +[Define tables for your files](https://thekevinscott.github.io/dirsql/howto/define-tables); +the index templates are in +[Add a search index to a table](https://thekevinscott.github.io/dirsql/howto/search-indexes). + ## Read-only by design **dirsql never modifies the directory it indexes.** It opens files for reading @@ -165,7 +214,7 @@ reinvent what SQLite expresses natively*. `validate_identifier` remains: `name` is spliced into `format!()`-built INSERT/DELETE SQL, so it is still the injection guard. -**Named tables are real; path-tables are virtual.** A declared `[[table]]` (or programmatic `Table`) is a real SQLite table whose rows are inserted on build and maintained by the watcher — the `db` module above. A [path-table](../docs/reference/path-tables.md) (`SELECT * FROM './'`, epic path-as-table) is a `dirsql_path` **virtual table** (`vtab.rs` / `path_table.rs`, rusqlite's `vtab` feature): no rows are stored, no reconcile or watcher runs, and the filesystem is walked live at query time (`xFilter`/`xNext` enumerate matched files; `xColumn` supplies stat values and lazily reads `content`). SQLite stays the entire query engine either way; the vtab only enumerates rows and supplies column values. Path-tables are registered on demand — see *Query execution* below. +**Named tables are real; path-tables are virtual.** A declared `[[table]]` (or programmatic `Table`) is a real SQLite table whose rows are inserted on build and maintained by the watcher — the `db` module above. A [path-table](https://thekevinscott.github.io/dirsql/reference/path-tables) (`SELECT * FROM './'`, epic path-as-table) is a `dirsql_path` **virtual table** (`vtab.rs` / `path_table.rs`, rusqlite's `vtab` feature): no rows are stored, no reconcile or watcher runs, and the filesystem is walked live at query time (`xFilter`/`xNext` enumerate matched files; `xColumn` supplies stat values and lazily reads `content`). SQLite stays the entire query engine either way; the vtab only enumerates rows and supplies column values. Path-tables are registered on demand — see *Query execution* below. ### `scanner` -- Directory traversal diff --git a/docs/.vitepress/config.ts b/docs/.vitepress/config.ts index 074fb437a..a03c00821 100644 --- a/docs/.vitepress/config.ts +++ b/docs/.vitepress/config.ts @@ -56,6 +56,7 @@ export default defineConfig({ { text: 'Parse your files into columns', link: '/howto/parse-files-into-columns' }, { text: 'Query JSON file contents', link: '/howto/query-json' }, { text: 'Search documents by meaning', link: '/howto/search-by-meaning' }, + { text: 'Add a search index to a table', link: '/howto/search-indexes' }, { text: "Skip files you don't want indexed", link: '/howto/skip-files' }, { text: 'Load a SQLite extension', link: '/howto/load-extension' }, { text: 'Keep the index across restarts', link: '/howto/persist' }, diff --git a/docs/howto/define-tables.md b/docs/howto/define-tables.md index ae0294e9c..a956b981d 100644 --- a/docs/howto/define-tables.md +++ b/docs/howto/define-tables.md @@ -107,7 +107,9 @@ maintain nothing. `name` still has to be the row table (`posts`); the virtual table lives beside it under its own name. The batch runs once, when the table is created, and editing any part of it rebuilds a [persistent cache](./persist.md) from scratch. Full rules: -[Batch `ddl`](../reference/config.md#batch-ddl). +[Batch `ddl`](../reference/config.md#batch-ddl); pasteable FTS5 and vector +templates, with the trigger mistakes that fail silently, are in +[Add a search index to a table](./search-indexes.md). ## Going further diff --git a/docs/howto/query-without-config.md b/docs/howto/query-without-config.md index e2981c6c1..23d34653e 100644 --- a/docs/howto/query-without-config.md +++ b/docs/howto/query-without-config.md @@ -55,7 +55,9 @@ question over a few hundred files, wasteful for a large tree you query repeatedly. When that day comes, [declare a table](./define-tables.md): it is indexed once, kept fresh by the watcher, and can persist across restarts. The zero-config query is the floor; a named table is the escalation path — nothing -you write here has to be thrown away to get there. +you write here has to be thrown away to get there. The two sides in full, and +why indexes belong to only one of them, are in +[how `dirsql` thinks](../explanation.md#two-table-kinds-opposite-trade-offs). ## Notes diff --git a/docs/howto/search-indexes.md b/docs/howto/search-indexes.md new file mode 100644 index 000000000..f3c137bac --- /dev/null +++ b/docs/howto/search-indexes.md @@ -0,0 +1,220 @@ +# Add a search index to a table + +Make a declared table answer search queries instead of scanning: a B-tree for +lookups, an FTS5 index for keywords, a `vec0` index for meaning. All three are +statements in the table's [`ddl` batch](../reference/config.md#batch-ddl), so +SQLite does the work and `dirsql` maintains nothing. + +The templates below are meant to be pasted and edited. Both are shaped around +one worked example — notes in `notes/*.md`, parsed into a `notes` table by an +`extract.py` that prints `{"slug": …, "title": …, "body": …}` per file, exactly +as in [Define tables for your files](./define-tables.md). + +## Why triggers, and why only two + +`dirsql` writes file rows with plain `INSERT` and `DELETE`; an update is a +delete and an insert in one transaction, and there is no `UPDATE` path on user +rows. So an insert trigger and a delete trigger cover every way a row can +change — during the initial build, on each [watcher](./react-to-changes.md) +event, and when a [persistent cache](./persist.md) reconciles a tree that +moved on. There is no third trigger to write and no maintenance command to run. + +The triggers are also the part that goes wrong, which is why the templates +spell them out rather than hiding them behind an abstraction. + +## Keyword search (FTS5) + +FTS5 ships inside SQLite — no extension, no install. Declare an +external-content index beside the row table and let the two triggers feed it: + +```toml +[[table]] +name = "notes" +glob = "notes/**/*.md" +on-file = "python3 extract.py {path}" +ddl = ''' +CREATE TABLE notes (slug TEXT, title TEXT, body TEXT); + +CREATE VIRTUAL TABLE notes_fts USING fts5( + body, + content='notes', + content_rowid='rowid', + tokenize='porter unicode61' +); +CREATE TRIGGER notes_ai AFTER INSERT ON notes BEGIN + INSERT INTO notes_fts(rowid, body) VALUES (new.rowid, new.body); +END; +CREATE TRIGGER notes_ad AFTER DELETE ON notes BEGIN + INSERT INTO notes_fts(notes_fts, rowid, body) + VALUES ('delete', old.rowid, old.body); +END; +''' +``` + +`content='notes'` makes the index store only its own search structures and +read column values back from `notes`, so the text is not duplicated. +`tokenize='porter unicode61'` stems English on top of the default +case-folding, so *deploying* matches *deploy*; drop the `porter` half to match +whole words only. + +Query the index and join back to the row table on `rowid`: + +```bash +dirsql query " +SELECT n.slug, + bm25(notes_fts) AS score, + snippet(notes_fts, 0, '[', ']', '…', 8) AS excerpt +FROM notes_fts JOIN notes AS n ON n.rowid = notes_fts.rowid +WHERE notes_fts MATCH 'deploy' +ORDER BY score" -c ./.dirsql.toml +``` + +```json +[{"excerpt":"Rebase before you [deploy]. Keep pull requests small.","score":-1.0476190476190478e-6,"slug":"branches"}, + {"excerpt":"…the spaghetti goes in. [Deploy] the garlic late.","score":-8.461538461538463e-7,"slug":"pasta"}] +``` + +**`bm25()` returns negative scores, and better matches are more negative** — +so `ORDER BY score` ascending is best-first, with no `DESC`. `snippet()`'s +arguments are the table, the column index, the open and close markers, the +ellipsis, and the token budget. + +### Check the delete trigger + +A wrong or missing delete trigger fails **silently**: the index keeps rows for +files that are gone, and they keep matching. Nothing errors. + +The symptom is a hit that has no row behind it, which a `LEFT JOIN` exposes: + +```bash +dirsql query " +SELECT n.slug +FROM notes_fts LEFT JOIN notes AS n ON n.rowid = notes_fts.rowid +WHERE notes_fts MATCH 'frost'" -c ./.dirsql.toml --persist +``` + +```json +[{"slug":null}] +``` + +A `null` slug is a stale index entry — the base row is gone and the index did +not hear about it. With the `notes_ad` trigger above, the deleted file's entry +is gone too and the query returns nothing. + +Note the `--persist` flag: without it every run rebuilds the index from +scratch, so a broken delete trigger cannot show itself. Deletes are visible +against a [warm cache](./persist.md) and through the watcher, which is exactly +where the staleness would have bitten. + +## Vector search (`vec0`) + +Two pieces beyond SQLite: the +[`sqlite-vec`](https://github.com/asg017/sqlite-vec) extension supplies the +`vec0` virtual table, and +[`dirsql-plugin-embeddings`](../plugins.md#dirsql-plugin-embeddings) supplies +`embed()`. Installing the plugin brings both — its config fragment declares +the extension too, and the launcher +[discovers it](../reference/cli.md#plugins): + +```bash +uvx --with dirsql-plugin-embeddings dirsql query "…" -c ./.dirsql.toml +``` + +Loading `sqlite-vec` by hand instead (another runtime, a pinned build) is +[Load a SQLite extension](./load-extension.md). + +### Find your model's dimension + +A `vec0` column declares a fixed width, so the template needs the number of +components your model emits. `embed()` returns the vector as JSON text, so ask +it: + +```bash +uvx --with dirsql-plugin-embeddings dirsql query \ + "SELECT json_array_length(embed('probe', 'minishlab/potion-retrieval-32M')) AS dims" +``` + +Run it once per model id you intend to use, and substitute the answer for the +`512` in the template below. (The first call for a model downloads it — see +[model](../plugins.md#model).) Getting it wrong is at least loud — the build fails naming both +numbers: + +``` +dirsql query: failed to load config: SQLite error: Dimension mismatch for +inserted vector for the "embedding" column. Expected 8 dimensions but received 4. +``` + +### The template + +```toml +[[table]] +name = "notes" +glob = "notes/**/*.md" +on-file = "python3 extract.py {path}" +ddl = ''' +CREATE TABLE notes (slug TEXT, title TEXT, body TEXT); + +-- Width must equal what the probe printed for the model id named below. +CREATE VIRTUAL TABLE notes_vec USING vec0(embedding float[512]); +CREATE TRIGGER notes_vi AFTER INSERT ON notes BEGIN + INSERT INTO notes_vec(rowid, embedding) + VALUES (new.rowid, embed(new.body, 'minishlab/potion-retrieval-32M')); +END; +CREATE TRIGGER notes_vd AFTER DELETE ON notes BEGIN + DELETE FROM notes_vec WHERE rowid = old.rowid; +END; +''' +``` + +The delete side is an ordinary `DELETE`, not FTS5's `'delete'` command row — +`vec0` is a normal-looking table that way. + +Query it with `sqlite-vec`'s KNN form, joined back on `rowid`: + +```bash +uvx --with dirsql-plugin-embeddings dirsql query " +SELECT n.slug, v.distance +FROM notes_vec AS v JOIN notes AS n ON n.rowid = v.rowid +WHERE v.embedding MATCH embed('how do I cook pasta?', 'minishlab/potion-retrieval-32M') + AND k = 3 +ORDER BY v.distance" -c ./.dirsql.toml +``` + +`MATCH` plus `k = 3` is what makes this a top-k scan rather than a full one — +`vec0` uses `k`, not `LIMIT`. `distance` is supplied by the virtual table, +closest first. + +**Name the same model id in the trigger and the query.** Nothing checks that +the two agree, and vectors from different models are not comparable. If the +models happen to share a width the mismatch is entirely silent — rankings just +get worse. Spelling the id out in both places, rather than leaning on the +default in either, is the cheap defence; it also puts the model in the +[config hash](../reference/config.md#batch-ddl), so changing it rebuilds the +index instead of mixing old vectors with new. + +Rows are embedded once, at ingest, and cached on disk by content and model +([vector cache](../plugins.md#vector-cache)). A search then costs one +`embed()` call for the query text plus an in-process scan. + +::: warning Editing `ddl` wedges a persisted `vec0` cache +Under `--persist`, editing any part of a `ddl` batch whose cache holds a +`vec0` table currently fails with `SQLite error: no such module: vec0`, and +keeps failing until the cache is deleted (`rm -rf /.dirsql`). Tracked in +[#1008](https://github.com/thekevinscott/dirsql/issues/1008); FTS5 is +unaffected. +::: + +## What a rebuild does + +`ddl` runs once, when the table is created. Editing any character of it +changes the config hash, which drops a persisted cache and re-ingests every +file — so a new index, a different tokenizer or a changed model id all rebuild +from scratch rather than leaving a half-migrated index behind. The full rules +are in [Batch `ddl`](../reference/config.md#batch-ddl). + +## Going further + +- Ranked semantic search over files with no config at all — + [Search documents by meaning](./search-by-meaning.md). +- Why indexes belong to declared tables and not to path-tables — + [how `dirsql` thinks](../explanation.md). diff --git a/docs/reference/path-tables.md b/docs/reference/path-tables.md index fd76dd138..1f155fb22 100644 --- a/docs/reference/path-tables.md +++ b/docs/reference/path-tables.md @@ -190,7 +190,13 @@ That is the right trade for a hundreds-of-files, run-it-once question. When the same tree is queried repeatedly, or is large, declare a [table](/reference/config) for it instead — a declared table is indexed on build, kept fresh by the watcher, and (with `--persist`) survives restarts, so -its rows are read from SQLite rather than re-walked each time. +its rows are read from SQLite rather than re-walked each time. It is also the +only place a keyword or vector index can live +([templates](/howto/search-indexes)). + +Neither kind is the better one: each buys what the other gives up, and the +choice is stated as one trade in +[how `dirsql` thinks](/explanation#two-table-kinds-opposite-trade-offs). ## Skip rules