Repository navigation
docs: state the path-table/declared-table trade-off, ship FTS5 and vector index templates - #1010
Merged
Merged
Conversation
…ates Two gaps, one page's worth of work. The trade-off between the two table kinds was true but scattered across the path-tables reference, the vtab's hidden `content` column and `db.rs`'s use of `temp`. Explanation now states it as one deliberate choice: a path-table stores nothing, which is exactly why it is always current, needs no setup, and can carry no index; a declared table costs a config entry and buys the watcher, persistence and a `ddl` batch to hang indexes off. The asymmetry is structural -- a table cannot both hold no state and index its state -- so the pages that used to gesture at it (path-tables, query-without-config, define-tables) now link there instead. The templates are the error-prone half. `ddl` has been a wide-open batch since #956, and the triggers are where it goes wrong: a malformed FTS5 'delete' trigger leaves stale rows in results and never errors. The new how-to ships both templates whole rather than abstracting them, plus the diagnostics -- a LEFT JOIN that exposes stale FTS entries as a null base row, and a `json_array_length(embed(...))` probe for the vec0 column width, since guessing it is the other easy way to paste something that does not work. Both were run against real dirsql and real SQLite while writing: FTS5 including delete propagation over a warm cache (and a no-delete-trigger control that stays stale, confirming the check discriminates), and vec0 against sqlite-vec 0.1.9 for trigger-time insert, KNN, and delete. Uncovered #1008 doing so, noted inline on the vector template. Links added inside ARCHITECTURE.md use absolute doc-site URLs. The `../docs/` form already there rendered as `./../docs/...` once included into the Explanation page, dropping the site's `/dirsql/` base -- that one existing occurrence is fixed the same way. Fixes #960 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SFEXWZPYdkXVkRjjy8tvdZ
thekevinbot
pushed a commit
that referenced
this pull request
Aug 21, 2026
…plate #1010 landed while this branch was open and put the vec0 template on its own page, search-indexes.md, which frames search-by-meaning.md as the no-config guide. Git merges the two cleanly because neither touched the other's file, but the result would have taught the same recipe twice, with different details. So the division of labour follows the page that landed: the index recipe lives once, on search-indexes.md, and search-by-meaning.md keeps the path-table shape and links out. What #1003 asked for survives that split. The two shapes and their trade are stated up front as a table -- setup, freshness, per-query cost, top-k form -- along with the caveat that an index only pays off when the table outlives the query, since a one-shot run against an ephemeral index re-embeds the corpus before answering. The "Top-k is LIMIT k" tip is retired: it existed to explain why MATCH ... AND k = did not apply, which is now a row in that table and one clause in the walk-through. Two fixes to the template on search-indexes.md, both found by running it: vec0 defaults to L2, which is not the metric the rest of dirsql's semantic search uses -- vec_distance_cosine() backs both the plugin one-liner and search-by-meaning. The two agree only for vectors of equal length, so documents that embed to different magnitudes rank differently: over three notes, L2 returned a, c, b where cosine returned a, b, c. The declaration now carries distance_metric=cosine. embed(NULL) is NULL and vec0 rejects a NULL vector, so one row with missing text failed the whole table load rather than its own insert. The insert trigger now carries WHEN new.body IS NOT NULL; the row still lands, only its vector is skipped. Closes #1003
3 of 8 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #960
Two docs gaps, both unblocked by #956's wide-open
ddlbatch.1. The trade-off, stated once
The facts were true but scattered —
path-tables.md§Freshness,db.rs:100ontemp,vtab.rs:33on the hiddencontentcolumn — so nothing said these are two mechanisms with opposite trade-offs rather than a general one and a limited one.Explanation (
ARCHITECTURE.md→/explanation) gains Two table kinds, opposite trade-offs: a path-table stores nothing, which is exactly why it is always current, needs no setup, and can carry no index; a declared table costs a[[table]]entry and anon-filehook and buys the watcher,--persist, and addlbatch to hang indexes off. The asymmetry is framed as structural rather than an unfinished path-table — indexes, persistence and the watcher all need rows that outlive the statement, and a path-table deliberately has none. A five-row table summarizes, and a closing line says which to reach for.The three pages that used to gesture at it now link there instead of half-restating it:
reference/path-tables.md(§When to promote),howto/query-without-config.md(§Graduate to a named table),howto/define-tables.md.2. Copy-pasteable index templates
New how-to: Add a search index to a table (
docs/howto/search-indexes.md), sidebar entry added after Search documents by meaning.Per #960's framing these are shipped as templates, not abstractions, because the triggers are the error-prone part and their failure mode is silent. Both are built on the same worked
notesexample asdefine-tables.md.porter unicode61, queried withMATCH+bm25()+snippet()joined back onrowid. Includes the two things a reader gets wrong:bm25()scores are negative (so best-first isORDER BY scorewith noDESC), and a §Check the delete trigger section giving the diagnostic — aLEFT JOINsurfaces a stale entry as anullbase row, and it only shows up under--persist, since a cold run rebuilds the index and hides the bug.sqlite-vec'svec0+embed()triggers, queried withMATCH … AND k = Njoined back onrowid, ordered bydistance. Includes ajson_array_length(embed(…))probe for the column width (guessing it is the other easy way to paste something that doesn't work; that failure at least errors loudly, and the message is quoted), and a warning that the model id must be named on both sides — a same-width model mismatch is entirely silent.Mechanics are linked to
Batch ddlinreference/config.md, not restated, per #960's note about reusing #994's section.search-by-meaning.mdis untouched — that rewrite is #1003.Verified by running it
Everything below was run against the real CLI (
cargo build -p dirsql --features cli), real SQLite, real files on disk — no mocks.FTS5, exactly as the template is written:
MATCH/bm25()/snippet()— the JSON output in the page is copied verbatim from the run.--persist, delete a file, re-run → index rows drop 2 → 1._adtrigger removed stays at 2, and theLEFT JOINreturns[{"slug":null}]. This is what makes the diagnostic in the page a real one rather than a plausible one — the check discriminates.Vector, against sqlite-vec 0.1.9:
vec0, KNN query joined back on rowid, delete propagation (3 → 2), the two-argembed(text, model)form in both trigger and query, and the dimension probe.Dimension mismatch … Expected 8 dimensions but received 4— the error quoted in the page.Two honest limits on that verification:
huggingface.cois blocked by this sandbox's proxy policy, so the real model could not run. The vector mechanics were exercised with a stub worker speaking the plugin's NDJSON protocol at a small fixed width. What this does not verify is the actual component count ofminishlab/potion-retrieval-32M— which is why the page makes the probe the source of truth and marks thefloat[512]in the template as a slot to substitute (-- Width must equal what the probe printed…) rather than asserting the number.Found a bug: #1008
Editing
ddlon a--persistcache holding avec0table wedges it permanently withSQLite error: no such module: vec0, recoverable only by deleting the cache directory.prepare_persist(lib.rs:1491) opens the cache and sweeps it at:1504without loading[[dirsql.extension]], sodrop_user_tablescan't drop a virtual table whose module came from an extension. #956'sa_rebuild_sweeps_a_cache_holding_virtual_tablescovers this exact flow but with FTS5, which is compiled into SQLite and so always registered.Filed as #1008 with a minimal repro rather than folded in here — it's a Rust core fix with its own red/green, changelog and attestation obligations. The vector template carries a warning box pointing at it, since a reader following that template under
--persistwill hit it.Drive-by fix
ARCHITECTURE.md's existing[path-table](../docs/reference/path-tables.md)link rendered as./../docs/reference/path-tables.htmlonce included into/dirsql/explanation.html, escaping the site's/dirsql/base and 404ing. There is no link form that works both on the site and when reading the file on GitHub, so that one — and the links this PR adds — use absolute doc-site URLs. Confirmed in the built output: no rendered page has a base-escaping href any more.E2E Verification
N/A - docs/lint/typo onlyChangelog / Migrations
changelog.d/(or:skip-changelogtrailer on a commit with reason) — not required: thechangelog-gatescopes to non-test source underpackages/*/orplugins/*/. This PR touches onlydocs/**and the rootARCHITECTURE.md, which is a root prose file.migrations.d/(or: not required -- additive/bugfix only) — not required: no public API change.PARITY.mdunchanged: no SDK public API is touched.Manual run
VitePress build passes (it fails on broken links), the new page renders at
/dirsql/howto/search-indexes.html, and the anchor the three cross-links point at resolves —id="two-table-kinds-opposite-trade-offs"is present indist/explanation.html.