Skip to content

docs: state the path-table/declared-table trade-off, ship FTS5 and vector index templates - #1010

Merged
thekevinscott merged 1 commit into
mainfrom
claude/tackle-960-lrxmr3
Aug 21, 2026
Merged

thekevinscott merged 1 commit into
mainfrom
claude/tackle-960-lrxmr3

Conversation

@thekevinbot

@thekevinbot thekevinbot commented Aug 21, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes #960

Two docs gaps, both unblocked by #956's wide-open ddl batch.

1. The trade-off, stated once

The facts were true but scattered — path-tables.md §Freshness, db.rs:100 on temp, vtab.rs:33 on the hidden content column — so nothing said these are two mechanisms with opposite trade-offs rather than a general one and a limited one.

Explanation (ARCHITECTURE.md → /explanation) gains Two table kinds, opposite trade-offs: a path-table stores nothing, which is exactly why it is always current, needs no setup, and can carry no index; a declared table costs a [[table]] entry and an on-file hook and buys the watcher, --persist, and a ddl batch to hang indexes off. The asymmetry is framed as structural rather than an unfinished path-table — indexes, persistence and the watcher all need rows that outlive the statement, and a path-table deliberately has none. A five-row table summarizes, and a closing line says which to reach for.

The three pages that used to gesture at it now link there instead of half-restating it: reference/path-tables.md (§When to promote), howto/query-without-config.md (§Graduate to a named table), howto/define-tables.md.

2. Copy-pasteable index templates

New how-to: Add a search index to a table (docs/howto/search-indexes.md), sidebar entry added after Search documents by meaning.

Per #960's framing these are shipped as templates, not abstractions, because the triggers are the error-prone part and their failure mode is silent. Both are built on the same worked notes example as define-tables.md.

  • FTS5 — external-content vtab + insert/delete triggers, porter unicode61, queried with MATCH + bm25() + snippet() joined back on rowid. Includes the two things a reader gets wrong: bm25() scores are negative (so best-first is ORDER BY score with no DESC), and a §Check the delete trigger section giving the diagnostic — a LEFT JOIN surfaces a stale entry as a null base row, and it only shows up under --persist, since a cold run rebuilds the index and hides the bug.
  • Vector — sqlite-vec's vec0 + embed() triggers, queried with MATCH … AND k = N joined back on rowid, ordered by distance. Includes a json_array_length(embed(…)) probe for the column width (guessing it is the other easy way to paste something that doesn't work; that failure at least errors loudly, and the message is quoted), and a warning that the model id must be named on both sides — a same-width model mismatch is entirely silent.

Mechanics are linked to Batch ddl in reference/config.md, not restated, per #960's note about reusing #994's section.

search-by-meaning.md is untouched — that rewrite is #1003.

Verified by running it

Everything below was run against the real CLI (cargo build -p dirsql --features cli), real SQLite, real files on disk — no mocks.

FTS5, exactly as the template is written:

  • Build + MATCH/bm25()/snippet() — the JSON output in the page is copied verbatim from the run.
  • Delete propagation: populate under --persist, delete a file, re-run → index rows drop 2 → 1.
  • Control: the same config with the _ad trigger removed stays at 2, and the LEFT JOIN returns [{"slug":null}]. This is what makes the diagnostic in the page a real one rather than a plausible one — the check discriminates.

Vector, against sqlite-vec 0.1.9:

  • Trigger-time insert of the JSON-text vector into vec0, KNN query joined back on rowid, delete propagation (3 → 2), the two-arg embed(text, model) form in both trigger and query, and the dimension probe.
  • Wrong width fails loudly at load: Dimension mismatch … Expected 8 dimensions but received 4 — the error quoted in the page.

Two honest limits on that verification:

  1. huggingface.co is blocked by this sandbox's proxy policy, so the real model could not run. The vector mechanics were exercised with a stub worker speaking the plugin's NDJSON protocol at a small fixed width. What this does not verify is the actual component count of minishlab/potion-retrieval-32M — which is why the page makes the probe the source of truth and marks the float[512] in the template as a slot to substitute (-- Width must equal what the probe printed…) rather than asserting the number.
  2. For the same reason the vector section shows no result rows; inventing distances would have been the only way to fill them.

Found a bug: #1008

Editing ddl on a --persist cache holding a vec0 table wedges it permanently with SQLite error: no such module: vec0, recoverable only by deleting the cache directory. prepare_persist (lib.rs:1491) opens the cache and sweeps it at :1504 without loading [[dirsql.extension]], so drop_user_tables can't drop a virtual table whose module came from an extension. #956's a_rebuild_sweeps_a_cache_holding_virtual_tables covers this exact flow but with FTS5, which is compiled into SQLite and so always registered.

Filed as #1008 with a minimal repro rather than folded in here — it's a Rust core fix with its own red/green, changelog and attestation obligations. The vector template carries a warning box pointing at it, since a reader following that template under --persist will hit it.

Drive-by fix

ARCHITECTURE.md's existing [path-table](../docs/reference/path-tables.md) link rendered as ./../docs/reference/path-tables.html once included into /dirsql/explanation.html, escaping the site's /dirsql/ base and 404ing. There is no link form that works both on the site and when reading the file on GitHub, so that one — and the links this PR adds — use absolute doc-site URLs. Confirmed in the built output: no rendered page has a base-escaping href any more.

E2E Verification

N/A - docs/lint/typo only

Changelog / Migrations

  • Changelog fragment added under each changed package's changelog.d/ (or: skip-changelog trailer on a commit with reason) — not required: the changelog-gate scopes to non-test source under packages/*/ or plugins/*/. This PR touches only docs/** and the root ARCHITECTURE.md, which is a root prose file.
  • Migration fragment added under the package's migrations.d/ (or: not required -- additive/bugfix only) — not required: no public API change.

PARITY.md unchanged: no SDK public API is touched.

Manual run

cd docs && pnpm install && pnpm build   # clean, 20 pages rendered

VitePress build passes (it fails on broken links), the new page renders at /dirsql/howto/search-indexes.html, and the anchor the three cross-links point at resolves — id="two-table-kinds-opposite-trade-offs" is present in dist/explanation.html.

…ates

Two gaps, one page's worth of work.

The trade-off between the two table kinds was true but scattered across the
path-tables reference, the vtab's hidden `content` column and `db.rs`'s use of
`temp`. Explanation now states it as one deliberate choice: a path-table stores
nothing, which is exactly why it is always current, needs no setup, and can
carry no index; a declared table costs a config entry and buys the watcher,
persistence and a `ddl` batch to hang indexes off. The asymmetry is structural
-- a table cannot both hold no state and index its state -- so the pages that
used to gesture at it (path-tables, query-without-config, define-tables) now
link there instead.

The templates are the error-prone half. `ddl` has been a wide-open batch since
#956, and the triggers are where it goes wrong: a malformed FTS5 'delete'
trigger leaves stale rows in results and never errors. The new how-to ships
both templates whole rather than abstracting them, plus the diagnostics --
a LEFT JOIN that exposes stale FTS entries as a null base row, and a
`json_array_length(embed(...))` probe for the vec0 column width, since guessing
it is the other easy way to paste something that does not work.

Both were run against real dirsql and real SQLite while writing: FTS5 including
delete propagation over a warm cache (and a no-delete-trigger control that
stays stale, confirming the check discriminates), and vec0 against sqlite-vec
0.1.9 for trigger-time insert, KNN, and delete. Uncovered #1008 doing so, noted
inline on the vector template.

Links added inside ARCHITECTURE.md use absolute doc-site URLs. The `../docs/`
form already there rendered as `./../docs/...` once included into the
Explanation page, dropping the site's `/dirsql/` base -- that one existing
occurrence is fixed the same way.

Fixes #960

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SFEXWZPYdkXVkRjjy8tvdZ
@thekevinscott
thekevinscott merged commit 9b23473 into main Aug 21, 2026
2 checks passed
@thekevinscott
thekevinscott deleted the claude/tackle-960-lrxmr3 branch August 21, 2026 02:46
thekevinbot pushed a commit that referenced this pull request Aug 21, 2026
…plate

#1010 landed while this branch was open and put the vec0 template on its own
page, search-indexes.md, which frames search-by-meaning.md as the no-config
guide. Git merges the two cleanly because neither touched the other's file,
but the result would have taught the same recipe twice, with different
details. So the division of labour follows the page that landed: the index
recipe lives once, on search-indexes.md, and search-by-meaning.md keeps the
path-table shape and links out.

What #1003 asked for survives that split. The two shapes and their trade are
stated up front as a table -- setup, freshness, per-query cost, top-k form --
along with the caveat that an index only pays off when the table outlives the
query, since a one-shot run against an ephemeral index re-embeds the corpus
before answering. The "Top-k is LIMIT k" tip is retired: it existed to explain
why MATCH ... AND k = did not apply, which is now a row in that table and one
clause in the walk-through.

Two fixes to the template on search-indexes.md, both found by running it:

vec0 defaults to L2, which is not the metric the rest of dirsql's semantic
search uses -- vec_distance_cosine() backs both the plugin one-liner and
search-by-meaning. The two agree only for vectors of equal length, so
documents that embed to different magnitudes rank differently: over three
notes, L2 returned a, c, b where cosine returned a, b, c. The declaration now
carries distance_metric=cosine.

embed(NULL) is NULL and vec0 rejects a NULL vector, so one row with missing
text failed the whole table load rather than its own insert. The insert
trigger now carries WHEN new.body IS NOT NULL; the row still lands, only its
vector is skipped.

Closes #1003
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Docs: path-table vs declared-table tradeoff; FTS5 and vector index templates

3 participants