Rewrite search-by-meaning around vec0; enforce the distcheck platform mirror

.github/workflows/platforms-mirror-check.yml

@@ -0,0 +1,44 @@
+name: Platforms Mirror Check
+
+# `packages/ts/src/platforms.ts` is the release source of truth for the published
+# `@dirsql/lib-*` sub-packages; the node distcheck flow keeps a hand-maintained
+# Python subset of it. Nothing checked the correspondence, so it drifted in both
+# directions -- a field on the Python side justified as mirroring a TypeScript
+# property that never existed, and TypeScript-only fields with no counterpart.
+# This holds the two tables to the subset on every change to either, so a new
+# published target cannot be half-added.
+#
+# Deliberately its own workflow rather than a job in internals-distcheck-ci.yml:
+# that lane's path filter does not watch packages/ts, which is exactly where the
+# half-added-target case starts.
+
+on:
+  pull_request:
+    branches: [main]
+    paths:
+      - 'packages/ts/src/platforms.ts'
+      - 'internals/distcheck/src/distcheck/node_flow/platforms.py'
+      - 'internals/checks/src/checks/platforms_mirror/**'
+      - '.github/workflows/platforms-mirror-check.yml'
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+
+permissions:
+  contents: read
+
+jobs:
+  platforms-mirror:
+    name: Platforms mirror
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v6
+
+      - name: Install uv
+        uses: astral-sh/setup-uv@v7
+        with:
+          python-version: "3.13"
+
+      - name: Assert the two platform tables agree on the fields they share
+        run: uv run --project internals/checks dirsql-checks platforms-mirror

docs/howto/search-by-meaning.md

@@ -3,10 +3,10 @@
 Ask a question in plain language and get the closest documents back — even
 when they share no keywords with it. Install
 [`dirsql-plugin-embeddings`](../plugins.md#dirsql-plugin-embeddings) and
-semantic search becomes plain SQL: the plugin's `embed()` function turns
-text into vectors, [`sqlite-vec`](https://github.com/asg017/sqlite-vec)'s
-`vec_distance_cosine()` measures distance, and `ORDER BY … LIMIT` does the
-ranking. No config, no API keys, no services — the model runs locally.
+semantic search becomes plain SQL: the plugin's `embed()` function turns text
+into vectors and loads [`sqlite-vec`](https://github.com/asg017/sqlite-vec),
+which does the distance math. No API keys, no services — the model runs
+locally.
 
 Suppose short notes live in `notes/*.md`:
 
@@ -40,7 +40,32 @@ The first-ever run downloads the model (on the order of a hundred megabytes,
 with progress on stderr); after that it loads from the local cache. Results
 print one `path<TAB>distance` line per match, closest first.
 
-## The SQL behind it
+## Which shape
+
+The one-liner embeds every matched file on every run. That is the right trade
+for a question you ask once, and the wrong one for a corpus you search
+repeatedly — where a stored [`vec0` index](./search-indexes.md#vector-search-vec0)
+embeds each file once, at ingest, and a query embeds only the question:
+
+| | This page: path-table | [Vector index](./search-indexes.md#vector-search-vec0) |
+|---|---|---|
+| Setup | none | a `[[table]]` with a `ddl` batch |
+| Freshness | always current — the walk is the read | watcher-maintained; survives restarts under [`--persist`](./persist.md) |
+| Per query | one `embed()` round trip **per matched file**, plus a walk that reads every file's content | one `embed()` round trip **total**, plus an in-process KNN scan |
+| Top-k | `ORDER BY … LIMIT k` | `MATCH … AND k = …` |
+| Good for | a one-off question, a small corpus, an ad-hoc glob | a corpus you query repeatedly |
+
+The index only pays off when the table outlives the query — under `--persist`,
+or inside a long-running [`dirsql server`](../reference/cli.md). A one-shot
+`dirsql query` against an ephemeral index rebuilds the table, and therefore
+re-embeds the corpus, before it answers; that is strictly more work than the
+subquery below. The full recipe — the width probe, the `ddl` batch, both
+triggers, and what a model-id edit costs — is
+[Add a search index to a table](./search-indexes.md#vector-search-vec0).
+
+The rest of this page is the zero-setup shape.
+
+## The SQL behind the one-liner
 
 The one-liner generates and runs ordinary `dirsql` SQL, and you can write it
 yourself when you want more than ranked paths — a different projection, a
@@ -79,7 +104,9 @@ Reading the query inside-out:
    and its distance are NULL too — and SQLite sorts NULLs *first* ascending,
    so without this line the unrankable files take the top-k slots.
 4. `vec_distance_cosine(...)` computes cosine distance between the two
-   vectors; `ORDER BY distance LIMIT 3` keeps the three nearest.
+   vectors; `ORDER BY distance LIMIT 3` keeps the three nearest. There is no
+   `vec0` table here, so `sqlite-vec`'s `MATCH … AND k = …` does not apply —
+   for a plain expression, `ORDER BY … LIMIT k` *is* its documented top-k.
 
 Structured files compose with SQL's JSON operators — embed one field instead
 of the whole file:
@@ -93,24 +120,19 @@ ORDER BY vec_distance_cosine(emb, embed('local private models'))
 LIMIT 10
 ```
 
-::: tip Top-k is `LIMIT k`
-If you know `sqlite-vec` you may reach for its `MATCH … AND k = 10` idiom.
-That syntax belongs to `sqlite-vec`'s `vec0` virtual table, which the table
-above does not use: a `[[table]]`'s own `name` is always a per-file row table.
-For plain expressions, `sqlite-vec`'s own documented pattern is exactly what
-this guide uses: `ORDER BY vec_distance_cosine(...) LIMIT k`. To get the `vec0`
-idiom instead, declare the `vec0` table alongside the row table in the same
-[`ddl` batch](../reference/config.md#batch-ddl) and fill it from a trigger.
-:::
+The same projection works as an `on-file` hook feeding the indexed shape:
+parse the field in the hook, store it as a column, and the trigger embeds it
+once instead of on every query.
 
 ## Repeat runs are cheap
 
 Computed vectors are cached on disk, keyed on content and model
 ([vector cache](../plugins.md#vector-cache)) — re-running a search over
-unchanged files skips the model entirely and re-embeds only what changed.
-And the plugin costs nothing when idle: a query that never calls `embed()`
-spawns no worker and loads no model
-([zero cost when unused](../plugins.md#zero-cost-when-unused)).
+unchanged files skips the model entirely and re-embeds only what changed. That
+takes the *inference* out of the shape above, but not the walk or the per-file
+round trip; only a stored index removes those. And the plugin costs nothing
+when idle: a query that never calls `embed()` spawns no worker and loads no
+model ([zero cost when unused](../plugins.md#zero-cost-when-unused)).
 
 ## How `embed()` gets into SQL
 

docs/howto/search-indexes.md

@@ -155,8 +155,10 @@ ddl     = '''
 CREATE TABLE notes (slug TEXT, title TEXT, body TEXT);
 
 -- Width must equal what the probe printed for the model id named below.
-CREATE VIRTUAL TABLE notes_vec USING vec0(embedding float[512]);
-CREATE TRIGGER notes_vi AFTER INSERT ON notes BEGIN
+CREATE VIRTUAL TABLE notes_vec
+  USING vec0(embedding float[512] distance_metric=cosine);
+CREATE TRIGGER notes_vi AFTER INSERT ON notes
+WHEN new.body IS NOT NULL BEGIN
   INSERT INTO notes_vec(rowid, embedding)
     VALUES (new.rowid, embed(new.body, 'minishlab/potion-retrieval-32M'));
 END;
@@ -169,6 +171,29 @@ END;
 The delete side is an ordinary `DELETE`, not FTS5's `'delete'` command row —
 `vec0` is a normal-looking table that way.
 
+Two clauses in there are load-bearing:
+
+- **`distance_metric=cosine`.** `vec0` defaults to L2, which is not the metric
+  the rest of dirsql's semantic search uses — `vec_distance_cosine()` backs
+  both the plugin's one-liner and
+  [Search documents by meaning](./search-by-meaning.md). The two agree only for
+  vectors of equal length, so as soon as documents embed to vectors of
+  different magnitude they rank differently: against three notes, L2 returned
+  `a, c, b` where cosine returned `a, b, c`. Declaring the metric makes
+  `distance` the number `vec_distance_cosine()` would compute.
+- **`WHEN new.body IS NOT NULL`.** `embed(NULL)` is `NULL`, and `vec0` rejects
+  a NULL vector outright — so without the guard a single row whose text is
+  missing fails the **whole** table load, not just its own insert:
+
+  ```
+  dirsql query: failed to load config: SQLite error: Inserted vector for the
+  "embedding" column is invalid: Input must have type BLOB (compact format) or
+  TEXT (JSON), found NULL
+  ```
+
+  With it, that row still lands in `notes`; only its vector is skipped. FTS5
+  needs no such guard — it indexes a NULL happily.
+
 Query it with `sqlite-vec`'s KNN form, joined back on `rowid`:
 
 ```bash

internals/checks/e2e-attestations/claude-issues-1003-1004-kfn8ug.json

@@ -0,0 +1,7 @@
+{
+  "command": "uv run python -m pytest tests/e2e -x -q",
+  "ran_at": 1787281290,
+  "exit_code": 0,
+  "commit": "c9a387f5ccf1974a5d0153252b5487c662f80e67",
+  "branch": "claude/issues-1003-1004-kfn8ug"
+}

internals/checks/src/checks/cli.py

@@ -12,6 +12,7 @@
 from checks.changelog_gate.cli import cli as changelog_gate
 from checks.declared_deps.cli import cli as declared_deps
 from checks.npm_binary_extension_load.cli import cli as npm_binary_extension_load
+from checks.platforms_mirror.cli import cli as platforms_mirror
 from checks.preflight.cli import cli as preflight
 from checks.pytest_gate.cli import cli as pytest_gate
 from checks.release_globs.cli import cli as release_globs
@@ -27,6 +28,7 @@ def main() -> None:
 main.add_command(changelog_gate, name="changelog-gate")
 main.add_command(declared_deps, name="declared-deps")
 main.add_command(npm_binary_extension_load, name="npm-binary-extension-load")
+main.add_command(platforms_mirror, name="platforms-mirror")
 main.add_command(preflight, name="preflight")
 main.add_command(pytest_gate, name="pytest-gate")
 main.add_command(release_globs, name="release-globs")

internals/checks/src/checks/cli_test.py

@@ -30,6 +30,10 @@ def test_declared_deps_is_registered():
     assert "declared-deps" in main.commands
 
 
+def test_platforms_mirror_is_registered():
+    assert "platforms-mirror" in main.commands
+
+
 def test_preflight_is_registered():
     assert "preflight" in main.commands
 

internals/checks/src/checks/platforms_mirror/cli.py

@@ -0,0 +1,29 @@
+"""The platforms-mirror check -- repo-only (#1004).
+
+Backs `dirsql-checks platforms-mirror`: the node distcheck flow's platform table
+and the TypeScript release source of truth must agree on the fields they share.
+"""
+
+from __future__ import annotations
+
+import click
+
+from checks.platforms_mirror.decide import PYTHON_FILE, TYPESCRIPT_FILE
+from checks.platforms_mirror.gate import run
+
+
+@click.command()
+@click.option(
+    "--python",
+    "python_path",
+    default=PYTHON_FILE,
+    help="The node distcheck flow's platform table.",
+)
+@click.option(
+    "--typescript",
+    "typescript_path",
+    default=TYPESCRIPT_FILE,
+    help="The release source of truth for published sub-packages.",
+)
+def cli(python_path: str, typescript_path: str) -> None:
+    raise SystemExit(run(python_path, typescript_path))

internals/checks/src/checks/platforms_mirror/cli_test.py

@@ -0,0 +1,32 @@
+"""Colocated unit tests for the platforms-mirror command (isolation -- no
+`CliRunner`). Driven through `.callback`; `run` is mocked at its import site.
+"""
+
+from unittest import mock
+
+import pytest
+
+from checks.platforms_mirror.cli import cli
+
+
+def invoke(**kwargs):
+    with mock.patch("checks.platforms_mirror.cli.run", **kwargs) as run:
+        with pytest.raises(SystemExit) as exc_info:
+            cli.callback(python_path="p.py", typescript_path="t.ts")
+    return run, exc_info.value.code
+
+
+def test_exits_with_runs_return_code():
+    run, code = invoke(return_value=0)
+    run.assert_called_once_with("p.py", "t.ts")
+    assert code == 0
+
+
+def test_propagates_a_drifted_mirror_as_exit_one():
+    assert invoke(return_value=1)[1] == 1
+
+
+def test_defaults_to_the_two_committed_platform_tables():
+    python_path, typescript_path = cli.params
+    assert python_path.default == "internals/distcheck/src/distcheck/node_flow/platforms.py"
+    assert typescript_path.default == "packages/ts/src/platforms.ts"