Declared tables: ddl becomes a wide-open multi-statement SQL batch

PARITY.md

@@ -140,6 +140,15 @@ This lives entirely in the shared Rust core (`create_table` runs DDL verbatim;
 `query` returns vanilla rows), so `PRAGMA table_info` and `SELECT *` report the
 same user-only columns across all three SDKs.
 
+**Multi-statement `ddl` batch (#956) — parity by construction, no drift.** A
+`[[table]]`'s `ddl` runs as a whole SQL batch (`execute_batch`, one
+transaction), so indexes, FTS5/vec0 virtual tables and triggers are declarable
+in config; what the batch produced is settled against SQLite's catalog
+(`pragma_table_list`), not by reading DDL text. The `ddl` value is an opaque
+string every SDK already forwards unchanged, so no binding signature moves and
+all three gain the batch, the catalog validation and the new load errors
+together.
+
 **Internal tables unreachable through `query()` (#378, epic #358) — parity by
 construction, no drift.** The internal bookkeeping tables (`_dirsql_internal_rows`,
 `_dirsql_files`, `_dirsql_meta`) are denied on the `query()` path by a SQLite
@@ -411,6 +420,9 @@ incl. #313).
 | `bytes`/`Vec<u8>`/`Buffer` → BLOB (list/array of ints does NOT) | Y (#465) | Y | Y (#343: `Buffer`/`Uint8Array` in, `Buffer` out) |
 | Invalid SQL raises         | Y      | Y    | Y          |
 | Invalid DDL raises         | Y      | Y    | Y          |
+| Multi-statement `ddl` batch (index / FTS5 vtab / triggers) runs whole (#956) | core | Y (`ddl_batch.rs`, `ddl_batch_e2e.rs`) | core |
+| `ddl` batch SQLite rejects: rolled back, error prefixed with the entry (#956) | core | Y (`ddl_batch.rs`, `sdk.rs`) | core |
+| Declared `name` resolving to a virtual table errors (#956) | core | Y (`ddl_batch.rs`) | core |
 | Query rejects writes       | Y      | Y    | Y          |
 | Write-rejection edge matrix (leading comments, mixed case, CTE/whitespace allowed) | core | Y (`readonly_query.rs`) | core |
 | Internal `_dirsql_*` columns hidden from `SELECT *` | Y | Y | Y |
@@ -473,6 +485,7 @@ incl. #313).
 | `[[table]]` missing `ddl` errors | Y | core | Y |
 | `[[table]]` missing `name` errors | Y | Y | Y |
 | `[[table]]` `name` absent from the catalog after `ddl` errors | Y | Y | Y |
+| `[[table]]` `ddl` batch: whole batch runs, catalog settles the result (#956) | core | Y | core |
 | Config `persist` / `persist_path` resolution | core | Y (`from_config.rs`) | core |
 
 ### Persistence

docs/howto/define-tables.md

@@ -65,6 +65,50 @@ Add one `[[table]]` entry per table — each with its own `glob`, `ddl`, and
 table — each table is an independent view. See
 [`[[table]]`](../reference/config.md#table) for the remaining key, `strict`.
 
+## Add indexes and full-text search
+
+`ddl` is a SQL batch, not a single statement — SQLite runs the whole thing. So
+a table can arrive with its own index, and with an FTS5 index kept current by
+two triggers:
+
+```toml
+[[table]]
+name    = "posts"
+glob    = "posts/**/*.md"
+on-file = "python3 extract.py {path}"
+ddl     = '''
+CREATE TABLE posts (title TEXT, slug TEXT, body TEXT);
+CREATE INDEX posts_slug ON posts(slug);
+
+CREATE VIRTUAL TABLE posts_fts
+  USING fts5(body, content='posts', content_rowid='rowid');
+CREATE TRIGGER posts_ai AFTER INSERT ON posts BEGIN
+  INSERT INTO posts_fts(rowid, body) VALUES (new.rowid, new.body);
+END;
+CREATE TRIGGER posts_ad AFTER DELETE ON posts BEGIN
+  INSERT INTO posts_fts(posts_fts, rowid, body)
+    VALUES ('delete', old.rowid, old.body);
+END;
+'''
+```
+
+```bash
+dirsql query "SELECT slug FROM posts JOIN posts_fts ON posts.rowid = posts_fts.rowid
+              WHERE posts_fts MATCH 'recursion'" -c ./.dirsql.toml
+```
+
+```json
+[{"slug":"again"}]
+```
+
+dirsql writes rows with plain `INSERT` and `DELETE`, so those triggers hold
+through the initial load and every [watcher](./react-to-changes.md) event — you
+maintain nothing. `name` still has to be the row table (`posts`); the virtual
+table lives beside it under its own name. The batch runs once, when the table
+is created, and editing any part of it rebuilds a
+[persistent cache](./persist.md) from scratch. Full rules:
+[Batch `ddl`](../reference/config.md#batch-ddl).
+
 ## Going further
 
 - The parser mechanics — placeholders, stdout protocol, per-file failure

docs/howto/load-extension.md

@@ -76,9 +76,11 @@ interpreter to resolve package names with
 
 ## Notes
 
-- Extensions add **functions**. An extension-backed virtual table cannot be
-  declared as a `[[table]]` — `dirsql` tables are per-file row tables
-  ([reference](../reference/config.md#dirsql-extension)).
+- Extensions add **functions**, and virtual tables a table's `ddl` batch can
+  create. A `[[table]]`'s own `name` may not be a virtual table — that one is
+  the per-file row table `dirsql` inserts into — so declare the virtual table
+  alongside it in the same batch
+  ([reference](../reference/config.md#batch-ddl)).
 - Loading happens before any table DDL runs, and the SQL
   `load_extension()` function is never exposed to queries
   ([reference](../reference/config.md#dirsql-extension)).

docs/howto/search-by-meaning.md

@@ -95,10 +95,12 @@ LIMIT 10
 
 ::: tip Top-k is `LIMIT k`
 If you know `sqlite-vec` you may reach for its `MATCH … AND k = 10` idiom.
-That syntax belongs to `sqlite-vec`'s `vec0` virtual table, which `dirsql`
-does not use — `dirsql` tables are per-file row tables. For plain
-expressions, `sqlite-vec`'s own documented pattern is exactly what this
-guide uses: `ORDER BY vec_distance_cosine(...) LIMIT k`.
+That syntax belongs to `sqlite-vec`'s `vec0` virtual table, which the table
+above does not use: a `[[table]]`'s own `name` is always a per-file row table.
+For plain expressions, `sqlite-vec`'s own documented pattern is exactly what
+this guide uses: `ORDER BY vec_distance_cosine(...) LIMIT k`. To get the `vec0`
+idiom instead, declare the `vec0` table alongside the row table in the same
+[`ddl` batch](../reference/config.md#batch-ddl) and fill it from a trigger.
 :::
 
 ## Repeat runs are cheap

docs/reference/config.md

@@ -87,10 +87,11 @@ name you installed:
   `path = "sqlite-vec-linux-x64"` (matching your platform), not
   `path = "sqlite-vec"`, whose meta-package ships no loadable.
 
-Extensions add **functions** callable in queries and in a table's DDL. An
-extension-backed **virtual table** cannot be declared as a `[[table]]` —
-`dirsql` tables are per-file row tables, so a `CREATE VIRTUAL TABLE` DDL is
-rejected; call the extension's functions in queries instead.
+Extensions add **functions** callable in queries and in a table's DDL, and
+**virtual tables** a table's [`ddl` batch](#batch-ddl) can create. What a
+`[[table]]`'s own `name` may not be is a virtual table: that one is the
+per-file row table `dirsql` inserts into. Create the virtual table alongside
+it, under its own name.
 
 ## `[[dirsql.function]]`
 
@@ -167,7 +168,7 @@ what its required `on-file` command emits — dirsql injects nothing (see
 | Key | Required | Description |
 |---|---|---|
 | `name` | yes | The table's SQL name — the name you query it by. Declared, never derived from `ddl`: dirsql does not read the DDL text. The `ddl` must create a table by this name; if it doesn't, loading fails. |
-| `ddl` | yes | A SQLite `CREATE TABLE` statement, run verbatim. Only the columns declared here are kept; keys the `on-file` command emits that are not declared are dropped. |
+| `ddl` | yes | A SQL batch, run verbatim — any number of statements. It must create a table called `name`; that table holds the file rows, and only the columns it declares are kept (keys the `on-file` command emits that are not declared are dropped). The rest of the batch is yours: indexes, virtual tables, triggers. See [Batch `ddl`](#batch-ddl). |
 | `glob` | yes | Glob pattern matched against root-relative paths. Every table whose glob matches a file receives that file's rows — a file can populate multiple tables. A `{name}` segment is rewritten to `*` (it matches one path segment but captures nothing). |
 | `on-file` | **yes** | A command run once per matched file; its stdout (a JSON array of row objects) becomes the file's rows. Must be non-empty. A `[[table]]` with no `on-file` is a load error (see [parse errors](#parse-errors)). See [Command hooks](./hooks.md#on-file). |
 | `strict` | no (default `false`) | When `true`, rows whose keys do not exactly match the declared columns are rejected with an error: extra keys error, and every declared column must be supplied by the `on-file` output. When `false`, extra keys are dropped and missing columns become `NULL`. |
@@ -194,6 +195,72 @@ on-file = "uv run python extract_papers.py {path}"
 strict  = true
 ```
 
+### Batch `ddl`
+
+`ddl` is handed to SQLite whole, so a table declaration is not limited to one
+statement:
+
+```toml
+[[table]]
+name    = "messages"
+glob    = "sessions/*/messages/*.json"
+on-file = "jq -c '.' {path}"
+ddl     = '''
+CREATE TABLE messages (session TEXT, idx INT, role TEXT, text TEXT);
+CREATE INDEX messages_session ON messages(session);
+
+CREATE VIRTUAL TABLE messages_fts
+  USING fts5(text, content='messages', content_rowid='rowid');
+CREATE TRIGGER messages_ai AFTER INSERT ON messages BEGIN
+  INSERT INTO messages_fts(rowid, text) VALUES (new.rowid, new.text);
+END;
+CREATE TRIGGER messages_ad AFTER DELETE ON messages BEGIN
+  INSERT INTO messages_fts(messages_fts, rowid, text)
+    VALUES ('delete', old.rowid, old.text);
+END;
+'''
+```
+
+```bash
+dirsql query "SELECT text FROM messages_fts WHERE messages_fts MATCH 'deploy'" \
+  -c ./.dirsql.toml
+```
+
+Those two triggers are all a keyword index needs. dirsql writes file rows with
+plain `INSERT` and `DELETE` — an update is a delete and an insert in one
+transaction, and there is no `UPDATE` path on user rows — so triggers you
+declare here stay current through the initial load and every
+[watcher](../howto/react-to-changes.md) event. The same shape with a `vec0`
+virtual table and an [`embed()`](#dirsql-function) call in the trigger gives
+you stored vectors.
+
+**dirsql never reads the batch.** After it runs, SQLite's own catalog
+(`pragma_table_list`) settles what it produced:
+
+- No table called `name` → a load error that lists what the batch *did*
+  create, so a typo is obvious.
+- `name` is a **virtual** table → a load error. The declared table is the one
+  dirsql inserts file rows into, so it has to be a real row table. Create the
+  virtual table alongside it, under its own name.
+- `name` is **`WITHOUT ROWID`** → a warning. Internal row bookkeeping is keyed
+  on rowid, so these will be rejected in a future release.
+
+The whole batch runs in one transaction: if any statement fails, none of them
+took effect, and the error is SQLite's own, prefixed with the config entry —
+`table 'messages': SQLite error: near "(": syntax error`. Context, never
+interpretation.
+
+Two consequences of `ddl` running **once, when the table is created**:
+
+- **Rows the batch inserts itself are not file-tracked.** No file owns them, so
+  they survive file deletions — and they vanish on any rebuild.
+- **Editing `ddl` at all rebuilds a
+  [persistent cache](../howto/persist.md).** The config hash covers the entire
+  batch, so a new index, a different FTS5
+  tokenizer or a changed embedding model id forces a full sweep and re-ingest.
+  That is the only invalidation lane: dirsql tracks no ownership of what the
+  batch made.
+
 ### `on-file` row mapping
 
 The command prints a JSON array of objects; each object becomes one row.
@@ -247,13 +314,24 @@ SDKs raise/reject) when:
 - A `[[table]]` entry omits `name`, `ddl`, or `glob` (or `name` is
   empty/whitespace).
 - A `[[table]]` entry's `ddl` runs but creates no table by its `name`. The
-  error carries the entry's name and points at the fix:
+  error carries the entry's name, lists what the batch did create, and points
+  at the fix:
 
-  > `table 'messages': its `ddl` ran but created no table called 'messages'. Set `name` to the table the `ddl` creates.`
+  > `table 'messages': its `ddl` ran but created no table called 'messages' (it created: mesages, mesages_fts). Set `name` to the table the `ddl` creates.`
 
   dirsql asks SQLite's catalog rather than interpreting the DDL, so quoted
   (`CREATE TABLE "messages"`), schema-qualified (`main.messages`) and
   `IF NOT EXISTS` forms all match a plain `name = "messages"`.
+- A `[[table]]` entry's `name` names a **virtual** table. The declared table
+  holds the file rows, so it must be a real row table:
+
+  > `table 'messages': its `ddl` created a virtual table called 'messages'. The declared table holds the file rows, so it must be a real row table; create the virtual table alongside it, under its own name.`
+
+- A `[[table]]` entry's `ddl` is rejected by SQLite. Nothing the batch did
+  takes effect, and SQLite's own message is passed through under the entry's
+  name:
+
+  > `table 'messages': SQLite error: near "(": syntax error`
 - A `[[table]]` entry omits `on-file` (or it is empty/whitespace). The error
   names the offending glob and points at the fix:
 

packages/python/e2e-attestations/claude-tackle-956-xzen74.json

@@ -0,0 +1,7 @@
+{
+  "command": "uv run python -m pytest tests/e2e/ -x -q",
+  "ran_at": 1787258633,
+  "exit_code": 0,
+  "commit": "9191ced6e960be0f079fd21957473441f5cd2455",
+  "branch": "claude/tackle-956-xzen74"
+}

packages/rust/changelog.d/2026-08-20-ddl-batch.md

@@ -0,0 +1,9 @@
+**Changed**
+
+- **A `[[table]]`'s `ddl` is now a wide-open multi-statement SQL batch.** It runs through `execute_batch` inside one transaction, so a table can arrive with its own indexes, virtual tables and triggers — `CREATE INDEX` for B-trees, an external-content FTS5 vtab plus two triggers for keyword search with `bm25()`/`snippet()`, a `vec0` vtab plus an `embed()` trigger for stored vectors. dirsql writes user rows with plain `INSERT`/`DELETE` (an upsert is a delete and an insert in one transaction, and there is no `UPDATE` path on user rows), so those triggers stay current through the initial load and every watcher event, with no bookkeeping on dirsql's side. The two string sniffs the single-statement rule needed — `is_virtual_table_ddl` and `is_without_rowid_ddl` — are gone: what the batch produced is settled against SQLite's catalog (`pragma_table_list`) instead. A declared `name` that ends up a virtual table is a load error; one that is `WITHOUT ROWID` keeps its existing warning. (#956)
+
+**Fixed**
+
+- **A `ddl` holding more than one statement no longer runs only its first.** `Connection::execute` prepares one statement and discards the rest of the string, so a second statement was silently dropped — a `CREATE INDEX` that never ran, and a batch whose tail SQLite could not even parse still loading successfully. The whole batch now runs, and a statement SQLite rejects fails the load with SQLite's own message under the config entry that caused it (`table 'messages': SQLite error: near "(": syntax error`), leaving nothing behind. (#956)
+
+- **The persistent-cache sweep now drops virtual tables before plain ones.** Drop order came from `sqlite_master`'s creation order, which happened to be safe; it is now derived from `pragma_table_list`'s type. Dropping a shadow table before the virtual table that owns it poisons the vtab's own drop even with `IF EXISTS` (probed against sqlite-vec 0.1.9), which a `ddl` batch creating a `vec0` table could now hit. (#956)

packages/rust/migrations.d/2026-08-20-ddl-batch.md

@@ -0,0 +1,88 @@
+### `ddl` becomes a multi-statement SQL batch, validated against the catalog (#956)
+
+#### Summary
+
+A `[[table]]`'s `ddl` had to be exactly one statement, and dirsql string-sniffed
+it for `CREATE VIRTUAL TABLE` and `WITHOUT ROWID`. It is now a whole SQL batch,
+run through `execute_batch` inside one transaction, and what it produced is
+settled afterwards against SQLite's catalog (`pragma_table_list`) rather than by
+reading the DDL text. `Db::create_table` keeps its `(name, ddl)` signature;
+`db::is_virtual_table_ddl`, `db::is_without_rowid_ddl` and `DbError::DdlParse`
+are deleted. `DirSqlError` gains `TableIsVirtual` and `TableDdl`, and
+`TableNotCreated` gains a `created` field carrying the before/after catalog diff.
+
+#### Required changes
+
+| Before | After | Fix |
+| ------ | ----- | --- |
+| `Err(DirSqlError::TableNotCreated { name })` as an exhaustive pattern | Does not compile — the variant gained `created: Vec<String>` | Match `TableNotCreated { name, .. }`, or bind `created` to read what the batch did create. |
+| `Err(DirSqlError::Core(_))` for a `ddl` SQLite rejects | No longer produced for that case | Match `DirSqlError::TableDdl { name, source }`; `source` is the same `DbError` as before, and `Display` prefixes it with `table '<name>': `. |
+| `DbError::DdlParse` constructed or matched | Does not compile | Nothing produces it any more. A rejected batch is `DbError::Sqlite`, reached through `DirSqlError::TableDdl`. |
+| `ddl = "CREATE VIRTUAL TABLE vss USING vec0(...)"` rejected before running | The batch runs; the declared table being virtual is caught after, as `DirSqlError::TableIsVirtual` | Keep `name` on a real row table and declare the virtual table alongside it in the same batch. |
+
+#### Deprecations removed
+
+- `DbError::DdlParse` — removed outright, with `db::is_virtual_table_ddl` and
+  `db::is_without_rowid_ddl`, the two whitespace-normalizing string sniffs
+  behind it.
+
+#### Behavior changes without code changes
+
+- **Every statement in `ddl` now runs.** `Connection::execute` prepared only the
+  first and discarded the tail, so a config with `CREATE TABLE t (...); CREATE
+  INDEX ...` silently created the table without the index, and one whose second
+  statement SQLite could not parse still loaded. Both now behave as written: the
+  index is created, and the unparseable batch is a load error.
+- **A batch SQLite rejects leaves nothing behind**, and its error carries the
+  config entry: `table 'messages': SQLite error: near "(": syntax error`.
+  Previously the first statement of a partly-invalid `ddl` took effect.
+- **A `[[table]]` whose `name` resolves to a virtual table is a load error**
+  raised after the batch runs, not a string match before it. A batch is now free
+  to create as many virtual tables as it likes under other names.
+- **`WITHOUT ROWID` is detected from the catalog's `wr` flag** rather than from
+  the DDL text. Same warning, same "will be rejected in a future release"
+  wording; a `WITHOUT ROWID` *shadow* table belonging to some other virtual
+  table no longer triggers it.
+- **The persistent-cache sweep drops virtual tables first**, derived from
+  `pragma_table_list`'s type instead of `sqlite_master`'s creation order, and
+  leaves shadow tables to the virtual table that owns them.
+- **Editing any part of `ddl` still invalidates a persistent cache** — the
+  config hash already covered the whole string, and now that the whole string
+  runs, that is the single invalidation lane for indexes, tokenizers and
+  embedding model ids alike. Rows a batch inserts itself are not file-tracked:
+  they survive file deletions and vanish on a rebuild.
+
+#### Verification
+
+```bash
+# .dirsql.toml
+#   [[table]]
+#   name    = "records"
+#   glob    = "data/*.json"
+#   on-file = "cat {path}"
+#   ddl     = '''
+#   CREATE TABLE records (id TEXT, body TEXT);
+#   CREATE INDEX records_id ON records(id);
+#   CREATE VIRTUAL TABLE records_fts
+#     USING fts5(body, content='records', content_rowid='rowid');
+#   CREATE TRIGGER records_ai AFTER INSERT ON records BEGIN
+#     INSERT INTO records_fts(rowid, body) VALUES (new.rowid, new.body);
+#   END;
+#   '''
+
+dirsql query "SELECT name FROM pragma_index_list('records')" -c ./.dirsql.toml
+# -> [{"name":"records_id"}]          (before: [], the CREATE INDEX was dropped)
+
+dirsql query "SELECT body FROM records_fts WHERE records_fts MATCH 'hello'" \
+  -c ./.dirsql.toml
+# -> [{"body":"hello world"}]
+
+# Truncate the batch's last statement:
+dirsql query "SELECT id FROM records" -c ./.dirsql.toml
+# -> error: table 'records': SQLite error: incomplete input
+#    (before: exit 0 — the tail was silently ignored)
+
+# Point `name` at a virtual table:
+dirsql query "SELECT id FROM records" -c ./.dirsql.toml
+# -> error: table 'records': its `ddl` created a virtual table called 'records'.
+```