Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -57,5 +57,7 @@ docs/blob-report/
docs/.playwright/
actionlint

# cargo-mutants run output (never commit)
# cargo-mutants run output (never commit). The tool rotates the previous
# run's directory to `.old` on every re-run, so both forms need ignoring.
mutants.out/
mutants.out.old/
10 changes: 10 additions & 0 deletions ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -155,6 +155,16 @@ The `dirsql` Rust crate handles all heavy lifting:

Wraps `rusqlite` with an anonymous disk-backed temp database (#402) — ephemeral like `:memory:`, but index pages spill to disk so resident memory does not scale with the corpus. Handles DDL execution (run verbatim -- no injected columns, epic #358), row insertion with per-file ownership recorded in the internal `_dirsql_internal_rows` table, querying, and row deletion by file path. The internal bookkeeping tables (`_dirsql_internal_rows`, `_dirsql_files`, `_dirsql_meta`) are a private surface: a SQLite authorizer installed on the `query()` path denies any read (or schema `PRAGMA`) targeting the reserved `_dirsql_*` namespace, so they are unreachable through the public query surface (issue #378) while the engine still writes them in the same transaction as the user rows.

**A table's name is declared, never derived.** Both `[[table]]` and the SDK
`Table` take a required `name`; nothing tokenizes the `CREATE TABLE` head of
`ddl` to work it out. After the DDL runs, SQLite's own catalog
(`pragma_table_list`) settles whether a table by that name exists — a load-time
error when it does not. That keeps quoted, schema-qualified and
`IF NOT EXISTS` DDL working without dirsql owning a SQL tokenizer, per *never
reinvent what SQLite expresses natively*. `validate_identifier` remains: `name`
is spliced into `format!()`-built INSERT/DELETE SQL, so it is still the
injection guard.

**Named tables are real; path-tables are virtual.** A declared `[[table]]` (or programmatic `Table`) is a real SQLite table whose rows are inserted on build and maintained by the watcher — the `db` module above. A [path-table](../docs/reference/path-tables.md) (`SELECT * FROM './'`, epic path-as-table) is a `dirsql_path` **virtual table** (`vtab.rs` / `path_table.rs`, rusqlite's `vtab` feature): no rows are stored, no reconcile or watcher runs, and the filesystem is walked live at query time (`xFilter`/`xNext` enumerate matched files; `xColumn` supplies stat values and lazily reads `content`). SQLite stays the entire query engine either way; the vtab only enumerates rows and supplies column values. Path-tables are registered on demand — see *Query execution* below.

### `scanner` -- Directory traversal
Expand Down
11 changes: 7 additions & 4 deletions PARITY.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ API surface comparison across the three language SDKs.

| Concept | Python | Rust | TypeScript |
|-------------|-------------------------|--------------------------|--------------------------|
| Table def | `Table(ddl, glob, on_file, strict)` | `Table::new(ddl, glob, on_file)` / `Table::strict(...)` / `Table::try_new(...)` | `new Table({...})` or `{ ddl, glob, onFile, strict? }` (plain object) |
| Table def | `Table(name, ddl, glob, on_file, strict)` | `Table::new(name, ddl, glob, on_file)` / `Table::strict(...)` / `Table::try_new(...)` | `new Table({...})` or `{ name, ddl, glob, onFile, strict? }` (plain object) |
| on-file callback | `(path) -> list[dict]` | `Fn(&str) -> Vec<Row>` | `(path) => Record<string, unknown>[]` |
| Row event | `RowEvent` (class, frozen attrs; `file_path` on all variants) | `RowEvent` (enum: Insert/Update/Delete/Error; `file_path` on all variants) | `RowEvent` (plain object with action string; `filePath` on all variants) |
| Row type | `dict[str, Any]` | `HashMap<String, Value>` | `Record<string, unknown>` |
Expand Down Expand Up @@ -339,7 +339,7 @@ public-API binding, so no drift.
### TypeScript
- Uses `camelCase` for method names.
- `RowEvent` field names use `camelCase` (`oldRow`, `filePath`), not `snake_case`.
- Table definitions may be written as `new Table({...})` or as a plain object literal (`{ ddl, glob, onFile, strict? }`); `Table` is a thin identity wrapper that exists for parity with the Python/Rust `Table` constructors. Both forms are interchangeable at every call site that takes `TableDef[]`.
- Table definitions may be written as `new Table({...})` or as a plain object literal (`{ name, ddl, glob, onFile, strict? }`); `Table` is a thin identity wrapper that exists for parity with the Python/Rust `Table` constructors. Both forms are interchangeable at every call site that takes `TableDef[]`.
- The constructor is overloaded: `new DirSQL(configPath: string)` or `new DirSQL(options: { root?, tables?, ignore?, config? })`. There is no separate `fromConfig` factory.
- No separate `AsyncDirSQL` — JS is async by default, so `DirSQL` has `ready: Promise<void>`, `query(): Promise<Record[]>`, and `watch(): AsyncIterable<RowEvent>` built in.
- `query()`, `startWatcher()`, and `pollEvents()` all return `Promise`s and run on the libuv threadpool so the JS event loop stays responsive (even for long poll timeouts).
Expand Down Expand Up @@ -371,7 +371,8 @@ Real-core file map:
| Config file | `from_config_test.py` | `from_config.rs`, `config.rs` | `from-config.test.ts` |
| Persistence | `persist_test.py` | `persist.rs`, `persist_parsed_path_table.rs` | `persist.test.ts` |
| Extensions | `extensions_test.py`, `extension_package_test.py`, `config_extension_package_test.py` | `extensions.rs` | `extensions.test.ts`, `extension-package.test.ts`, `config-extension-package.test.ts` |
| Table-name resolution (#204) | `table_name_resolution_test.py` | `table_name_resolution.rs` | `table-name-resolution.test.ts` |
| Quoted-identifier DDL (#204) | `table_name_resolution_test.py` | `table_name_resolution.rs` | (covered by `declared-table-name.test.ts`) |
| Declared table `name` (#962) | `declared_table_name_test.py` | `declared_table_name.rs`, `declared_table_name_e2e.rs` | `declared-table-name.test.ts` |
| Docs examples | `docs_examples_test.py` | `docs_examples.rs` | `docs-examples.test.ts` |
| Docs gap-fills | `docs_gaps_test.py` | `docs_gaps.rs` | `docs-gaps.test.ts` |
| Path-tables (#627/#628/#629) | `path_table_test.py` | `path_table_query.rs`, `path_table_globs.rs`, `vtab.rs` | `path-table.test.ts` |
Expand Down Expand Up @@ -412,7 +413,7 @@ incl. #313).
| Strict mode (extra keys) | Y | Y | Y |
| Strict mode (missing keys) | Y | Y | Y |
| Strict mode (exact match) | Y | Y | Y |
| `Table` construction + `ddl`/`glob` attributes | Y | Y | Y (`Table` class + plain-object interchangeability) |
| `Table` construction + `name`/`ddl`/`glob` attributes | Y | Y | Y (`Table` class + plain-object interchangeability) |
| Quoted-identifier DDL registers/queries by bare name (#204) | Y | Y | Y |
| Path-table query (`FROM './'`, glob scoping, zero-match rows, live re-scan, JOIN vs named) | Y (#629) | Y | Y (#629) |
| Path-table hidden `content` (excluded from `*`, selectable by name, non-UTF-8 → NULL) | Y (#629) | Y | Y (#629) |
Expand Down Expand Up @@ -461,6 +462,8 @@ incl. #313).
| Missing config file errors | Y | Y | Y |
| Invalid TOML errors | Y | core | Y |
| `[[table]]` missing `ddl` errors | Y | core | Y |
| `[[table]]` missing `name` errors | Y | Y | Y |
| `[[table]]` `name` absent from the catalog after `ddl` errors | Y | Y | Y |
| Config `persist` / `persist_path` resolution | core | Y (`from_config.rs`) | core |

### Persistence
Expand Down
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,7 @@ async def main():
"./my-blog",
tables=[
Table(
name="posts",
ddl="CREATE TABLE posts (title TEXT, author TEXT)",
glob="posts/*.json",
on_file=lambda path: [json.loads(open(path, encoding="utf-8").read())],
Expand Down
7 changes: 7 additions & 0 deletions ROADMAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,7 @@ Format is **inferrable from glob extension**: `.json` -> `json`, `.jsonl`/`.ndjs

```toml
[[table]]
name = "items"
ddl = "CREATE TABLE items (name TEXT, price REAL)"
glob = "catalog/*.json"
each = "data.items"
Expand All @@ -45,6 +46,7 @@ each = "data.items"

```toml
[[table]]
name = "comments"
ddl = "CREATE TABLE comments (thread_id TEXT, body TEXT, author TEXT)"
glob = "comments/{thread_id}/index.jsonl"
# thread_id captured from glob path, body and author from JSONL content
Expand All @@ -67,6 +69,7 @@ columns.display_name = "metadata.author.name" # dot-path into nested object

```toml
[[table]]
name = "posts"
ddl = "CREATE TABLE posts (title TEXT, author TEXT)"
glob = "posts/*.json"
# format inferred from .json, columns match JSON keys 1:1
Expand All @@ -79,19 +82,23 @@ glob = "posts/*.json"
ignore = ["node_modules/**", ".git/**"]

[[table]]
name = "comments"
ddl = "CREATE TABLE comments (thread_id TEXT, body TEXT, author TEXT, resolved INTEGER)"
glob = "_comments/{thread_id}/index.jsonl"

[[table]]
name = "documents"
ddl = "CREATE TABLE documents (title TEXT, draft INTEGER, body TEXT)"
glob = "**/index.md"

[[table]]
name = "items"
ddl = "CREATE TABLE items (name TEXT, price REAL)"
glob = "catalog/*.json"
each = "data.items"

[[table]]
name = "metrics"
ddl = "CREATE TABLE metrics (date TEXT, requests INTEGER, errors INTEGER)"
glob = "logs/*.csv"
```
Expand Down
1 change: 1 addition & 0 deletions docs/getting-started.md
Original file line number Diff line number Diff line change
Expand Up @@ -164,6 +164,7 @@ Now create a `.dirsql.toml` that points a table at it:
```bash
cat > .dirsql.toml <<'EOF'
[[table]]
name = "notes"
ddl = "CREATE TABLE notes (title TEXT, author TEXT)"
glob = "notes/**/*.md"
on-file = "sh note.sh {path}"
Expand Down
1 change: 1 addition & 0 deletions docs/howto/columns-from-paths.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@ Point a table at it in `.dirsql.toml`:

```toml
[[table]]
name = "photos"
ddl = "CREATE TABLE photos (year TEXT, month TEXT, basename TEXT)"
glob = "photos/*/*/*.jpg"
on-file = "python3 pathcols.py {path}"
Expand Down
1 change: 1 addition & 0 deletions docs/howto/define-tables.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ Then declare the table in `.dirsql.toml`:

```toml
[[table]]
name = "posts"
ddl = "CREATE TABLE posts (title TEXT, slug TEXT)"
glob = "posts/**/*.md"
on-file = "python3 extract.py {path}"
Expand Down
3 changes: 3 additions & 0 deletions docs/howto/embed.md
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,7 @@ async def main() -> None:
"./comments-root",
tables=[
Table(
name="comments",
ddl="CREATE TABLE comments (thread TEXT, author TEXT, body TEXT)",
glob="comments/*/*.json",
on_file=on_file,
Expand All @@ -83,6 +84,7 @@ const db = new DirSQL({
root: "./comments-root",
tables: [
{
name: "comments",
ddl: "CREATE TABLE comments (thread TEXT, author TEXT, body TEXT)",
glob: "comments/*/*.json",
onFile: (path) => [
Expand Down Expand Up @@ -129,6 +131,7 @@ fn main() -> Result<(), Box<dyn std::error::Error>> {
let db = DirSQL::builder()
.root("./comments-root")
.table(Table::new(
"comments",
"CREATE TABLE comments (thread TEXT, author TEXT, body TEXT)",
"comments/*/*.json",
on_file,
Expand Down
2 changes: 2 additions & 0 deletions docs/howto/extract-from-contents.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ stdout works. With [`jq`](https://jqlang.org/):

```toml
[[table]]
name = "books"
ddl = "CREATE TABLE books (title TEXT, author TEXT, year INTEGER)"
glob = "books/*.json"
on-file = "jq -c '[{title, author, year}]' {path}"
Expand Down Expand Up @@ -53,6 +54,7 @@ row per line, slurp it:

```toml
[[table]]
name = "events"
ddl = "CREATE TABLE events (event TEXT, user TEXT)"
glob = "logs/*.jsonl"
on-file = "jq -c -s '.' {path}"
Expand Down
1 change: 1 addition & 0 deletions docs/howto/parse-files-into-columns.md
Original file line number Diff line number Diff line change
Expand Up @@ -84,6 +84,7 @@ command in verbatim:

```toml
[[table]]
name = "posts"
ddl = "CREATE TABLE posts (title TEXT, author TEXT)"
glob = "posts/*.md"
on-file = "python3 extract.py {path}"
Expand Down
1 change: 1 addition & 0 deletions docs/howto/react-to-changes.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ prints each file's basename:
```toml
# .dirsql.toml
[[table]]
name = "files"
ddl = "CREATE TABLE files (basename TEXT)"
glob = "**/*"
on-file = '''sh -c 'printf "[{\"basename\":\"%s\"}]" "${1##*/}"' sh {path}'''
Expand Down
1 change: 1 addition & 0 deletions docs/howto/skip-files.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ Exclude the noise in `.dirsql.toml`:
ignore = ["notes/drafts/**", "**/*.tmp"]

[[table]]
name = "notes"
ddl = "CREATE TABLE notes (basename TEXT)"
glob = "notes/**/*"
on-file = '''sh -c 'printf "[{\"basename\":\"%s\"}]" "${1##*/}"' sh {path}'''
Expand Down
1 change: 1 addition & 0 deletions docs/howto/write-a-plugin.md
Original file line number Diff line number Diff line change
Expand Up @@ -108,6 +108,7 @@ path = "sqlite_vec"
entrypoint = "sqlite3_vec_init"

[[table]]
name = "notes"
ddl = "CREATE TABLE notes (path TEXT, text TEXT, embedding TEXT)"
glob = "notes/*.md"
on-file = "uv run --with model2vec python embed.py {path} {root}"
Expand Down
3 changes: 3 additions & 0 deletions docs/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@ db = DirSQL(
"./my-project",
tables=[
Table(
name="records",
ddl="CREATE TABLE records (name TEXT, size INTEGER, type TEXT)",
glob="data/*.json",
on_file=lambda path: [json.loads(open(path, encoding="utf-8").read())],
Expand All @@ -52,6 +53,7 @@ let db = DirSQL::new(
"./my-project",
vec![
Table::new(
"records",
"CREATE TABLE records (name TEXT, size INTEGER, type TEXT)",
"data/*.json",
|path| vec![serde_json::from_str(&std::fs::read_to_string(path).unwrap()).unwrap()],
Expand All @@ -70,6 +72,7 @@ const db = new DirSQL({
root: './my-project',
tables: [
new Table({
name: 'records',
ddl: 'CREATE TABLE records (name TEXT, size INTEGER, type TEXT)',
glob: 'data/*.json',
onFile: (path) => [JSON.parse(readFileSync(path, 'utf8'))],
Expand Down
18 changes: 16 additions & 2 deletions docs/reference/config.md
Original file line number Diff line number Diff line change
Expand Up @@ -166,7 +166,8 @@ what its required `on-file` command emits — dirsql injects nothing (see

| Key | Required | Description |
|---|---|---|
| `ddl` | yes | A SQLite `CREATE TABLE` statement. The table name is parsed from it. Only the columns declared here are kept; keys the `on-file` command emits that are not declared are dropped. |
| `name` | yes | The table's SQL name — the name you query it by. Declared, never derived from `ddl`: dirsql does not read the DDL text. The `ddl` must create a table by this name; if it doesn't, loading fails. |
| `ddl` | yes | A SQLite `CREATE TABLE` statement, run verbatim. Only the columns declared here are kept; keys the `on-file` command emits that are not declared are dropped. |
| `glob` | yes | Glob pattern matched against root-relative paths. Every table whose glob matches a file receives that file's rows — a file can populate multiple tables. A `{name}` segment is rewritten to `*` (it matches one path segment but captures nothing). |
| `on-file` | **yes** | A command run once per matched file; its stdout (a JSON array of row objects) becomes the file's rows. Must be non-empty. A `[[table]]` with no `on-file` is a load error (see [parse errors](#parse-errors)). See [Command hooks](./hooks.md#on-file). |
| `strict` | no (default `false`) | When `true`, rows whose keys do not exactly match the declared columns are rejected with an error: extra keys error, and every declared column must be supplied by the `on-file` output. When `false`, extra keys are dropped and missing columns become `NULL`. |
Expand All @@ -180,11 +181,13 @@ instead of declaring a table.

```toml
[[table]]
name = "comments"
ddl = "CREATE TABLE comments (path TEXT, author TEXT, body TEXT)"
glob = "_comments/*/*.jsonl"
on-file = "jq -c -s '.' {path}"

[[table]]
name = "papers"
ddl = "CREATE TABLE papers (paper_id TEXT, title TEXT)"
glob = "**/meta.json"
on-file = "uv run python extract_papers.py {path}"
Expand Down Expand Up @@ -241,7 +244,16 @@ SDKs raise/reject) when:
- The TOML is malformed.
- Any table contains an unknown key (top level, `[dirsql]`, `[[table]]`, or
`[[dirsql.extension]]`). The error names the offending key.
- A `[[table]]` entry omits `ddl` or `glob`.
- A `[[table]]` entry omits `name`, `ddl`, or `glob` (or `name` is
empty/whitespace).
- A `[[table]]` entry's `ddl` runs but creates no table by its `name`. The
error carries the entry's name and points at the fix:

> `table 'messages': its `ddl` ran but created no table called 'messages'. Set `name` to the table the `ddl` creates.`

dirsql asks SQLite's catalog rather than interpreting the DDL, so quoted
(`CREATE TABLE "messages"`), schema-qualified (`main.messages`) and
`IF NOT EXISTS` forms all match a plain `name = "messages"`.
- A `[[table]]` entry omits `on-file` (or it is empty/whitespace). The error
names the offending glob and points at the fix:

Expand Down Expand Up @@ -275,11 +287,13 @@ deterministic = true
timeout = "600s"

[[table]]
name = "comments"
ddl = "CREATE TABLE comments (author TEXT, body TEXT)"
glob = "_comments/*/*.jsonl"
on-file = "jq -c -s '.' {path}"

[[table]]
name = "documents"
ddl = "CREATE TABLE documents (title TEXT, summary TEXT)"
glob = "**/index.md"
on-file = "uv run python extract_doc.py {path}"
Expand Down
19 changes: 11 additions & 8 deletions docs/reference/sdk.md
Original file line number Diff line number Diff line change
Expand Up @@ -317,27 +317,30 @@ rather than waiting — an intentional, language-idiomatic difference.
::: code-group

```python [Python]
Table(*, ddl: str, glob: str, on_file: Callable[[str], list[dict]], strict: bool = False)
Table(*, name: str, ddl: str, glob: str, on_file: Callable[[str], list[dict]], strict: bool = False)
```

```typescript [TypeScript]
new Table({ ddl, glob, onFile, strict? })
new Table({ name, ddl, glob, onFile, strict? })
// or a plain object — TableDef and Table are interchangeable:
{ ddl: string, glob: string, onFile: (path: string) => Record<string, unknown>[], strict?: boolean }
{ name: string, ddl: string, glob: string, onFile: (path: string) => Record<string, unknown>[], strict?: boolean }
```

```rust [Rust]
Table::new(ddl, glob, on_file) // on_file: Fn(&str) -> Vec<Row>, infallible
Table::try_new(ddl, glob, on_file) // on_file: Fn(&str) -> Result<Vec<Row>, _>
Table::strict(ddl, glob, on_file) // Table::new with strict = true
Table::new(name, ddl, glob, on_file) // on_file: Fn(&str) -> Vec<Row>, infallible
Table::try_new(name, ddl, glob, on_file) // on_file: Fn(&str) -> Result<Vec<Row>, _>
Table::strict(name, ddl, glob, on_file) // Table::new with strict = true
```

:::

Maps files to table rows.

- `ddl` — A SQLite `CREATE TABLE` statement; the table name is parsed from
it. Table names must be unique across all tables.
- `name` — The table's SQL name, declared rather than derived from `ddl`.
Must be unique across all tables. Building fails if the `ddl` creates no
table by this name (checked against SQLite's catalog, before any file is
indexed).
- `ddl` — A SQLite `CREATE TABLE` statement, run verbatim.
- `glob` — Glob pattern matched against root-relative paths. Every table whose
glob matches a file receives that file's rows — a file can populate multiple
tables. A `{name}` segment is rewritten to `*` (it matches one path segment
Expand Down
3 changes: 3 additions & 0 deletions packages/python/changelog.d/2026-08-20-declared-table-name.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
**Changed**

- **A table's name is now declared, not derived from `ddl`.** `Table` takes a required keyword-only `name` (`Table(name=..., ddl=..., glob=..., on_file=...)`), and a `[[table]]` entry in a `.dirsql.toml` requires a `name` key. The core no longer parses the DDL text to find the name; a `name` its `ddl` does not create fails at load with an error naming the entry. `Table.name` is now the declared `str` rather than an `Optional[str]` parsed from the DDL. (#962)
2 changes: 2 additions & 0 deletions packages/python/dirsql/_dirsql.pyi
Original file line number Diff line number Diff line change
Expand Up @@ -30,13 +30,15 @@ class ExtensionSpec(TypedDict):
class Table:
"""A table definition. Construct via keyword arguments only."""

name: str
ddl: str
glob: str
strict: bool

def __init__(
self,
*,
name: str,
ddl: str,
glob: str,
on_file: Callable[[str], list[Row]],
Expand Down
8 changes: 6 additions & 2 deletions packages/python/dirsql/resolve_config_extensions_test.py
Original file line number Diff line number Diff line change
Expand Up @@ -70,7 +70,9 @@ def it_returns_none_on_malformed_toml(cfg_dir):

def it_returns_none_when_there_is_no_dirsql_section(cfg_dir):
path = os.path.join(cfg_dir, ".dirsql.toml")
_write(path, '[[table]]\nddl = "CREATE TABLE t (x TEXT)"\nglob = "*"\n')
_write(
path, '[[table]]\nname = "t"\nddl = "CREATE TABLE t (x TEXT)"\nglob = "*"\n'
)
assert rce.resolve_config_extension_specs(path) is None

def it_returns_none_when_no_extensions_are_declared(cfg_dir):
Expand Down Expand Up @@ -146,7 +148,9 @@ def it_returns_none_when_no_config_uses_a_bare_name(cfg_dir):
a = os.path.join(cfg_dir, "a.toml")
b = os.path.join(cfg_dir, "b.toml")
_write(a, '[[dirsql.extension]]\npath = "ext/a.so"\n')
_write(b, '[[table]]\nddl = "CREATE TABLE t (x TEXT)"\nglob = "*"\n')
_write(
b, '[[table]]\nname = "t"\nddl = "CREATE TABLE t (x TEXT)"\nglob = "*"\n'
)
with _patch() as resolver:
assert rce.resolve_configs_extension_specs([a, b]) is None
resolver.assert_not_called()
Expand Down
Loading
Loading