Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions docs/.vitepress/config.ts
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,9 @@ export default defineConfig({
description: 'Ephemeral SQL index over a local directory. Watches a filesystem, ingests structured files into an ephemeral SQLite database, and exposes a SQL query interface.',
base: '/',

// The guide `dirsql context` prints, compiled into the binary; not site pages.
srcExclude: ['context/**'],

// Vite blocks dev-server requests whose Host header isn't allowlisted, so
// `pnpm dev --host` reached by a network hostname (e.g. a Tailscale
// `*.ts.net` name) 400s. Allow extra hosts from the environment so a
Expand Down
2 changes: 2 additions & 0 deletions docs/context/.gitattributes
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
# include_str! embeds these verbatim; a CRLF checkout would print a mixed-ending guide.
*.md text eol=lf
103 changes: 103 additions & 0 deletions docs/context/recipes.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,103 @@
## Recipes

Each query runs as written from the directory in question and prints a JSON array of row objects.

### Largest files

```sh
uvx dirsql "SELECT path, size FROM './**' ORDER BY size DESC LIMIT 10" --format json
```

### Count and total size by extension

```sh
uvx dirsql "SELECT ext, COUNT(*) AS files, SUM(size) AS bytes FROM './**' GROUP BY ext ORDER BY files DESC" --format json
```

`ext` is `NULL` for files with no extension and keeps its case; group by `LOWER(ext)` to fold `JPG` into `jpg`.

### Files per directory

```sh
uvx dirsql "SELECT dir, COUNT(*) AS files FROM './**' GROUP BY dir ORDER BY files DESC" --format json
```

`dir` is the empty string for files at the top level.

### Stale files

Not modified in the last 90 days:

```sh
uvx dirsql "SELECT path, datetime(mtime, 'unixepoch') AS modified FROM './**' WHERE mtime < strftime('%s', 'now') - 90 * 86400 ORDER BY mtime" --format json
```

Most recently modified markdown:

```sh
uvx dirsql "SELECT path, datetime(mtime, 'unixepoch') AS modified FROM './**/*.md' ORDER BY mtime DESC LIMIT 10" --format json
```

### The same JSON field from many files

```sh
uvx dirsql "SELECT path, content ->> 'name' AS name, content ->> 'version' AS version FROM './**/metadata.json' WHERE json_valid(content) ORDER BY name" --format json
```

`json_valid(content)` keeps one empty or malformed file from failing the whole query. Nested keys are JSON paths, still quoted: `content ->> '$.runtime.engine'`. Booleans arrive as `1` and `0`:

```sh
uvx dirsql "SELECT path FROM './**/metadata.json' WHERE json_valid(content) AND content ->> 'disabled' = 0" --format json
```

Count across an array field with `json_each`, guarded the same way:

```sh
uvx dirsql "SELECT t.value AS tag, COUNT(*) AS n FROM './**/metadata.json' AS m, json_each(m.content, '$.tags') AS t WHERE json_valid(m.content) GROUP BY tag ORDER BY n DESC, tag" --format json
```

### Frontmatter

Files whose frontmatter contains a given line (`%` matches across newlines):

```sh
uvx dirsql "SELECT path FROM './**/*.md' WHERE content LIKE '---%status: draft%'" --format json
```

One frontmatter field from every file:

```sh
uvx dirsql "WITH fm AS (SELECT path, substr(content, instr(content, 'title:') + 6) AS rest FROM './**/*.md' WHERE content LIKE '---%title:%') SELECT path, trim(substr(rest, 1, instr(rest, char(10)) - 1)) AS title FROM fm ORDER BY path" --format json
```

### Occurrences of a string per file

grep stops at matching lines; SQL can count them:

```sh
uvx dirsql "SELECT path, (length(content) - length(replace(content, 'TODO', ''))) / length('TODO') AS todos FROM './**/*.md' WHERE todos > 0 ORDER BY todos DESC" --format json
```

### Join two directories

Source modules with no matching doc page:

```sh
uvx dirsql "SELECT s.path FROM './src/*.py' AS s LEFT JOIN './docs/**/*.md' AS d ON replace(d.basename, '.md', '') = replace(s.basename, '.py', '') WHERE d.path IS NULL ORDER BY s.path" --format json
```

### Top-k by meaning

Needs the embeddings plugin. The first run downloads a model of roughly a hundred megabytes, with progress on stderr:

```sh
uvx --with dirsql-plugin-embeddings dirsql "
SELECT path,
vec_distance_cosine(emb, embed('how do I cook pasta?')) AS distance
FROM (SELECT path, embed(content) AS emb FROM './notes/*.md')
WHERE emb IS NOT NULL
ORDER BY distance
LIMIT 3" --format json
```

`WHERE emb IS NOT NULL` matters: an unreadable file embeds to `NULL`, and SQLite sorts `NULL` first. For ranked paths only, `uvx dirsql-plugin-embeddings './notes/*.md' "how do I cook pasta?" -k 3` prints `path<TAB>distance` lines.
64 changes: 64 additions & 0 deletions docs/context/troubleshooting.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
## Troubleshooting

Diagnostics go to stderr and the exit code is `1`.

### `no such table: **/*.md; did you mean './**/*.md'?`

A bare glob. Paths relative to the directory you run in start with `./`:

```sh
uvx dirsql "SELECT path FROM './**/*.md'" --format json
```

### `near ".": syntax error` and `hint: paths used as table names must be quoted`

`FROM ./` reaches SQLite's parser as punctuation. Quote the path:

```sh
uvx dirsql "SELECT COUNT(*) AS files FROM './'" --format json
```

### `no such column: disabled`

`content ->> disabled` names a column. The key is a string literal:

```sh
uvx dirsql "SELECT path, content ->> 'disabled' AS disabled FROM './**/metadata.json' WHERE json_valid(content)" --format json
```

### `SQLite error: malformed JSON`

One matched file is not valid JSON (an empty file counts), and that fails the whole query. Guard with `json_valid`:

```sh
uvx dirsql "SELECT path, content ->> 'name' AS name FROM './**/*.json' WHERE json_valid(content)" --format json
```

### `no such table: files; did you mean FROM './**'?`

There is no implicit table. Query the tree directly:

```sh
uvx dirsql "SELECT path FROM './**'" --format json
```

### Empty `[]` for files you can see

- **You named a directory**, which lists one level only. `'./**'` and `'./docs/**'` are the recursive forms.
- **The files are gitignored.** `--no-ignore` scans them:

```sh
uvx dirsql "SELECT path FROM './**'" --format json --no-ignore
```

Naming the directory outright also works: `'./build'` scans `build/` even when it is ignored.
- **The files are under `node_modules`**, which is skipped unless the path names it: `'./node_modules/**'`.
- **The files start with a dot, or sit under a directory that does.** Spell the dot: `'./.git/**'`, `'./**/.env'`.

### `table ./ may not be modified`

dirsql is read-only. Mutations belong in a script that consumes the query's JSON.

### A table appeared, not JSON

`--format json` was not passed and stdout was a terminal. Pass it.
54 changes: 54 additions & 0 deletions docs/context/usage.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
## Usage

dirsql answers a question about a directory tree with SQL. Nothing to install; run it from the directory in question:

```sh
uvx dirsql "SELECT basename, size FROM './**' ORDER BY size DESC LIMIT 5" --format json
```

Always pass `--format json`. Without it the output depends on stdout: a terminal gets a table, a pipe gets JSON, and you cannot know which you have. With it, stdout is one JSON array of row objects. Errors go to stderr with exit code `1`.

### Path-tables

A quoted path stands in for a table name. Paths are relative to the directory you run in.

| You write | dirsql scans |
| --- | --- |
| `'./'` | files directly here, no deeper |
| `'./docs'` | files directly inside `docs/`, no deeper |
| `'./*.md'` | markdown directly here |
| `'./**'` | every file, any depth |
| `'./docs/**'` | every file under `docs/`, any depth |
| `'./**/*.md'` | markdown at any depth |
| `'./notes/today.md'` | that one file |

`*` matches within one directory level and `**` matches any depth. A directory name lists one level, like `ls docs`, so write `'./**'` to scan the whole tree. Rows are files only, never directories. `'/var/log/*.log'`, `'../notes'` and `'~/notes/*.md'` also resolve; those report absolute `path` values.

Two mistakes fail every time:

1. **The path must be quoted.** `FROM ./` is a syntax error; write `FROM './'`.
2. **`content ->> 'key'` takes a string literal.** `content ->> key` fails with `no such column: key`.

### Columns

Every row has `path`, `basename`, `dir`, `ext` (no dot, `NULL` when absent), `size` (bytes), `mtime` and `ctime` (Unix seconds). A hidden `content` column holds the file's text: it is left out of `SELECT *`, read only when you name it, and `NULL` for unreadable or non-UTF-8 files.

### What a scan skips

- **`.gitignore` is respected.** Pass `--no-ignore` to scan gitignored files too, or name the ignored directory outright: `'./build'` scans `build/`.
- **`node_modules` is skipped** unless the path names it: `'./node_modules/**'`.
- **`'./**'` does not enter `.git`.** Naming it scans it: `'./.git/**'`.
- **To match dotfiles, spell the dot:** `'./.github/**'`, `'./**/.env'`.

### Escalation

1. A path-table, as above. Fits a question you ask once.
2. The same tree queried repeatedly, or a parser you want to keep: declare a table in a `.dirsql.toml` and pass it with `-c .dirsql.toml`. See [config](https://dirsql.dev/reference/config).
3. One row per record inside each file, inline: `--on-file <command>` attaches a parser to every path-table in the query, and its output replaces the stat columns. See [hooks](https://dirsql.dev/reference/hooks).
4. Search by meaning: `uvx --with dirsql-plugin-embeddings dirsql "<sql>" --format json` adds `embed()` and `vec_distance_cosine()`. See [search by meaning](https://dirsql.dev/howto/search-by-meaning).

### Read-only

dirsql never writes to, moves or changes the files it scans, and it rejects mutating SQL. Do not pass `--persist`: it writes a cache database into the directory.

Full docs: [dirsql.dev](https://dirsql.dev/) ([path-tables](https://dirsql.dev/reference/path-tables), [columns](https://dirsql.dev/reference/columns), [CLI](https://dirsql.dev/reference/cli), [query JSON](https://dirsql.dev/howto/query-json)).
12 changes: 12 additions & 0 deletions docs/reference/cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ The `dirsql` binary has these modes:
| `dirsql query "<sql>"` | Explicit synonym for the default one-shot query. |
| `dirsql server` | Start a long-lived HTTP server exposing a SQL view of a directory. See [HTTP API](./http-api.md). |
| `dirsql init` | Generate a `.dirsql.toml`. |
| `dirsql context` | Print the usage guide for this version, for an agent to read. |
| `dirsql` (bare) | Open a [REPL](#the-repl) over the current directory, reading statements until EOF. |

Bare `dirsql` does **not** start the server — that is `dirsql server`.
Expand Down Expand Up @@ -439,6 +440,17 @@ All failures exit `1` with a message on stderr:

On success, `init` exits `0`.

## `dirsql context`

```sh
uvx dirsql context
```

Prints a markdown guide for an agent that is about to query files: the
first line is `dirsql <version>`, then sections on usage, recipes and
troubleshooting. The guide is compiled into the binary, so it describes the
version that prints it. It takes no flags and exits `0`.

## Plugins

A **plugin** is an ordinary Python package that ships a `dirsql.toml` config
Expand Down
3 changes: 3 additions & 0 deletions packages/python/changelog.d/2026-10-05-cli-context.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
**Added**

- **`dirsql context` prints a usage guide for agents.** The first line is `dirsql <version>`, then markdown sections on usage, recipes, and fixes for common errors, matched to the installed version. The launcher passes `context` through without adding plugin or extension flags, as it does for `init`. (#1264)
Original file line number Diff line number Diff line change
Expand Up @@ -22,8 +22,8 @@
def with_discovered_plugins(argv: list[str]) -> list[str]:
"""Return ``argv`` with each installed plugin's fragment appended as ``-c``
(plus ``--include-default`` when the user passed no ``-c``). ``--no-plugin``
/ ``DIRSQL_NO_PLUGIN`` skip discovery, consuming the flag. ``init`` takes no
config, so it is left untouched. Raises if a declared plugin is missing its
/ ``DIRSQL_NO_PLUGIN`` skip discovery, consuming the flag. ``init`` and ``context`` take no
config, so they are left untouched. Raises if a declared plugin is missing its
module or fragment (the launcher surfaces a clean error).

Appending is safe because config flags are subcommand-local (#609): the
Expand All @@ -34,7 +34,7 @@ def with_discovered_plugins(argv: list[str]) -> list[str]:
"""
if discovery_disabled(argv):
return [a for a in argv if a != NO_PLUGIN_FLAG]
if argv and argv[0] == "init":
if argv and argv[0] in ("init", "context"):
return argv
fragments = discovered_fragments()
if not fragments:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,15 @@ def it_leaves_init_untouched():
):
assert with_discovered_plugins([init, "--force"]) == [init, "--force"]

def it_leaves_context_untouched():
context = "".join(["con", "text"])
with (
patch.object(module, "discovery_disabled", return_value=False),
patch.object(module, "discovered_fragments", side_effect=AssertionError),
patch.object(module, "user_passed_config", side_effect=AssertionError),
):
assert with_discovered_plugins([context]) == [context]

def it_is_a_no_op_when_no_plugins_are_installed():
with (
patch.object(module, "discovery_disabled", return_value=False),
Expand Down
2 changes: 1 addition & 1 deletion packages/python/dirsql/cli/resolve_config_extensions.py
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ def with_resolved_extensions(argv: list[str]) -> list[str]:
"""Return ``argv`` plus ``--extension`` flags when a TOML config names an
extension by package name; otherwise return ``argv`` unchanged. Raises if a
package name cannot be resolved (the launcher surfaces a clean error)."""
if argv and argv[0] == "init":
if argv and argv[0] in ("init", "context"):
return argv
config_paths = [
p for p in config_paths_from_argv(argv) if not p.endswith(_NATIVE_SUFFIXES)
Expand Down
7 changes: 7 additions & 0 deletions packages/python/dirsql/cli/resolve_config_extensions_test.py
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,13 @@ def it_passes_init_through_untouched():
scan.assert_not_called()
resolver.assert_not_called()

def it_passes_context_through_untouched():
argv = ["".join(["con", "text"])]
with _patch([{"path": "R:pkg", "entrypoint": None}]) as (scan, resolver):
assert rce.with_resolved_extensions(argv) is argv
scan.assert_not_called()
resolver.assert_not_called()

def it_matches_init_by_value_not_identity_or_ordering():
# Only the exact first argument "init" skips resolution: a
# runtime-built "init" still skips; other subcommands (sorting above
Expand Down
7 changes: 7 additions & 0 deletions packages/python/e2e-attestations/claude-1264-context.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
{
"command": "just test-e2e",
"ran_at": 1791216630,
"exit_code": 0,
"commit": "886e5785c2a90371f02bc2683417b9c73892b5c5",
"branch": "claude/1264-context"
}
26 changes: 26 additions & 0 deletions packages/python/migrations.d/2026-10-05-cli-context.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
### `dirsql context` is a subcommand (#1264)

#### Summary

`dirsql context` now prints the agent usage guide and exits 0. Before, the
word `context` as the first argument was run as SQL.

#### Required changes

_None._

#### Deprecations removed

_None._

#### Behavior changes without code changes

- `dirsql context` used to be read as the SQL statement `context` and fail
with a syntax error, exit 1. It now prints the guide.

#### Verification

```bash
dirsql context | head -n 1
# -> dirsql <version>
```
3 changes: 3 additions & 0 deletions packages/rust/changelog.d/2026-10-05-cli-context.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
**Added**

- **`dirsql context` prints a usage guide for agents.** The first line is `dirsql <version>`, then markdown sections on usage (path-table syntax, columns, what a scan skips), recipes, and fixes for common errors. The guide is compiled into the binary, so it always describes the version that prints it. It takes no flags and exits 0. (#1264)
26 changes: 26 additions & 0 deletions packages/rust/migrations.d/2026-10-05-cli-context.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
### `dirsql context` is a subcommand (#1264)

#### Summary

`dirsql context` now prints the agent usage guide and exits 0. Before, the
word `context` as the first argument was run as SQL.

#### Required changes

_None._

#### Deprecations removed

_None._

#### Behavior changes without code changes

- `dirsql context` used to be read as the SQL statement `context` and fail
with a syntax error, exit 1. It now prints the guide.

#### Verification

```bash
dirsql context | head -n 1
# -> dirsql <version>
```
Loading
Loading