Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 14 additions & 8 deletions docs/reference/path-tables.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,8 +47,13 @@ beneath it; the non-recursive form is spelled explicitly with `*`.
| `'./docs/**/*.md'` | markdown files at any depth under `docs/` |
| `'./notes/today.md'` | exactly that one file — one file is one row |

A path containing `*`, `?` or `[` is a glob and is used exactly as written: `*`
matches within a single directory, `**` crosses directories.
A path containing `*`, `?`, `[` or `{` is a glob and is used exactly as
written: `*` matches within a single directory, `**` crosses directories.

The scan starts at the last directory named outright before the first glob
component -- `'./small/*.md'` walks `small/` and nothing else -- so a query
over one directory costs what `find ./small` costs, however large the
directories beside it.

A path naming a single file yields exactly one row. dirsql never splits a file
into rows on its own — that is what a table's `on_file` hook is for.
Expand Down Expand Up @@ -186,8 +191,9 @@ bug to design around.
The table itself is per-connection: it lives in `temp`, so it cannot leak into
`sqlite_master` or survive a restart. Under `--persist` a *parsed* table's rows
outlive the connection in the cache (above), but the table is still minted
fresh each run and the scan still decides what exists. The reserved top-level
`.dirsql/` directory is excluded from the scan, as everywhere else.
fresh each run and the scan still decides what exists. A `.dirsql/` directory
at the top of the directory the scan starts in is reserved and excluded, as
everywhere else.

### When to promote to a declared table

Expand Down Expand Up @@ -234,17 +240,17 @@ defaults and configured `ignore` patterns still apply.

### Naming a skipped directory

Skip rules are judged on the part of the path *below* what you named outright,
so pointing at a skipped directory — built-in or gitignored — still scans it:
Skip rules are judged from the directory the scan starts in, so pointing at a
skipped directory — built-in or gitignored — still scans it:

```sql
SELECT path FROM './'; -- no node_modules rows
SELECT path FROM './node_modules/*/package.json'; -- scans it anyway
SELECT path FROM './dist'; -- scans dist/ even when gitignored
```

A `.gitignore` at or below the directory you named still filters beneath it;
only rules inherited from above it are set aside.
A `.gitignore` at or below the directory the scan starts in still filters
beneath it; one above it is never read.

### Hidden files

Expand Down
7 changes: 7 additions & 0 deletions packages/python/e2e-attestations/claude-1199-glob-prefix.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
{
"command": "just test-e2e",
"ran_at": 1790974761,
"exit_code": 0,
"commit": "80332ac7891d42541409c3544f1bf1b41644b17c",
"branch": "claude/1199-glob-prefix"
}
1 change: 1 addition & 0 deletions packages/rust/changelog.d/2026-10-02-glob-prefix-walk.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
**Fixed** A `./` path-table that names a directory before its first wildcard -- `'./small/*.md'`, `'./docs'`, `'./src/**/*.rs'` -- now starts its walk at that directory instead of at the index root, so it costs what `find ./small` costs rather than what `find .` costs, however large the siblings. `path` is reported exactly as before. Two consequences follow from where the walk starts: a `.dirsql/` directly under the named directory is now the reserved one (`'./.dirsql'` scans it outright, `'./x/.dirsql'` is pruned), and a symlinked directory named outright is now entered, both as absolute tables already behave. A parser (`--on-file`) over any path-table now runs in the index root with `{root}` naming it, which absolute tables previously got wrong. The scanner's `scan_glob` drops its `ignore_base` argument, `path_table::ignore_base` is gone, and the `dirsql_parsed` module takes the index root as an argument; see the migration note.
47 changes: 47 additions & 0 deletions packages/rust/migrations.d/2026-10-02-glob-prefix-walk.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
### Path-tables walk from their literal prefix

**Summary**

A `./` path-table is now resolved like an absolute one: the literal directory
chain ahead of the first wildcard becomes the scan root, and the glob is the
remainder. The walker therefore no longer needs to know which part of a glob
was named outright, so `dirsql::scanner::scan_glob` drops its `ignore_base`
parameter and `dirsql::path_table::ignore_base` is removed. Because the scan
root can now sit below the index root, the `dirsql_parsed` module takes the
index root as a sixth fixed argument and spawns the parser there, with
`{root}` naming it. Only Rust callers of those functions and hand-written
`CREATE VIRTUAL TABLE ... USING dirsql_parsed(...)` statements are affected;
the CLI, the Python and TypeScript SDKs, and every query are unchanged in what
they return.

**Required changes**

| Surface | Before | After |
|---|---|---|
| `scanner::scan_glob` | `scan_glob(&root, &glob, &ignore, &ignore_base, gitignore)` | `scan_glob(&root, &glob, &ignore, gitignore)` |
| `path_table::ignore_base(glob)` | returned the glob's leading literal directories | removed; `path_table::resolve` now puts them in `PathTable::root` and `PathTable::path_prefix` |
| `dirsql_parsed` module arguments | `(root, glob, parser, gitignore, cache, ignores...)` | `(root, glob, parser, gitignore, cache, index root, ignores...)` |

**Deprecations removed**

_None._

**Behavior changes without code changes**

| Surface | Before | After |
|---|---|---|
| `'./small/*.md'` beside a large sibling tree | walked the whole index root | walks `small/` only |
| `'./.dirsql'` | zero rows | scans the reserved directory, as naming it outright does for every other skipped directory |
| `'./x/.dirsql/**'` | scanned | pruned, as the reserved directory is at any scan root |
| a symlinked directory named ahead of the first wildcard | never entered | entered, as for an absolute table |
| `--on-file` over an absolute path-table (`'/var/log/*.log'`) | the parser ran in `/var/log`, `{root}` = `/var/log` | the parser runs in the index root, `{root}` = the index root, as the `on-file` hook documents |

**Verification**

```bash
mkdir -p /tmp/t/small /tmp/t/big
touch /tmp/t/small/a.md /tmp/t/big/b.md
cd /tmp/t
dirsql "SELECT path FROM './small/*.md'"
# [{"path":"small/a.md"}]
```
69 changes: 58 additions & 11 deletions packages/rust/src/db.rs
Original file line number Diff line number Diff line change
Expand Up @@ -145,15 +145,17 @@ fn no_home_path_table(name: &str) -> String {
///
/// A parsed table also carries `cache` — the persistent cache path when the
/// index has one — so it can serve an unchanged file's rows without re-running
/// the parser. The stat module takes no such argument: its columns are the
/// stat tuple the scan already has, so there is nothing to save.
/// the parser, and `index_root`, where the parser is spawned. The stat module
/// takes neither: its columns are the stat tuple the scan already has, so
/// there is nothing to save and no process to place.
fn path_table_ddl(
name: &str,
table: &PathTable,
ignore: &[String],
gitignore: bool,
parser: Option<&str>,
cache: Option<&Path>,
index_root: &Path,
) -> String {
let (module, mut args) = match parser {
Some(command) => (
Expand Down Expand Up @@ -184,6 +186,7 @@ fn path_table_ddl(
.map(|p| p.to_string_lossy().into_owned())
.unwrap_or_default(),
));
args.push(quote_literal(&index_root.to_string_lossy()));
}
args.extend(ignore.iter().map(|p| quote_literal(p)));

Expand Down Expand Up @@ -1027,13 +1030,18 @@ impl Db {
/// straight on the connection rather than through [`query`](Self::query),
/// which classifies `CREATE VIRTUAL TABLE` as a write and would reject it.
fn create_path_table(&self, name: &str, table: &PathTable) -> Result<()> {
let index_root = self
.path_table_root
.as_deref()
.expect("a path-table resolves only under an index root");
self.conn.execute_batch(&path_table_ddl(
name,
table,
&self.path_table_ignore,
self.path_table_gitignore,
self.path_table_parser.as_deref(),
self.path_table_cache.as_deref(),
index_root,
))?;
Ok(())
}
Expand Down Expand Up @@ -2639,7 +2647,15 @@ mod tests {

#[test]
fn path_table_ddl_creates_in_temp_if_not_exists() {
let ddl = path_table_ddl("./docs/*.md", &docs_path_table(), &[], true, None, None);
let ddl = path_table_ddl(
"./docs/*.md",
&docs_path_table(),
&[],
true,
None,
None,
Path::new("/root"),
);
assert_eq!(
ddl,
"CREATE VIRTUAL TABLE IF NOT EXISTS temp.\"./docs/*.md\" \
Expand All @@ -2655,16 +2671,40 @@ mod tests {
path_prefix: "/var/log".to_string(),
};
assert!(
path_table_ddl("/var/log/*.log", &table, &[], true, None, None)
.contains("'/var/log', '*.log', '/var/log', 'gitignore')"),
path_table_ddl(
"/var/log/*.log",
&table,
&[],
true,
None,
None,
Path::new("/root"),
)
.contains("'/var/log', '*.log', '/var/log', 'gitignore')"),
"got: {}",
path_table_ddl("/var/log/*.log", &table, &[], true, None, None)
path_table_ddl(
"/var/log/*.log",
&table,
&[],
true,
None,
None,
Path::new("/root"),
)
);
}

#[test]
fn path_table_ddl_carries_the_no_gitignore_switch() {
let ddl = path_table_ddl("./", &docs_path_table(), &[], false, None, None);
let ddl = path_table_ddl(
"./",
&docs_path_table(),
&[],
false,
None,
None,
Path::new("/root"),
);
assert!(
ddl.ends_with("'', 'no-gitignore')"),
"gitignore off must emit the no-gitignore switch, got: {ddl}"
Expand All @@ -2680,6 +2720,7 @@ mod tests {
true,
None,
None,
Path::new("/root"),
);
assert!(
ddl.ends_with("'', 'gitignore', 'node_modules/**', '*.tmp')"),
Expand All @@ -2696,11 +2737,12 @@ mod tests {
true,
Some("cat {path}"),
None,
Path::new("/root"),
);
assert_eq!(
ddl,
"CREATE VIRTUAL TABLE IF NOT EXISTS temp.\"./docs/*.md\" \
USING dirsql_parsed('/root', 'docs/*.md', 'cat {path}', 'gitignore', '')"
USING dirsql_parsed('/root', 'docs/*.md', 'cat {path}', 'gitignore', '', '/root')"
);
}

Expand All @@ -2713,9 +2755,10 @@ mod tests {
true,
Some("cat {path}"),
Some(Path::new("/cache/dirsql.db")),
Path::new("/root"),
);
assert!(
ddl.ends_with("'cat {path}', 'gitignore', '/cache/dirsql.db')"),
ddl.ends_with("'cat {path}', 'gitignore', '/cache/dirsql.db', '/root')"),
"a persisted index points the parsed module at its cache, got: {ddl}"
);
}
Expand All @@ -2729,6 +2772,7 @@ mod tests {
true,
None,
Some(Path::new("/cache/dirsql.db")),
Path::new("/root"),
);
assert!(
ddl.ends_with("'', 'gitignore')"),
Expand All @@ -2750,13 +2794,15 @@ mod tests {
true,
Some("parse.py {path}"),
None,
Path::new("/index"),
);
assert!(
ddl.ends_with(
"USING dirsql_parsed('/var/log', '*.log', 'parse.py {path}', 'gitignore', \
'', 'node_modules/**')"
'', '/index', 'node_modules/**')"
),
"the parser form drops the path prefix and keeps ignore rules, got: {ddl}"
"the parser form drops the path prefix, names the index root and keeps \
ignore rules, got: {ddl}"
);
}

Expand All @@ -2769,6 +2815,7 @@ mod tests {
true,
Some("sh -c 'echo hi'"),
None,
Path::new("/root"),
);
assert!(
ddl.contains("'sh -c ''echo hi'''"),
Expand Down
Loading
Loading