Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 12 additions & 2 deletions docs/reference/config.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ the config file's location. See [`--config`](./cli.md#flags).

| Key | Type | Default | Description |
|---|---|---|---|
| `ignore` | array of strings | `[]` | Glob patterns matched against root-relative paths. Matched files are skipped entirely — excluded from the initial scan and from watch events. |
| `ignore` | array of strings | `[]` | Glob patterns matched against root-relative paths, under the [one glob rule](#glob-rule): `*` matches one level, `**` any depth. Matched files are skipped entirely — excluded from the initial scan and from watch events. |

There is no timeout key. `on-file` hook runs are unbounded; to bound one, wrap
its command in `timeout(1)` (see [Command hooks](./hooks.md#bounding-a-hook)).
Expand All @@ -43,6 +43,16 @@ directory.
ignore = ["node_modules/**", ".git/**"]
```

### Glob rule

Every glob in dirsql — `ignore`, a `[[table]]`'s `glob`, and a
[path-table](./path-tables.md#writing-the-path) — reads as the shell does:
`*` and `?` match within one path segment and never cross `/`; `**` matches
any depth. So `ignore = ["*"]` hides only the files directly inside the root,
`ignore = ["build/*"]` hides `build/a.o` but not `build/sub/b.o`, and
`glob = "*.json"` selects only the top-level `.json` files; write
`**/*.json` to select them at every depth.

Persistence is not a config key. Keep the SQLite index on disk between runs
with the [`--persist [PATH]` CLI flag](./cli.md#dirsql-server) — a machine-local
operational choice that belongs to the runner, not to shareable config.
Expand Down Expand Up @@ -220,7 +230,7 @@ what its required `on-file` command emits — dirsql injects nothing (see
|---|---|---|
| `name` | yes | The table's SQL name — the name you query it by. Declared, never derived from `ddl`: dirsql does not read the DDL text. The `ddl` must create a table by this name; if it doesn't, loading fails. |
| `ddl` | yes | A SQL batch, run verbatim — any number of statements. It must create a table called `name`; that table holds the file rows, and only the columns it declares are kept (keys the `on-file` command emits that are not declared are dropped). The rest of the batch is yours: indexes, virtual tables, triggers. See [Batch `ddl`](#batch-ddl). |
| `glob` | yes | Glob pattern matched against root-relative paths. Every table whose glob matches a file receives that file's rows — a file can populate multiple tables. A `{name}` segment is rewritten to `*` (it matches one path segment but captures nothing). |
| `glob` | yes | Glob pattern matched against root-relative paths, under the [one glob rule](#glob-rule): `*` matches one level, `**` any depth. Every table whose glob matches a file receives that file's rows — a file can populate multiple tables. A `{name}` segment is rewritten to `*` (it matches one path segment but captures nothing). |
| `on-file` | **yes** | A command run once per table, with every matched file's absolute path appended as a trailing argument; its stdout (one JSON array of row objects) is the table's rows. Must be non-empty. A `[[table]]` with no `on-file` is a load error (see [parse errors](#parse-errors)). See [Command hooks](./hooks.md#on-file). |
| `strict` | no (default `false`) | When `true`, rows whose keys do not exactly match the declared columns are rejected with an error: extra keys error, and every declared column must be supplied by the `on-file` output. When `false`, extra keys are dropped and missing columns become `NULL`. |

Expand Down
4 changes: 3 additions & 1 deletion docs/reference/path-tables.md
Original file line number Diff line number Diff line change
Expand Up @@ -207,7 +207,9 @@ choice is stated as one trade in
## Skip rules

A path-table scan applies the same [`ignore`](/reference/config) patterns your
declared tables use, plus two built-in defaults so a zero-config
declared tables use — matched against root-relative paths under the same
[glob rule](/reference/config#glob-rule) as the path itself, `*` one level and
`**` any depth — plus two built-in defaults so a zero-config
`SELECT * FROM './'` does not drown in machinery:

- `**/node_modules/**`
Expand Down
5 changes: 3 additions & 2 deletions docs/reference/sdk.md
Original file line number Diff line number Diff line change
Expand Up @@ -116,8 +116,9 @@ shortcut was removed in #603 — use
- `root` — Directory to index. When omitted, the index roots at the process
cwd (even when `config` is supplied).
- `tables` — Programmatic [`Table`](#table) definitions.
- `ignore` — Glob patterns matched against root-relative paths; matched
files are skipped entirely (scan and watch).
- `ignore` — Glob patterns matched against root-relative paths under the
[glob rule](./config.md#glob-rule) (`*` one level, `**` any depth);
matched files are skipped entirely (scan and watch).
- `no_ignore` / `noIgnore` — Scan files a `.gitignore` would hide.
[Path-tables](./path-tables.md#skip-rules) respect `.gitignore` files by
default; the built-in `node_modules`/`.git` skips and any `ignore` patterns
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
**Changed** `Table(glob=...)`, `DirSQL(ignore=...)` and the `[[table]] glob` /
`[dirsql] ignore` keys of a `.dirsql.toml` now follow the same rule as
path-tables: `*` and `?` match within one path segment and stop at `/`, `**`
matches any depth. `glob="*.json"` selects only the top-level `.json` files
and `ignore=["folder/*"]` hides only the files directly inside `folder/`.
Patterns written with `**` are unaffected.
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
{
"command": "just test-e2e",
"ran_at": 1791065836,
"exit_code": 0,
"commit": "62a7c434b5efe4630dbd52818c4c8ede7ca0725d",
"branch": "claude/1223-config-glob-rule"
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,67 @@
### Python: `*` stops at `/` in `Table(glob=...)`, `DirSQL(ignore=...)` and `.dirsql.toml` globs

#### Summary

Declared-table globs and ignore patterns were compiled with `globset`
defaults, where `*` and `?` also match `/`, so `glob = "*.json"` quietly
selected `.json` files at every depth and `ignore = ["folder/*"]` hid the
whole `folder/` subtree. Path-tables already compiled with
`literal_separator`, so one spelling meant two things. Every surface now
compiles the same way: `*` and `?` match within one path segment, `**` matches
any depth. Affected: the `glob` key of a `[[table]]`, the `[dirsql] ignore`
list, and the `tables` / `ignore` builder parameters of the Rust, Python and
TypeScript SDKs, which share this matcher.

#### Required changes

A pattern that relied on `*` crossing `/` must say `**`.

| Surface | Before | After |
| ------- | ------ | ----- |
| `[[table]] glob`, every depth | `glob = "*.json"` | `glob = "**/*.json"` |
| `[[table]] glob`, a subtree | `glob = "folder/*"` | `glob = "folder/**"` |
| `[dirsql] ignore`, a subtree | `ignore = ["folder/*"]` | `ignore = ["folder/**"]` |
| `[dirsql] ignore`, an extension at every depth | `ignore = ["*.tmp"]` | `ignore = ["**/*.tmp"]` |
| `Table(glob=...)` | `Table(name="t", ddl=..., glob="*.json", on_file=...)` | `Table(name="t", ddl=..., glob="**/*.json", on_file=...)` |
| `DirSQL(ignore=...)` | `DirSQL(root, ignore=["folder/*"])` | `DirSQL(root, ignore=["folder/**"])` |

A pattern already written with `**` (`**/*.md`, `folder/**`, the built-in
`**/node_modules/**` and `**/.git/**`) matches exactly what it did before. A
pattern meant to stay flat (`glob = "data/*.csv"` over a flat `data/`) needs
no change.

#### Deprecations removed

_None._

#### Behavior changes without code changes

- `glob = "*"` matches the files directly inside the root; it used to match
every file at every depth.
- `glob = "folder/*"` matches `folder/a.json` but no longer `folder/sub/b.json`.
- `ignore = ["*"]` hides the top-level files only; `ignore = ["folder/*"]`
hides `folder/a.json` but no longer `folder/sub/b.json`.
- `?` matches one character that is not `/`.
- A `{name}` placeholder matches exactly one path segment, as documented.

#### Verification

```bash
cd "$(mktemp -d)"
mkdir -p folder/sub
printf '{}' > root.json
printf '{}' > folder/a.json
printf '{}' > folder/sub/b.json
cat > .dirsql.toml <<'TOML'
[[table]]
name = "files"
ddl = "CREATE TABLE files (path TEXT)"
glob = "*.json"
on-file = '''sh -c 'printf "["; sep=""; for p; do printf "%s{\"path\":\"%s\"}" "$sep" "$p"; sep=","; done; printf "]"' sh'''
TOML
dirsql query "SELECT count(*) AS n FROM files" -c .dirsql.toml
# expected: [{"n":1}] (root.json only; before: 3)
sed -i 's|glob = "\*.json"|glob = "**/*.json"|' .dirsql.toml
dirsql query "SELECT count(*) AS n FROM files" -c .dirsql.toml
# expected: [{"n":3}]
```
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
**Changed** `[[table]] glob` and `[dirsql] ignore` patterns now follow the
same rule as path-tables: `*` and `?` match within one path segment and stop
at `/`, `**` matches any depth. `glob = "*.json"` selects only the top-level
`.json` files, `glob = "folder/*"` no longer reaches `folder/sub/`, and
`ignore = ["folder/*"]` hides only the files directly inside `folder/`. A
`{name}` placeholder matches exactly one segment. Patterns written with `**`
(including the built-in `**/node_modules/**` and `**/.git/**` skips) are
unaffected. Applies to the SDK `tables` and `ignore` parameters alike.
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
### Core: `*` stops at `/` in `[[table]] glob` and `[dirsql] ignore`

#### Summary

Declared-table globs and ignore patterns were compiled with `globset`
defaults, where `*` and `?` also match `/`, so `glob = "*.json"` quietly
selected `.json` files at every depth and `ignore = ["folder/*"]` hid the
whole `folder/` subtree. Path-tables already compiled with
`literal_separator`, so one spelling meant two things. Every surface now
compiles the same way: `*` and `?` match within one path segment, `**` matches
any depth. Affected: the `glob` key of a `[[table]]`, the `[dirsql] ignore`
list, and the `tables` / `ignore` builder parameters of the Rust, Python and
TypeScript SDKs, which share this matcher.

#### Required changes

A pattern that relied on `*` crossing `/` must say `**`.

| Surface | Before | After |
| ------- | ------ | ----- |
| `[[table]] glob`, every depth | `glob = "*.json"` | `glob = "**/*.json"` |
| `[[table]] glob`, a subtree | `glob = "folder/*"` | `glob = "folder/**"` |
| `[dirsql] ignore`, a subtree | `ignore = ["folder/*"]` | `ignore = ["folder/**"]` |
| `[dirsql] ignore`, an extension at every depth | `ignore = ["*.tmp"]` | `ignore = ["**/*.tmp"]` |
| SDK `tables` / `ignore` parameters | same patterns | same rewrites |

A pattern already written with `**` (`**/*.md`, `folder/**`, the built-in
`**/node_modules/**` and `**/.git/**`) matches exactly what it did before. A
pattern meant to stay flat (`glob = "data/*.csv"` over a flat `data/`) needs
no change.

#### Deprecations removed

_None._

#### Behavior changes without code changes

- `glob = "*"` matches the files directly inside the root; it used to match
every file at every depth.
- `glob = "folder/*"` matches `folder/a.json` but no longer `folder/sub/b.json`.
- `ignore = ["*"]` hides the top-level files only; `ignore = ["folder/*"]`
hides `folder/a.json` but no longer `folder/sub/b.json`.
- `?` matches one character that is not `/`.
- A `{name}` placeholder matches exactly one path segment, as documented.

#### Verification

```bash
cd "$(mktemp -d)"
mkdir -p folder/sub
printf '{}' > root.json
printf '{}' > folder/a.json
printf '{}' > folder/sub/b.json
cat > .dirsql.toml <<'TOML'
[[table]]
name = "files"
ddl = "CREATE TABLE files (path TEXT)"
glob = "*.json"
on-file = '''sh -c 'printf "["; sep=""; for p; do printf "%s{\"path\":\"%s\"}" "$sep" "$p"; sep=","; done; printf "]"' sh'''
TOML
dirsql query "SELECT count(*) AS n FROM files" -c .dirsql.toml
# expected: [{"n":1}] (root.json only; before: 3)
sed -i 's|glob = "\*.json"|glob = "**/*.json"|' .dirsql.toml
dirsql query "SELECT count(*) AS n FROM files" -c .dirsql.toml
# expected: [{"n":3}]
```
128 changes: 124 additions & 4 deletions packages/rust/src/matcher.rs
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
use globset::{Glob, GlobSet, GlobSetBuilder};
use globset::{Glob, GlobBuilder, GlobSet, GlobSetBuilder};
use std::path::Path;

/// Result of matching a file path against a glob pattern.
Expand Down Expand Up @@ -83,6 +83,12 @@ fn glob_with_placeholders_as_star(pattern: &str) -> String {
out
}

/// One glob rule on every surface: `*` and `?` stop at `/`, `**` crosses it,
/// exactly as `scanner::compile_glob` reads a path-table.
fn compile(pattern: &str) -> Result<Glob, globset::Error> {
GlobBuilder::new(pattern).literal_separator(true).build()
}

impl TableMatcher {
/// Build a new matcher from (glob_pattern, table_name) pairs and ignore patterns.
/// Glob patterns may contain `{name}` placeholders, which match like `*`.
Expand All @@ -94,7 +100,7 @@ impl TableMatcher {
for (pattern, table_name) in mappings {
let glob_pattern = glob_with_placeholders_as_star(pattern);
let mut builder = GlobSetBuilder::new();
builder.add(Glob::new(&glob_pattern)?);
builder.add(compile(&glob_pattern)?);
entries.push(PatternEntry {
glob_set: builder.build()?,
table_name: table_name.to_string(),
Expand All @@ -104,9 +110,9 @@ impl TableMatcher {
let mut ignore_builder = GlobSetBuilder::new();
let mut ignore_dir_builder = GlobSetBuilder::new();
for pattern in ignore_patterns {
ignore_builder.add(Glob::new(pattern)?);
ignore_builder.add(compile(pattern)?);
if let Some(subtree) = pattern.strip_suffix("/**") {
ignore_dir_builder.add(Glob::new(subtree)?);
ignore_dir_builder.add(compile(subtree)?);
}
}
let ignore_set = ignore_builder.build()?;
Expand Down Expand Up @@ -273,6 +279,7 @@ mod tests {
assert_eq!(names(&matcher, "file1.txt"), vec!["t"]);
assert_eq!(names(&matcher, "fileA.txt"), vec!["t"]);
assert!(matcher.match_all(Path::new("file.txt")).is_empty());
assert!(matcher.match_all(Path::new("file/.txt")).is_empty());
}

#[test]
Expand Down Expand Up @@ -380,4 +387,117 @@ mod tests {
assert_eq!(placeholder_names("{a{b}"), vec!["b".to_string()]);
assert_eq!(glob_with_placeholders_as_star("{a{b}"), "{a*");
}

/// Two files at depth 0, two under `folder/`, two under `folder/sub/`, one
/// under a sibling directory.
const FIXTURE: [&str; 7] = [
"root.md",
"root.txt",
"folder/a.md",
"folder/a.txt",
"folder/sub/b.md",
"folder/sub/b.txt",
"sibling/c.md",
];

fn matched(glob: &str) -> Vec<&'static str> {
let matcher = TableMatcher::new(&[(glob, "t")], &[]).unwrap();
FIXTURE
.into_iter()
.filter(|p| !matcher.match_all(Path::new(p)).is_empty())
.collect()
}

fn hidden(ignore: &str) -> Vec<&'static str> {
let matcher = TableMatcher::new(&[], &[ignore]).unwrap();
FIXTURE
.into_iter()
.filter(|p| matcher.is_ignored(Path::new(p)))
.collect()
}

#[test]
fn a_lone_star_glob_matches_depth_zero_only() {
assert_eq!(matched("*"), ["root.md", "root.txt"]);
}

#[test]
fn a_bare_directory_name_glob_matches_nothing() {
assert!(matched("folder").is_empty());
}

#[test]
fn a_directory_slash_star_glob_stops_at_the_next_separator() {
assert_eq!(matched("folder/*"), ["folder/a.md", "folder/a.txt"]);
}

#[test]
fn a_directory_slash_double_star_glob_matches_any_depth_below() {
assert_eq!(
matched("folder/**"),
[
"folder/a.md",
"folder/a.txt",
"folder/sub/b.md",
"folder/sub/b.txt"
]
);
}

#[test]
fn a_leading_double_star_glob_matches_the_suffix_at_any_depth() {
assert_eq!(
matched("**/*.md"),
["root.md", "folder/a.md", "folder/sub/b.md", "sibling/c.md"]
);
}

#[test]
fn a_double_star_slash_star_glob_matches_every_file() {
assert_eq!(matched("**/*"), FIXTURE);
}

#[test]
fn a_directory_slash_star_ignore_hides_only_that_directory_s_files() {
assert_eq!(hidden("folder/*"), ["folder/a.md", "folder/a.txt"]);
}

#[test]
fn a_lone_star_ignore_hides_depth_zero_only() {
assert_eq!(hidden("*"), ["root.md", "root.txt"]);
}

#[test]
fn the_default_ignores_still_hide_at_every_depth() {
let matcher = TableMatcher::new(&[], &["**/node_modules/**", "**/.git/**"]).unwrap();
for path in [
"node_modules/pkg/index.js",
"apps/site/node_modules/pkg/dist/index.js",
".git/config",
"vendor/lib/.git/HEAD",
] {
assert!(
matcher.is_ignored(Path::new(path)),
"{path} must stay hidden"
);
}
}

#[test]
fn a_subtree_ignore_whose_prefix_has_a_star_marks_one_level_of_directories() {
let matcher = TableMatcher::new(&[], &["folder/*/**"]).unwrap();
assert!(matcher.is_ignored_dir(Path::new("folder/sub")));
assert!(!matcher.is_ignored_dir(Path::new("folder/sub/deeper")));
}

#[test]
fn a_placeholder_matches_exactly_one_path_segment() {
let matcher = TableMatcher::new(&[("data/{id}/metadata.json", "a")], &[]).unwrap();
assert_eq!(names(&matcher, "data/x/metadata.json"), vec!["a"]);
assert!(
matcher
.match_all(Path::new("data/x/y/metadata.json"))
.is_empty()
);
}
}
Loading
Loading