diff --git a/docs/reference/config.md b/docs/reference/config.md index 733cf24a5..8543f0516 100644 --- a/docs/reference/config.md +++ b/docs/reference/config.md @@ -24,7 +24,7 @@ the config file's location. See [`--config`](./cli.md#flags). | Key | Type | Default | Description | |---|---|---|---| -| `ignore` | array of strings | `[]` | Glob patterns matched against root-relative paths. Matched files are skipped entirely — excluded from the initial scan and from watch events. | +| `ignore` | array of strings | `[]` | Glob patterns matched against root-relative paths, under the [one glob rule](#glob-rule): `*` matches one level, `**` any depth. Matched files are skipped entirely — excluded from the initial scan and from watch events. | There is no timeout key. `on-file` hook runs are unbounded; to bound one, wrap its command in `timeout(1)` (see [Command hooks](./hooks.md#bounding-a-hook)). @@ -43,6 +43,16 @@ directory. ignore = ["node_modules/**", ".git/**"] ``` +### Glob rule + +Every glob in dirsql — `ignore`, a `[[table]]`'s `glob`, and a +[path-table](./path-tables.md#writing-the-path) — reads as the shell does: +`*` and `?` match within one path segment and never cross `/`; `**` matches +any depth. So `ignore = ["*"]` hides only the files directly inside the root, +`ignore = ["build/*"]` hides `build/a.o` but not `build/sub/b.o`, and +`glob = "*.json"` selects only the top-level `.json` files; write +`**/*.json` to select them at every depth. + Persistence is not a config key. Keep the SQLite index on disk between runs with the [`--persist [PATH]` CLI flag](./cli.md#dirsql-server) — a machine-local operational choice that belongs to the runner, not to shareable config. @@ -220,7 +230,7 @@ what its required `on-file` command emits — dirsql injects nothing (see |---|---|---| | `name` | yes | The table's SQL name — the name you query it by. Declared, never derived from `ddl`: dirsql does not read the DDL text. The `ddl` must create a table by this name; if it doesn't, loading fails. | | `ddl` | yes | A SQL batch, run verbatim — any number of statements. It must create a table called `name`; that table holds the file rows, and only the columns it declares are kept (keys the `on-file` command emits that are not declared are dropped). The rest of the batch is yours: indexes, virtual tables, triggers. See [Batch `ddl`](#batch-ddl). | -| `glob` | yes | Glob pattern matched against root-relative paths. Every table whose glob matches a file receives that file's rows — a file can populate multiple tables. A `{name}` segment is rewritten to `*` (it matches one path segment but captures nothing). | +| `glob` | yes | Glob pattern matched against root-relative paths, under the [one glob rule](#glob-rule): `*` matches one level, `**` any depth. Every table whose glob matches a file receives that file's rows — a file can populate multiple tables. A `{name}` segment is rewritten to `*` (it matches one path segment but captures nothing). | | `on-file` | **yes** | A command run once per table, with every matched file's absolute path appended as a trailing argument; its stdout (one JSON array of row objects) is the table's rows. Must be non-empty. A `[[table]]` with no `on-file` is a load error (see [parse errors](#parse-errors)). See [Command hooks](./hooks.md#on-file). | | `strict` | no (default `false`) | When `true`, rows whose keys do not exactly match the declared columns are rejected with an error: extra keys error, and every declared column must be supplied by the `on-file` output. When `false`, extra keys are dropped and missing columns become `NULL`. | diff --git a/docs/reference/path-tables.md b/docs/reference/path-tables.md index b7be587a0..9bf09ca33 100644 --- a/docs/reference/path-tables.md +++ b/docs/reference/path-tables.md @@ -207,7 +207,9 @@ choice is stated as one trade in ## Skip rules A path-table scan applies the same [`ignore`](/reference/config) patterns your -declared tables use, plus two built-in defaults so a zero-config +declared tables use — matched against root-relative paths under the same +[glob rule](/reference/config#glob-rule) as the path itself, `*` one level and +`**` any depth — plus two built-in defaults so a zero-config `SELECT * FROM './'` does not drown in machinery: - `**/node_modules/**` diff --git a/docs/reference/sdk.md b/docs/reference/sdk.md index f26a4103f..9a5e3e67b 100644 --- a/docs/reference/sdk.md +++ b/docs/reference/sdk.md @@ -116,8 +116,9 @@ shortcut was removed in #603 — use - `root` — Directory to index. When omitted, the index roots at the process cwd (even when `config` is supplied). - `tables` — Programmatic [`Table`](#table) definitions. -- `ignore` — Glob patterns matched against root-relative paths; matched - files are skipped entirely (scan and watch). +- `ignore` — Glob patterns matched against root-relative paths under the + [glob rule](./config.md#glob-rule) (`*` one level, `**` any depth); + matched files are skipped entirely (scan and watch). - `no_ignore` / `noIgnore` — Scan files a `.gitignore` would hide. [Path-tables](./path-tables.md#skip-rules) respect `.gitignore` files by default; the built-in `node_modules`/`.git` skips and any `ignore` patterns diff --git a/packages/python/changelog.d/2026-10-03-one-glob-rule-for-config-globs.md b/packages/python/changelog.d/2026-10-03-one-glob-rule-for-config-globs.md new file mode 100644 index 000000000..803a88ac2 --- /dev/null +++ b/packages/python/changelog.d/2026-10-03-one-glob-rule-for-config-globs.md @@ -0,0 +1,6 @@ +**Changed** `Table(glob=...)`, `DirSQL(ignore=...)` and the `[[table]] glob` / +`[dirsql] ignore` keys of a `.dirsql.toml` now follow the same rule as +path-tables: `*` and `?` match within one path segment and stop at `/`, `**` +matches any depth. `glob="*.json"` selects only the top-level `.json` files +and `ignore=["folder/*"]` hides only the files directly inside `folder/`. +Patterns written with `**` are unaffected. diff --git a/packages/python/e2e-attestations/claude-1223-config-glob-rule.json b/packages/python/e2e-attestations/claude-1223-config-glob-rule.json new file mode 100644 index 000000000..e85e943c5 --- /dev/null +++ b/packages/python/e2e-attestations/claude-1223-config-glob-rule.json @@ -0,0 +1,7 @@ +{ + "command": "just test-e2e", + "ran_at": 1791065836, + "exit_code": 0, + "commit": "62a7c434b5efe4630dbd52818c4c8ede7ca0725d", + "branch": "claude/1223-config-glob-rule" +} diff --git a/packages/python/migrations.d/2026-10-03-one-glob-rule-for-config-globs.md b/packages/python/migrations.d/2026-10-03-one-glob-rule-for-config-globs.md new file mode 100644 index 000000000..30093a756 --- /dev/null +++ b/packages/python/migrations.d/2026-10-03-one-glob-rule-for-config-globs.md @@ -0,0 +1,67 @@ +### Python: `*` stops at `/` in `Table(glob=...)`, `DirSQL(ignore=...)` and `.dirsql.toml` globs + +#### Summary + +Declared-table globs and ignore patterns were compiled with `globset` +defaults, where `*` and `?` also match `/`, so `glob = "*.json"` quietly +selected `.json` files at every depth and `ignore = ["folder/*"]` hid the +whole `folder/` subtree. Path-tables already compiled with +`literal_separator`, so one spelling meant two things. Every surface now +compiles the same way: `*` and `?` match within one path segment, `**` matches +any depth. Affected: the `glob` key of a `[[table]]`, the `[dirsql] ignore` +list, and the `tables` / `ignore` builder parameters of the Rust, Python and +TypeScript SDKs, which share this matcher. + +#### Required changes + +A pattern that relied on `*` crossing `/` must say `**`. + +| Surface | Before | After | +| ------- | ------ | ----- | +| `[[table]] glob`, every depth | `glob = "*.json"` | `glob = "**/*.json"` | +| `[[table]] glob`, a subtree | `glob = "folder/*"` | `glob = "folder/**"` | +| `[dirsql] ignore`, a subtree | `ignore = ["folder/*"]` | `ignore = ["folder/**"]` | +| `[dirsql] ignore`, an extension at every depth | `ignore = ["*.tmp"]` | `ignore = ["**/*.tmp"]` | +| `Table(glob=...)` | `Table(name="t", ddl=..., glob="*.json", on_file=...)` | `Table(name="t", ddl=..., glob="**/*.json", on_file=...)` | +| `DirSQL(ignore=...)` | `DirSQL(root, ignore=["folder/*"])` | `DirSQL(root, ignore=["folder/**"])` | + +A pattern already written with `**` (`**/*.md`, `folder/**`, the built-in +`**/node_modules/**` and `**/.git/**`) matches exactly what it did before. A +pattern meant to stay flat (`glob = "data/*.csv"` over a flat `data/`) needs +no change. + +#### Deprecations removed + +_None._ + +#### Behavior changes without code changes + +- `glob = "*"` matches the files directly inside the root; it used to match + every file at every depth. +- `glob = "folder/*"` matches `folder/a.json` but no longer `folder/sub/b.json`. +- `ignore = ["*"]` hides the top-level files only; `ignore = ["folder/*"]` + hides `folder/a.json` but no longer `folder/sub/b.json`. +- `?` matches one character that is not `/`. +- A `{name}` placeholder matches exactly one path segment, as documented. + +#### Verification + +```bash +cd "$(mktemp -d)" +mkdir -p folder/sub +printf '{}' > root.json +printf '{}' > folder/a.json +printf '{}' > folder/sub/b.json +cat > .dirsql.toml <<'TOML' +[[table]] +name = "files" +ddl = "CREATE TABLE files (path TEXT)" +glob = "*.json" +on-file = '''sh -c 'printf "["; sep=""; for p; do printf "%s{\"path\":\"%s\"}" "$sep" "$p"; sep=","; done; printf "]"' sh''' +TOML +dirsql query "SELECT count(*) AS n FROM files" -c .dirsql.toml +# expected: [{"n":1}] (root.json only; before: 3) +sed -i 's|glob = "\*.json"|glob = "**/*.json"|' .dirsql.toml +dirsql query "SELECT count(*) AS n FROM files" -c .dirsql.toml +# expected: [{"n":3}] +``` diff --git a/packages/rust/changelog.d/2026-10-03-one-glob-rule-for-config-globs.md b/packages/rust/changelog.d/2026-10-03-one-glob-rule-for-config-globs.md new file mode 100644 index 000000000..11d8db728 --- /dev/null +++ b/packages/rust/changelog.d/2026-10-03-one-glob-rule-for-config-globs.md @@ -0,0 +1,8 @@ +**Changed** `[[table]] glob` and `[dirsql] ignore` patterns now follow the +same rule as path-tables: `*` and `?` match within one path segment and stop +at `/`, `**` matches any depth. `glob = "*.json"` selects only the top-level +`.json` files, `glob = "folder/*"` no longer reaches `folder/sub/`, and +`ignore = ["folder/*"]` hides only the files directly inside `folder/`. A +`{name}` placeholder matches exactly one segment. Patterns written with `**` +(including the built-in `**/node_modules/**` and `**/.git/**` skips) are +unaffected. Applies to the SDK `tables` and `ignore` parameters alike. diff --git a/packages/rust/migrations.d/2026-10-03-one-glob-rule-for-config-globs.md b/packages/rust/migrations.d/2026-10-03-one-glob-rule-for-config-globs.md new file mode 100644 index 000000000..3d1d223db --- /dev/null +++ b/packages/rust/migrations.d/2026-10-03-one-glob-rule-for-config-globs.md @@ -0,0 +1,66 @@ +### Core: `*` stops at `/` in `[[table]] glob` and `[dirsql] ignore` + +#### Summary + +Declared-table globs and ignore patterns were compiled with `globset` +defaults, where `*` and `?` also match `/`, so `glob = "*.json"` quietly +selected `.json` files at every depth and `ignore = ["folder/*"]` hid the +whole `folder/` subtree. Path-tables already compiled with +`literal_separator`, so one spelling meant two things. Every surface now +compiles the same way: `*` and `?` match within one path segment, `**` matches +any depth. Affected: the `glob` key of a `[[table]]`, the `[dirsql] ignore` +list, and the `tables` / `ignore` builder parameters of the Rust, Python and +TypeScript SDKs, which share this matcher. + +#### Required changes + +A pattern that relied on `*` crossing `/` must say `**`. + +| Surface | Before | After | +| ------- | ------ | ----- | +| `[[table]] glob`, every depth | `glob = "*.json"` | `glob = "**/*.json"` | +| `[[table]] glob`, a subtree | `glob = "folder/*"` | `glob = "folder/**"` | +| `[dirsql] ignore`, a subtree | `ignore = ["folder/*"]` | `ignore = ["folder/**"]` | +| `[dirsql] ignore`, an extension at every depth | `ignore = ["*.tmp"]` | `ignore = ["**/*.tmp"]` | +| SDK `tables` / `ignore` parameters | same patterns | same rewrites | + +A pattern already written with `**` (`**/*.md`, `folder/**`, the built-in +`**/node_modules/**` and `**/.git/**`) matches exactly what it did before. A +pattern meant to stay flat (`glob = "data/*.csv"` over a flat `data/`) needs +no change. + +#### Deprecations removed + +_None._ + +#### Behavior changes without code changes + +- `glob = "*"` matches the files directly inside the root; it used to match + every file at every depth. +- `glob = "folder/*"` matches `folder/a.json` but no longer `folder/sub/b.json`. +- `ignore = ["*"]` hides the top-level files only; `ignore = ["folder/*"]` + hides `folder/a.json` but no longer `folder/sub/b.json`. +- `?` matches one character that is not `/`. +- A `{name}` placeholder matches exactly one path segment, as documented. + +#### Verification + +```bash +cd "$(mktemp -d)" +mkdir -p folder/sub +printf '{}' > root.json +printf '{}' > folder/a.json +printf '{}' > folder/sub/b.json +cat > .dirsql.toml <<'TOML' +[[table]] +name = "files" +ddl = "CREATE TABLE files (path TEXT)" +glob = "*.json" +on-file = '''sh -c 'printf "["; sep=""; for p; do printf "%s{\"path\":\"%s\"}" "$sep" "$p"; sep=","; done; printf "]"' sh''' +TOML +dirsql query "SELECT count(*) AS n FROM files" -c .dirsql.toml +# expected: [{"n":1}] (root.json only; before: 3) +sed -i 's|glob = "\*.json"|glob = "**/*.json"|' .dirsql.toml +dirsql query "SELECT count(*) AS n FROM files" -c .dirsql.toml +# expected: [{"n":3}] +``` diff --git a/packages/rust/src/matcher.rs b/packages/rust/src/matcher.rs index 28071a70e..47a386303 100644 --- a/packages/rust/src/matcher.rs +++ b/packages/rust/src/matcher.rs @@ -1,4 +1,4 @@ -use globset::{Glob, GlobSet, GlobSetBuilder}; +use globset::{Glob, GlobBuilder, GlobSet, GlobSetBuilder}; use std::path::Path; /// Result of matching a file path against a glob pattern. @@ -83,6 +83,12 @@ fn glob_with_placeholders_as_star(pattern: &str) -> String { out } +/// One glob rule on every surface: `*` and `?` stop at `/`, `**` crosses it, +/// exactly as `scanner::compile_glob` reads a path-table. +fn compile(pattern: &str) -> Result { + GlobBuilder::new(pattern).literal_separator(true).build() +} + impl TableMatcher { /// Build a new matcher from (glob_pattern, table_name) pairs and ignore patterns. /// Glob patterns may contain `{name}` placeholders, which match like `*`. @@ -94,7 +100,7 @@ impl TableMatcher { for (pattern, table_name) in mappings { let glob_pattern = glob_with_placeholders_as_star(pattern); let mut builder = GlobSetBuilder::new(); - builder.add(Glob::new(&glob_pattern)?); + builder.add(compile(&glob_pattern)?); entries.push(PatternEntry { glob_set: builder.build()?, table_name: table_name.to_string(), @@ -104,9 +110,9 @@ impl TableMatcher { let mut ignore_builder = GlobSetBuilder::new(); let mut ignore_dir_builder = GlobSetBuilder::new(); for pattern in ignore_patterns { - ignore_builder.add(Glob::new(pattern)?); + ignore_builder.add(compile(pattern)?); if let Some(subtree) = pattern.strip_suffix("/**") { - ignore_dir_builder.add(Glob::new(subtree)?); + ignore_dir_builder.add(compile(subtree)?); } } let ignore_set = ignore_builder.build()?; @@ -273,6 +279,7 @@ mod tests { assert_eq!(names(&matcher, "file1.txt"), vec!["t"]); assert_eq!(names(&matcher, "fileA.txt"), vec!["t"]); assert!(matcher.match_all(Path::new("file.txt")).is_empty()); + assert!(matcher.match_all(Path::new("file/.txt")).is_empty()); } #[test] @@ -380,4 +387,117 @@ mod tests { assert_eq!(placeholder_names("{a{b}"), vec!["b".to_string()]); assert_eq!(glob_with_placeholders_as_star("{a{b}"), "{a*"); } + + /// Two files at depth 0, two under `folder/`, two under `folder/sub/`, one + /// under a sibling directory. + const FIXTURE: [&str; 7] = [ + "root.md", + "root.txt", + "folder/a.md", + "folder/a.txt", + "folder/sub/b.md", + "folder/sub/b.txt", + "sibling/c.md", + ]; + + fn matched(glob: &str) -> Vec<&'static str> { + let matcher = TableMatcher::new(&[(glob, "t")], &[]).unwrap(); + FIXTURE + .into_iter() + .filter(|p| !matcher.match_all(Path::new(p)).is_empty()) + .collect() + } + + fn hidden(ignore: &str) -> Vec<&'static str> { + let matcher = TableMatcher::new(&[], &[ignore]).unwrap(); + FIXTURE + .into_iter() + .filter(|p| matcher.is_ignored(Path::new(p))) + .collect() + } + + #[test] + fn a_lone_star_glob_matches_depth_zero_only() { + assert_eq!(matched("*"), ["root.md", "root.txt"]); + } + + #[test] + fn a_bare_directory_name_glob_matches_nothing() { + assert!(matched("folder").is_empty()); + } + + #[test] + fn a_directory_slash_star_glob_stops_at_the_next_separator() { + assert_eq!(matched("folder/*"), ["folder/a.md", "folder/a.txt"]); + } + + #[test] + fn a_directory_slash_double_star_glob_matches_any_depth_below() { + assert_eq!( + matched("folder/**"), + [ + "folder/a.md", + "folder/a.txt", + "folder/sub/b.md", + "folder/sub/b.txt" + ] + ); + } + + #[test] + fn a_leading_double_star_glob_matches_the_suffix_at_any_depth() { + assert_eq!( + matched("**/*.md"), + ["root.md", "folder/a.md", "folder/sub/b.md", "sibling/c.md"] + ); + } + + #[test] + fn a_double_star_slash_star_glob_matches_every_file() { + assert_eq!(matched("**/*"), FIXTURE); + } + + #[test] + fn a_directory_slash_star_ignore_hides_only_that_directory_s_files() { + assert_eq!(hidden("folder/*"), ["folder/a.md", "folder/a.txt"]); + } + + #[test] + fn a_lone_star_ignore_hides_depth_zero_only() { + assert_eq!(hidden("*"), ["root.md", "root.txt"]); + } + + #[test] + fn the_default_ignores_still_hide_at_every_depth() { + let matcher = TableMatcher::new(&[], &["**/node_modules/**", "**/.git/**"]).unwrap(); + for path in [ + "node_modules/pkg/index.js", + "apps/site/node_modules/pkg/dist/index.js", + ".git/config", + "vendor/lib/.git/HEAD", + ] { + assert!( + matcher.is_ignored(Path::new(path)), + "{path} must stay hidden" + ); + } + } + + #[test] + fn a_subtree_ignore_whose_prefix_has_a_star_marks_one_level_of_directories() { + let matcher = TableMatcher::new(&[], &["folder/*/**"]).unwrap(); + assert!(matcher.is_ignored_dir(Path::new("folder/sub"))); + assert!(!matcher.is_ignored_dir(Path::new("folder/sub/deeper"))); + } + + #[test] + fn a_placeholder_matches_exactly_one_path_segment() { + let matcher = TableMatcher::new(&[("data/{id}/metadata.json", "a")], &[]).unwrap(); + assert_eq!(names(&matcher, "data/x/metadata.json"), vec!["a"]); + assert!( + matcher + .match_all(Path::new("data/x/y/metadata.json")) + .is_empty() + ); + } } diff --git a/packages/rust/tests/from_config.rs b/packages/rust/tests/from_config.rs index dd6aa8a7b..468e2277b 100644 --- a/packages/rust/tests/from_config.rs +++ b/packages/rust/tests/from_config.rs @@ -422,3 +422,65 @@ on-file = '''sh -c 'r=$(printf %s "$1" | tr "\\\\" /); shift; printf "["; sep="" assert_eq!(rows[0]["path"], Value::Text("data.csv".into())); assert_eq!(rows[0]["basename"], Value::Text("data.csv".into())); } + +const PATH_HOOK: &str = r#"on-file = '''sh -c 'r=$(printf %s "$1" | tr "\\\\" /); shift; printf "["; sep=""; for p; do p=$(printf %s "$p" | tr "\\\\" /); rel=${p#"$r"/}; printf "%s{\"path\":\"%s\"}" "$sep" "$rel"; sep=","; done; printf "]"' sh {root}'''"#; + +/// root.json at depth 0, folder/a.json at depth 1, folder/sub/b.json at depth 2. +fn nested_json_fixture() -> TempDir { + let root = TempDir::new().unwrap(); + fs::create_dir_all(root.path().join("folder").join("sub")).unwrap(); + fs::write(root.path().join("root.json"), "{}").unwrap(); + fs::write(root.path().join("folder").join("a.json"), "{}").unwrap(); + fs::write(root.path().join("folder").join("sub").join("b.json"), "{}").unwrap(); + root +} + +fn paths(root: &TempDir, config: &str) -> Vec { + fs::write(root.path().join(".dirsql.toml"), config).unwrap(); + let db = DirSQL::builder() + .root(root.path()) + .config(root.path().join(".dirsql.toml")) + .build() + .unwrap(); + db.query("SELECT path FROM files ORDER BY path") + .unwrap() + .into_iter() + .map(|r| match &r["path"] { + Value::Text(s) => s.clone(), + other => panic!("path must be text, got {other:?}"), + }) + .collect() +} + +#[test] +fn from_config_a_star_glob_ingests_depth_zero_only() { + let root = nested_json_fixture(); + let config = format!( + r#" +[[table]] +name = "files" +ddl = "CREATE TABLE files (path TEXT)" +glob = "*.json" +{PATH_HOOK} +"# + ); + assert_eq!(paths(&root, &config), ["root.json"]); +} + +#[test] +fn from_config_a_directory_slash_star_ignore_hides_only_that_directory_s_files() { + let root = nested_json_fixture(); + let config = format!( + r#" +[dirsql] +ignore = ["folder/*"] + +[[table]] +name = "files" +ddl = "CREATE TABLE files (path TEXT)" +glob = "**/*.json" +{PATH_HOOK} +"# + ); + assert_eq!(paths(&root, &config), ["folder/sub/b.json", "root.json"]); +} diff --git a/packages/ts/changelog.d/2026-10-03-one-glob-rule-for-config-globs.md b/packages/ts/changelog.d/2026-10-03-one-glob-rule-for-config-globs.md new file mode 100644 index 000000000..965253e52 --- /dev/null +++ b/packages/ts/changelog.d/2026-10-03-one-glob-rule-for-config-globs.md @@ -0,0 +1,6 @@ +**Changed** `Table.glob`, the `ignore` constructor option and the +`[[table]] glob` / `[dirsql] ignore` keys of a `.dirsql.toml` now follow the +same rule as path-tables: `*` and `?` match within one path segment and stop +at `/`, `**` matches any depth. `glob: "*.json"` selects only the top-level +`.json` files and `ignore: ["folder/*"]` hides only the files directly inside +`folder/`. Patterns written with `**` are unaffected. diff --git a/packages/ts/e2e-attestations/claude-1223-config-glob-rule.json b/packages/ts/e2e-attestations/claude-1223-config-glob-rule.json new file mode 100644 index 000000000..b9e6b5ca9 --- /dev/null +++ b/packages/ts/e2e-attestations/claude-1223-config-glob-rule.json @@ -0,0 +1,7 @@ +{ + "command": "pnpm test:e2e", + "ran_at": 1791065880, + "exit_code": 0, + "commit": "c7393ed1e19747906e942707e759b671a6e9490e", + "branch": "claude/1223-config-glob-rule" +} diff --git a/packages/ts/migrations.d/2026-10-03-one-glob-rule-for-config-globs.md b/packages/ts/migrations.d/2026-10-03-one-glob-rule-for-config-globs.md new file mode 100644 index 000000000..e3ac670a9 --- /dev/null +++ b/packages/ts/migrations.d/2026-10-03-one-glob-rule-for-config-globs.md @@ -0,0 +1,67 @@ +### TypeScript: `*` stops at `/` in `Table.glob`, the `ignore` option and `.dirsql.toml` globs + +#### Summary + +Declared-table globs and ignore patterns were compiled with `globset` +defaults, where `*` and `?` also match `/`, so `glob = "*.json"` quietly +selected `.json` files at every depth and `ignore = ["folder/*"]` hid the +whole `folder/` subtree. Path-tables already compiled with +`literal_separator`, so one spelling meant two things. Every surface now +compiles the same way: `*` and `?` match within one path segment, `**` matches +any depth. Affected: the `glob` key of a `[[table]]`, the `[dirsql] ignore` +list, and the `tables` / `ignore` builder parameters of the Rust, Python and +TypeScript SDKs, which share this matcher. + +#### Required changes + +A pattern that relied on `*` crossing `/` must say `**`. + +| Surface | Before | After | +| ------- | ------ | ----- | +| `[[table]] glob`, every depth | `glob = "*.json"` | `glob = "**/*.json"` | +| `[[table]] glob`, a subtree | `glob = "folder/*"` | `glob = "folder/**"` | +| `[dirsql] ignore`, a subtree | `ignore = ["folder/*"]` | `ignore = ["folder/**"]` | +| `[dirsql] ignore`, an extension at every depth | `ignore = ["*.tmp"]` | `ignore = ["**/*.tmp"]` | +| `Table.glob` | `new Table({ name: "t", ddl, glob: "*.json", onFile })` | `new Table({ name: "t", ddl, glob: "**/*.json", onFile })` | +| `ignore` option | `new DirSQL({ root, ignore: ["folder/*"] })` | `new DirSQL({ root, ignore: ["folder/**"] })` | + +A pattern already written with `**` (`**/*.md`, `folder/**`, the built-in +`**/node_modules/**` and `**/.git/**`) matches exactly what it did before. A +pattern meant to stay flat (`glob = "data/*.csv"` over a flat `data/`) needs +no change. + +#### Deprecations removed + +_None._ + +#### Behavior changes without code changes + +- `glob = "*"` matches the files directly inside the root; it used to match + every file at every depth. +- `glob = "folder/*"` matches `folder/a.json` but no longer `folder/sub/b.json`. +- `ignore = ["*"]` hides the top-level files only; `ignore = ["folder/*"]` + hides `folder/a.json` but no longer `folder/sub/b.json`. +- `?` matches one character that is not `/`. +- A `{name}` placeholder matches exactly one path segment, as documented. + +#### Verification + +```bash +cd "$(mktemp -d)" +mkdir -p folder/sub +printf '{}' > root.json +printf '{}' > folder/a.json +printf '{}' > folder/sub/b.json +cat > .dirsql.toml <<'TOML' +[[table]] +name = "files" +ddl = "CREATE TABLE files (path TEXT)" +glob = "*.json" +on-file = '''sh -c 'printf "["; sep=""; for p; do printf "%s{\"path\":\"%s\"}" "$sep" "$p"; sep=","; done; printf "]"' sh''' +TOML +dirsql query "SELECT count(*) AS n FROM files" -c .dirsql.toml +# expected: [{"n":1}] (root.json only; before: 3) +sed -i 's|glob = "\*.json"|glob = "**/*.json"|' .dirsql.toml +dirsql query "SELECT count(*) AS n FROM files" -c .dirsql.toml +# expected: [{"n":3}] +```