diff --git a/docs/.vitepress/config.ts b/docs/.vitepress/config.ts index 464102e0d..790ddb57f 100644 --- a/docs/.vitepress/config.ts +++ b/docs/.vitepress/config.ts @@ -77,6 +77,7 @@ export default defineConfig({ { text: 'Command Hooks', link: '/reference/hooks' }, { text: 'Columns', link: '/reference/columns' }, { text: 'Path-tables', link: '/reference/path-tables' }, + { text: 'Globs', link: '/reference/glob' }, { text: 'HTTP API', link: '/reference/http-api' }, { text: 'SDK', link: '/reference/sdk' } ] diff --git a/docs/reference/config.md b/docs/reference/config.md index d7edc0253..3cbbddc21 100644 --- a/docs/reference/config.md +++ b/docs/reference/config.md @@ -47,13 +47,12 @@ ignore = ["node_modules/**", ".git/**"] ### Glob rule -Every glob in dirsql — `ignore`, a `[[table]]`'s `glob`, and a -[path-table](./path-tables.md#writing-the-path) — reads as the shell does: -`*` and `?` match within one path segment and never cross `/`; `**` matches -any depth. So `ignore = ["*"]` hides only the files directly inside the root, -`ignore = ["build/*"]` hides `build/a.o` but not `build/sub/b.o`, and -`glob = "*.json"` selects only the top-level `.json` files; write -`**/*.json` to select them at every depth. +Every glob in dirsql, whether `ignore`, a `[[table]]`'s `glob` or a +[path-table](./path-tables.md#writing-the-path), reads as bash does with +`globstar` on. The [glob page](./glob.md) has the rules, examples and every +divergence. In short, `*` and `?` stay within one path segment, `**` crosses +directories, and `glob = "*.json"` selects only top-level `.json` files; write +`**/*.json` for every depth. Persistence is not a config key. Keep the SQLite index on disk between runs with the [`--persist [PATH]` CLI flag](./cli.md#dirsql-server) — a machine-local @@ -232,7 +231,7 @@ what its required `on-file` command emits — dirsql injects nothing (see |---|---|---| | `name` | yes | The table's SQL name — the name you query it by. Declared, never derived from `ddl`: dirsql does not read the DDL text. The `ddl` must create a table by this name; if it doesn't, loading fails. | | `ddl` | yes | A SQL batch, run verbatim — any number of statements. It must create a table called `name`; that table holds the file rows, and only the columns it declares are kept (keys the `on-file` command emits that are not declared are dropped). The rest of the batch is yours: indexes, virtual tables, triggers. See [Batch `ddl`](#batch-ddl). | -| `glob` | yes | Glob pattern, [anchored](#glob-anchor) at the config file's directory, under the [one glob rule](#glob-rule): `*` matches one level, `**` any depth. Every table whose glob matches a file receives that file's rows — a file can populate multiple tables. A `{name}` segment is rewritten to `*` (it matches one path segment but captures nothing). | +| `glob` | yes | Glob pattern, [anchored](#glob-anchor) at the config file's directory, under the [one glob rule](#glob-rule): `*` matches one level, `**` any depth. Every table whose glob matches a file receives that file's rows — a file can populate multiple tables. A `{name}` is a literal, as in bash and a path-table; the [glob page](./glob.md) has the syntax. | | `on-file` | **yes** | A command run once per table, with every matched file's absolute path appended as a trailing argument; its stdout (one JSON array of row objects) is the table's rows. Must be non-empty. A `[[table]]` with no `on-file` is a load error (see [parse errors](#parse-errors)). See [Command hooks](./hooks.md#on-file). | | `strict` | no (default `false`) | When `true`, rows whose keys do not exactly match the declared columns are rejected with an error: extra keys error, and every declared column must be supplied by the `on-file` output. When `false`, extra keys are dropped and missing columns become `NULL`. | diff --git a/docs/reference/glob.md b/docs/reference/glob.md new file mode 100644 index 000000000..1c001acf4 --- /dev/null +++ b/docs/reference/glob.md @@ -0,0 +1,140 @@ +# Globs + +A glob is the string you write where a table name goes (`FROM './docs/*.md'`) +or in a config `[[table]]`'s `glob`. dirsql reads it the way `bash` does with +`globstar` on and `dotglob` off, and every place it does not is listed under +[Divergences from bash](#divergences-from-bash). + +## Why bash + +Bash with `globstar` is the most popular glob behavior. Bash, zsh, node-glob +and minimatch (npm, ESLint, VS Code, GitHub Actions) and Python's `glob` agree +on one core: `*`, `?` and `[...]`, recursive `**`, `{a,b}` braces, `*` hiding +dotfiles, and `**` not entering symlinked directories. Python has no braces +and follows symlinks; bash breaks that tie. POSIX has no `**` or braces. So +there is one standard nobody has to learn, the same string works in a +[path-table](./path-tables.md) and in a [config](./config.md) `glob`, and each +divergence is a written decision. + +## The fixture + +Every example below runs against this tree. A name ending in `/` is an empty +directory, `a -> b` is a symlink, and `.gitignore` holds `*.log`. + + +```text +top.md +a.md +b.md +c.md +1.md +2.md +3.md +4.md +x.txt +ignored.log +{q}.md +[a].md +[z.md +my file.md +.env +.git/ +.config/app.toml +.config/.secret +.config/sub/deep.md +docs/guide.md +docs/api.md +docs/.draft.md +docs/nested/deep.md +docs/nested/.cache/x.md +node_modules/pkg/index.js +real/r.md +linkdir -> real +link.md -> top.md +broken -> nowhere +``` + +## Examples + +Each row lists the files the pattern returns, sorted. Every row without a +divergence was produced by `bash -O globstar -O nullglob` with `dotglob` off, +run from the fixture's root, keeping only files. The `Divergence` column names +the rule from [below](#divergences-from-bash) that makes a row differ from +bash's raw output. `$ROOT` is the fixture's directory and `$BASE` its parent. + +A test runs every row against dirsql as a path-table and, with the leading +`./` dropped, as a config `glob`, so this table cannot drift from the code. + + +| Pattern | Rows | Divergence | +| --- | --- | --- | +| `./*.md` | `1.md`, `2.md`, `3.md`, `4.md`, `[a].md`, `[z.md`, `a.md`, `b.md`, `c.md`, `link.md`, `my file.md`, `top.md`, `{q}.md` | | +| `./**/*.md` | `1.md`, `2.md`, `3.md`, `4.md`, `[a].md`, `[z.md`, `a.md`, `b.md`, `c.md`, `docs/api.md`, `docs/guide.md`, `docs/nested/deep.md`, `link.md`, `linkdir/r.md`, `my file.md`, `real/r.md`, `top.md`, `{q}.md` | | +| `./**` | `1.md`, `2.md`, `3.md`, `4.md`, `[a].md`, `[z.md`, `a.md`, `b.md`, `c.md`, `docs/api.md`, `docs/guide.md`, `docs/nested/deep.md`, `link.md`, `my file.md`, `real/r.md`, `top.md`, `x.txt`, `{q}.md` | | +| `./?.md` | `1.md`, `2.md`, `3.md`, `4.md`, `a.md`, `b.md`, `c.md` | | +| `./[ab].md` | `a.md`, `b.md` | | +| `./[!a].md` | `1.md`, `2.md`, `3.md`, `4.md`, `b.md`, `c.md` | | +| `./\[a\].md` | `[a].md` | | +| `./[z.md` | `[z.md` | | +| `./{a,b}.md` | `a.md`, `b.md` | | +| `./{q}.md` | `{q}.md` | | +| `./{a.md` | none | | +| `./{1..3}.md` | `1.md`, `2.md`, `3.md` | | +| `./my file.md` | `my file.md` | | +| `./.*` | `.env`, `.gitignore` | | +| `./**/.*` | `.env`, `.gitignore`, `docs/.draft.md` | | +| `./.env` | `.env` | | +| `./.config/**` | `.config/app.toml`, `.config/sub/deep.md` | | +| `./docs/.*` | `docs/.draft.md` | | +| `./link.md` | `link.md` | | +| `./linkdir/*` | `linkdir/r.md` | | +| `./*/r.md` | `linkdir/r.md`, `real/r.md` | | +| `./**/r.md` | `linkdir/r.md`, `real/r.md` | | +| `./real/**` | `real/r.md` | | +| `./docs` | `docs/api.md`, `docs/guide.md` | directory-name | +| `./docs/` | `docs/api.md`, `docs/guide.md` | directory-name | +| `./*/` | `docs/api.md`, `docs/guide.md`, `linkdir/r.md`, `real/r.md` | directory-name | +| `./d*` | none | files-only | +| `./*.log` | none | gitignore | +| `./**/*.js` | none | node-modules | +| `./node_modules/*/index.js` | `node_modules/pkg/index.js` | | +| `./docs/**/*.md` | `docs/api.md`, `docs/guide.md`, `docs/nested/deep.md` | | +| `./docs/*.md` | `docs/api.md`, `docs/guide.md` | | +| `$ROOT/docs/*.md` | `$ROOT/docs/api.md`, `$ROOT/docs/guide.md` | absolute | +| `~/docs/*.md` | `$ROOT/docs/api.md`, `$ROOT/docs/guide.md` | absolute | +| `../proj/docs/*.md` | `$BASE/proj/docs/api.md`, `$BASE/proj/docs/guide.md` | absolute | + +## Divergences from bash + +| Tag | Rule | Why | +| --- | --- | --- | +| `gitignore` | Inside a git repo, `.gitignore` files hide matching files and directories. `--no-ignore` turns it off. | A query over a repo should not drown in build output. | +| `directory-name` | A pattern's last component that names a directory lists the files directly inside it, like `ls docs`. A trailing `/` is `*` appended. | A table is rows of files, and `ls` is the shell habit. | +| `files-only` | A pattern that matches a directory but no file returns nothing. | Every row is a file. | +| `node-modules` | A `node_modules` directory is skipped unless the pattern spells its name. | A zero-config `./**` should not walk dependencies. | +| `absolute` | `/`, `~/` and `../` patterns report absolute `path` values, with `..` folded out textually. | The scanned root is derived from the pattern, so there is no relative base. | + +`.git` needs no rule of its own: the dotfile rule hides it. + +## Where the glob is anchored + +A path-table pattern is relative to the index root and starts with `./`. A config `[[table]] glob` +is relative to the directory holding its config file, whatever directory you +run `dirsql` from, and is written without the `./`: `./docs/*.md` in a path-table +is `glob = "docs/*.md"` in a config. A config table whose glob matches no files prints +`dirsql: table 'name': glob 'pattern' matched no files under ` to +stderr; the exit code is unchanged. Path-tables never warn. + +## Edge cases + +- A `{`, `}` or `[` with no partner is a literal character. +- A brace group with no comma and no `..` is literal (`{q}`); `{1..3}` and + `{a..c}` expand, as do zero-padded + `{01..10}` and stepped `{1..10..2}` ranges. Groups nest. +- `\[` is a literal bracket. +- A dot-named file or directory is listed only where a component spells the + dot at that depth: `./**/.*` finds `docs/.draft.md` but does not enter + `.config/`. +- A symlinked file is a file. A symlinked directory is entered by any component + except `**`. +- A broken link lists nothing. diff --git a/docs/reference/path-tables.md b/docs/reference/path-tables.md index 0b316f8cc..851efcb89 100644 --- a/docs/reference/path-tables.md +++ b/docs/reference/path-tables.md @@ -52,16 +52,8 @@ depth, as in the shell. A trailing `/` is `*` appended, so `'./*/'` is | `'./notes/today.md'` | exactly that one file — one file is one row | A path containing `*`, `?`, `[` or `{` is a glob and is used exactly as -written: `*` matches within a single directory, `**` crosses directories. - -Braces expand as bash expands them. A group holding a comma is an -alternation: `'./docs/{a,b}.md'` is `docs/a.md` and `docs/b.md`, and groups -nest. A group holding a `..` range counts: `'./log/{1..3}.txt'`, -`'./{a..c}.md'`, `'./{01..10}.md'` (zero-padded) and `'./{1..10..2}.md'` -(every second value). Any other brace is literal text, so `'./br/{q}.md'` -names the file `br/{q}.md`, and `\{` is always a literal brace. A `{`, `}` -or `[` with no partner is literal too, in a path-table and in a config -`[[table]] glob`. +written. The [glob page](./glob.md) has the syntax, examples and every +divergence from bash, including braces and unbalanced brackets. The scan starts at the last directory named outright before the first glob component, and descends only as deep as the pattern can match -- diff --git a/packages/rust/tests/glob_conformance.rs b/packages/rust/tests/glob_conformance.rs new file mode 100644 index 000000000..8fcc58807 --- /dev/null +++ b/packages/rust/tests/glob_conformance.rs @@ -0,0 +1,177 @@ +//! The example table in `docs/reference/glob.md` is the specification: every +//! row runs against dirsql as a path-table and as a config `[[table]] glob`. +#![cfg(all(unix, feature = "cli"))] + +use std::collections::HashMap; +use std::fs; +use std::os::unix::fs::symlink; +use std::path::Path; + +use assert_cmd::prelude::*; +use dirsql::{DirSQL, Table, Value}; +use tempfile::TempDir; + +struct Row { + pattern: String, + expected: Vec, + divergence: String, +} + +fn page() -> String { + let path = Path::new(env!("CARGO_MANIFEST_DIR")).join("../../docs/reference/glob.md"); + fs::read_to_string(path).unwrap() +} + +fn after<'a>(page: &'a str, marker: &str) -> &'a str { + let at = page.find(marker).unwrap_or_else(|| panic!("no {marker}")); + &page[at + marker.len()..] +} + +fn tree_entries(page: &str) -> Vec { + after(page, "") + .lines() + .skip(2) + .take_while(|line| !line.starts_with("```")) + .map(str::to_string) + .collect() +} + +fn rows(page: &str) -> Vec { + after(page, "") + .lines() + .skip(1) + .take_while(|line| line.starts_with('|')) + .skip(2) + .map(|line| { + let cells: Vec<&str> = line.trim_matches('|').split(" | ").map(str::trim).collect(); + let backticked = |cell: &str| -> Vec { + cell.split('`') + .skip(1) + .step_by(2) + .map(str::to_string) + .collect() + }; + Row { + pattern: backticked(cells[0]).remove(0), + expected: backticked(cells[1]), + divergence: cells[2].to_string(), + } + }) + .collect() +} + +fn build(root: &Path, entries: &[String]) { + for entry in entries { + let at = root.join(entry.split(" -> ").next().unwrap()); + if let Some((_, target)) = entry.split_once(" -> ") { + symlink(target, at).unwrap(); + } else if entry.ends_with('/') { + fs::create_dir_all(at).unwrap(); + } else { + fs::create_dir_all(at.parent().unwrap()).unwrap(); + fs::write(at, "x\n").unwrap(); + } + } + fs::write(root.join(".gitignore"), "*.log\n").unwrap(); +} + +fn texts(db: &DirSQL, sql: &str, column: &str) -> Vec { + let mut out: Vec = db + .query(sql) + .unwrap() + .iter() + .map(|r| match r.get(column) { + Some(Value::Text(s)) => s.clone(), + other => panic!("{column} was not text: {other:?}"), + }) + .collect(); + out.sort(); + out +} + +fn path_table_paths(root: &Path, pattern: &str) -> Vec { + let output = std::process::Command::cargo_bin("dirsql") + .unwrap() + .args([ + "query", + &format!("SELECT path FROM '{pattern}'"), + "--format", + "json", + ]) + .current_dir(root) + .env("HOME", root) + .output() + .unwrap(); + if !output.status.success() { + return vec![String::from_utf8_lossy(&output.stderr).trim().to_string()]; + } + let rows: Vec = serde_json::from_slice(&output.stdout).unwrap(); + let mut out: Vec = rows + .iter() + .map(|row| row["path"].as_str().unwrap().to_string()) + .collect(); + out.sort(); + out +} + +fn config_paths(root: &Path, glob: &str) -> Vec { + let base = root.to_path_buf(); + let table = Table::new("t", "CREATE TABLE t (p TEXT)", glob, move |path| { + let rel = Path::new(path) + .strip_prefix(&base) + .unwrap_or(Path::new(path)); + vec![HashMap::from([( + "p".to_string(), + Value::Text(rel.to_string_lossy().into_owned()), + )])] + }); + let db = match DirSQL::builder().root(root).table(table).build() { + Ok(db) => db, + Err(err) => return vec![err.to_string()], + }; + texts(&db, "SELECT p FROM t", "p") +} + +#[test] +fn every_row_of_the_glob_page_holds() { + let page = page(); + let base = TempDir::new().unwrap(); + let root = base.path().join("proj"); + fs::create_dir(&root).unwrap(); + build(&root, &tree_entries(&page)); + let (root_text, base_text) = ( + root.to_string_lossy().into_owned(), + base.path().to_string_lossy().into_owned(), + ); + + let mut failures = Vec::new(); + for row in rows(&page) { + let pattern = row.pattern.replace("$ROOT", &root_text); + let mut expected: Vec = row + .expected + .iter() + .filter(|cell| *cell != "none") + .map(|cell| { + cell.replace("$ROOT", &root_text) + .replace("$BASE", &base_text) + }) + .collect(); + expected.sort(); + + let got = path_table_paths(&root, &pattern); + if got != expected { + failures.push(format!( + "path-table {}: {got:?} != {expected:?}", + row.pattern + )); + } + if row.divergence != "absolute" { + let glob = pattern.strip_prefix("./").unwrap(); + let got = config_paths(&root, glob); + if got != expected { + failures.push(format!("config {}: {got:?} != {expected:?}", row.pattern)); + } + } + } + assert!(failures.is_empty(), "{}", failures.join("\n")); +}