Repository navigation
rust: share the vtab scaffolding between the two path tables - #1112
Conversation
`vtab.rs` and `parsed_vtab.rs` are two `dirsql_*` virtual tables built the same way, and each carried its own copy of the way: module-argument decoding through `sql_literal::unquote`, the arity error, glob and ignore-pattern compilation, the gitignore switch, cursor arithmetic, and the whole `VTab` / `CreateVTab` / `VTabCursor` shape. That shape now lives once in `vtab_scaffold`, as a generic `ScaffoldTab<S>` / `ScaffoldCursor<S>` over a `TableSource` trait. Each table supplies only the three things that actually differ: how its module arguments decode, how it produces a row set, and how a row becomes a cell. One behavior-preserving consequence worth naming: the parsed table used to clone its whole row vector into every cursor at `xOpen`; the scaffold hands out an `Arc` per `xFilter` instead, so a join over a parsed table no longer copies the rows per outer row. Public surface is unchanged -- `MODULE_NAME`, `STAT_COLUMNS`, `declared_schema` and both `load_module` functions keep their paths and signatures, and the scaffold module is crate-private.
Two integration assertions over the shared scaffold: rowids count up from zero in scan order, and a join against a small ordinary table plans as a plain nested loop rather than an automatic index. The second is what the declared flat scan cost buys; left at SQLite's near-infinite default every join path costs the same and the planner reshapes the join around it.
|
CI status: everything green except Not fixing this here by weakening or deleting a test. Holding for the maintainer to decide between a memory cap on the gate (#1113, preferred), a re-run, or a test-side bound on the row collection. |
…y-root queries Five cargo-mutants mutants in vtab_scaffold.rs (at_eof -> false, >= -> <, next -> Ok(()), += -> *=, eof -> false) make the cursor never reach EOF. Nothing killed them except the OOM killer: the lib tests that run a real SELECT through Db::query collected an infinite row stream until the runner died, which cargo-mutants counted as caught. Those four lib tests query an empty root, so only the eof mutants looped there; the next mutants passed the lib binary and hung in the integration binaries instead. A direct ScaffoldCursor test over a two-row fake source now fails all five in the lib binary in microseconds, and cargo stops before the integration binaries run. LIMIT 1 on the empty-root queries bounds them under the eof mutants either way. Refs #1113
|
The red |
|
Status after 0f12a5a:
|
The stale release-ci check runs left on the previous sha carry version numbers willfire no longer predicts, so CI Gate reports them as unpredicted checks and cannot confirm the set is complete.
Fixes #1080. Part of #1072.
vtab.rsandparsed_vtab.rscarried parallel copies of the same virtual-table scaffolding. This moves the shared part into a new internal module,packages/rust/src/vtab_scaffold.rs: module-argument decoding (user_args,arity_error), glob and ignore-pattern compilation, the gitignore switch, cursor arithmetic (at_eof,rowid_of), and the read-onlyVTab/VTabCursorimpls (xBestIndex/xFilter/xNext/xEof/xColumn/xRowid). Each table now implements one trait,TableSource, that supplies what differs: its own argument parsing, how it produces a row set, and how a row becomes a cell.Public surface is unchanged:
MODULE_NAME,STAT_COLUMNS,declared_schema, and bothload_modulefunctions keep their paths and signatures. No SQL surface or observable behavior changes. Inline#[cfg(test)]modules moved with the code they cover;tests/vtab.rsgains assertions for the scan cost and rowids the scaffold declares.Measured delta
The issue estimated about −100 net source lines. The measured result is −13:
vtab.rs−160,parsed_vtab.rs−114,vtab_scaffold.rs+331 (of which ~130 are the moved inline unit tests),lib.rs+1. The two tables shared less than the estimate assumed once the per-tableconnect/rows/columnbodies and the generic plumbing were accounted for. I did not hunt for more lines to cut; the duplication the issue names is gone, which was the point.Verification
cargo fmt --check,cargo clippy,cargo test --workspace --features cli: green.cargo +nightly llvm-cov --lib --features cli --branch): up on all four dimensions, above the floors intesting-conventions.toml; no floor change needed.unit mutation(rust, diff-scoped againstmain): 30 mutants, 0 survivors. Run locally once. Five of the thirty (at_eof -> false,>=-><inat_eof,next -> Ok(()),+=->*=innext,eof -> false) make the cursor never reach EOF, so the lib unit tests that run a real query through a path table collect rows without bound; each was "caught" only by the kernel OOM-killing the test binary at 19–39 GB RSS. Filed separately as a gate-infrastructure problem (Local mutation gate has no memory cap; runaway mutants OOM the host #1113); not a test weakness to fix here. CI runs the same gate again on the GitHub runner.Benchmarks
cargo bench -p dirsql(criterion:db_bench,scanner_bench,differ_bench,matcher_bench) with--save-baseline mainon35f34a2, then--baseline mainon this branch, same target dir. None of the four exercise the virtual tables, so this is a no-regression check on the rest of the crate, not a measurement of the change. Everything within ±5% except twodbcases that got faster (create_table−30%,insert_rows/1−29%: SQLite open / first insert, untouched code, first-run warmup on the baseline side) and one first-pass outlier,matcher/is_ignored_20_patterns+90%. A rerun ofmatcher_benchalone against the same baseline gave −0.07% (p = 0.89) for that case and every other matcher case within ±2%, so it was machine noise.For the vtab paths themselves, hyperfine on the debug CLI, main vs branch:
dirsql_pathquery 3.1 ± 0.2 ms vs 3.2 ± 0.6 ms;dirsql_parsedquery (--on-file) 23.0 ± 0.8 ms vs 22.9 ± 0.9 ms. No measurable difference.Manual run
Debug CLI built from this branch (
cargo build --features cli -p dirsql), run in a fixture dir withdocs/a.md(hello) anddocs/b.md(world!!):The first goes through
dirsql_path(stat columns), the second throughdirsql_parsed(the--on-fileparser emits[{"title": <first line>, "words": <wc -w>}]). Both exit 0.Mutation gate
Unit mutation — changed lines (rust)had been going red on this branch since #1113: fivevtab_scaffold.rsmutants (at_eof -> false,>= -> <,next -> Ok(()),+= -> *=,eof -> false) make the cursor never reach EOF, and the only thing "catching" them was the OOM killer on the lib tests that ran a realSELECTthroughDb::query.51555d8adds a directScaffoldCursorunit test over a two-row fake source that fails all five in the lib binary in microseconds, and putsLIMIT 1on the four empty-root queries so they stay bounded under the eof mutants either way. Localcargo mutants --file packages/rust/src/vtab_scaffold.rsunder a 24G cap: 32 mutants, 24 caught, 8 unviable, no SIGKILL; the five target mutants each finish their test phase in under 1.5s.E2E Verification
packages/python/e2e-attestations/<branch>.jsonreceipt written ifpackages/pythonchanged (just e2e-attest-python)packages/ts/e2e-attestations/<branch>.jsonreceipt written ifpackages/tschanged (just e2e-attest-ts)cargo test -p dirsql --features cli(55 binaries, 1311 passed);just preflight(all lanes green on committed state);just e2e-attest-python(37 passed);just e2e-attest-ts(16 passed)packages/python/e2e-attestations/claude-1080-share-vtab-scaffolding.jsonandpackages/ts/e2e-attestations/claude-1080-share-vtab-scaffolding.json(both bindings are in scope becausepackages/rust/srcchanged).Changelog / Migrations
<root>/<pkg>/changelog.d/for each changed package:packages/rust/changelog.d/2026-09-09-shared-vtab-scaffold.md<root>/<pkg>/migrations.d/: not required -- internal refactor, no public API or behavior changeParity
No public API change in any SDK;
PARITY.mduntouched. No drift introduced.