epic: cut source without changing the public API or touching tests #1072

Cut as much source as possible under two hard constraints: the public API surface does not change, and no existing test is rewritten or weakened. Everything that is not the product itself is on the table.

The deterministic gates already in place — one_function_per_file (ratcheting toward max_lines = 1 in #1020), 100% unit coverage, unit mutation at zero survivors, colocated-test, unit lint isolation, declared-deps, changelog-gate, e2e attestation — are constraints every candidate must satisfy, not goals competing with the cut. A candidate that only shrinks the number by breaking a gate is not a candidate. That rules the ranking below: the changes worth doing delete whole logic (cross-SDK duplication, hand-rolled infrastructure a dependency covers, dead checks) rather than reshaping it, because those cut source without fighting any gate.

Current size

Measured on origin/main at f575c06e1e53fd59d1653e0963f079a7d9a8b6ca, 2026-09-05. Metric: non-test, non-comment, non-blank source lines. Rust's inline #[cfg(test)] modules are stripped before counting — a naive per-file count includes them and is wrong by 8,450 lines in the core alone.

Test lines outside the scan roots, for context (they count toward nothing here): packages/rust/tests 10,942 · packages/python/tests 3,802 · packages/ts/tests 3,373 · internals/checks/tests 1,065 · plugins/…/tests 756 · internals/distcheck/tests 28. Total test lines: 35,252, a 3.4:1 ratio against source.

Two roots sit outside the metric because they are not src: packages/ts/tools (195 source lines of release staging) and packages/rust/benches (330). packages/ts/tools/stage-platform.ts matters to candidate 2 anyway — it is a third consumer of the platform table.

Measurement

The file list was verified two independent ways before any of the above was written down, because willfire#51's headline lesson is that its canonical command silently undercounted for months (git reads src/**/*.ts as needing two slashes without :(glob) pathspec magic, so every top-level file vanished).

Method 1 — file list and raw lines, straight from git:

for r in packages/rust/src packages/python/dirsql packages/python/src packages/ts/src \
         packages/ts/napi/src plugins/dirsql-plugin-embeddings/src \
         internals/checks/src internals/distcheck/src; do
  printf '%-45s ' "$r"
  git grep -c '' origin/main \
    -- ":(glob)$r/**/*.rs" ":(glob)$r/**/*.py" ":(glob)$r/**/*.ts" \
       ":(glob)$r/*.rs"    ":(glob)$r/*.py"    ":(glob)$r/*.ts" \
    | awk -F: '{s+=$NF; n+=1} END {print n" files, "s" raw lines"}'
done

Method 2 — the classifier below, which walks a git archive of the named sha and splits every line into code / comment / blank.

The two reconcile exactly. Method 1 reports 11 fewer files across the tree, all of them zero-byte __init__.py (git grep -c '' does not list a file with no lines); every other file count matches. Every line of every file lands in exactly one of the classifier's three buckets, and those buckets sum to Method 1's raw totals for all eight roots with no residue. Reproduce with:

uv run --no-project srcline.py f575c06e1e53fd59d1653e0963f079a7d9a8b6ca

#!/usr/bin/env python3
"""Non-test, non-comment, non-blank source lines per scan root.

Rust: inline `#[cfg(test)]` modules are stripped before counting.
Python: docstrings count as comments.
"""
import ast, io, os, re, subprocess, sys, tempfile, tokenize
from pathlib import Path

ROOTS = [
    ("packages/rust/src", {".rs"}),
    ("packages/python/src", {".rs"}),
    ("packages/ts/napi/src", {".rs"}),
    ("packages/python/dirsql", {".py"}),
    ("packages/ts/src", {".ts"}),
    ("plugins/dirsql-plugin-embeddings/src", {".py"}),
    ("internals/checks/src", {".py"}),
    ("internals/distcheck/src", {".py"}),
]

def py_lines(text):
    lines = text.splitlines(); kind = [None] * len(lines)
    for i, l in enumerate(lines):
        if not l.strip(): kind[i] = "blank"
    try: tree = ast.parse(text)
    except SyntaxError: tree = None
    if tree:
        for node in ast.walk(tree):
            if isinstance(node, (ast.Module, ast.ClassDef, ast.FunctionDef, ast.AsyncFunctionDef)):
                b = getattr(node, "body", [])
                if b and isinstance(b[0], ast.Expr) and isinstance(b[0].value, ast.Constant) \
                   and isinstance(b[0].value.value, str):
                    for i in range(b[0].lineno - 1, b[0].end_lineno):
                        if kind[i] is None: kind[i] = "comment"
    stripped = list(lines)
    try: toks = list(tokenize.generate_tokens(io.StringIO(text).readline))
    except Exception: toks = []
    for t in toks:
        if t.type == tokenize.COMMENT:
            r, col = t.start[0] - 1, t.start[1]
            if stripped[r][:col].strip() == "":
                if kind[r] is None: kind[r] = "comment"
            else: stripped[r] = stripped[r][:col]
    for i, l in enumerate(lines):
        if kind[i] is None:
            kind[i] = "code" if stripped[i].strip() else ("comment" if l.strip() else "blank")
    return kind.count("code"), kind.count("comment"), kind.count("blank")

def strip_c(text):
    out, i, n, state, quote = [], 0, len(text), "code", None
    while i < n:
        c, nx = text[i], text[i + 1] if i + 1 < n else ""
        if state == "code":
            if c == "/" and nx == "/": state = "line"; i += 2; continue
            if c == "/" and nx == "*": state = "block"; i += 2; continue
            if c in "\"'`": quote = c; state = "str"; out.append(c); i += 1; continue
            out.append(c); i += 1; continue
        if state == "str":
            if c == "\\":
                out.append(c)
                if i + 1 < n: out.append(text[i + 1])
                i += 2; continue
            out.append(c)
            if c == quote: state, quote = "code", None
            i += 1; continue
        if state == "line":
            if c == "\n": out.append("\n"); state = "code"
            i += 1; continue
        if c == "*" and nx == "/": state = "code"; i += 2; continue
        if c == "\n": out.append("\n")
        i += 1
    return "".join(out)

def c_lines(text):
    orig, strp = text.splitlines(), strip_c(text).splitlines()
    strp += [""] * (len(orig) - len(strp))
    code = comment = blank = 0
    for o, s in zip(orig, strp):
        if not o.strip(): blank += 1
        elif s.strip(): code += 1
        else: comment += 1
    return code, comment, blank

CFG_TEST = re.compile(r"^\s*#\[cfg\(test\)\]\s*$")
def rust_split(text):
    lines, keep, test, i = text.splitlines(keepends=True), [], [], 0
    while i < len(lines):
        if CFG_TEST.match(lines[i]):
            depth, started, j = 0, False, i
            while j < len(lines):
                test.append(lines[j])
                for ch in strip_c(lines[j]):
                    if ch == "{": depth += 1; started = True
                    elif ch == "}": depth -= 1
                j += 1
                if started and depth <= 0: break
            i = j; continue
        keep.append(lines[i]); i += 1
    return "".join(keep), "".join(test)

def is_test(p):
    if "tests" in p.parts: return True
    return bool(p.name.endswith("_test.py") or re.search(r"\.(test|spec)\.(ts|tsx|js|mjs)$", p.name))

def main(tree):
    tot = totf = totc = tott = 0
    print(f"{'root':45s} {'files':>6s} {'src':>7s} {'comment':>8s} {'blank':>6s} {'test':>7s}")
    for rel, exts in ROOTS:
        code = com = blank = nf = tst = 0
        for dp, dns, fns in os.walk(Path(tree) / rel):
            dns[:] = [d for d in dns if d not in ("node_modules", "target", "__pycache__", ".venv", "dist")]
            for fn in sorted(fns):
                p = Path(dp) / fn
                if p.suffix not in exts: continue
                text = p.read_text(encoding="utf-8", errors="replace")
                if is_test(p.relative_to(tree)):
                    tst += (py_lines(text) if p.suffix == ".py" else c_lines(text))[0]; continue
                if p.suffix == ".rs":
                    keep, t = rust_split(text); c, m, b = c_lines(keep); tst += c_lines(t)[0]
                elif p.suffix == ".py": c, m, b = py_lines(text)
                else: c, m, b = c_lines(text)
                code += c; com += m; blank += b; nf += 1
        print(f"{rel:45s} {nf:6d} {code:7d} {com:8d} {blank:6d} {tst:7d}")
        tot += code; totf += nf; totc += com; tott += tst
    print(f"{'TOTAL':45s} {totf:6d} {tot:7d} {totc:8d} {'':6s} {tott:7d}")

if __name__ == "__main__":
    rev = sys.argv[1] if len(sys.argv) > 1 else "origin/main"
    with tempfile.TemporaryDirectory() as d:
        subprocess.run(f"git archive {rev} | tar -x -C {d}", shell=True, check=True)
        main(d)

The decomposition overhead the target has to absorb

Every one_function_per_file extraction adds a module header and an import block to source, so #1020's walk toward max_lines = 1 pushes the number up. That is a known fixed cost the target is set around, not an argument against the campaign. It was measured, not guessed, on the three rung-9 PRs that landed at 01:41 UTC on 2026-09-05:

Nine extractions, +45 source lines: 5.0 lines per extraction. An AST sweep of the four Python roots and the TypeScript SDK at max_lines = 1 finds ~107 module-scope functions still sharing a file (internals/checks/src 72, plugins/… 10, packages/python/dirsql 8, internals/distcheck/src 8, packages/ts/src ~9), so the remaining ratchet costs roughly +535 source lines.

The cost is elastic, and the two interact in our favour: 72 of those 107 extractions live in internals/checks/src, which is also where the largest deletions below are. Deleting platforms_mirror (11 files) and npm_binary_extension_load (4 files) removes their extractions along with them, so landing the cut before the last rungs lowers the overhead rather than fighting it.

Target

Superseded — the target is now to hold flat at 10,194 source lines (no net growth), measured by srcline.py on main. See the re-measurement comment for why: all six sub-issues landed for a net −180, because consolidations relocate lines into the core rather than deleting them, and the remaining candidate pool is ~70 lines. The original target follows, for the record.

~~8,500 source lines — a cut of 1,874 (−18%), with the +535 of decomposition overhead already priced in. In gross terms that is ~2,400 lines to find and delete.~~

willfire's 50% is not reachable here, and saying so is more useful than adopting the number. willfire could take 50% because its single largest subsystem — the execution engine, 1,150 of 3,599 lines — was on the table as a scope cut. This epic forbids scope cuts: the public API surface does not change. 50% of dirsql is 5,187 lines, and packages/rust/src alone is 6,274 of the total — reaching it means deleting the product.

For calibration, here is what the feature tier would buy if it were ever on the table (it is not part of this epic): the REPL 280 · the HTTP server (cli/server.rs + cli/router.rs) 155 · worker-backed SQL scalar functions (functions.rs) 410 · the parsed path-table (parsed_vtab.rs + parsed_cache.rs + infer.rs) 610 · the embeddings plugin 227 — 1,682 lines. Even deleting all of it and landing every candidate below reaches −33%, not −50%. The number to judge this epic against is 18%, not 50%.

Candidates

Roughly biggest first. Current line counts are measured; net deltas are estimates and are marked as such.

1. Fold the language-agnostic half of extension resolution into the core (~−185 net)

The one place where the one-implementation principle is visibly broken. packages/python/dirsql/cli/config_paths_from_argv.py (16) and packages/ts/src/cli/config-paths-from-argv.ts (21) hand-roll the same argv scanner twice, and clap parses the same -c / --config flags a third time in Rust. Three parsers, no parity test against the real binary. Python advances i += 2 past a consumed value while TypeScript carries an expectValue flag.

Correction (2026-09-04): this issue first claimed the two scanners already disagree on dirsql -c -c x. They do not. Run against 24 argv shapes they agree on all 24 (-c -c x yields ["-c"] from both), and that input is not an error case for this code — the scan runs before clap, only to locate config files for extension loading. The defect is latent rather than live: two hand-written parsers with nothing binding them together, and a third (clap) neither is checked against. See #1078 for the full correction.

The scanner is the visible tip. The whole cluster is a Python/TypeScript port of one another:

Only ~90 of those 367 lines are genuinely host-specific: importlib.util.find_spec in Python and require.resolve in Node cannot move to Rust. Everything else — the argv scan, the init short-circuit, the native-suffix filter (.py/.js/.mjs/.cjs), the TOML read, the is_bare_name test, the base-dir rule, the path::entrypoint spec encoding — is language-neutral logic the core already has clap and toml for. The shape: the core exposes a resolution plan over argv, each launcher answers only "where is package X installed", the core builds the specs.

Tradeoff. It adds a core surface and one binding call to a launcher path that today needs neither, and it is the only candidate that could regress startup latency (the TypeScript launcher currently defers smol-toml behind an existsSync guard specifically to keep the parser off the common path, #720). It is also the only candidate that makes cross-SDK divergence structurally impossible rather than policed.

2. One source of truth for the platform table; delete the check that polices the copy (~−276 net)

packages/ts/src/platforms.ts (67) is the release source of truth for the published @dirsql/lib-* sub-packages. internals/distcheck/src/distcheck/node_flow/platforms.py (44) is a hand-maintained Python subset of it. Because those two can drift — and did, in both directions — #1004 added internals/checks/src/checks/platforms_mirror/ (262 lines across 11 files, plus its own platforms-mirror-check.yml workflow), which parses both files with ast and a TypeScript reader and compares them field by field.

That is 306 lines of duplicated data plus 262 lines of machinery to prove the duplicate is faithful. A single declarative table (JSON or TOML at the repo root) that platforms.ts, platforms.py and packages/ts/tools/stage-platform.ts all read deletes the check, its workflow, and one of the two tables.

Tradeoff. The TypeScript Platform interface currently carries types the data file cannot (NodeJS.Platform, NodeJS.Architecture); those become a cast at the read boundary. And a JSON import in the published SDK bundle is a packaging detail to get right — platforms.ts ships to npm.

3. Put the napi binding's value marshaling on napi-rs's safe API (~−200 net)

packages/ts/napi/src/lib.rs is 777 source lines against the PyO3 binding's 381 for the same surface. The gap is one region: lines 547–882, 277 source lines of raw napi_sys FFI — parse_js_array_of_objects, js_val_to_value, read_js_string, coerce_js_to_string, extract_exception_message, get_u8_array_bytes — each carrying its own #[expect(unsafe_code, reason = "raw napi_sys FFI")]. napi-rs's own safe API (Unknown, Object, ValueType, BigInt) covers essentially all of it; PyO3's equivalent conversions are why the Python binding is half the size.

Tradeoff. Scope this to the marshaling half only. The adjacent FnRef + make_on_file_closure region (103 lines) invokes the JS on_file callback synchronously off a raw napi_value, which napi-rs's safe Function handle is not Send enough to express and ThreadsafeFunction is async; that part is probably genuinely irreducible and should stay. Also: this is the binding boundary, so the unit mutation and coverage gates over it are the ones most worth re-reading before and after.

4. Delete npm-binary-extension-load, which probes an artifact that no longer exists (−144)

internals/checks/src/checks/npm_binary_extension_load/ (144 lines) downloads dirsql-npm-bundled-cli-linux-x64-gnu* and probes whether the bundled CLI binary can dlopen a SQLite extension. putitoutthere.toml has carried no [package.bundle_cli] for the npm lane since #739 ("the npm lane ships no bundled binary … matrix.bundle_cli have nothing to do for this package"). The artifact pattern matches nothing, the check's documented "skips cleanly when the plan built no npm bundled-cli row" branch fires, and the npm-extension-load job in release-ci.yml has been passing on an empty directory ever since.

Tradeoff. There is a real question underneath — can the shipped .node addon dlopen an extension — and the answer may be worth keeping. So this is verify, then either delete or retarget at the addon, not a blind delete. The same review should look at wheel_extension_load (122), whose "statically-linked bundled binary" rationale is equally obsolete post-#738 even though the probe itself still exercises a live path (the wheel's extension module).

5. Share the vtab scaffolding between the two path-table modules (~−100 net)

vtab.rs (221) and parsed_vtab.rs (327) are two dirsql_* virtual tables built the same way, and they carry parallel copies of FIXED_ARGS, compile_ignore, parse_module_args, at_eof, rowid_of, and the CreateVTab/VTabCursor scaffolding — differing only in which module arguments they take and how a row is produced. A shared module holding the argument decoding and cursor arithmetic leaves each file its actual difference.

Tradeoff. The smallest and safest item here, and the only core-internal one. vtab.rs is also the file the Rust coverage floors were nudged down for twice (#626, #737); moving code between the two files moves coverage between them, so expect to re-pin [rust.coverage].

6. Replace the hand-rolled progress reporter with indicatif (~−200, needs a decision)

packages/rust/src/progress.rs is 239 source lines of \r-rewritten counter, throttle, TTY detection and phase summary. Its own module docs state the constraint explicitly: "No new dependencies. The line is a \r-rewritten counter, not a drawn bar: a bar wants the terminal width, and every portable way to ask for it is a crate." indicatif is that crate, and it covers the whole module.

Tradeoff. This reverses a decision #957 made deliberately, and it is the one candidate that trades source lines for a dependency in the published crate. indicatif pulls console, which pulls terminal_size / libc / windows-sys. The rule of thumb elsewhere in this repo is never hand-roll infrastructure a real tool covers — this is a clear instance of it, but it is Kevin's call whether the dependency is worth 200 lines, so it is filed unassigned.

Hygiene, not really a candidate

The root testing-conventions.toml [typescript] table (one_function_per_file, [typescript.coverage]) is read by no scan root — the file's own header says the TypeScript lane passes packages/ts/testing-conventions.toml and does not read the default. Zero source lines, but it is live-looking configuration that lies about what governs the TS gates. Delete it with whichever candidate lands first.

Settled

• Comments stay. They are excluded from the metric by definition, so a comment sweep moves the number by zero and is not part of this epic. Same for blank lines. (3,481 comment lines across the scan roots, if anyone was wondering.)
• No test is rewritten or weakened. The narrow reading needs stating: deleting a source file deletes its colocated unit test, because the colocation gate demands exactly one test per source file and a test for a file that no longer exists is not a test. That is the deletion completing, not a test being touched. What is out of bounds is changing an assertion to accommodate a refactor. Any candidate that needs that is not a candidate.
• The gates are constraints, not obstacles. ci: ratchet one-function-per-file from max_lines = 100 toward the default of 1 #1020 continues to max_lines = 1 on its own schedule; every candidate here lands green under it, under 100% unit coverage, under zero mutation survivors, and with a changelog fragment and an e2e receipt. Nothing in this epic asks for an exemption, and the exemption count stays at zero.
• No scope cuts. The REPL, the HTTP server, worker-backed SQL functions, the parsed path-table and the embeddings plugin (1,682 lines together) are priced above for calibration only. Deleting any of them changes the public surface, which this epic forbids.
• packages/ts/tools (195) and packages/rust/benches (330) are out of the metric. They are not src. stage-platform.ts is still in scope for candidate 2 as a consumer of the platform table.
• Investigated and rejected: making the core's internal modules private. All 16 packages/rust/src modules are pub mod, so the crate's nominal API surface is everything. Tightening that to pub(crate) is worth doing on its own merits, but it deletes zero source lines and it is a public API change — the exact thing this epic rules out.
• Investigated and rejected: barrel and forwarder deletion. packages/ts/src/cli/dirsql.ts (3), run-cli.ts (9), die.ts (4), index.ts (9), the Python __init__.py barrels. willfire#51 got ≈40 lines from this; here it is worth under 30, each one has a colocated test that would go with it, and one_function_per_file at max_lines = 1 wants files this shape. Not worth the churn.
• Investigated and rejected: internals/checks as redundant with testing-conventions. The obvious suspicion — that a repo-local check now duplicates an upstream gate — does not hold up. preflight (307) derives the CI gate matrix from the workflow files and is the reason just preflight covers 40 pairs instead of 6; release_globs (142), artifact_completeness (102), changelog_gate (133) and attestation_guard (99) each encode a putitoutthere or repo-policy invariant with no upstream equivalent; pytest_gate (28) exists so that "no tests collected" is not a CI failure, and inlining it would put logic back in run:. The redundancy in this package is not check-vs-gate, it is the platform-table duplication of candidate 2.

Sub-issues

• checks: delete or retarget npm-binary-extension-load, which probes an artifact that is no longer built #1073 — Delete or retarget npm-binary-extension-load, which probes an artifact that is no longer built (−144)
• release: one source of truth for the platform table, and delete the platforms-mirror check #1074 — One source of truth for the platform table; delete the platforms-mirror check (~−276)
• napi: replace the hand-rolled raw napi_sys marshaling with napi-rs's safe API #1075 — Put the napi binding's value marshaling on napi-rs's safe API (~−200)
• core: fold the language-agnostic half of extension resolution into Rust (three argv parsers today) #1078 — Fold the language-agnostic half of extension resolution into the core (~−185)
• rust: share the vtab scaffolding between vtab.rs and parsed_vtab.rs #1080 — Share the vtab scaffolding between vtab.rs and parsed_vtab.rs (~−100)
• decide: replace the hand-rolled progress reporter with indicatif #1081 — Decide: replace progress.rs with indicatif (Kevin's call, unassigned, ~−200)

Six items, ~1,105 net lines. The target of 8,500 needs ~2,400 gross, so this list is a start, not the whole epic — the remaining ~1,300 has not been found yet and the candidates above are the ones that survived a first pass. Re-measure after each lands and add to the list.

Sequencing. #1073 first: verify-and-delete, no design content, lands immediately, and it removes four files from #1020's remaining extraction list. Then #1074, the largest single deletion, touching nothing the SDKs publish. Then #1075, self-contained inside one binding crate. #1078 carries the real design content (a new core surface, a launcher-latency question) and should follow the mechanical ones rather than block them. #1080 any time. #1081 waits on Kevin.

Landing #1074 and #1073 before #1020's last rungs is worth doing in that order — together they remove 15 files from internals/checks/src, which is where 72 of the ~107 remaining extractions live.