Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/internals-checks-ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ jobs:
source: internals/checks/src
config: internals/checks/testing-conventions.toml
base: ${{ github.event.pull_request.base.sha || 'origin/main' }}
gates: '["colocated-test", "unit-lint", "integration-lint", "unit-coverage", "mutation", "e2e-verify"]'
gates: '["colocated-test", "one-function-per-file", "unit-lint", "integration-lint", "unit-coverage", "mutation", "e2e-verify"]'

internals-checks-integration:
# The conventions job runs this package's colocated unit tier and checks its
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/internals-distcheck-ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -29,4 +29,4 @@ jobs:
languages: '["python"]'
source: internals/distcheck/src
base: ${{ github.event.pull_request.base.sha || 'origin/main' }}
gates: '["colocated-test", "unit-lint", "integration-lint", "unit-coverage", "mutation"]'
gates: '["colocated-test", "one-function-per-file", "unit-lint", "integration-lint", "unit-coverage", "mutation"]'
2 changes: 1 addition & 1 deletion .github/workflows/plugin-dirsql-embedding-ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ jobs:
source: plugins/dirsql-plugin-embeddings/src
config: plugins/dirsql-plugin-embeddings/testing-conventions.toml
base: ${{ github.event.pull_request.base.sha || 'origin/main' }}
gates: '["colocated-test", "unit-lint", "integration-lint", "unit-coverage", "mutation", "e2e-verify"]'
gates: '["colocated-test", "one-function-per-file", "unit-lint", "integration-lint", "unit-coverage", "mutation", "e2e-verify"]'

embeddings-integration:
# The conventions job runs the plugin's colocated unit tier and checks its
Expand Down
2 changes: 1 addition & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -119,7 +119,7 @@ A feature is not done until integration tests pass and cover the new functionali

### Enforcing Colocation (testing-conventions)

The colocation rule is a blocking CI gate ([`testing-conventions`](https://github.com/thekevinscott/testing-conventions), wired in the per-domain CI workflows + `testing-conventions.toml`): every source file needs a colocated unit test (Rust: inline `#[cfg(test)]`), a PR-only co-change check flags modified sources whose test didn't change, and `unit lint` enforces **isolation** -- a unit test must mock every collaborator (Python `patch(...)`, TS `vi.mock`, Rust trait double or relocate the effectful test), never weaken the test. The gates also cover the two binding crates and the repo-tooling `internals/checks` / `internals/distcheck` uv packages. **The exemption count is zero and stays there**: barrels get a colocated surface test, logic gets extracted, dead shells get deleted -- an exemption is never the escape hatch. Full wiring, per-package gate map, and history: `agents/reference/testing-gates.md`.
The colocation rule is a blocking CI gate ([`testing-conventions`](https://github.com/thekevinscott/testing-conventions), wired in the per-domain CI workflows + `testing-conventions.toml`): every source file needs a colocated unit test (Rust: inline `#[cfg(test)]`), a PR-only co-change check flags modified sources whose test didn't change, and `unit lint` enforces **isolation** -- a unit test must mock every collaborator (Python `patch(...)`, TS `vi.mock`, Rust trait double or relocate the effectful test), never weaken the test. The gates also cover the two binding crates and the repo-tooling `internals/checks` / `internals/distcheck` uv packages. **There is exactly one exemption**: `no-first-party-patch` on the two `packages/python/tests/integration/hermetic/` suites, waived because patching the first-party `_RustDirSQL` core boundary *is* what makes that tier hermetic -- there is no third-party target the rule could redirect it to, so the rule does not describe that seam. Everywhere else the count stays zero and an exemption is never the escape hatch: barrels get a colocated surface test, logic gets extracted, dead shells get deleted. Full wiring, per-package gate map, and history: `agents/reference/testing-gates.md`.

**Run every gate CI declares before pushing** with `just preflight` (#781), which *derives* the (source, gate) pairs from the CI workflows instead of restating them -- 40 pairs across 8 scan roots, versus the 6 across 3 the four hand-written recipes it replaced covered. It reads every `.github/workflows/*.yml` and folds together each job that calls the reusable workflow, so a lane added in a new per-domain file is a pair it runs. It also encodes the invocations that differ from the naive one (python suites via the package venv, typescript via `npx`, per-gate option support), each of which used to pass locally while failing in CI. `--dry-run` prints the matrix; `--gate <name>` and `--conventions <workflow>` narrow it (both repeatable); `packaging` reports SKIP since it needs a built artifact (`just test-packaging`).

Expand Down
4 changes: 2 additions & 2 deletions agents/reference/testing-gates.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ flags. `packaging` is reported **SKIP**: it inspects a built artifact, so use

**Exemptions.** The goal is **zero** exemptions, and barrels are no longer an excuse for one. A re-export barrel gets a **colocated test that asserts its public surface** (TS `src/index.ts` ↔ `index.test.ts`, python `dirsql/__init__.py` ↔ `__init___test.py`), exactly as any module would -- it is *tested*, not exempted, so a broken re-export is caught. An `__init__.py` carrying no executable logic is made **truly empty** (0 bytes), which the tool auto-skips with no config entry. A package shell left dead by a feature removal is **deleted**, not parked behind an exemption. When a "barrel" actually holds logic, the fix remains to **extract that logic into colocated-tested modules** (#239). The npm `bin` shim `src/cli/dirsql.ts` is likewise *not* exempt: its error-handling lives in the unit-tested `cli/run-cli.ts`, leaving a trivial `runCli()` shim covered by a mocked distcheck-test.

The exemption count is again **zero**. The three-tier conformance work (#517) restored it: the binding suites moved into the recognized tier `tests/integration/binding/` (#519) and the packaging distcheck flows moved to the `internals/distcheck` package (#520), so nothing trips `unknown-tier` anymore and the temporary `unknown-tier` waivers #518 added are all removed. Before that the last standing entry -- the python barrel-test isolation waiver (`[[python.exempt]] path = "__init___test.py" rules = ["unmocked-collaborator"]`) -- was removed once testing-conventions#382 / PR #384 brought the python `unmocked-collaborator` rule to parity with TS (a bare package-relative import `from . import <names>` in a barrel test now resolves to the SUT and passes). Exemption entries carry a `path` (relative to the scanned source dir for source-file rules, but the derived package root for the suite-tier `unknown-tier` rule), the `rules` waived, and a required `reason`; the CLI **rejects a stale entry whose `path` matches no file**, so it self-cleans. Adding a *new* untested source file fails the gate -- an exemption is never the escape hatch.
The exemption count is **one**: `no-first-party-patch` waived on `tests/integration/hermetic/dirsql_test.py` and `tests/integration/hermetic/extensions_test.py` in `packages/python/testing-conventions.toml`. Patching the first-party `_RustDirSQL` core boundary is the hermetic tier's definition -- it is why that subdir needs no native build -- and there is no third-party target to redirect the patch to, so the rule has nothing to say about that seam. It is not a way past a real failure: the two stdlib patches reported alongside it (`asyncio.to_thread`, `os.getcwd`, both reached through `dirsql._async`) were re-rooted at their own modules rather than waived, and everything else remains at zero. Before this entry the count was zero. The three-tier conformance work (#517) restored it then: the binding suites moved into the recognized tier `tests/integration/binding/` (#519) and the packaging distcheck flows moved to the `internals/distcheck` package (#520), so nothing trips `unknown-tier` anymore and the temporary `unknown-tier` waivers #518 added are all removed. Before that the last standing entry -- the python barrel-test isolation waiver (`[[python.exempt]] path = "__init___test.py" rules = ["unmocked-collaborator"]`) -- was removed once testing-conventions#382 / PR #384 brought the python `unmocked-collaborator` rule to parity with TS (a bare package-relative import `from . import <names>` in a barrel test now resolves to the SUT and passes). Exemption entries carry a `path` (relative to the scanned source dir for source-file rules, but the derived package root for `integration lint`'s suite subjects -- both `unknown-tier` and the mocking lints), the `rules` waived, and a required `reason`; the CLI **rejects a stale entry whose `path` matches no file**, so it self-cleans. Adding a *new* untested source file fails the gate -- an exemption is never the escape hatch.
## Mutation

The rung above coverage is the **`unit mutation`** gate (#235 / epic #231): testing-conventions mutates the source and fails on any **surviving** mutant -- one no unit test caught. Engines: **cosmic-ray** (Python), **Stryker** (TypeScript), **cargo-mutants** (Rust). It is **PR-only and diff-scoped** (`--base <base.sha>...HEAD`): only the lines a PR added/modified are mutated, so each PR's surface stays bounded. A PR that changes no SDK source has nothing to mutate and passes trivially.
Expand Down Expand Up @@ -91,7 +91,7 @@ Keep the job count low enough that the build dirs fit: each *unscoped* job dir r
**What the audit found.** Unit scope: 330 caught, **61 survivors**, 75 unviable, 4 timeouts. Re-running those 61 under CI's scoping, **40 are killed by the integration tier** -- the #233 split (effectful filesystem / subprocess / HTTP / signal code lives in the integration tier by design) showing up as mutation survivors rather than gate escapes -- 3 time out, and **18 escape every tier**. Those 18 divide three ways, and the split is what matters for anyone repeating this:

- **7 were real gaps and are now closed** -- six operator mutations in `matcher.rs`'s `placeholder_spans` (its own grammar counter-examples were untested) and `lib.rs`'s `prepare_persist` reading `meta_current` as `compatible && lengths match`, where `&&` -> `||` survived. The tests added for them kill 14 mutants at those sites.
- **7 are equivalent mutants, proved rather than argued.** `|` -> `^` over disjoint bit fields is always a no-op: `function_flags`' `SQLITE_UTF8|SQLITE_DETERMINISTIC` (bits `0x1` and `0x800`), `base64_encode`'s `(b0<<16)|(b1<<8)|b2` (exhaustive over all 16,777,216 inputs), and `base64_decode`'s `(triple<<6)|value_of(b)` (all 67,108,864). In `placeholder_spans`, `j += 1` -> `j *= 1` is re-consumed by the following `while`, and `i = j + 1` -> `i = j * 1` / `i = j - 1` land on the closing brace or a name byte, never a `{` -- zero differing results over all 335,923 strings up to length 7 on `{ } a 1 _ -`. **No exemptions were added**: the gate is diff-scoped, so these only surface if someone edits those lines, and the exemption count stays zero.
- **7 are equivalent mutants, proved rather than argued.** `|` -> `^` over disjoint bit fields is always a no-op: `function_flags`' `SQLITE_UTF8|SQLITE_DETERMINISTIC` (bits `0x1` and `0x800`), `base64_encode`'s `(b0<<16)|(b1<<8)|b2` (exhaustive over all 16,777,216 inputs), and `base64_decode`'s `(triple<<6)|value_of(b)` (all 67,108,864). In `placeholder_spans`, `j += 1` -> `j *= 1` is re-consumed by the following `while`, and `i = j + 1` -> `i = j * 1` / `i = j - 1` land on the closing brace or a name byte, never a `{` -- zero differing results over all 335,923 strings up to length 7 on `{ } a 1 _ -`. **No exemptions were added**: the gate is diff-scoped, so these only surface if someone edits those lines, and the `mutation` exemption count stays zero.
- **4 sat in effectful code no tier reached** -- `RootedFs::hash` (two), `cli/run.rs`'s `load_state` guard and `wait_for_shutdown`. Tracked as #1012 rather than re-litigating #233 there, and closed by it at the integration and e2e tiers (below).

The headline: nothing merged behind the blind gate turned out to be a silent behavior regression. What it hid was **test debt** -- two pure functions whose own documented edge cases had no assertions.
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
{
"command": "uv run python -m pytest tests/e2e -x -q",
"ran_at": 1788368546,
"exit_code": 0,
"commit": "f032f0fa05bf1ba089f1a71197a4bca124c2ad36",
"branch": "claude/1018-preflight-one-function-gate"
}
34 changes: 23 additions & 11 deletions internals/checks/src/checks/preflight/gate.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,10 +12,10 @@

Three pairs cannot be run the naive way, and each is encoded rather than skipped:

* python `mutation` / `unit-coverage` -- `uvx` supplies its own environment, so
the suite's `python3 -m pytest` resolves to an interpreter with no pytest
(#706 saw it as a cosmic-ray baseline failure naming no cause). Both run via
the package's own venv instead.
* python `mutation` / `unit-coverage` -- these execute the suite, whose
`python3 -m pytest` must resolve to the package's own venv rather than an
ambient interpreter with no pytest (#706 saw it as a cosmic-ray baseline
failure naming no cause). Both run under `uv run` from the package root.
* rust `colocated-test` -- the co-change variant rides only the
python/typescript language set (rust units are inline, so no sibling can go
stale), and the CLI rejects `--base` for it.
Expand All @@ -36,13 +36,25 @@

MANIFESTS = ("pyproject.toml", "package.json", "Cargo.toml")

# The CLI, resolved the way CI resolves it. Two silent drifts made the local run
# enforce a different ruleset, and naming the tag closes both: the PyPI wheel
# `uvx` fetches lags npm by several releases, and a BARE `npx testing-conventions`
# resolves engine-aware, so a shim on node older than `engines.node` picks the
# last release below that floor -- and exits 0 with no banner to say so.
CLI = ["npx", "-y", "testing-conventions@latest"]

# How to launch a suite-executing gate, per language. Both run from the package
# root: python so `python3 -m pytest` resolves to that package's venv, typescript
# because only the npm CLI appends `--ts-mutation-adapter`, which the rule needs.
# Rust is absent -- it has no suite environment to enter, so it stays on `uvx`.
# root: python so the suite's `python3 -m pytest` resolves to that package's
# venv, typescript because only the npm CLI appends `--ts-mutation-adapter`,
# which the rule needs. Rust is absent -- it has no suite environment to enter.
#
# The python arm needs BOTH distributions. `--with` is not how the CLI is
# resolved; it puts the wheel's `testing_conventions` package on the interpreter
# the gate hands to `python3 -m`, which is the only form the mutation adapter
# ships in.
LAUNCHERS = {
"python": ["uv", "run", "--with", "testing-conventions", "testing-conventions"],
"typescript": ["npx", "-y", "testing-conventions"],
"python": ["uv", "run", "--with", "testing-conventions", *CLI],
"typescript": CLI,
}


Expand Down Expand Up @@ -95,7 +107,7 @@ def invocation(
home = package_root(root.source, exists)
return Invocation(
[
*["uvx", "testing-conventions", *gate.command, *options],
*[*CLI, *gate.command, *options],
*["--scope", root.source, *e2e_flags(e2e), home],
],
".",
Expand All @@ -107,7 +119,7 @@ def invocation(
options[-1] = os.path.relpath(root.config, cwd)
source = os.path.relpath(root.source, cwd)
return Invocation([*LAUNCHERS[language], *gate.command, *options, source], cwd)
return Invocation(["uvx", "testing-conventions", *gate.command, *options, root.source], ".")
return Invocation([*CLI, *gate.command, *options, root.source], ".")


def default_runner(argv: Sequence[str], cwd: str) -> int:
Expand Down
26 changes: 14 additions & 12 deletions internals/checks/src/checks/preflight/gate_test.py
Original file line number Diff line number Diff line change
Expand Up @@ -78,21 +78,21 @@ def it_returns_nothing_for_an_absent_table():


def describe_invocation():
def it_runs_an_ordinary_gate_through_uvx_from_the_repo_root():
def it_runs_an_ordinary_gate_through_the_npm_cli_from_the_repo_root():
assert call(PY, "python", "unit-lint") == Invocation(
[
*["uvx", "testing-conventions", "unit", "lint"],
*["npx", "-y", "testing-conventions@latest", "unit", "lint"],
*["--language", "python"],
*["--config", "testing-conventions.toml", "packages/python/dirsql"],
],
".",
)

def it_takes_the_ordinary_path_for_a_gate_name_sorting_below_e2e_verify():
# Both paths start `uvx testing-conventions`, so `--scope` is what tells
# Both paths start with the same CLI prefix, so `--scope` is what tells
# them apart -- an `<=` here would route colocated-test into e2e's branch.
assert call(PY, "python", "colocated-test").argv == [
*["uvx", "testing-conventions", "unit", "colocated-test"],
*["npx", "-y", "testing-conventions@latest", "unit", "colocated-test"],
*["--language", "python", "--base", "origin/main"],
*["--config", "testing-conventions.toml", "packages/python/dirsql"],
]
Expand All @@ -117,7 +117,7 @@ def it_omits_config_for_a_root_that_declares_none():
def it_targets_the_package_root_for_e2e_verify_scoped_to_the_source():
assert call(PY, "python", "e2e-verify", {"extra_scope": ["packages/rust/src"]}) == Invocation(
[
*["uvx", "testing-conventions", "e2e", "verify"],
*["npx", "-y", "testing-conventions@latest", "e2e", "verify"],
*["--base", "origin/main"],
*["--scope", "packages/python/dirsql"],
*["--extra-scope", "packages/rust/src", "packages/python"],
Expand All @@ -131,8 +131,9 @@ def it_omits_language_for_e2e_verify_which_does_not_accept_it():
def it_runs_python_mutation_through_the_packages_own_venv():
mutation = call(PY, "python", "mutation")
assert mutation.cwd == "packages/python"
assert mutation.argv[:5] == [
*["uv", "run", "--with", "testing-conventions", "testing-conventions"]
assert mutation.argv[:7] == [
*["uv", "run", "--with", "testing-conventions"],
*["npx", "-y", "testing-conventions@latest"],
]

def it_rewrites_the_config_and_source_paths_relative_to_that_cwd():
Expand All @@ -154,10 +155,10 @@ def it_runs_python_unit_coverage_through_that_venv_too():
def it_runs_a_typescript_suite_gate_through_npx_from_the_package_root():
root = Root(job="ts", source="packages/ts/src", languages=["typescript"], gates=["mutation"])
mutation = call(root, "typescript", "mutation")
assert mutation.argv[:3] == ["npx", "-y", "testing-conventions"]
assert mutation.argv[:3] == ["npx", "-y", "testing-conventions@latest"]
assert (mutation.cwd, mutation.argv[-1]) == (".", "packages/ts/src")

def it_keeps_a_rust_mutation_gate_on_uvx():
def it_keeps_a_rust_mutation_gate_on_the_default_cli():
assert call(RUST, "rust", "mutation").cwd == "."


Expand Down Expand Up @@ -228,7 +229,7 @@ def it_runs_the_drift_guards_first_then_every_pair():
assert drive(runner=lambda argv, cwd: calls.append((argv, cwd)) or 0) == 0
assert [argv[:2] for argv, _cwd in calls] == [
*[["uv", "sync"], ["uv", "run"]],
*[["uvx", "testing-conventions"], ["uv", "run"]],
*[["npx", "-y"], ["uv", "run"]],
]
assert [cwd for _argv, cwd in calls] == [".", ".", ".", "packages/python"]

Expand Down Expand Up @@ -283,7 +284,7 @@ def it_echoes_the_argv_it_is_about_to_run():
lines = []
drive(only=["unit-lint"], echo=lines.append)
assert lines[0] == (
"==> python-sdk [python] unit-lint: uvx testing-conventions unit lint "
"==> python-sdk [python] unit-lint: npx -y testing-conventions@latest unit lint "
"--language python packages/python/dirsql"
)

Expand All @@ -292,7 +293,8 @@ def it_runs_only_the_gates_named_by_the_filter():
drive(only=["mutation"], runner=lambda _argv, _cwd: 0, echo=lines.append)
assert [line for line in lines if line.startswith("==>")] == [
"==> python-sdk [python] mutation: uv run --with testing-conventions "
"testing-conventions unit mutation --language python --base origin/main dirsql"
"npx -y testing-conventions@latest unit mutation --language python "
"--base origin/main dirsql"
]

def it_prints_every_pair_without_running_any_when_dry_run():
Expand Down
Loading
Loading