[thekevinbot]

Fixes #1001. Closes the second half of #957.

Stacked on #1000 (claude/tackle-957-lrm0z6), which adds the Progress reporter this reuses. GitHub retargets this to main when #1000 merges.

Why this is in core and not in the plugin

#957 asks for progress "on the embedding path in the plugin". I probed that design before writing it and it does not work. A worker drawing its own \r counter on its inherited stderr leaves the line un-erased, and the query result lands on top of it — measured with a fake worker wired in as an ordinary [[dirsql.function]]:

fakeworker: embedded 13 values[{"v":1},{"v":2},…,{"v":13}]

The worker cannot fix that itself, for two independent reasons: ProcessTransport::drop SIGKILLs it (so no atexit, signal handler or tqdm.close() ever runs), and it is torn down after println! prints the result anyway — so it could never know when to erase. An idle-timer thread inside the worker races the host's own stdout write and does not close the gap either. Full repro in #1001.

The process that owns stdout has to own the line, and that process is dirsql. It also already has the number worth showing: functions.rs performs exactly one round trip per call. So core counts them — no protocol change, no plugin change, and it works for any worker-backed function rather than just embed(), which fits "dirsql knowing nothing about the domain" better than a domain-specific bar would.

What it looks like

dirsql: running 9204 worker calls        ← rewritten in place during the query
dirsql: ran 41231 worker calls in 2m41s  ← after the live line is erased

Same DIRSQL_PROGRESS gate as #1000 (auto / always / never), same warmup, same silence on a pipe. A query that calls no worker prints nothing even under always: the reporter speaks only when there is something to say.

Notes for review

• Db::query opens the phase with an RAII guard, so an error path still ends it and the live line is always gone before the caller prints rows. The third e2e test is the regression test for exactly that.
• One reporter shared by every declared function. What a user waiting on a query wants to know is how many round trips it is paying for, not which function made them.
• Progress gains a noun — worker round trips are not files, and a line calling them files would be lying about what it counted.
• Progress::restart reuses a reporter for a fresh phase instead of constructing one per query. That is not just an optimization: reconstructing discarded an injected sink and silently reverted to stderr, which is how the phase-guard unit test first failed.
• A poisoned reporter mutex costs the counter, never the call — reporting is not worth failing a query over.
• Db::query is not reentrant here (create_path_table runs execute_batch, not a nested query), so phases never nest.

Scope

Core only, again — but the plugin gets it for free, and that is verified rather than assumed: dirsql-plugin-embeddings builds embed(content)-per-row SQL and runs it through the Python SDK's db.query, which is the same Db::query this brackets. Driving that path directly through the PyO3 binding (see the manual run below) reports identically to the CLI.

The cache-hit split (… (8811 cached)) stays out, filed as #1034. It is the plugin's private knowledge and surfacing it needs an optional metadata field in the worker response, i.e. a change to the response contract documented in docs/reference/config.md's "Worker protocol" section and restated in functions.rs's module doc — worth doing only if it turns out to matter. It is also the only part of #1001 that lands in plugins/dirsql-plugin-embeddings/.

Red/green

f8b77e7 pushed the e2e first. CI went red on Integration tests (run 32420619521): test result: FAILED. 3 passed; 2 failed, with the summary carries the count and the elapsed time: "" — the two reporting assertions failing on the empty stderr they captured, the three guard assertions (no calls means no counter, a pipe stays silent, stdout stays clean) passing. 863948c is the implementation, which flips those two green.

Manual run

Through the CLI, 14 files and a worker sleeping 50 ms per call, under a real pty with no env var set (so this is the default auto path, warmup and all):

$ dirsql query "SELECT fake(basename) AS v FROM './*.txt'" -c .dirsql.toml
dirsql: running 11 worker calls          ← first draw, half a second in
dirsql: running 13 worker calls
dirsql: ran 14 worker calls in 0.7s
[{"v":"f1.txt"},{"v":"f10.txt"}, …]      ← on its own line, nothing glued to it

Through the Python SDK, the path the plugin actually takes:

$ DIRSQL_PROGRESS=always python drive.py     # dirsql.DirSQL(config=…) + await db.query(…)
dirsql: running 15 worker calls
dirsql: ran 15 worker calls in 0.8s
SDK returned 15 rows

E2E Verification

• Ran e2e suites locally for every affected SDK
• Python SDK e2e: pass (32 passed)
• TypeScript SDK e2e: pass (16 passed)
• Rust core e2e: pass (--tests runs all 48 suites under packages/rust/tests/, including the new worker_call_progress_e2e)
• packages/python/e2e-attestations/claude-tackle-957-lrm0z6-worker-calls.json receipt written
• packages/ts/e2e-attestations/claude-tackle-957-lrm0z6-worker-calls.json receipt written
• Command(s) run:
  • cargo test --manifest-path packages/rust/Cargo.toml --features cli --lib → 711 passed
  • cargo test --manifest-path packages/rust/Cargo.toml --features cli --tests → 48 suites, all ok
  • cd packages/python && uv run maturin develop --release && uvx testing-conventions e2e attest 'uv run python -m pytest tests/e2e/ -x -q'
  • cd packages/ts && pnpm build && uvx testing-conventions e2e attest 'pnpm test:e2e'
  • cargo clippy --workspace -- -D warnings, cargo fmt --all -- --check → clean
• Result summary: green everywhere. Both receipts committed — a packages/rust/src change stales both bindings' attestations via [e2e].extra_scope.
• Re-run after the main merge: cargo test --features cli --lib → 807 passed; --tests → all suites pass except persist_parsed_path_table (pre-existing, machine-speed-dependent, see above). just test-e2e → 32 passed, 5 pre-existing repl_interactive_test.py failures reproduced identically on clean main. uvx testing-conventions e2e verify passes for both packages against the committed receipts.

Rebase onto main (2026-09-02)

main moved (#1000 merged, so this retargeted; #1014 landed after). Merged in, and re-ran everything against the moving-tag toolchain, which had advanced to testing-conventions 0.0.111. Three things changed:

• rust / Unit mutation — changed lines now actually runs. The earlier 0 mutant(s) tested note is obsolete — 0.0.111 tests this diff and found one survivor: replace <impl CallProgress for Progress>::restart with (). Real gap, not an equivalent mutant: the only test touching a real Progress went through CallReporter::new(), whose reporter is disabled off a terminal and so has nothing to erase. Killed with an assertion.
• CallProgress moved to progress.rs. Killing that mutant from functions.rs needed use crate::progress::{Mode, SystemClock} in a unit test, which unit lint's no-out-of-module-import rule rejects. The trait describes Progress, so it now lives beside it; functions.rs imports it and still injects its double.
• conn() got its doc comment back. call_reporter had landed between conn()'s docs and conn() itself, taking over both the prose and the #[doc(hidden)].

Merge main (2026-09-04)

#1023 landed the fixes for the two testing-conventions 0.0.111 gates that were reding this PR, so main (612bcf1, all five workflows green) was merged in. The merge was clean — no conflicts. It brings only gate config and internals/checks; no Rust source moved, so the implementation is untouched.

one-function-per-file needed no work here: this PR's source diff is Rust-only, and the gate is Python/TypeScript-only ([python]/[typescript] one_function_per_file = { max_lines = 100 }). No exemption was added.

Re-verified against the merged tree:

• just preflight --gate one-function-per-file --gate colocated-test --gate unit-lint --gate integration-lint --gate e2e-verify --gate declared-deps → 0 failing pair(s) across all 33 pairs.
• just preflight --gate unit-coverage → 0 failing pair(s).
• just preflight --gate mutation → python / typescript / internals / plugin arms all no mutatable changed lines; the rust arm cannot run on this machine — cargo-mutants aborts with cargo test failed in an unmutated tree because persist_parsed_path_table asserts a wall-clock floor and fails on a fast machine #1028's persist_parsed_path_table timing assertion fails before any mutant is built. CI's slower runner is the authority and it passes there (rust / Unit mutation — changed lines, 9m47s).
• dirsql-checks changelog-gate → pass. Fragments unchanged and still correct: packages/rust is the only changed package, and it carries both a changelog and a migration fragment.
• PARITY.md unchanged by the merge; still accurate, still no drift.

E2E receipts were not rewritten. The gate is diff-based and does not compare a receipt's commit against HEAD, both receipts are already in the main...HEAD diff, and e2e verify passes for both packages against the merged tree. Re-attesting python would in fact fail — just e2e-attest-python only records a receipt when the command exits 0, and just test-e2e is red on #1027's five pre-existing REPL failures, which are not this PR's to fix.

Suites re-run in full on the merged tree:

• cargo test --workspace --features cli --no-fail-fast → every suite passes except persist_parsed_path_table asserts a wall-clock floor and fails on a fast machine #1028's unchanged_second_run_reuses_the_cache_instead_of_reparsing (cold run measured 2.30s against a ≥3s assertion).
• cargo test --features cli --test worker_call_progress_e2e → 5 passed, this PR's own e2e suite.
• uv run python -m pytest tests/e2e/ -q (packages/python) → 32 passed, 5 failed — the five are exactly REPL e2e tests still assert JSON output after table rendering landed #1027's repl_interactive_test.py cases, all failing on table-vs-JSON output.
• pnpm test:e2e (packages/ts) → 16 passed, 5 files.

Not from this PR

These failures are pre-existing on main and are handled elsewhere:

• python-sdk / Static checks and typescript-sdk / Static checks fail on the new unit one-function-per-file and integration lint (no-first-party-patch) gates. Resolved — ci: adopt the testing-conventions one-function-per-file gate at max_lines = 100 #1017 / ci: Python integration lint is red -- no-first-party-patch now resolves the patch.object form #1024 are fixed by ci: adopt one-function-per-file at max_lines = 100 #1023, merged to main and merged in here.
• REPL e2e tests still assert JSON output after table rendering landed #1027 — packages/python/tests/e2e/repl_interactive_test.py has 5 failures on clean main: the REPL e2e tests (cb75f4e, 21:38 UTC) predate table rendering (060576f, 22:33 UTC), so they still assert JSON output while the REPL now draws a table. E2E is not in CI, so nothing caught it. persist_parsed_path_table asserts a wall-clock floor and fails on a fast machine #1028 — packages/rust/tests/persist_parsed_path_table.rs also fails on a fast machine (asserts a cold run takes ≥3s; measured 2.3s).

Changelog / Migrations

• Changelog fragment: packages/rust/changelog.d/2026-08-20-worker-call-progress.md
• Migration fragment: packages/rust/migrations.d/2026-08-20-worker-call-progress.md — required, because this changes runtime behavior without an API change: a query that used to write nothing to stderr may now write to it. DIRSQL_PROGRESS=never is the documented restore.

Parity

PARITY.md updated. No drift: the reporter sits in Db::query, below every SDK's query / await db.query, so all three change at once and none carries a copy.

[thekevinbot]

CI Gate is red on 793f0ea, and it is not this PR's diff. Reporting rather than fixing, because the fix belongs in a file this PR has no business touching.

What the gate says

Required: [changelog-check.yml, dirsql-python-ci.yml, dirsql-rust-ci.yml,
           dirsql-typescript-ci.yml, docs-ci.yml]
error: Unpredicted workflow runs: [".github/workflows/release-ci.yml"].

job log. Not a failing check — an unpredicted one, which the gate treats as "the check set cannot be confirmed complete".

Why

release-ci.yml is the only workflow here that declares pull_request: branches: [main]. This PR is stacked, so its base is claude/tackle-957-lrm0z6, not main. willfire reads the filter, sees base ≠ main, and leaves release-ci out of Required. GitHub dispatched it anyway (run 32421684955, event: pull_request, pull_requests: [1002]), so the gate sees a run nothing predicted and fails.

The asymmetry is visible within this PR's own history: on f8b77e7 release-ci did not dispatch, and on 793f0ea it did. Every other workflow declares no branches: filter, which is why only this one diverges.

Worth flagging beyond this PR: any stacked PR whose diff touches a release glob will hit this, and that is most of them. #991 does not, only because it is the bottom of its stack and based on main.

Proposed patch (not applied here)

Dropping branches: [main] from release-ci.yml's pull_request trigger makes willfire's prediction match what GitHub actually does. It is a widening, not a narrowing — main-based PRs already run it, and stacked PRs would gain a build gate they currently get without the gate knowing about it. The paths: list and its subset relationship with putitoutthere.toml's globs (#944, dirsql-checks release-globs) are untouched by that.

The alternative fix is upstream in willfire, modelling whatever GitHub is doing with branches: on a non-main base.

Either way it is a CI-config change with release-path implications, so it should not ride along on a progress-reporting PR.

What is not blocked

The substantive checks are unaffected by this and were green locally before the push — 711 unit tests, all 48 integration/e2e suites, clippy, fmt, both binding e2e suites plus their attestations. They are still in flight on this head; if any of them comes back red, that one is mine and I will fix it.

This should also resolve itself: when #1000 merges, GitHub retargets this PR to main, release-ci becomes predicted, and the gate has nothing to complain about.

---

Generated by Claude Code