Progress reporting on the embedding path (dirsql-plugin-embeddings)

Second half of #957. The core half — progress on the startup scan — is #1000.

Problem

The plugin's only progress today is the Hugging Face model download bar (embedding/progress.py), TTY-gated. Nothing reports the embedding itself, and embedding is where the wall time is:

Today, dirsql-plugin-embeddings '<glob>' '<query>' builds SQL that calls embed(content) once per corpus row (search/sql.py:build_search_sql) inside a single db.query. On the reference corpus that is 41k worker round trips in one silent call.
After Declared tables: wide-open multi-statement ddl, catalog validation #956, a first vector-index build calls embed() once per ingested row from a trigger. Same cost, different phase.

The protocol embeds one value per round trip, so the worker cannot batch and the cost is irreducible. Per #957 the answer is visibility, not policy.

Proposed

A running counter from embedding/worker.py, whose stderr is already inherited straight through to dirsql's (functions.rs spawns it with Stdio::inherit()). The worker knows nothing about a total, so a count is what it can honestly report — the same shape core's walk phase uses:

dirsql-embeddings: embedded 9204 values (8811 cached)

The cache split is the number worth showing: the disk cache is what makes re-runs cheap, and a user watching a slow first build wants to know the second one will not cost this.

Gate it on the same DIRSQL_PROGRESS variable #1000 introduces (auto / always / never, terminal-only and warmup-gated under auto) rather than a plugin-specific variable — one knob for "show me progress" across the product.

The open design question

Two writers, one stderr. During a #956 triggered build, core draws dirsql: indexing 9204/41231 files (22%) while the worker would be drawing its own counter. Both rewrite their line with \r, so interleaved they produce garbage.

The two do not collide on today's query path — core finishes its scan before db.query runs — so this is specifically about the post-#956 build. Options, none evaluated yet:

Parent wins. Core sets DIRSQL_PROGRESS=never in the environment of the function workers it spawns while its own reporting is enabled. Simple, but the worker is spawned lazily at the first embed() call, which may be either phase, so the decision has to be made per spawn and functions.rs does not currently know whether a scan reporter is live.
Worker reports only when core is not drawing, via an explicit signal the core sets on the child. Same mechanism as (1), finer-grained.
One line, two counters — core includes worker-reported counts in its own line. Needs a protocol addition; almost certainly not worth it.

Resolve this before the worker draws anything; a garbled terminal is worse than a silent one.

Not in scope

The model download bar stays as-is — it already measures something real and is already TTY-gated.

Probed the worker-draws-its-own-bar design before writing any of it, and it does not work. Revising the proposal above.

What was measured

A fake worker that writes a \r-rewritten counter to its inherited stderr, one line per round trip, wired in as an ordinary [[dirsql.function]]:

# fakeworker.py
n = 0
for line in sys.stdin:
    n += 1
    sys.stderr.write(f"\rfakeworker: embedded {n} values"); sys.stderr.flush()
    sys.stdout.write(json.dumps({"ok": n}) + "\n"); sys.stdout.flush()

[[dirsql.function]]
name    = "fake"
args    = [1]
command = "python3 fakeworker.py"

Run under a real pty over 13 files, the tail of the captured terminal is:

fakeworker: embedded 13 values[{"v":1},{"v":2},…,{"v":13}]

The query result is glued onto the end of the worker's un-erased progress line. stdout lands on the stderr line the worker left behind, so a dirsql "…" | jq user who also watches the terminal sees corrupted output.

Why the worker cannot fix this itself

Two independent reasons, and each is sufficient:

It is SIGKILLed. ProcessTransport::drop (functions.rs:426) is child.kill() then child.wait(). No atexit, no signal handler, no tqdm.close() runs, so nothing erases the line.
Even a graceful shutdown would be too late. The worker is torn down when the index drops, which is after println!("{value}") in run_query. The line has to be erased when the query ends, and the worker has no idea when that is — it only ever sees "another request arrived" and silence, and silence is indistinguishable from a slow caller.

An idle-timer thread in the worker races the host's own stdout write, so it does not close the gap either.

Revised proposal: core counts the round trips

The process that owns stdout has to own the line. That process is dirsql, and it already has the number worth showing: functions.rs performs exactly one round trip per call, so core can count them without any protocol change and without the plugin knowing progress exists.

dirsql: embed() 9204 calls

drawn and erased by the same Progress #1000 adds, bracketed by the phase that owns the calls — the query for today's embed(content)-per-row search, the ingest loop for a post-#956 triggered build. That also dissolves the two-writers problem from the original description: both counters become core-side, so "only one reporter draws at a time" is a local decision rather than a cross-process protocol.

Sketch: an Arc<AtomicU64> per registered function, incremented in the transport's call(), read by a reporter around the query / ingest phase.

What this gives up

The cache-hit split (… (8811 cached)) is the plugin's private knowledge — core cannot see which calls the disk cache served. Showing it needs an optional metadata field in the worker response, i.e. a change to a documented protocol contract. Worth doing only if the split turns out to matter in practice; filing it separately rather than bundling it here.

Navigation Menu

Repository navigation

Progress reporting on the embedding path (dirsql-plugin-embeddings) #1001

Problem

Proposed

The open design question

Not in scope

What was measured

Why the worker cannot fix this itself

Revised proposal: core counts the round trips

What this gives up

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Progress reporting on the embedding path (dirsql-plugin-embeddings) #1001

Description

Problem

Proposed

The open design question

Not in scope

Activity

thekevinbot commented on Aug 20, 2026

What was measured

Why the worker cannot fix this itself

Revised proposal: core counts the round trips

What this gives up

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions