Skip to content

Stream the CLI's JSON rows - #1327

Merged
thekevinscott merged 8 commits into
claude/1305-glob-speedfrom
claude/1326-stream-rows
Oct 6, 2026
Merged

thekevinscott merged 8 commits into
claude/1305-glob-speedfrom
claude/1326-stream-rows

Conversation

@thekevinbot

@thekevinbot thekevinbot commented Oct 5, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes #1326

Stacked on #1324.

With JSON output (the default when stdout is not a terminal), dirsql query now writes each row as SQLite yields it. Before, it built a HashMap for each row, then a serde_json::Value tree, and then formatted that. The bytes printed are unchanged. A column named twice still gives one key, at its first position, holding the last value.

These keep the old collected path:

  • table output
  • the server
  • statements that call a batched worker, because the worker can force a second run

The text is written once the statement finishes, so a query that fails part-way still prints no rows.

Red commit 0d5aa4b: streamed_rows_keep_one_key_per_repeated_column_name failed in CI because a repeated column's key was written twice.

Measurements

Wall time, best of 3, /usr/bin/time, release builds, #1324 compared with this PR:

Tree Query #1324 this PR find
1M files, flat SELECT path FROM './*' 0.81s 0.44s 0.43s
1M files, flat SELECT count(*) FROM './*' 0.38s 0.38s
1.6M files, recursive SELECT path FROM './**/*.md' 1.32s 0.80s 0.58s
1.6M files, recursive SELECT count(*) FROM './**/*.md' 0.71s 0.71s

glob_speed_one_level_test.py (#1305) now passes all 3 cases locally.

I compared output from the #1324 binary and this one on 15 queries:

All 15 are byte-identical.

E2E Verification

  • Python: just e2e-attest-python (pytest tests/e2e/): 43 passed. That includes glob_speed_one_level_test.py, so this time nothing was deselected.
  • TypeScript: just e2e-attest-ts (pnpm test:e2e) passed.
  • Rust: cargo test -p dirsql --features cli --no-fail-fast had one failure, the watcher test server_drops_the_rows_of_a_directory_moved_out_of_the_root. It does not touch this code path and passes when its binary is rerun on its own (47/47).
  • By hand:
    • the 15 identical-output queries above
    • the table output on a tty
    • the error path, which prints no rows and exits 1
    • | head -c 20, which closes the pipe without a panic

Changelog / Migrations

  • packages/rust/changelog.d/2026-10-05-cli-stream-rows.md
  • packages/rust/migrations.d/2026-10-05-cli-stream-rows.md (nothing breaks)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants