Skip to content

Start per-table hooks while the walk is still running - #1502

Merged
thekevinbot merged 6 commits into
mainfrom
bot/speed-parse-overlap-1011
Oct 13, 2026
Merged

thekevinbot merged 6 commits into
mainfrom
bot/speed-parse-overlap-1011

Conversation

@thekevinbot

@thekevinbot thekevinbot commented Oct 12, 2026 •

Copy link
Copy Markdown
Collaborator

Refs #1483

Per-table on_file hooks now start on the first path the startup walk matches and are fed paths as it finds them, instead of after the walk ends. Hook-spawn latency and path copying overlap the walk. Output is unchanged; row order across invocations was already unspecified.

Measurements (65k-note vault, GROUP BY status, release binary, A/B interleaved vs PR A head)

  • Unrestricted CPUs (load ~7-13): min 82.4 -> 79.6ms, median 85.7 -> 88.9ms (within noise).
  • taskset -c 8-15 (load ~16): median 103.5 -> 87.2ms.
  • taskset -c 8-11: median 150 -> 132ms.
  • strace: first awk spawn at 53ms -> 23ms after process start.
  • Path-table query (no hook): 18.6 -> 19.1ms (noise). jsonl 692 -> 683ms (noise).
  • Harness ratio vs fd+xargs -P stays ~2.5x, so the bar is not flipped: the remaining cost is dirsql's own walk (~30ms), SQLite ingest and process start, not the spawn.

Decisions

Red/green

Red run 38190251372 (Unit tests job): prepare_starts_a_per_table_hook_while_the_walk_is_still_running failed at assert!(hook_saw_it_mid_walk.get()), the asserted reason.

E2E Verification

  • uv run python -m pytest tests/e2e/jsonl_parse_speed_test.py tests/e2e/frontmatter_parse_speed_test.py tests/e2e/on_file_query_test.py tests/e2e/query_contract_test.py -x -q: 14 passed (python receipt).
  • pnpm test:e2e: passed (ts receipt).
  • cargo test -p dirsql --lib: 1152 passed. Gated preflight: 0 failing.

@thekevinbot
thekevinbot merged commit 08dd34f into main Oct 13, 2026
89 checks passed
@thekevinbot
thekevinbot deleted the bot/speed-parse-overlap-1011 branch October 13, 2026 21:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant