Skip to content

perf: one equality lookup no longer indexes every row - #1530

Merged
thekevinbot merged 6 commits into
mainfrom
bot/1524-path-eq-1016
Oct 19, 2026
Merged

thekevinbot merged 6 commits into
mainfrom
bot/1524-path-eq-1016

Conversation

@thekevinbot

Copy link
Copy Markdown
Collaborator

Fixes #1524

A path-table answered every equality lookup by first building a HashMap<String, Vec<usize>> over all rows (one owned key per row, on one thread). For a single WHERE path = '...' that cost far more than the scan. Now the first probe of a column compares each row in place; the index is built only when the same cursor probes that column again, as a join's inner loop does.

Timings

Tree from glob_speed_recursive_test (320k .md under a/b), release CLI, median of 7 by hand:

query before after
SELECT count(*) FROM './a/b/**/*.md' 73ms 32ms (machine load differed)
... WHERE path = 'x' 305ms 26ms
... WHERE path = 'a/b/x1/y0/z0/n4.md' 259ms 24ms

New e2e case it_finds_one_path_by_equality_within_the_bar (15 paired runs, startup subtracted):

  • before: 1.477x find (red)
  • after: 0.167x find, 0.523x fd (fastest)

Red/green

E2E Verification

  • Ran e2e suites locally for every affected SDK
  • Python SDK e2e: pass (impacted subset)
  • TypeScript SDK e2e: pass
  • Rust core e2e (if applicable): pass (cargo test --workspace --features cli)
  • packages/python/e2e-attestations/<branch>.json receipt written if packages/python changed (just e2e-attest-python)
  • packages/ts/e2e-attestations/<branch>.json receipt written if packages/ts changed (just e2e-attest-ts)
  • Command(s) run: testing-conventions e2e attest 'uv run python -m pytest tests/e2e/glob_speed_recursive_test.py tests/e2e/use_case_join_test.py -x -q'; testing-conventions e2e attest 'pnpm test:e2e'; cargo test --workspace --features cli
  • Result summary: all pass. The Rust run needed a local, uncommitted fix to the scan_glob_rows wide-directory test that breaks main (main fails to compile tests: scan_glob_rows test from #1515 predates #1513's visit argument #1525).

@thekevinbot
thekevinbot merged commit 1c974b4 into main Oct 19, 2026
89 checks passed
@thekevinbot
thekevinbot deleted the bot/1524-path-eq-1016 branch October 19, 2026 10:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Speed: one equality lookup on a path-table builds an index over every row

1 participant