Skip to content

perf: each reference to a path table walks the tree again, and one walk is ~4x find at 400k files #1201

Closed

Description

@thekevinbot

use_case_dossier_test.py (#1200) is red at 17x find on a 400k-file tree. Two separate costs, measured on that tree (launcher included, warm page cache):

query wall
find . -name '*.md' -printf '%T@\t%P\n' | sort -rn | head -10 0.90 s
SELECT count(*) FROM './' 3.83 s
SELECT count(*) FROM './planner/internal/notes/*/*.md' 3.27 s
SELECT count(*) FROM (SELECT path FROM './' UNION ALL SELECT path FROM './' UNION ALL SELECT path FROM './') 10.53 s
  1. Every reference to a path table walks the tree again. Three CTEs over ./ cost three walks. One walk per distinct path table per statement would take the article's dossier query from 11 s to under 4 s on this tree.
  2. One walk is ~4x find at this size (1.4x at 50k files, Path-table glob walks the entire root instead of the glob's literal directory prefix #1199). find stats every file too, so the gap is on dirsql's side of the stat: row building and insertion, not the filesystem.

Bar from #1198: dirsql wall minus launcher startup <= 1.1x native. The test sizes its corpus until native takes a second, so it holds the same ratio on any machine.

Done when

packages/python/tests/e2e/use_case_dossier_test.py passes with its pytest.mark.xfail(strict=True) marker removed in the same PR: identical rows, and dirsql wall time minus launcher startup within 1.1x the native pipeline. Part of epic #1205.

Activity

  1. self-assigned this
    on Oct 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions