Skip to content

Path-table walk: readdir-level speed for the count(*) floor #1322

Closed

Description

@thekevinbot

Part of #1304.

SELECT count(*) FROM './*' on a 1M-file flat directory takes 0.61s with #1321 applied. find -maxdepth 1 takes 0.42s, so the budget is 0.46s. Even with no output at all, dirsql misses the 1.1x bar. The only cost native doesn't have is the SQLite insert, so the walk has to come down to native's cost.

How

  1. Profile the count(*) path. Break it into readdir, per-entry allocations, sort, judging, row build and SQLite stepping. If what remains after readdir is SQLite cost, report those numbers.
  2. Try raw getdents64/openat with reused buffers, and no per-entry PathBuf/String allocation before an entry is known to match.
  3. Walk sibling subdirectories in parallel.

What each glob returns, and the order, must not change.

Activity

  1. self-assigned this
    on Oct 5, 2026
  2. thekevinbot commented on Oct 6, 2026

    @thekevinbot
    CollaboratorAuthor

    Landed on main via #1317 (and #1356 for the dot-dir test); its PR merged into a stacked branch, so it didn't auto-close.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions