Skip to content

Speed up path matching and add label-only traversal - #4

Merged
andrew merged 2 commits into
mainfrom
perf/path-matching
Sep 14, 2026
Merged

andrew merged 2 commits into
mainfrom
perf/path-matching

Conversation

@andrew

@andrew andrew commented Sep 14, 2026

Copy link
Copy Markdown
Contributor

Reduce path-matching overhead by storing pointers to corpus rules and rejecting invalid prefix boundaries before case folding. On Go 1.27.1, darwin/arm64, the median million-path benchmark fell from 274 ms to 209 ms across three runs. Matching the brief path inventory fell from 228 to 177 ns/path, with zero allocations. Corpus labels and evidence ordering are unchanged.

Add WalkMatch and Classifier.WalkMatch for visitors that only need a compact Set, plus CLI -labels-only. Both walk modes share traversal, limits and pruning semantics. For 10,000 files in a deep vendor/source/test layout, label-only traversal used 6.65 MB versus 34.05 MB for owned evidence results. Wide and monorepo disk timings were similar despite lower allocation, so this does not imply a speedup for every filesystem scan.

An isolated brief integration called Match on accepted paths inside its existing indexing loop, without adding file reads or another traversal. Six alternating pairs of full Engine.Run measurements over 10,000 synthetic files showed median scan overhead of 0.37%, with individual pairs between -0.37% and +1.03%. Import initialization was measured separately at roughly 1 ms. This experiment used minimal aggregation; storing richer per-file results would have additional costs. No brief integration is shipped here.

Update API documentation and CLI examples, add concurrent matching benchmarks, and cover label-only traversal through public APIs and the CLI, including vendor-root context, pruning, limits and errors.

@andrew
andrew merged commit f36e5b8 into main Sep 14, 2026
5 checks passed
@andrew
andrew deleted the perf/path-matching branch September 14, 2026 15:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant