Skip to content

perf(pkg): parse git ls-tree output by index instead of regex - #15990

Closed
Alizter wants to merge 5 commits into
ocaml:mainfrom
Alizter:push-krosrsyvkzsk
Closed

Alizter wants to merge 5 commits into
ocaml:mainfrom
Alizter:push-krosrsyvkzsk

Conversation

@Alizter

@Alizter Alizter commented Aug 17, 2026

Copy link
Copy Markdown
Collaborator

Stacked on #15989. Only the final commit is new in this draft.

Summary

  • Parse the fixed git ls-tree -z --long record shape with delimiter indices.
  • Remove per-entry regex compilation and matching from the package-lock hot path.
  • Preserve blob, submodule, size, and path handling.

Tests

  • dune runtest test/blackbox-tests/test-cases/pkg/git-repo.t test/blackbox-tests/test-cases/pkg/submodules.t test/expect-tests/dune_pkg
  • dune build @fmt @check

@Alizter Alizter added the package management Dune's package management — `(pkg)` stanza, lockdirs, `dune pkg` commands label Aug 17, 2026
@Alizter
Alizter force-pushed the push-krosrsyvkzsk branch 4 times, most recently from c582f82 to d531fe0 Compare August 17, 2026 11:50
@Alizter
Alizter force-pushed the push-krosrsyvkzsk branch 7 times, most recently from c2d3131 to 3172785 Compare August 17, 2026 23:06
Signed-off-by: Ali Caglayan <alizter@gmail.com>
@Alizter
Alizter force-pushed the push-krosrsyvkzsk branch from 3172785 to 443b431 Compare August 17, 2026 23:19
Build one SAT universe for every requested platform instead of solving each
platform independently. Enforce one version per package name across platforms
while allowing different package-name alternatives, evaluate availability and
local-package constraints in each platform environment, and minimize distinct
avoid-version package versions.

Keep the single-platform SAT encoding unchanged, preserve each selected
platform package when extracting lock data, and only copy extra files for
selected package versions.

Signed-off-by: Ali Caglayan <alizter@gmail.com>
Use the conditional lock-file representation for every newly generated lock directory and remove the portable-lock-dir feature flag and split code paths.

Keep old lock directories readable: bare atom-form dependency fields decode as all-platform dependencies, and legacy bare build/install actions continue to run on every platform.

Signed-off-by: Ali Caglayan <alizter@gmail.com>
Signed-off-by: Ali Caglayan <alizter@gmail.com>
@Alizter
Alizter force-pushed the push-krosrsyvkzsk branch 2 times, most recently from a114ad7 to 0ae5b8d Compare August 18, 2026 10:07
rev_store Entry.parse compiled a Re regex and ran Re.exec_opt per
line of every git ls-tree -z --long listing. pkg lock lists the
whole repository tree, putting this on the lock hot path. Replace
it with index-based parsing of the fixed ls-tree shape
(perm type hash<TAB>size<TAB>path).

Re-verified with dune pkg lock on a 4-platform solve against the
full opam-repository (5,983 SAT variables, 14,394 clauses):
wall unchanged within noise (regex 259ms vs index 262ms, median
of 10); the Re parse frames disappear from the perf profile
(Re.Core.entry 0.5% -> gone); ~380K words of transient
allocations removed (alloc-profiler measurement on the original
ocaml-re benchmark).

Signed-off-by: Ali Caglayan <alizter@gmail.com>
@Alizter
Alizter force-pushed the push-krosrsyvkzsk branch from 0ae5b8d to 3b306a8 Compare August 18, 2026 10:42
@Alizter Alizter closed this Aug 18, 2026
@Alizter
Alizter deleted the push-krosrsyvkzsk branch August 18, 2026 10:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

no changelog package management Dune's package management — `(pkg)` stanza, lockdirs, `dune pkg` commands

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant