feat(artifacts): narrow binary lookups to declared dependencies by punchagan · Pull Request #15381 · ocaml/dune

punchagan · 2026-06-29T15:39:18Z

Description

Binary lookups made from a directory owned by a package (via the package's
(dir ...) field) are narrowed to the packages that package explicitly depends on,
(depopts ..) included. This covers %{bin:...}, %{bin-available:...}, the PATH
given to actions, and the binaries staged by (deps %{bin:...}).

Directories with no owning package are unaffected: nothing is narrowed and
every package is searched, as before. Binaries found on the ambient PATH,
and those bound by (env (binaries ...)) are also unaffected.

Related Issue and Motivation

To know which binaries a lock directory package provides, dune has to build
it since the metadata currently available does not provide that information.
So resolving a single %{bin:X} meant building every package in the lock
directory to find out which one provides X. Narrowing the search to the
owning package's dependencies restricts the build to only those packages
(and their build dependency packages).

Shrinking the built set is also a prerequisite for the in-and-out work
(#8652), where forcing every package's install cookie causes a cycle.

Known limitations

Dependency filters such as {with-test} are not interpreted, so a
dependency is visible whatever its filter says.

## Description Extend the test as a precursor to fixing PATH to include only the bin layouts of packages that the owning package of a stanza explicitly depend on. ## Related Issue and Motivation    This change is a preparatory improvement to the test for the changes in #15381

## Description This commit extends the resolve-program-from-undeclared-pkg.t cram test in preparation for narrowing of the lockdir packages on PATH based on the dependency closure of a stanza's owning package. This PR is similar to #15222 ## Related Issue and Motivation    This is a preparatory change in the resolve-program test case for narrowing of the lockdir package binaries available on PATH implemented in #15381.

## Description  This is another preparatory commit with tests for #15381. It captures current behavior w.r.t bin resolution from both workspace deps, lockdir deps and a mix of deps. ## Checklist - [x] Tests added, if applicable. - [ ] [Change log entry added](../CONTRIBUTING.md#updating-the-changelog) for any user-facing changes. - [ ] Documentation added for any user-facing changes.

Signed-off-by: Puneeth Chaganti <punchagan@muse-amuse.in>

This commit improves the bin-narrowing/transitive-deps.t test to list the lockdir packages being built for bin pform lookups. Signed-off-by: Puneeth Chaganti <punchagan@muse-amuse.in>

Signed-off-by: Puneeth Chaganti <punchagan@muse-amuse.in>

Co-authored-by: Puneeth Chaganti <punchagan@muse-amuse.in> Signed-off-by: Ali Caglayan <alizter@gmail.com>

Signed-off-by: Puneeth Chaganti <punchagan@muse-amuse.in>

To discover the binaries provided by a lockdir package, dune currently has to build it -- the currently available metadata on packages does not provide this information. So, searching for a binary meant building all the lockdir packages to see which one provides a particular binary. This commit narrows the search space to the set of packages that the directory's owning package depends upon. A directory has an owning package when a (package ...) stanza has it as the (dir ...) field value. When a directory has no owning package, the search is not narrowed and all the lockdir packages are built as before. The dependency closure is included in the narrowed search to match what opam users see: installing a package into an opam switch also installs the transitive dependencies, and so their binaries are available on PATH, even when those dependencies aren't explicitly depended upon. Binaries not found in the narrowed set still fallback to the ambient PATH, as before. Signed-off-by: Puneeth Chaganti <punchagan@muse-amuse.in>

%{bin:...} lookups have been narrowed to the owning package's dependencies. But, actions can also look up binaries by their names through PATH, for instance through (system ...). Previously, every lockdir package's install prefix contributed its bin directory to PATH, so a binary not visible to %{bin:...} could still be reached via PATH. This commit puts only the bin directories of the same narrowed dependency closure on PATH, so that both lookups are consistent with each other. PATH is now added via Env_node rather than the context-wide lock directory env, since the visible packages could be different per directory. To facilitate this, Env_node's env is now split into two parts, the inherited part and the node's own PATH, which cannot be inherited. Signed-off-by: Puneeth Chaganti <punchagan@muse-amuse.in>

We restricted the search space when looking up binaries provided by lockdir packages to the dependencies declared on the owning package. This commit extends this restriction to binaries provided by local workspace packages too. Under `dune build -p`, sibling install stanzas are masked away, so finding a sibling binary provided by an undeclared dependency fails at release time during `opam install`, or in the opam-repository CI, while a plain `dune build` resolves it happily. This narrowing promotes this validation to dev time and surfaces the failure earlier. The `(dir ...)` field takes on an additional meaning as a result: the search for sibling binaries is now narrowed to the transitive closure of the packages declared in the `(depends ...)` field. This narrowing depends on package management being enabled. This would mean that projects newly adopting package management may need to fix their project metadata for everything to work correctly. This dependency on package management has been retained to keep the blast radius of this change as small as possible. Signed-off-by: Puneeth Chaganti <punchagan@muse-amuse.in>

Naming a binary in `(deps %{bin:...})` also stages it into a `.binaries` directory that is prepended to the action's PATH, so that the action can invoke it by bare name. This staging looked the binary up in the context-wide artifacts, which are not narrowed. So when the directory's own lookup picked a different file -- an undeclared sibling shadowing a declared lockdir package of the same name -- the staged copy came first on PATH, and `(system ...)` and `%{bin:...}` named different files within a single action. This commit passes the directory's own artifacts to the staging instead, so that both name the same file. They are obtained through `Super_context.artifacts_host`, since the staging is given the host context but a directory of the target context. This only matters once local binary lookups are narrowed. Until then the narrowing only applied to the lockdir fallthrough, whose results the staging discards anyway, so either set of artifacts staged the same file. Signed-off-by: Puneeth Chaganti <punchagan@muse-amuse.in>

Previously, creation of bin layouts had two look-ups for binaries, once in Bin_layout.create where it recorded the lookup name and install names, and the second time when actually creating the symlinks. This commit collapses them into a single look-up by storing the original path for the binary to create the symlink. Signed-off-by: Puneeth Chaganti <punchagan@muse-amuse.in>

Signed-off-by: Puneeth Chaganti <punchagan@muse-amuse.in>

Only the binaries provided by packages which are listed as depends (or depopts) are visible to %{bin} or %{bin-available} pform look-ups when narrowing is enabled. Narrowing is only enabled when using Dune package management and for packages whose stanza has a `(dir ...)` field. Signed-off-by: Puneeth Chaganti <punchagan@muse-amuse.in>

rgrinberg · 2026-09-28T20:10:36Z

+    packages are included, not their dependencies. [None] means the whole lock
+    directory. Empty when the context has no lock directory. *)
+val env_for_packages
+  :  packages:Package.Name.Set.t option


It's weird to repurpose None for this purpose.

rgrinberg · 2026-09-28T20:14:47Z

+let installed_env t =
+  let* env = t.builder.env in
+  let+ bin_path = Pkg_rules.bin_path_env ~packages:None t.builder.name in
+  Env_path.extend_env_concat_path env bin_path


Is it ok to constantly recompute this?

rgrinberg · 2026-09-28T20:17:10Z

-         Findlib_config.discover_from_env ~env ~which ~ocamlpath ~findlib_toolchain)
+         Findlib_config.discover_from_env
+           ~env
+           ~which:(which ~packages:None)


This seems wrong. Shouldn't this be the closure of packages starting at findlib (if findlib is in the dep closure)

rgrinberg · 2026-09-28T20:19:03Z

+         | None -> visible
+         | Some pkg ->
+           List.fold_left
+             (Package.depends pkg @ Package.depopts pkg)


Seems wrong as well. Why would we just select all the depopts like this? We know the ones that are effective.

rgrinberg · 2026-09-28T20:19:52Z

+      List.filter all_project_deps ~f:(fun (pkg : Pkg.t) ->
+        Package.Name.Set.mem packages pkg.info.name)
+    in
+    filtered


Useless variable

punchagan force-pushed the narrow-pkg-which branch 5 times, most recently from 344e923 to e258dde Compare July 1, 2026 10:12

shonfeder assigned punchagan Jul 1, 2026

punchagan requested a review from Alizter July 1, 2026 17:03

punchagan force-pushed the narrow-pkg-which branch 2 times, most recently from 8aaa6fd to e186b0f Compare July 2, 2026 09:12

This was referenced Jul 3, 2026

test: extend pkg/resolve-program cram test for narrowing #15412

Merged

test: extend bin-pform-from-undeclared-pkg for PATH changes #15413

Merged

punchagan force-pushed the narrow-pkg-which branch 2 times, most recently from 2ea1389 to 985e8af Compare July 3, 2026 15:11

rgrinberg reviewed Jul 5, 2026

View reviewed changes

Comment thread src/dune_rules/pkg_rules.mli Outdated

punchagan force-pushed the narrow-pkg-which branch 2 times, most recently from 8c4b3eb to 4845db6 Compare July 8, 2026 07:34

punchagan mentioned this pull request Jul 8, 2026

Overlapping Dependencies in Lock Directory ("in and out problem") #8652

Open

punchagan mentioned this pull request Jul 17, 2026

Add more bin narrowing tests #15513

Merged

1 of 3 tasks

punchagan force-pushed the narrow-pkg-which branch 6 times, most recently from 10e2ec4 to c88489f Compare July 23, 2026 10:41

Alizter added the needs-rebase label Jul 27, 2026

punchagan force-pushed the narrow-pkg-which branch 2 times, most recently from 102e137 to 48132a8 Compare July 29, 2026 14:25

punchagan force-pushed the narrow-pkg-which branch from bd167ed to abcd061 Compare September 9, 2026 02:57

punchagan added a commit to punchagan/dune that referenced this pull request Sep 9, 2026

doc: add changelog entry for ocaml#15381

848db4d

Signed-off-by: Puneeth Chaganti <punchagan@muse-amuse.in>

punchagan force-pushed the narrow-pkg-which branch from abcd061 to 848db4d Compare September 9, 2026 03:36

punchagan requested a review from Alizter September 9, 2026 11:30

punchagan added a commit to punchagan/dune that referenced this pull request Sep 10, 2026

doc: add changelog entry for ocaml#15381

ba886f8

Signed-off-by: Puneeth Chaganti <punchagan@muse-amuse.in>

punchagan force-pushed the narrow-pkg-which branch from 848db4d to ba886f8 Compare September 10, 2026 11:17

punchagan added a commit to punchagan/dune that referenced this pull request Sep 11, 2026

doc: add changelog entry for ocaml#15381

635851d

Signed-off-by: Puneeth Chaganti <punchagan@muse-amuse.in>

punchagan force-pushed the narrow-pkg-which branch from ba886f8 to 635851d Compare September 11, 2026 15:59

punchagan added a commit to punchagan/dune that referenced this pull request Sep 11, 2026

doc: add changelog entry for ocaml#15381

9353774

Signed-off-by: Puneeth Chaganti <punchagan@muse-amuse.in>

punchagan and others added 16 commits September 22, 2026 17:32

test(pkg): list lockdir packages being built for bin pform lookups

fded791

This commit improves the bin-narrowing/transitive-deps.t test to list the lockdir packages being built for bin pform lookups. Signed-off-by: Puneeth Chaganti <punchagan@muse-amuse.in>

test(pkg): add test for lockdir binaries used by cross-compiled actions

cb55325

Signed-off-by: Puneeth Chaganti <punchagan@muse-amuse.in>

test(pkg): check that executables can launch companion executables

fef57a5

Signed-off-by: Puneeth Chaganti <punchagan@muse-amuse.in>

test(pkg): assert precedence of env binaries over lockdir binaries

bacac05

Signed-off-by: Puneeth Chaganti <punchagan@muse-amuse.in>

test(pkg): assert (deps (package ...)) on an action works correctly

895346a

Signed-off-by: Puneeth Chaganti <punchagan@muse-amuse.in>

test(pkg): assert that cross-project binaries are visible

4119119

Co-authored-by: Puneeth Chaganti <punchagan@muse-amuse.in> Signed-off-by: Ali Caglayan <alizter@gmail.com>

test(pkg): assert PATH made available to dune file plugin

c0bf416

Signed-off-by: Puneeth Chaganti <punchagan@muse-amuse.in>

test(pkg): tests for run command to lockdir-deps.t

725a8ee

Signed-off-by: Puneeth Chaganti <punchagan@muse-amuse.in>

test(pkg): capture behavior of bin-available with depopts

9ec143f

Signed-off-by: Puneeth Chaganti <punchagan@muse-amuse.in>

doc: add changelog entry for ocaml#15381

ee1fc24

Signed-off-by: Puneeth Chaganti <punchagan@muse-amuse.in>

rgrinberg reviewed Sep 28, 2026

View reviewed changes

Navigation Menu

feat(artifacts): narrow binary lookups to declared dependencies - #15381

feat(artifacts): narrow binary lookups to declared dependencies#15381
punchagan wants to merge 16 commits into
ocaml:mainfrom
punchagan:narrow-pkg-which

punchagan commented Jun 29, 2026 •

edited

Loading

Uh oh!

Uh oh!

rgrinberg Sep 28, 2026

Uh oh!

rgrinberg Sep 28, 2026

Uh oh!

rgrinberg Sep 28, 2026

Uh oh!

rgrinberg Sep 28, 2026

Uh oh!

rgrinberg Sep 28, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

punchagan commented Jun 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related Issue and Motivation

Known limitations

Uh oh!

Uh oh!

rgrinberg Sep 28, 2026

Choose a reason for hiding this comment

Uh oh!

rgrinberg Sep 28, 2026

Choose a reason for hiding this comment

Uh oh!

rgrinberg Sep 28, 2026

Choose a reason for hiding this comment

Uh oh!

rgrinberg Sep 28, 2026

Choose a reason for hiding this comment

Uh oh!

rgrinberg Sep 28, 2026

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

punchagan commented Jun 29, 2026 •

edited

Loading