feat(engine): select an exec environment, and put it in the cache key - #409
Open
raphaelvigee wants to merge 2 commits into
Open
feat(engine): select an exec environment, and put it in the cache key#409raphaelvigee wants to merge 2 commits into
raphaelvigee wants to merge 2 commits into
Conversation
raphaelvigee
marked this pull request as ready for review
August 23, 2026 11:19
raphaelvigee
force-pushed
the
raphaelvigee/exec-runners-phase1
branch
from
August 23, 2026 11:29
fb6cccd to
cc2d15c
Compare
Phase 1a of docs/EXEC_RUNNERS.md. Adds the authored surface and the identity half of exec runners: a target can say which environment it builds in, and that choice reaches `hashin`. Nothing executes differently yet — Phase 1b opens sessions and Phase 2 carries them across the plugin ABI. Landing the key change on its own is deliberate. It is the part that can silently produce a wrong build, and it is provable without a session existing: a runner target's hashout reaches its consumers, `runner = None` hashes byte-identically to today, the artifact is never materialized, and a runner that describes nothing is refused. Surface (§6): per-target `runner = "//pkg:name"`, workspace `defaultRunner:`, and `runner = None` as the opt-out. Resolution is per-target → default → local. Three levels, not four: `runner` is deliberately NOT accepted in `transitive`, which is dependency-directional — as specified there, adding a dep would silently change a target's execution environment, and a scalar in an additive merge would pick an iteration-order winner. `runner =` names a *target*, not a driver-style name. That is the one place it differs from `driver =` sitting beside it, so a bare word is rejected with an error saying why rather than guessed at. The reason it must be a target is the whole design: only a target has a hashout, and only a hashout reaches the key without inventing a second hash component. The runner is wired in as a `hashed: true, runtime: false` input — the existing `hash_deps` shape. That pair is the trick: the environment's hashout reaches the consumer's `hashin` while its bytes are never materialized into the consumer's sandbox. Materializing would cost a symlink, a list file and an `SRC_*` entry per target — 20k redundant file operations on a 20k-target build — and would change what an in-sandbox glob matches, which nobody asked for. Injected AFTER `apply_transitive`, not before. A driver's `apply_transitive` may rebuild `def.inputs` rather than push to it, and the runner is an engine-level concern no driver is told about. Injecting earlier would leave its presence in the cache key at the mercy of a driver that does not know it exists — and its absence is silent, because a target that hashes as though it built on the host is exactly what it would then be. Nothing is lost by being late: `collect_transitive_deps` filters on `runtime`, which this input is not. A runner target with **no outputs is rejected**. `hashin` folds input *hashouts*, and a zero-output target has none — so two runners describing completely different environments would give their consumers byte-identical keys, and an artifact built in one would be served for the other. Folding the runner's own `hashin` instead would have been a second, runner-shaped hash component; refusing is the option that keeps exactly one way for anything to reach the key. The runner target is excluded from the workspace default. It has to build under something, and letting it inherit makes it its own dependency — which the cycle checker does catch, but reports as a graph problem for what is really a config one. `TargetSpec.runner` crosses the provider ABI as an additive string field (empty = unauthored, `"local"` = opt-out, else an addr), so a provider that knows nothing about runners is unaffected. `defaultRunner:` needs no rejection path: `ConfigYaml` is already `deny_unknown_fields`.
Phase 1b of docs/EXEC_RUNNERS.md, on top of 1a's selection and hashing. The resolved runner now becomes a live `ExecSession` that the target's processes are actually created in. `ExecRunner::open` turns a runner target's artifacts into a session. Which implementation runs is decided by the **driver name of the runner target**: a plugin exporting a `devenv` driver (which builds the environment artifact) exports a `devenv` runner beside it (which reads that artifact back). One name, two halves. `open` must be a pure function of its request, and the request carries the runner target's artifact bytes rather than a path. That is not tidiness: `open` runs after `hashin` is computed, and does not run at all on a fully-cached build, so anything it reads that is not in the request is unhashed input that cannot be validated on the build where a stale artifact is served. The pool is keyed by **content** — the runner target's hashouts — not by address. Two runner targets whose artifacts are byte-identical describe the same environment and share one session, which is the intended behaviour and falls out of the same reasoning that keeps only the hashout in the cache key. It is deliberately not `hmemoizer`. That gives single-flight and panic-guarding and nothing else this needs: no TTL, no refcount, errors memoized for the process lifetime, and every instance in the tree request-scoped. A pool that cached "devenv.nix doesn't evaluate" forever would keep failing after the user fixed it, so a failed open evicts its cell and the next request retries. `EnvSession` is the `Direct` mode: a base environment merged **underneath** the caller's own entries. The caller wins on a collision, because by the time a driver hands over a spec it has already resolved the target's `env`/`pass_env` and the sandbox's `$OUT`/`$SRC` routing, and none of that may be silently replaced by an environment the target did not write. Built as a pre-sized `Vec` rather than routed through a `HashMap<String, String>`, which would clone every key and value and convert each again — hundreds of allocations per target for a 60–150-variable dev shell. **An ABI-served driver refuses a runner outright.** `plugin-go` and `plugin-oci` are cdylib-only, and a driver behind the seam cannot yet be told which environment to build in — that is Phase 2's `runner_*` fields and their positive ack. Degrading is the dangerous option, not the safe one: the runner's hashout is already folded into the target's `hashin`, so a plugin quietly building in the host environment would write an artifact whose key asserts an environment its bytes do not reflect, and push it to the shared remote cache. An older cdylib could not even warn — prost drops fields it was not compiled with before guest code runs. The session reaches the driver on `RunRequest`, and `ShellFallback` forwards it to the synthetic shell target, so `heph run --shell` shows the environment the target actually sees rather than the host's. Tests (crates/e2e/tests/exec_runner.rs, 4 new on top of 1a's 7): - the session's environment reaches the target, is derived from the runner target's artifact, and does not overwrite the target's own `env` - **one open serves many targets** — the "spawn the shell once" requirement, asserted with a counter, which is the only thing that can show it - byte-identical runner artifacts share one session - an ABI-served driver refuses a runner, naming the driver and why
raphaelvigee
force-pushed
the
raphaelvigee/exec-runners-phase1
branch
from
August 23, 2026 23:25
cc2d15c to
4b3589f
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Phase 1a of #404, on top of #405. The identity half of exec runners: a target can say which environment it builds in, and that choice reaches
hashin. Nothing executes differently yet — #413 opens sessions, #411 carries them across the plugin ABI.Landing the key change alone is deliberate. It is the part that can silently produce a wrong build, and it is provable without a session existing.
Surface
Resolution: per-target →
defaultRunner→ local. Three levels, not four —runneris deliberately not accepted intransitive, which is dependency-directional: as specified there, adding a dep would silently change a target's execution environment, and a scalar in an additive merge would pick an iteration-order winner.runner =names a target, not a driver-style name — the one place it differs fromdriver =sitting beside it, so a bare word is rejected with an error saying why. That it must be a target is the load-bearing choice of the whole feature: only a target has a hashout, and only a hashout reaches the key without inventing a second hash component.Three decisions worth reviewing
The runner is a
hashed: true, runtime: falseinput — the existinghash_depsshape. That pair is the trick: the environment's hashout reaches the consumer'shashinwhile its bytes are never materialized into the consumer's sandbox. Materializing would cost a symlink, a list file and anSRC_*entry per target — 20k redundant file ops on a 20k-target build — and would change what an in-sandbox glob matches.Injected after
apply_transitive, not before. A driver'sapply_transitivemay rebuilddef.inputsrather than push to it, and the runner is an engine-level concern no driver is told about. Injecting earlier would leave its presence in the cache key at the mercy of a driver that doesn't know it exists — and its absence is silent, because a target that hashes as though it built on the host is exactly what it would then be. Nothing is lost by being late:collect_transitive_depsfilters onruntime, which this input is not.A runner target with no outputs is rejected.
hashinfolds input hashouts, and a zero-output target has none — so two runners describing completely different environments would give their consumers byte-identical keys, and an artifact built in one would be served for the other. Folding the runner's ownhashininstead would have been a second, runner-shaped hash component; refusing keeps exactly one way for anything to reach the key.Compatibility
TargetSpec.runnercrosses the provider ABI as an additive string (empty = unauthored,"local"= opt-out, else an addr). A provider that knows nothing about runners is unaffected.defaultRunner:needs no rejection path —ConfigYamlis alreadydeny_unknown_fields.Tests (
crates/e2e/tests/exec_runner.rs, 7 new)runner = Noneand no-runner hash identically — the compatibility promiseenv.jsonin the sandbox,SRC_/LIST_empty)runner = Nonesurvives itFull local run green: 545 e2e, 183 engine, 91 plugin-exec, plus buildfile/abi.
lintclean.Not in this PR
ExecRunner/open, the session pool, wiring the resolved runner into execution,inspectoutput, and the devenv cdylib — those are #411 and #413.Stack created with GitHub Stacks CLI • Give Feedback 💬