Skip to content

feat(engine): select an exec environment, and put it in the cache key - #409

Open
raphaelvigee wants to merge 2 commits into
raphaelvigee/exec-runners-phase0from
raphaelvigee/exec-runners-phase1
Open

feat(engine): select an exec environment, and put it in the cache key#409
raphaelvigee wants to merge 2 commits into
raphaelvigee/exec-runners-phase0from
raphaelvigee/exec-runners-phase1

Conversation

@raphaelvigee

@raphaelvigee raphaelvigee commented Aug 22, 2026

Copy link
Copy Markdown
Member

Phase 1a of #404, on top of #405. The identity half of exec runners: a target can say which environment it builds in, and that choice reaches hashin. Nothing executes differently yet — #413 opens sessions, #411 carries them across the plugin ABI.

Landing the key change alone is deliberate. It is the part that can silently produce a wrong build, and it is provable without a session existing.

Surface

defaultRunner: //:devenv          # .hephconfig — the expected way to use this
target(name = "build", ..., runner = "//:devenv")   # per-target
target(name = "bootstrap", ..., runner = None)      # opt out

Resolution: per-target → defaultRunner → local. Three levels, not fourrunner is deliberately not accepted in transitive, which is dependency-directional: as specified there, adding a dep would silently change a target's execution environment, and a scalar in an additive merge would pick an iteration-order winner.

runner = names a target, not a driver-style name — the one place it differs from driver = sitting beside it, so a bare word is rejected with an error saying why. That it must be a target is the load-bearing choice of the whole feature: only a target has a hashout, and only a hashout reaches the key without inventing a second hash component.

Three decisions worth reviewing

The runner is a hashed: true, runtime: false input — the existing hash_deps shape. That pair is the trick: the environment's hashout reaches the consumer's hashin while its bytes are never materialized into the consumer's sandbox. Materializing would cost a symlink, a list file and an SRC_* entry per target — 20k redundant file ops on a 20k-target build — and would change what an in-sandbox glob matches.

Injected after apply_transitive, not before. A driver's apply_transitive may rebuild def.inputs rather than push to it, and the runner is an engine-level concern no driver is told about. Injecting earlier would leave its presence in the cache key at the mercy of a driver that doesn't know it exists — and its absence is silent, because a target that hashes as though it built on the host is exactly what it would then be. Nothing is lost by being late: collect_transitive_deps filters on runtime, which this input is not.

A runner target with no outputs is rejected. hashin folds input hashouts, and a zero-output target has none — so two runners describing completely different environments would give their consumers byte-identical keys, and an artifact built in one would be served for the other. Folding the runner's own hashin instead would have been a second, runner-shaped hash component; refusing keeps exactly one way for anything to reach the key.

Compatibility

  • TargetSpec.runner crosses the provider ABI as an additive string (empty = unauthored, "local" = opt-out, else an addr). A provider that knows nothing about runners is unaffected.
  • defaultRunner: needs no rejection path — ConfigYaml is already deny_unknown_fields.
  • No cache entry is invalidated: a target with no runner hashes byte-identically to today, asserted by a test.

Tests (crates/e2e/tests/exec_runner.rs, 7 new)

  • runner identity reaches the key; two runners ⇒ two keys
  • runner = None and no-runner hash identically — the compatibility promise
  • the runner artifact is not materialized (no env.json in the sandbox, SRC_/LIST_ empty)
  • zero-output runner ⇒ typed error naming the actual problem
  • bare name ⇒ rejected with an error explaining a runner is a target
  • workspace default applies, and runner = None survives it
  • the runner target does not inherit the default (no cycle)

Full local run green: 545 e2e, 183 engine, 91 plugin-exec, plus buildfile/abi. lint clean.

Not in this PR

ExecRunner/open, the session pool, wiring the resolved runner into execution, inspect output, and the devenv cdylib — those are #411 and #413.

Stack created with GitHub Stacks CLIGive Feedback 💬

Phase 1a of docs/EXEC_RUNNERS.md. Adds the authored surface and the identity
half of exec runners: a target can say which environment it builds in, and
that choice reaches `hashin`. Nothing executes differently yet — Phase 1b
opens sessions and Phase 2 carries them across the plugin ABI.

Landing the key change on its own is deliberate. It is the part that can
silently produce a wrong build, and it is provable without a session existing:
a runner target's hashout reaches its consumers, `runner = None` hashes
byte-identically to today, the artifact is never materialized, and a runner
that describes nothing is refused.

Surface (§6): per-target `runner = "//pkg:name"`, workspace `defaultRunner:`,
and `runner = None` as the opt-out. Resolution is per-target → default →
local. Three levels, not four: `runner` is deliberately NOT accepted in
`transitive`, which is dependency-directional — as specified there, adding a
dep would silently change a target's execution environment, and a scalar in an
additive merge would pick an iteration-order winner.

`runner =` names a *target*, not a driver-style name. That is the one place it
differs from `driver =` sitting beside it, so a bare word is rejected with an
error saying why rather than guessed at. The reason it must be a target is the
whole design: only a target has a hashout, and only a hashout reaches the key
without inventing a second hash component.

The runner is wired in as a `hashed: true, runtime: false` input — the
existing `hash_deps` shape. That pair is the trick: the environment's hashout
reaches the consumer's `hashin` while its bytes are never materialized into
the consumer's sandbox. Materializing would cost a symlink, a list file and an
`SRC_*` entry per target — 20k redundant file operations on a 20k-target build
— and would change what an in-sandbox glob matches, which nobody asked for.

Injected AFTER `apply_transitive`, not before. A driver's `apply_transitive`
may rebuild `def.inputs` rather than push to it, and the runner is an
engine-level concern no driver is told about. Injecting earlier would leave
its presence in the cache key at the mercy of a driver that does not know it
exists — and its absence is silent, because a target that hashes as though it
built on the host is exactly what it would then be. Nothing is lost by being
late: `collect_transitive_deps` filters on `runtime`, which this input is not.

A runner target with **no outputs is rejected**. `hashin` folds input
*hashouts*, and a zero-output target has none — so two runners describing
completely different environments would give their consumers byte-identical
keys, and an artifact built in one would be served for the other. Folding the
runner's own `hashin` instead would have been a second, runner-shaped hash
component; refusing is the option that keeps exactly one way for anything to
reach the key.

The runner target is excluded from the workspace default. It has to build
under something, and letting it inherit makes it its own dependency — which
the cycle checker does catch, but reports as a graph problem for what is
really a config one.

`TargetSpec.runner` crosses the provider ABI as an additive string field
(empty = unauthored, `"local"` = opt-out, else an addr), so a provider that
knows nothing about runners is unaffected. `defaultRunner:` needs no rejection
path: `ConfigYaml` is already `deny_unknown_fields`.
Phase 1b of docs/EXEC_RUNNERS.md, on top of 1a's selection and hashing. The
resolved runner now becomes a live `ExecSession` that the target's processes
are actually created in.

`ExecRunner::open` turns a runner target's artifacts into a session. Which
implementation runs is decided by the **driver name of the runner target**: a
plugin exporting a `devenv` driver (which builds the environment artifact)
exports a `devenv` runner beside it (which reads that artifact back). One name,
two halves.

`open` must be a pure function of its request, and the request carries the
runner target's artifact bytes rather than a path. That is not tidiness: `open`
runs after `hashin` is computed, and does not run at all on a fully-cached
build, so anything it reads that is not in the request is unhashed input that
cannot be validated on the build where a stale artifact is served.

The pool is keyed by **content** — the runner target's hashouts — not by
address. Two runner targets whose artifacts are byte-identical describe the
same environment and share one session, which is the intended behaviour and
falls out of the same reasoning that keeps only the hashout in the cache key.

It is deliberately not `hmemoizer`. That gives single-flight and panic-guarding
and nothing else this needs: no TTL, no refcount, errors memoized for the
process lifetime, and every instance in the tree request-scoped. A pool that
cached "devenv.nix doesn't evaluate" forever would keep failing after the user
fixed it, so a failed open evicts its cell and the next request retries.

`EnvSession` is the `Direct` mode: a base environment merged **underneath** the
caller's own entries. The caller wins on a collision, because by the time a
driver hands over a spec it has already resolved the target's `env`/`pass_env`
and the sandbox's `$OUT`/`$SRC` routing, and none of that may be silently
replaced by an environment the target did not write. Built as a pre-sized `Vec`
rather than routed through a `HashMap<String, String>`, which would clone every
key and value and convert each again — hundreds of allocations per target for a
60–150-variable dev shell.

**An ABI-served driver refuses a runner outright.** `plugin-go` and
`plugin-oci` are cdylib-only, and a driver behind the seam cannot yet be told
which environment to build in — that is Phase 2's `runner_*` fields and their
positive ack. Degrading is the dangerous option, not the safe one: the runner's
hashout is already folded into the target's `hashin`, so a plugin quietly
building in the host environment would write an artifact whose key asserts an
environment its bytes do not reflect, and push it to the shared remote cache.
An older cdylib could not even warn — prost drops fields it was not compiled
with before guest code runs.

The session reaches the driver on `RunRequest`, and `ShellFallback` forwards it
to the synthetic shell target, so `heph run --shell` shows the environment the
target actually sees rather than the host's.

Tests (crates/e2e/tests/exec_runner.rs, 4 new on top of 1a's 7):
- the session's environment reaches the target, is derived from the runner
  target's artifact, and does not overwrite the target's own `env`
- **one open serves many targets** — the "spawn the shell once" requirement,
  asserted with a counter, which is the only thing that can show it
- byte-identical runner artifacts share one session
- an ABI-served driver refuses a runner, naming the driver and why
@raphaelvigee
raphaelvigee force-pushed the raphaelvigee/exec-runners-phase1 branch from cc2d15c to 4b3589f Compare August 23, 2026 23:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant