Skip to content

docs(exec-runners): what an exec runner is, and where the cache key is drawn - #404

Open
raphaelvigee wants to merge 1 commit into
masterfrom
raphaelvigee/exec-runners-design
Open

docs(exec-runners): what an exec runner is, and where the cache key is drawn#404
raphaelvigee wants to merge 1 commit into
masterfrom
raphaelvigee/exec-runners-design

Conversation

@raphaelvigee

@raphaelvigee raphaelvigee commented Aug 22, 2026

Copy link
Copy Markdown
Member

Doc only — no code. Phase 0 is #405 on top of this.

What

docs/EXEC_RUNNERS.md: a short reference for ExecRunner / ExecSession, a driver-independent answer to "in what environment is this target's process created?", with devenv.sh as the first non-trivial implementation (shipped as a cdylib plugin).

The ask was: let a target run inside the devenv shell and inherit its tools; make it general rather than a pluginexec feature; spawn the shell once and have many targets run in it.

It covers the model and the surface, how a runner reaches the cache key, the environment layering rules, the three session modes, devenv, Go, and diagnosability. It documents what the stack ships — the phasing lives in the PRs.

The shape

A runner reference is a target reference. Its identity then reaches the cache key through the existing dependency mechanism instead of a new hash component — and ordering, failure attribution and cycle checking all come along for free. In particular the runner target is built during hashin, before any worker permit is held, so "opening a session may require building a target" cannot deadlock against the execute semaphore.

Selection is a workspace-level defaultRunner: plus a per-target runner =.

Reviewed by the standing board

product-vision SHIP WITH CHANGES · compatibility COMPATIBLE 6/7 · feature-quality BLOCKED · hermeticity NOT HERMETIC · code-quality BLOCKED.

The model survived; the execution details did not. The findings that most altered it:

  • prepare(spec) -> Spec is the seam, not a trait-objectified process factory. Direct and Wrap are pure spec transformations, so proc_exec keeps its synchronous spawn, its Handle invariants and its PTY handling untouched.
  • Six of eight process-creation sites are proc_exec::output, which cannot be built on spawn — its drain is unbounded because nothing consumes until the wait returns. A go list over 512 KiB routed through spawn + wait wedges on darwin and passes on linux.
  • The "content-addressed toolchain" claim for snapshot mode was empirically false. Measured in this repo's own devenv shell: 38 of 107 PATH entries are outside /nix/store, and DEVENV_ROOT / $HOME / DEVENV_CMDLINE would all have entered the snapshot — under-hashing mutable host tools while over-hashing the checkout path and the literal command line used to enter the shell. (feat(plugin-devenv): capture a devenv environment and run targets in it #413 corrects the source of that measurement; the rule it produced stayed.)
  • runtime_pass_env = ["*"] sat above the runner's PATH, so a target could build with host tools under a key asserting devenv, then push that to the shared cache. Now a hard error.
  • A runner target with zero outputs is invisible to hashin — two different environments would share a cache key. Now rejected at resolution.
  • runner in transitive was removed: transitive is dependency-directional, so as specified, adding a dep would silently change a target's execution environment, with an iteration-order winner when two deps disagreed.

Decisions recorded as the user's

  • The snapshot's PATH is store-only and fails loudly. Host tools must become explicit tools = deps.
  • Snapshot artifacts are remote_enabled = false, matching the plugin-nix precedent ("wrappers point at host-local /nix/store").

Left open and flagged rather than assumed: manifest provenance, which is a cache-format change needing its own compatibility consult. (Whether enterShell / shell functions / services are load-bearing in your devenv.nix was settled by shipping both modes — see #414.)

Stack created with GitHub Stacks CLIGive Feedback 💬

@raphaelvigee raphaelvigee changed the title raphaelvigee/exec runners design docs(exec-runners): design for a driver-independent exec environment Aug 22, 2026
@raphaelvigee
raphaelvigee marked this pull request as ready for review August 23, 2026 11:19
@raphaelvigee
raphaelvigee force-pushed the raphaelvigee/exec-runners-design branch from 96c88cf to d33f464 Compare August 23, 2026 11:29
@raphaelvigee raphaelvigee changed the title docs(exec-runners): design for a driver-independent exec environment docs(exec-runners): what an exec runner is, and where the cache key is drawn Aug 23, 2026
raphaelvigee added a commit that referenced this pull request Aug 23, 2026
…e dangling doc refs

`crates/plugin-sdk` covers the lane at the seam — a driver's `open_session` /
`prepare_spec` / `close_session` crossing a real stabby vtable. What that
cannot show is the half above it: that the engine resolves a runner target to
its driver, opens one session for the environment, and routes every target's
actual process through it.

`crates/e2e/tests/exec_runner_spec.rs` runs a real BUILD graph against a
`DriverExecRunner` over a `ManagedDriver`, which is exactly what a loaded
cdylib becomes host-side:

- a driver-served session reaches the target's real process
- the runner is consulted **per target** while the environment opens **once** —
  the property that justifies the ABI lane over describing the environment as
  data
- a driver that serves no sessions is refused, not degraded to the host
  environment under a key asserting the runner's
- the runner's identity still reaches the cache key

Writing these surfaced a semantic worth stating: under this lane the host does
**not** merge `base_env` for anyone. The runner owns the whole transformation,
because the runner is what starts the process; `ExecSession::base_env` is what
the host *reports* for diagnostics, not what it applies. The first draft of the
fixture assumed otherwise and failed with `FROM_RUNNER: unbound variable`.

Also removes the `§N.N` references to the design doc's numbered sections, which
have pointed at nothing since #404 made it a short reference. Each now names a
real heading ("Environment layering", "How it reaches the cache key") or drops
the section. `GHA_REPORTING.md` and `CONCURRENCY_MEASUREMENTS.md` refs are
untouched — those docs still have those sections.

e2e 147 across 18 files, plus plugin-sdk 36 and the crate suites. lint clean.
raphaelvigee added a commit that referenced this pull request Aug 23, 2026
…op the dangling doc refs

`crates/plugin-sdk` covers the component at the seam — a runner's `open_session`
/ `prepare_spec` / `close_session` crossing a real stabby vtable. What that
cannot show is the half above it: that the engine resolves a runner target to
its registered runner, opens one session for the environment, and routes every
target's actual process through it.

`crates/e2e/tests/exec_runner_spec.rs` runs a real BUILD graph against a
`PluginExecRunner` over an `ExecRunnerPlugin` — exactly what a loaded cdylib
becomes host-side:

- a plugin-served session reaches the target's real process
- the runner is consulted **per target** while the environment opens **once** —
  the property that justifies a runner owning process creation
- a runner target with no registered runner is refused, not degraded to the host
  environment under a key asserting the runner's
- the runner's identity still reaches the cache key

The fixture is worth a look for what it does *not* contain: no `parse`, no
`run`, no schema. A runner is its own component kind, so the mock implements
three methods. Under the earlier draft — the lane bolted onto `ManagedDriver` —
the same mock needed three `bail!("unused")` stubs.

Writing these surfaced a semantic worth stating: the host does **not** merge
`base_env` for anyone. The runner owns the whole transformation, because the
runner is what starts the process; `ExecSession::base_env` is what the host
*reports* for diagnostics, not what it applies. The first draft of the fixture
assumed otherwise and failed with `FROM_RUNNER: unbound variable`.

Also removes the `§N.N` references to the design doc's numbered sections, which
have pointed at nothing since #404 made it a short reference. Each now names a
real heading ("Environment layering", "How it reaches the cache key") or drops
the section. `GHA_REPORTING.md` and `CONCURRENCY_MEASUREMENTS.md` refs are
untouched — those docs still have those sections.

e2e 147 across 18 files, plus plugin-sdk 36 and the crate suites. lint clean.
…s drawn

`docs/EXEC_RUNNERS.md`: a page on exec runners — the model, the authoring
surface, how a runner reaches the cache key, the environment layering, the
three session modes, the devenv and Go knobs, and what `heph inspect` will
tell you.

Reference rather than a design record: the rules and the reason each exists,
without the phasing, the review history or the corrections log that only
mattered while it was being built. That reasoning lives in the commits and
pull requests that made each change, which is where it is useful when someone
runs `git blame` on the line that surprised them.
raphaelvigee added a commit that referenced this pull request Aug 23, 2026
…op the dangling doc refs

`crates/plugin-sdk` covers the component at the seam — a runner's `open_session`
/ `prepare_spec` / `close_session` crossing a real stabby vtable. What that
cannot show is the half above it: that the engine resolves a runner target to
its registered runner, opens one session for the environment, and routes every
target's actual process through it.

`crates/e2e/tests/exec_runner_spec.rs` runs a real BUILD graph against a
`PluginExecRunner` over an `ExecRunnerPlugin` — exactly what a loaded cdylib
becomes host-side:

- a plugin-served session reaches the target's real process
- the runner is consulted **per target** while the environment opens **once** —
  the property that justifies a runner owning process creation
- a runner target with no registered runner is refused, not degraded to the host
  environment under a key asserting the runner's
- the runner's identity still reaches the cache key

The fixture is worth a look for what it does *not* contain: no `parse`, no
`run`, no schema. A runner is its own component kind, so the mock implements
three methods. Under the earlier draft — the lane bolted onto `ManagedDriver` —
the same mock needed three `bail!("unused")` stubs.

Writing these surfaced a semantic worth stating: the host does **not** merge
`base_env` for anyone. The runner owns the whole transformation, because the
runner is what starts the process; `ExecSession::base_env` is what the host
*reports* for diagnostics, not what it applies. The first draft of the fixture
assumed otherwise and failed with `FROM_RUNNER: unbound variable`.

Also removes the `§N.N` references to the design doc's numbered sections, which
have pointed at nothing since #404 made it a short reference. Each now names a
real heading ("Environment layering", "How it reaches the cache key") or drops
the section. `GHA_REPORTING.md` and `CONCURRENCY_MEASUREMENTS.md` refs are
untouched — those docs still have those sections.

e2e 147 across 18 files, plus plugin-sdk 36 and the crate suites. lint clean.
raphaelvigee added a commit that referenced this pull request Aug 23, 2026
…op the dangling doc refs

`crates/plugin-sdk` covers the component at the seam — a runner's `open_session`
/ `prepare_spec` / `close_session` crossing a real stabby vtable. What that
cannot show is the half above it: that the engine resolves a runner target to
its registered runner, opens one session for the environment, and routes every
target's actual process through it.

`crates/e2e/tests/exec_runner_spec.rs` runs a real BUILD graph against a
`PluginExecRunner` over an `ExecRunnerPlugin` — exactly what a loaded cdylib
becomes host-side:

- a plugin-served session reaches the target's real process
- the runner is consulted **per target** while the environment opens **once** —
  the property that justifies a runner owning process creation
- a runner target with no registered runner is refused, not degraded to the host
  environment under a key asserting the runner's
- the runner's identity still reaches the cache key

The fixture is worth a look for what it does *not* contain: no `parse`, no
`run`, no schema. A runner is its own component kind, so the mock implements
three methods. Under the earlier draft — the lane bolted onto `ManagedDriver` —
the same mock needed three `bail!("unused")` stubs.

Writing these surfaced a semantic worth stating: the host does **not** merge
`base_env` for anyone. The runner owns the whole transformation, because the
runner is what starts the process; `ExecSession::base_env` is what the host
*reports* for diagnostics, not what it applies. The first draft of the fixture
assumed otherwise and failed with `FROM_RUNNER: unbound variable`.

Also removes the `§N.N` references to the design doc's numbered sections, which
have pointed at nothing since #404 made it a short reference. Each now names a
real heading ("Environment layering", "How it reaches the cache key") or drops
the section. `GHA_REPORTING.md` and `CONCURRENCY_MEASUREMENTS.md` refs are
untouched — those docs still have those sections.

e2e 147 across 18 files, plus plugin-sdk 36 and the crate suites. lint clean.
raphaelvigee added a commit that referenced this pull request Aug 23, 2026
…op the dangling doc refs

`crates/plugin-sdk` covers the component at the seam — a runner's `open_session`
/ `prepare_spec` / `close_session` crossing a real stabby vtable. What that
cannot show is the half above it: that the engine resolves a runner target to
its registered runner, opens one session for the environment, and routes every
target's actual process through it.

`crates/e2e/tests/exec_runner_spec.rs` runs a real BUILD graph against a
`PluginExecRunner` over an `ExecRunnerPlugin` — exactly what a loaded cdylib
becomes host-side:

- a plugin-served session reaches the target's real process
- the runner is consulted **per target** while the environment opens **once** —
  the property that justifies a runner owning process creation
- a runner target with no registered runner is refused, not degraded to the host
  environment under a key asserting the runner's
- the runner's identity still reaches the cache key

The fixture is worth a look for what it does *not* contain: no `parse`, no
`run`, no schema. A runner is its own component kind, so the mock implements
three methods. Under the earlier draft — the lane bolted onto `ManagedDriver` —
the same mock needed three `bail!("unused")` stubs.

Writing these surfaced a semantic worth stating: the host does **not** merge
`base_env` for anyone. The runner owns the whole transformation, because the
runner is what starts the process; `ExecSession::base_env` is what the host
*reports* for diagnostics, not what it applies. The first draft of the fixture
assumed otherwise and failed with `FROM_RUNNER: unbound variable`.

Also removes the `§N.N` references to the design doc's numbered sections, which
have pointed at nothing since #404 made it a short reference. Each now names a
real heading ("Environment layering", "How it reaches the cache key") or drops
the section. `GHA_REPORTING.md` and `CONCURRENCY_MEASUREMENTS.md` refs are
untouched — those docs still have those sections.

e2e 147 across 18 files, plus plugin-sdk 36 and the crate suites. lint clean.
@raphaelvigee
raphaelvigee force-pushed the raphaelvigee/exec-runners-design branch from d33f464 to 38183ea Compare August 23, 2026 23:25
raphaelvigee added a commit that referenced this pull request Aug 24, 2026
…op the dangling doc refs

`crates/plugin-sdk` covers the component at the seam — a runner's `open_session`
/ `prepare_spec` / `close_session` crossing a real stabby vtable. What that
cannot show is the half above it: that the engine resolves a runner target to
its registered runner, opens one session for the environment, and routes every
target's actual process through it.

`crates/e2e/tests/exec_runner_spec.rs` runs a real BUILD graph against a
`PluginExecRunner` over an `ExecRunnerPlugin` — exactly what a loaded cdylib
becomes host-side:

- a plugin-served session reaches the target's real process
- the runner is consulted **per target** while the environment opens **once** —
  the property that justifies a runner owning process creation
- a runner target with no registered runner is refused, not degraded to the host
  environment under a key asserting the runner's
- the runner's identity still reaches the cache key

The fixture is worth a look for what it does *not* contain: no `parse`, no
`run`, no schema. A runner is its own component kind, so the mock implements
three methods. Under the earlier draft — the lane bolted onto `ManagedDriver` —
the same mock needed three `bail!("unused")` stubs.

Writing these surfaced a semantic worth stating: the host does **not** merge
`base_env` for anyone. The runner owns the whole transformation, because the
runner is what starts the process; `ExecSession::base_env` is what the host
*reports* for diagnostics, not what it applies. The first draft of the fixture
assumed otherwise and failed with `FROM_RUNNER: unbound variable`.

Also removes the `§N.N` references to the design doc's numbered sections, which
have pointed at nothing since #404 made it a short reference. Each now names a
real heading ("Environment layering", "How it reaches the cache key") or drops
the section. `GHA_REPORTING.md` and `CONCURRENCY_MEASUREMENTS.md` refs are
untouched — those docs still have those sections.

e2e 147 across 18 files, plus plugin-sdk 36 and the crate suites. lint clean.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant