Skip to content

feat(exec-runner): runner-declared env, Go knobs, and examples - #416

Open
raphaelvigee wants to merge 3 commits into
raphaelvigee/exec-runners-phase2bfrom
raphaelvigee/exec-runners-env
Open

feat(exec-runner): runner-declared env, Go knobs, and examples#416
raphaelvigee wants to merge 3 commits into
raphaelvigee/exec-runners-phase2bfrom
raphaelvigee/exec-runners-env

Conversation

@raphaelvigee

@raphaelvigee raphaelvigee commented Aug 23, 2026

Copy link
Copy Markdown
Member

On top of #414. Two commits, reviewable separately: a runner can now declare its own environment, and the Go plugin gets the two knobs that make runners usable there.

1. A runner declares env / pass_env / runtime_env / runtime_pass_env

The same four keys a target has. The split between them is where the cache key is drawn:

key resolved in the key
env at capture, literal yes
pass_env at capture, from the host yes — the value is baked in
runtime_env at spawn, literal the declaration only
runtime_pass_env at spawn, from the host the name only

The last row is the one that matters. SSH_AUTH_SOCK and DOCKER_HOST differ per machine and per login; passing them with pass_env bakes one machine's value into the environment's identity and hands every other machine a cache key that lies about what produced the artifact. runtime_pass_env reads them at spawn, so they reach the process and never the key. "*" is the same escape hatch a target has.

The declarations themselves are hashed either way — they live in the runner target's artifact, and that artifact is the environment's identity. heph will not claim two differently-declared environments are the same one, and will not put ambient host state in the key either.

Everything a runner declares sits underneath what the target set itself, wildcard included — a runner that could overwrite $OUT, $SRC or a target's own env would silently change what the target builds. Asserted directly.

SessionEnv lives in the contract, so every runner gets it; devenv populates it from four new spec keys. SNAPSHOT_FORMAT_VERSION → 3.

2. Go: goenv, and a runner for test execution

Two knobs, deliberately separate, because how it was built and how it runs are different questions.

goenv (provider option) — environment for the Go toolchain: go list, the compiler, the assembler. GOPRIVATE, GOFLAGS, a corporate GOPROXY.

Injected into each target's spec where it is hashed, not applied behind the drivers' backs. A knob that changed compiler input without moving the key would hand every machine artifacts built under settings the key does not record. GO_COMPILE_FORMAT_VERSION → 5, GO_GOLIST_FORMAT_VERSION → 18; both frozen def-hash goldens failed on the bump, which is what they are for.

One choke point (the provider's get) rather than the two dozen places a spec is built — a knob that reached most of them would be worse than one that reached none, because the gap surfaces as a cache hit on an archive built under different settings.

An allowlist decides who gets it: go_compile, go_golist. go_toolchain downloads an SDK and go_testmain writes a file — giving them an environment they never use would key them on it for nothing and re-key everything downstream. Lint/format exec heph-govet, not go.

Within a driver goenv is applied first, so GOROOT/GOOS/GOARCH/GOWORK=off win — those make the build hermetic and match the variant, and letting config override them would build something other than the address asked for, under that address's key.

provider_state(provider="go", test={"runner": "//:devenv"}) — the exec environment the test binary runs in. Joins the existing test = {...} map beside env/pass_env/pre_run, same allowlist that rejects typos, rather than inventing a second place to configure tests.

Only test/xtest take it. A test binary often wants a database client or a browser on PATH; the build wants the hermetic toolchain and nothing else. One environment for both would either leak the runtime's tools into every compile's key or force the build environment onto the test.

Examples

example/exec_runner/BUILD exercises what this stack added: selection and runner = None; a tool that resolves only under the runner and the same target failing without it; the four declared keys; a target overriding its runner; session mode providing a shell function snapshot mode cannot — with the snapshot counterpart kept as an expected failure, so the difference is visible rather than described.

Also fixes the example workspace, which has not loaded since #397 removed the sh driver: .hephconfig2 still listed builtin: sh, so every heph command run there failed at config load.

Where a runner comes from, per target type

Target type How a runner is set
BUILD-authored (bash, exec, http_fetch, nix, …) runner = //pkg:name, or runner = None
The runner target itself n/a — excluded from defaultRunner so it cannot become its own dependency
Go test / xtest (+_race) provider_state(test={"runner": …})
Go go_compile / go_golist inherits defaultRunner; environment shaped by goenv
Go go_toolchain / go_testmain / lint / format inherits defaultRunner; no goenv
Other provider-emitted (@heph/fs, group, hostbin, oci_*) inherits defaultRunner

runner: None on a TargetSpec means unauthored, not local — which is why provider-emitted targets inherit defaultRunner. That reaches go_compile, and plugin-go is cdylib-only, so that path works only because of #411's positive ack.

Known gap: go_lint/go_format inherit defaultRunner but get no goenv, since they exec heph-govet rather than go. If heph-govet needs a corporate GOPROXY, that wants its own knob.

Tests

26 in exec-runner (the four-key layering, "*", and the target-wins-over-everything case), 10 in plugin-devenv, 481 in plugin-go, plus e2e. lint clean.

Stack created with GitHub Stacks CLIGive Feedback 💬

@raphaelvigee raphaelvigee changed the title raphaelvigee/exec runners env feat(exec-runner): runner-declared env, Go knobs, and examples Aug 23, 2026
@raphaelvigee
raphaelvigee marked this pull request as ready for review August 23, 2026 11:19
@raphaelvigee
raphaelvigee force-pushed the raphaelvigee/exec-runners-env branch from ad5d85d to 9fd61f7 Compare August 23, 2026 11:29
@raphaelvigee
raphaelvigee force-pushed the raphaelvigee/exec-runners-env branch from 9fd61f7 to cf67110 Compare August 23, 2026 17:55
…me pair

A runner can now shape its environment with the same four keys a target has,
rather than only whatever it captured.

The split between them is where the cache key is drawn, and it is the whole
reason there are four rather than two:

  env               literal, captured    -> in the key
  pass_env          host VALUE captured  -> in the key; changing it re-keys
  runtime_env       literal, at spawn    -> the declaration is in the key
  runtime_pass_env  host value at SPAWN  -> only the NAME is ever hashed

The last row is the one that matters. `SSH_AUTH_SOCK` and `DOCKER_HOST` differ
per machine and per login; passing them with `pass_env` would bake one
machine's value into the environment's identity and hand every other machine a
cache key that lies about what produced the artifact. `runtime_pass_env` reads
them at spawn, so they reach the process and never the key. `"*"` is the same
escape hatch a target has.

The declarations themselves are hashed either way, because they live in the
runner target's artifact and that artifact *is* the environment's identity.
That is the honest line: heph will not claim two differently-declared
environments are the same one, and it will not put ambient host state in the
key either.

Everything a runner declares still sits UNDERNEATH what the target set itself —
a runner that could overwrite `$OUT`, `$SRC` or a target's own `env` would
silently change what the target builds. Asserted directly, including for the
`"*"` wildcard.

`SessionEnv` lives in the contract rather than in the devenv plugin, so every
runner gets the capability; the devenv driver populates it from four new spec
keys. `runtime_*` are resolved on every `prepare` rather than once at open, so
a host variable that changes mid-build is seen.

`SNAPSHOT_FORMAT_VERSION` -> 3.

Examples (example/exec_runner/BUILD), covering the paths this stack added:
selection and `runner = None`; a tool that resolves only under the runner and
the same target failing without it; the four declared keys; a target overriding
its runner; session mode providing a shell function that snapshot mode cannot,
with the snapshot counterpart kept as an expected failure so the difference is
visible rather than described.

Also fixes the example workspace, which has not loaded since #397 removed the
`sh` driver: `.hephconfig2` still listed `builtin: sh`, so every `heph` command
run there failed at config load.
Two knobs, deliberately separate, because "how it was built" and "how it runs"
are different questions and conflating them is what makes a cache key lie.

**`goenv`** (provider option) is environment for the **Go toolchain** — `go
list` and the compiler/assembler. `GOPRIVATE`, `GOFLAGS`, a corporate `GOPROXY`
and its `GONOSUMDB` belong here.

It is injected into each target's spec, where it is **hashed**, rather than
applied behind the drivers' backs at run time. A knob that changed compiler
input without moving the cache key would hand every machine artifacts built
under settings the key does not record — the same failure this stack has been
closing everywhere else. `GO_COMPILE_FORMAT_VERSION` -> 5 and
`GO_GOLIST_FORMAT_VERSION` -> 18, with both frozen def-hash goldens updated;
they failed on the bump, which is what they are for.

Injected at one choke point (the provider's `get`) rather than in each of the
two dozen places a spec is built: a knob that reached most of them would be
worse than one that reached none, because the gap would surface as a cache hit
on an archive built under different settings.

An allowlist decides who gets it — `go_compile` and `go_golist`. `go_toolchain`
downloads an SDK and `go_testmain` writes a file, so handing them an
environment they never use would put it in their key for nothing and re-key
everything downstream of them. The lint/format drivers exec `heph-govet`, not
`go`; stretching `goenv` to cover a different binary would make the name a lie.

Within a driver, `goenv` is applied FIRST so the driver's own settings win:
`GOROOT`, `GOOS`/`GOARCH` and `GOWORK=off` are what make the build hermetic and
match the target's variant, and letting config override them would build
something other than the address asked for, under that address's key.

**`provider_state(provider="go", test={"runner": "//:devenv"})`** sets the exec
environment the *test binary* runs in. It joins the existing `test = {...}` map
next to `env`/`pass_env`/`pre_run` — the same shape, the same allowlist that
rejects typos — rather than inventing a second place to configure tests.

Only `test`/`xtest` take it. A test binary often wants a database client, a
browser or a service on PATH; the build wants the hermetic toolchain and
nothing else. One environment for both would either leak the runtime's tools
into every compile's cache key or force the build environment onto the test.

The value is parsed where the rest of the map is validated, so a bare name
fails there with the other state errors — and it is rejected rather than
guessed at, because only a target has a hashout and a hashout is what carries
the environment into the key. Relative addresses resolve against the package
that declared the state.
…vironment

Caught by CI on both Linux legs, where the image sets `CC`: a runner declaring
`CC=clang` alongside `runtime_pass_env = ["*"]` produced `CC=gcc`.

The test was right and the ordering was wrong. I had mirrored a target's own
sequence, where `runtime_pass_env` sits on top of `env`. For a runner that is
backwards: a runner exists to *provide* an environment, so a passed-through
host variable outranking the captured one means `"*"` silently replaces that
environment with the developer's own — while the build stays keyed as though it
ran in the runner's. That is the ambient leak this whole design is built to
prevent, reintroduced by the one layer meant to be an escape hatch.

Weakest to strongest is now:

  host passthrough  <  captured environment  <  declared literals  <  the target

A runner that genuinely wants to override its own capture says so with
`runtime_env`, which is explicit rather than whatever the host happened to have.

The failing assertion is replaced by three that name the rule rather than
stumble into it: the capture outranks a `"*"` passthrough, `runtime_env` can
override the capture where the host cannot, and the target still wins over all
of it. They use uniquely-named variables, so they reproduce the CI condition
without depending on whether the machine sets `CC`.

§4.4 records the asymmetry with a target's own ordering, since a reader who
knows one would otherwise reasonably assume the other.
@raphaelvigee
raphaelvigee force-pushed the raphaelvigee/exec-runners-env branch from cf67110 to 94da999 Compare August 23, 2026 22:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant