RE-02.01 HermeticBuild: evolve beyond grep-based heuristic toward verifiable hermeticity

[mlieberman85]

Goal

Evolve the RE-02.01 HermeticBuild control beyond the v0.1 grep-based heuristic toward verifiable hermeticity. Track scope, roadmap, and what we will and won't support out of the gate.

Originating discussion: PR #138 review.

What v0.1 does

repro_hermetic_build_handler (in darnit-reproducibility) reads .github/workflows/*.{yml,yaml} line-by-line and flags lines containing suspicious substrings (curl , wget , pip install , npm install, yarn install, apt-get install, brew install) unless the same line contains a known-safe pattern (uv sync, uv pip install, pip install --no-index, npm ci, yarn --frozen-lockfile).

Result: PASS / FAIL / INCONCLUSIVE based on whether any suspicious line is found.

What v0.1 misses

The grep heuristic only sees workflow YAML. Real builds frequently fetch deps from places it can't see:

• Makefile — make build looks hermetic at the workflow level even if the Makefile does curl ... | bash.
• setup.py / pyproject.toml build hooks — setup.py can run arbitrary network code at build time.
• Build scripts — build.sh, scripts/setup, scripts/install-deps, etc.
• Composite GitHub Actions — action.yml files in .github/actions/*/ can run any commands.
• Reusable workflows — uses: org/repo/.github/workflows/foo.yml@ref is opaque.
• Containerfile / Dockerfile — RUN apt-get install inside the build container is invisible to workflow scanning.
• Pre-commit / lefthook / similar — hooks that fetch tools mid-build.
• Other CI systems — GitLab CI, CircleCI, Jenkinsfile, Buildkite, Drone are not scanned.

Even within workflows, the heuristic has known false-positive and false-negative shapes:

• FN: apt-get install -y curl on one line + curl ... | bash further down: caught by the per-line fix, but make install-deps invoking the same curl from a Makefile is not.
• FP: legitimate post-build uses (curl to upload artifacts, pip install of a release tool unrelated to the build itself).
• FP: comments referencing the suspicious patterns.

What "true" hermeticity verification needs

Grep on CI files is a smell test, not a verification. Real hermeticity comes from one of:

1. Runtime attestation — observe what the build actually does at runtime and attest to it. Witness collects in-toto attestations (network calls, file reads, subprocess invocations) during the build and produces signed evidence. We could (a) detect Witness in CI as a strong PASS signal, or (b) require Witness output as evidence for the control.
2. Build-system-enforced sandboxing:
   • Bazel with --sandbox_network=false and --remote_cache — network is structurally unavailable to actions.
   • Nix flakes with --pure-eval — builds run in a network-isolated sandbox by default.
   • Buck2, Pants — similar sandbox guarantees.
     Detecting these (presence of BUILD.bazel, flake.nix with pure = true, etc.) is a much stronger signal than grepping shell commands.
3. Reproducible-builds-style verification — running the build twice in different environments and diffing artifacts (reprotest, diffoscope). Already partially covered by RE-03.01 BitForBitReproducible — worth thinking about how the two controls compose.

Proposed roadmap

v0.2 — broaden the grep, lower false-positive rate

• Extend file scanning beyond workflow YAML: Makefile, **/Makefile, **/build.sh, setup.py, **/Containerfile, **/Dockerfile.
• Add support for non-GitHub CI: .gitlab-ci.yml, .circleci/config.yml, Jenkinsfile, azure-pipelines.yml.
• Strip YAML/shell comments before pattern matching to cut FPs.
• Treat apt-get install -y --no-install-recommends inside RUN of a Containerfile/Dockerfile as DEFERRED (image-build network is acceptable; build-step network in the resulting container is not).

v0.3 — strong PASS signals

• PASS the control if Bazel is detected with --sandbox_network=false (or no override).
• PASS the control if a flake.nix is present and used for the build (gated on RE-01.02 BuildEnvDeclared evidence).
• PASS the control if Witness attestations are present in the artifacts and assert no network access during the build step.
• These signals should short-circuit the grep heuristic, not stack with it — they're stronger evidence.

v1.0 — verifiable hermeticity

• Witness integration: emit a darnit MCP tool that returns the consultation prompt for the calling agent to run a Witness-instrumented rebuild and attach the attestation as evidence.
• Move the heuristic-only path from PASS/FAIL into INCONCLUSIVE-with-WARN unless one of the strong signals above is present. Grep alone should not produce a PASS at L2+.

Open questions

• Should v0.1 keep PASS/FAIL semantics on grep-only evidence, or downgrade to INCONCLUSIVE/WARN until v0.3 ships strong signals? Argument for downgrading: avoids falsely declaring a build hermetic on weak evidence (matches the project's conservative-by-default principle in CLAUDE.md).
• Where does this control sit relative to RE-03.01 BitForBitReproducible? They overlap — bit-for-bit is the ground truth, hermeticity is one prerequisite. Consider whether they should compose (RE-02.01 evidence feeds RE-03.01 signal) or remain independent.
• For non-GitHub CI, is it worth shipping in v0.2, or should that wait until at least one user requests it?

Acceptance for closing this issue

• v0.2 scope landed and tested against real-world repos (at least 3 distinct languages + 1 monorepo).
• v0.3 strong-signal detectors landed.
• v1.0 Witness integration design doc reviewed.

cc @Marc-cn

[mlieberman85]

Status update (2026-10-01). Checked against branch 042-candidate-integrity, which carries features 040, 041 and 042 and is not on main yet.

Done:

• v0.2 scan scope
• Witness detection (v0.3)
• Bazel/Nix signals (handlers.py ~L701)

Remaining:

• the v1.0 Witness rebuild tool
• downgrading grep-only evidence

Paths change with PR #532 (rename to darnit-amber).