From b52085b0acf6b339696d2499ddb0ed128bd6cb82 Mon Sep 17 00:00:00 2001 From: Michael Lieberman Date: Wed, 7 Oct 2026 17:16:12 -0400 Subject: [PATCH 1/2] docs(spec): register repro_witness_attestation as a FAIL-only step (#553) Witness / in-toto runtime-trace verification moves out of repro_hermetic_build into its own step type with ceiling {fail}, bound to the audited commit and reading only the runtime-trace predicate. A verified empty network log is evidence, not PASS: an absent optional list equals an empty one, and a monitor that does not trace sockets records the same empty log. PASS waits for an allowlist of monitor types. Assisted-by: Claude:claude-opus-5-5 Signed-off-by: Michael Lieberman --- docs/architecture/framework-design.md | 14 ++- .../contracts/hermeticity-scan.md | 2 + .../contracts/step-contract.md | 2 + .../contracts/witness-step-addendum-553.md | 100 ++++++++++++++++++ 4 files changed, 115 insertions(+), 3 deletions(-) create mode 100644 specs/044-false-pass-paths/contracts/witness-step-addendum-553.md diff --git a/docs/architecture/framework-design.md b/docs/architecture/framework-design.md index f97d0d1a..59677352 100644 --- a/docs/architecture/framework-design.md +++ b/docs/architecture/framework-design.md @@ -1,8 +1,8 @@ # Darnit Framework Design Specification -> **Version**: 1.0.0-alpha.13 +> **Version**: 1.0.0-alpha.14 > **Status**: Authoritative -> **Last Updated**: 2026-10-06 +> **Last Updated**: 2026-10-07 This specification defines the authoritative design of the Darnit framework, including the sieve orchestrator, TOML schema, built-in pass types, remediation actions, and plugin protocol. @@ -224,7 +224,7 @@ Every step type registers a **ceiling**: the set of outcomes (`pass`, `fail`) it | `manual`, `manual_steps` | `{}` | -- | | remediation handlers (`file_create`, `platform_setting`, `project_update`, `yaml_inject`) | `{}` | -- | -A plugin handler registers its ceiling with the handler (`registry.register(..., ceiling={"pass", "fail"})`). A plugin handler that registers no ceiling has the ceiling `{}`: its results are evidence only. A step type that decides from text or file-presence signals registers `{fail}`; the reproducibility framework's five step types (`repro_deps_pinned`, `repro_build_env_declared`, `repro_hermetic_build`, `repro_provenance_exists`, `repro_bit_for_bit`) do, so they conclude PASS only with a promotion, and their signals reach the control's later steps as evidence (feature 044, section 12). +A plugin handler registers its ceiling with the handler (`registry.register(..., ceiling={"pass", "fail"})`). A plugin handler that registers no ceiling has the ceiling `{}`: its results are evidence only. A step type that decides from text or file-presence signals registers `{fail}`; the reproducibility framework's five such step types (`repro_deps_pinned`, `repro_build_env_declared`, `repro_hermetic_build`, `repro_provenance_exists`, `repro_bit_for_bit`) do, so they conclude PASS only with a promotion, and their signals reach the control's later steps as evidence (feature 044, section 12). Its sixth, `repro_witness_attestation`, registers `{fail}` for a different reason: a Sigstore-verified in-toto runtime-trace attestation for the audited commit can show that the build accessed the network, but cannot yet show that it did not. Under the in-toto parsing rules an absent optional list is equivalent to an empty one, and what `monitorLog.network` records depends on the monitor's type and trace policy, so a monitor that does not trace sockets produces the same empty log as a build with no network access. A verified clean trace is therefore evidence, and PASS waits for an allowlist of monitor types backed by real attestations (#553). **Step fields** (on any `[[controls."ID".passes]]` entry; none are passed to the handler except `fail_on_miss` and `fail_on_status`): @@ -264,6 +264,12 @@ The orchestrator computes the effective set from the registry at dispatch time a - **WHEN** a step declares `concludes = ["pass"]` on a step type whose ceiling does not include `pass`, without a promotion - **THEN** loading the framework configuration MUST fail with an error naming the framework, control, step index, and outcome +#### Scenario: A runtime trace proves only network access +- **WHEN** a `repro_witness_attestation` step verifies an attestation for the audited commit whose runtime-trace `monitorLog.network` records one or more events +- **THEN** its FAIL MUST conclude the control +- **AND** when the verified trace records no network events, the step MUST NOT report PASS; its result MUST be recorded as evidence and evaluation MUST continue +- **AND** an attestation whose signing identity is not bound to the audited repository and the audited commit MUST NOT be used + ### 3.0.2 Step Disposition Table The disposition applied to each step, by handler outcome and whether the outcome is in the step's effective set: @@ -1934,6 +1940,7 @@ Implementation-registered sieve handlers (non-exhaustive): | `generate_threat_model` | `darnit-baseline` (remediation) | `{pass, fail}` | | `gittuf_verify_policy`, `gittuf_commits_signed` | `darnit-gittuf` | `{pass, fail}` (cryptographic verification) | | `repro_deps_pinned`, `repro_build_env_declared`, `repro_hermetic_build`, `repro_provenance_exists`, `repro_bit_for_bit` | `darnit-reproducibility` | `{fail}`: they decide from text and file-presence signals, so PASS needs a corpus-backed promotion (section 3.0.1) | +| `repro_witness_attestation` | `darnit-reproducibility` | `{fail}`: a verified runtime-trace attestation can show network access but, until monitor types are allowlisted, cannot show its absence (section 3.0.1, #553) | | `csl_llm_if_present` | `darnit-csl` | `{fail}` | The reproducibility ceilings are part of that framework's own registration, not its package name, so they hold under a rename of the package. @@ -2318,6 +2325,7 @@ The following requirements have been superseded: by the handler dispatch archite | Version | Date | Changes | |---------|------|---------| +| 1.0.0-alpha.14 | 2026-10-07 | `repro_witness_attestation`: Witness/in-toto runtime-trace verification moves out of `repro_hermetic_build` into its own step type with ceiling `{fail}`, bound to the audited commit and reading only the runtime-trace predicate; a verified clean trace is evidence, not PASS (Sections 3.0.1, 12; #553) | | 1.0.0-alpha.13 | 2026-10-06 | Error class `unexpected_exit`: an `exec` step whose undeclared exit code has no identified cause no longer reports `network`; exit code 127 reports `missing_tool` (Sections 3.3, 5.2; #562) | | 1.0.0-alpha.12 | 2026-10-04 | Repository-level `.baseline.toml` is no longer read: one notice points at `darnit config migrate`, framework selection only by `--framework` or a tool argument (Sections 2.3, 10.5, 14.4, 15.1; Appendix C) | | 1.0.0-alpha.11 | 2026-10-04 | Close remaining false-PASS paths (feature 044): an expression that cannot be evaluated or is not boolean makes the step ERROR, expression names per step type with usable `project` values and a repository-aware `file_exists`, load-time expression reference check (Section 3.7); step registration declares `settings` and `expression_names`, plugins cannot replace a registered step type, and unknown control keys, unknown step keys, and unregistered step types fail loading (Sections 2.3, 3.0.3); reproducibility step types conclude only FAIL (Sections 3.0.1, 12); `expr_decides`, an expression that decides on a handler PASS (Sections 3.0.3, 3.7); `gh_api` `evidence_fields`, required for personal records (Section 3.8); `file_must_exist` replaced by the registered `file_exists` (Section 3.2); field tables corrected to what each handler reads (Sections 3.3-3.5) | diff --git a/specs/038-repro-false-pass/contracts/hermeticity-scan.md b/specs/038-repro-false-pass/contracts/hermeticity-scan.md index 8851d145..6f7b363a 100644 --- a/specs/038-repro-false-pass/contracts/hermeticity-scan.md +++ b/specs/038-repro-false-pass/contracts/hermeticity-scan.md @@ -2,6 +2,8 @@ **Feature**: 038 (#430, #432) | **Package**: `darnit-reproducibility` | **Status**: proposed +> **Amended by #553**: Witness attestation verification no longer runs in `repro_hermetic_build`. It is the separate step `repro_witness_attestation`, which can only conclude FAIL; a verified clean network log is no longer a PASS signal. See `specs/044-false-pass-paths/contracts/witness-step-addendum-553.md`. Statements below about a verified Witness attestation describe the behavior at feature 038. + ## Problem `repro_hermetic_build` is careful about its PASS -- it requires a verified Witness attestation with a clean network log, a Nix flake build, or Bazel with a blocking flag, and explicitly rejects a mere mention of `witness run` in CI text. The defect is coverage, not credulity. diff --git a/specs/044-false-pass-paths/contracts/step-contract.md b/specs/044-false-pass-paths/contracts/step-contract.md index cb1b362d..07e68ea1 100644 --- a/specs/044-false-pass-paths/contracts/step-contract.md +++ b/specs/044-false-pass-paths/contracts/step-contract.md @@ -97,3 +97,5 @@ PASS and FAIL still conclude only within the step's effective set. The OSPS-BR-0 ## 5. Reproducibility framework All five reproducibility step types have ceiling `{"fail"}`. A PASS needs a corpus-backed `promotion` (041) declared on the step. + +`repro_witness_attestation` (#553), split out of `repro_hermetic_build`, also has ceiling `{"fail"}`: a verified runtime trace can show network access but not its absence. See [witness-step-addendum-553.md](witness-step-addendum-553.md). diff --git a/specs/044-false-pass-paths/contracts/witness-step-addendum-553.md b/specs/044-false-pass-paths/contracts/witness-step-addendum-553.md new file mode 100644 index 00000000..1fad6986 --- /dev/null +++ b/specs/044-false-pass-paths/contracts/witness-step-addendum-553.md @@ -0,0 +1,100 @@ +# Addendum: `repro_witness_attestation`, a FAIL-only runtime-trace step + +**Amends**: [step-contract.md](step-contract.md) section 5 and FR-010 (reproducibility step ceilings) +**Issue**: #553 +**Authoritative text**: `docs/architecture/framework-design.md` section 3.0.1 (scenario "A runtime trace proves only network access") and section 12 + +## Why + +`repro_hermetic_build` fetched and Sigstore-verified Witness / in-toto attestations from the repository's latest successful CI run on the default branch and treated a verified, empty network log as a PASS strong signal. Three things were wrong with that: + +1. **An empty log is not proof of no network access.** The in-toto runtime-trace v0.1 parsing rules make an absent or null optional list equivalent to an empty one, and what `monitorLog.network` records depends on `monitor.type` and its `tracePolicy`. A monitor that does not trace sockets produces the same empty log as a build that made no connection; the spec's own Tetragon example, with a `connect` policy, has no `network` field at all. +2. **The attestation was not bound to what was audited.** The latest successful run on the default branch need not have built the checked-out commit. +3. **The predicate was matched loosely.** Any statement with a top-level `network` key, or any payload containing `"network"`, was read as a runtime trace. + +Splitting the check into its own step gives it its own ceiling and its own setting, and leaves `repro_hermetic_build` a filesystem-only scan. + +## 1. Registration + +| Field | Value | +|---|---| +| Step type | `repro_witness_attestation` | +| Registered by | `darnit-reproducibility`, in `register_sieve_handlers` | +| Phase | `deterministic` | +| Ceiling | `{fail}` | +| Settings | `verify_witness_attestations` (bool, default `true`) | +| Expression names | none | + +`repro_hermetic_build` keeps ceiling `{fail}` and now declares no settings. Under strict loading (section 3.0.3) a step that sets `verify_witness_attestations` on `repro_hermetic_build`, in shipped TOML or an operator pass override, fails loading. + +## 2. Inputs and binding + +| Input | Source | +|---|---| +| Repository identity | `ctx.owner`, `ctx.repo` | +| Audited commit | `git rev-parse HEAD` in `ctx.local_path` | +| Runs | `gh run list --repo / --commit --status success --limit 5 --json databaseId` | +| Artifacts | `gh run download --repo / --pattern '*witness*'` for each run; `*.json` files, those ending `.att.json`, `.bundle.json`, `.sigstore.json` first, at most 20 | +| Verification | Sigstore bundle, DSSE payload type containing `in-toto`, policy `AllOf([OIDCIssuer("https://token.actions.githubusercontent.com"), GitHubWorkflowRepository("/"), GitHubWorkflowSHA("")])` | + +A file that does not verify under that policy is not read. Unsigned DSSE envelopes, bundles signed by another repository's workflow, and bundles signed by a workflow run for another commit are all ignored. + +## 3. What is read from a verified statement + +| Source | Read | Used as | +|---|---|---| +| Statement with `predicateType` exactly `https://in-toto.io/attestation/runtime-trace/v0.1` | `predicate.monitorLog.network`, `predicate.monitor.type` | Network events; monitor type for evidence | +| Entry of a Witness `https://witness.dev/attestation-collection/v0.1` whose `type` is exactly the runtime-trace URI | `attestation.monitorLog.network`, `attestation.monitor.type` | Same | +| Entry of a Witness collection whose `type` names `command-run` | `processes[].program`, `processes[].cmdline` | Matched against the installer patterns (`curl `, `wget `, `pip install `, ...) | + +Nothing else is read. A top-level `network` key on any other predicate type is ignored. A `network` value that is not a list is treated as unrecognized and decides nothing. + +## 4. Outcomes + +Every verified file is examined; one that records network access decides the step even if an earlier one was clean. + +| Situation | Outcome | Message / evidence | +|---|---|---| +| `verify_witness_attestations = false` | INCONCLUSIVE | Verification is disabled. No `git` or `gh` call is made. | +| `sigstore` not installed | INCONCLUSIVE | Names the `attestation` extra. | +| No `owner`/`repo` | INCONCLUSIVE | Repository identity unavailable. | +| `ctx.local_path` has no commit | INCONCLUSIVE | No commit to bind to. | +| `gh` missing, unauthenticated, timed out, or failed | INCONCLUSIVE | The specific `gh` reason. | +| No successful run for the commit | INCONCLUSIVE | Names the commit. | +| No `*witness*` artifacts | INCONCLUSIVE | Names the runs. | +| Artifacts found, none verified | INCONCLUSIVE | Lists the files checked. | +| Verified runtime trace with a non-empty `monitorLog.network` | **FAIL** | Names the artifact and the event count. | +| Verified command-run process matching an installer pattern | **FAIL** | Names the artifact, the pattern, and the command line. | +| Verified, network log empty or absent, nothing suspicious | INCONCLUSIVE | A clean runtime trace is recorded as evidence but cannot by itself establish that the build had no network access. Evidence names the artifact(s) and monitor type(s). | + +Evidence is under the single key `witness_attestation` (commit, runs, checked files, verified artifacts, monitor types, the deciding artifact) so it cannot collide with `repro_hermetic_build`'s keys in the control's gathered evidence. + +The step never raises. + +## 5. Why missing evidence is INCONCLUSIVE, not ERROR + +An ERROR records a broken measurement and, when no later step concludes, makes the control ERROR (section 3.0.2). This step can only add FAIL evidence. Turning "no `gh` login" or "no attestation published" into ERROR would change an RE-02.01 that would otherwise end WARN (no step concluded) into ERROR, on a repository that never claimed to publish runtime traces, for an optional evidence source. So every missing prerequisite is INCONCLUSIVE with the specific reason, as the check did inside `repro_hermetic_build`. + +## 6. RE-02.01 order + +```toml +[[controls."RE-02.01".passes]] +handler = "repro_witness_attestation" + +[[controls."RE-02.01".passes]] +handler = "repro_hermetic_build" + +[[controls."RE-02.01".passes]] +handler = "manual" +``` + +The attestation step runs first: a verified record of network access concludes FAIL even when the repository also carries a Nix or Bazel strong signal, which `repro_hermetic_build` reports only as evidence. + +## Follow-up (not in this change) + +PASS from a runtime trace needs a built-in allowlist of monitor types (and trace policies) known to record socket activity, each backed by real attestations in the corpus, and a corpus-backed `promotion` on the step (section 5.5). Until then no verified trace concludes PASS. + +## Unchanged + +- The ceilings of the other five reproducibility step types and their promotion rule (step-contract.md section 5). +- `repro_hermetic_build`'s CI-text scan, its Nix (gated on RE-01.02) and Bazel strong signals, and their evidence-only PASS. From 9748a3e7bafb80cf3ba2f12e156a10141fa7d156 Mon Sep 17 00:00:00 2001 From: Michael Lieberman Date: Wed, 7 Oct 2026 17:29:33 -0400 Subject: [PATCH 2/2] fix(reproducibility): split Witness verification out of repro_hermetic_build (#553) New step type repro_witness_attestation (ceiling {fail}) runs first in RE-02.01. It verifies the attestations of the successful CI runs for the audited commit with a Sigstore policy bound to the repository and that commit, reads network events only from the in-toto runtime-trace v0.1 predicate (monitorLog.network), and keeps the Witness command-run installer scan. Recorded network access concludes FAIL; a verified clean trace is evidence only. Missing prerequisites stay INCONCLUSIVE. repro_hermetic_build no longer calls gh or reads attestations, and verify_witness_attestations moves to the new step. Assisted-by: Claude:claude-opus-5-5 Signed-off-by: Michael Lieberman --- CHANGELOG.md | 21 + .../src/darnit_reproducibility/handlers.py | 186 +++-- .../darnit_reproducibility/implementation.py | 17 +- .../reproducibility.toml | 28 +- .../witness_attestation.py | 343 +++++++--- .../sieve/test_handler_registry_metadata.py | 1 + .../test_plugin_handler_registration.py | 1 + tests/darnit_reproducibility/conftest.py | 7 - tests/darnit_reproducibility/test_handlers.py | 255 +++---- .../test_implementation.py | 4 +- .../test_no_pass_from_signals.py | 116 +++- .../test_repro_corpus.py | 83 ++- .../test_witness_attestation.py | 641 ++++++++++++------ 13 files changed, 1127 insertions(+), 576 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 4d77c248..21a1311e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -456,6 +456,27 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 the command, and the start of stderr. Before, `grep` exiting 2 on a missing directory or `git` exiting 128 outside a repository was reported as a network failure (#562). +- **BREAKING:** Witness / in-toto attestation verification moves out of + `repro_hermetic_build` into a new step type, `repro_witness_attestation`, + which runs first in RE-02.01 and can conclude only FAIL. It verifies the + attestations of the successful CI runs for the audited commit (`git + rev-parse HEAD`), with a signing policy bound to the repository and that + commit, instead of the latest run on the default branch. It reads network + events only from a statement whose predicate type is exactly + `https://in-toto.io/attestation/runtime-trace/v0.1` (`monitorLog.network`), + plus the existing Witness `command-run` installer scan; a `network` key on + any other predicate is ignored. A verified attestation that records + network access concludes RE-02.01 FAIL even when the repository also has a + Nix or Bazel strong signal. A verified clean trace is evidence only, no + longer a PASS signal: an empty network log is also what a monitor that + does not trace sockets records. A missing prerequisite (no `gh`, no + login, no run, no attestation, `sigstore` not installed) leaves the step + INCONCLUSIVE with the reason, as before (#553). +- **BREAKING:** the `verify_witness_attestations` step setting moves from + `repro_hermetic_build` to `repro_witness_attestation`. A pass override or + custom control that sets it on `repro_hermetic_build` now fails loading + under strict step settings; set it on the `repro_witness_attestation` step + instead (#553). ### Fixed diff --git a/packages/darnit-reproducibility/src/darnit_reproducibility/handlers.py b/packages/darnit-reproducibility/src/darnit_reproducibility/handlers.py index fd09993f..7dc79b81 100644 --- a/packages/darnit-reproducibility/src/darnit_reproducibility/handlers.py +++ b/packages/darnit-reproducibility/src/darnit_reproducibility/handlers.py @@ -603,27 +603,6 @@ def _iter_container_files(path: Path) -> list[Path]: ) -def _maybe_check_witness_attestation( - ctx: HandlerContext, - config: dict[str, Any], -) -> WitnessCheckResult: - """Call check_witness_attestation() unless disabled via config. - - ``verify_witness_attestations = false`` in the TOML pass config opts out - of the network round-trip entirely — for air-gapped audits or - environments without a usable `gh` login for the audited repo. There is - deliberately no automatic "skip if CI text doesn't mention witness" - heuristic: that would reintroduce exactly the kind of unreliable text-only - guess this real verification replaced (e.g. a reusable/composite workflow - can produce a valid attestation without the calling repo's own CI files - ever spelling out "witness"). - """ - if not config.get("verify_witness_attestations", True): - return WitnessCheckResult(attempted=False, detail="witness attestation verification disabled via config") - - return check_witness_attestation(ctx) - - _NIX_CI_COMMANDS: tuple[str, ...] = ("nix build", "nix develop", "nix run", "nix flake") @@ -658,38 +637,30 @@ def _detect_strong_hermeticity_signal( path: Path, ci_files: list[Path], dependency_results: dict[str, Any], - ctx: HandlerContext, - config: dict[str, Any], -) -> tuple[str | None, WitnessCheckResult]: - """Return (signal description, witness check result). Signal is None if no - build-system-enforced hermeticity guarantee was found. +) -> str | None: + """Return a signal description, or None if no build-system-enforced + hermeticity guarantee was found. + + Witness runtime attestations are not checked here: that is the separate + ``repro_witness_attestation`` step (#553), and a mention of "witness run" + in CI text proves only that the tool ran, not what it observed. Checks in priority order: - 1. Witness runtime attestation — a Sigstore-verified DSSE envelope from the - repo's latest CI run, bound to its GitHub Actions OIDC identity, asserting - no network access occurred during the build (see witness_attestation.py). - Merely mentioning "witness run" in CI text is NOT sufficient — that only - proves the tool ran, not what it observed, so text mentions alone are no - longer treated as a strong signal. - 2. Nix flake used in CI — fixed-output derivations run network-isolated by default. + 1. Nix flake used in CI — fixed-output derivations run network-isolated by default. Gated on RE-01.02 (BuildEnvDeclared) having PASSED: a bare flake.nix that isn't the project's confirmed, declared build environment isn't a strong signal on its own. If RE-01.02 hasn't run (e.g. this control invoked standalone), the signal is withheld rather than assumed — conservative-by-default. - 3. Bazel with explicit network sandbox — Bazel allows network by default, so the + 2. Bazel with explicit network sandbox — Bazel allows network by default, so the blocking flag must be present to count as a strong signal. Checked in both CI files (where "bazel" must also appear, to avoid matching an unrelated tool that happens to share a flag name) and .bazelrc (the canonical place to set it, where the file itself is the bazel signal). Comments are stripped before matching (same as ``_scan_line``) so a - commented-out reference (e.g. ``# TODO: add witness run``) can't be + commented-out reference (e.g. ``# TODO: nix build``) can't be mistaken for the real thing. """ - witness_result = _maybe_check_witness_attestation(ctx, config) - if witness_result.verified and witness_result.network_clean is True: - return f"Witness attestation verified — {witness_result.detail}", witness_result - ci_content: dict[str, str] = {} for f in ci_files: try: @@ -703,7 +674,7 @@ def _detect_strong_hermeticity_signal( name for name, content in ci_content.items() if any(cmd in content for cmd in _NIX_CI_COMMANDS) ) if nix_hits: - return f"Nix flake build in CI ({', '.join(nix_hits)})", witness_result + return f"Nix flake build in CI ({', '.join(nix_hits)})" has_bazel = any((path / f).exists() for f in ("WORKSPACE", "WORKSPACE.bazel", "MODULE.bazel", "BUILD.bazel")) if has_bazel: @@ -724,9 +695,9 @@ def _detect_strong_hermeticity_signal( bazel_hits.append(".bazelrc") if bazel_hits: - return f"Bazel with network sandbox ({', '.join(sorted(bazel_hits))})", witness_result + return f"Bazel with network sandbox ({', '.join(sorted(bazel_hits))})" - return None, witness_result + return None def repro_hermetic_build_handler( @@ -742,27 +713,15 @@ def repro_hermetic_build_handler( inside Dockerfiles are DEFERRED — building the image environment is fine; fetching application dependencies at build time is not. - v0.3: Adds a Sigstore-verified Witness/runtime-trace attestation check - (see witness_attestation.py) — fetches attestation artifacts from the - repo's latest successful CI run via the `gh` CLI, cryptographically - verifies them against the repo's GitHub Actions OIDC identity, and only - treats an empty, verified network log as a strong PASS signal. A verified - attestation that *does* record network activity is fed into the violation - list — stronger evidence than the CI-text grep below it. Requires the - `gh` CLI and `darnit-core[attestation]`; any missing prerequisite (no gh, - not authenticated, no matching CI run/artifact, sigstore not installed, - verification failure) degrades to a specific "no attestation evidence" - reason in ``evidence["strong_signal"]``/the witness result's ``detail``, - never to failing the audit. Set ``verify_witness_attestations = false`` in - the TOML pass config to skip the network round-trip entirely (air-gapped - audits, or repos with no usable `gh` login). + Reads only the local checkout: Witness runtime attestations are checked by + the separate ``repro_witness_attestation`` step (#553), which runs before + this one in RE-02.01. Result semantics (conservative-by-default): - - PASS: strong hermeticity signal (verified Witness attestation, - Nix flake CI, Bazel sandbox). Evidence only: this step - type's ceiling is `{fail}` (feature 044, FR-010) - - FAIL: suspicious live network-fetch pattern in any scanned file, - or a verified Witness attestation that recorded network activity + - PASS: strong hermeticity signal (Nix flake CI, Bazel sandbox). + Evidence only: this step type's ceiling is `{fail}` + (feature 044, FR-010) + - FAIL: suspicious live network-fetch pattern in any scanned file - INCONCLUSIVE: files scanned, no violations, no strong signal (grep absence ≠ proof of hermeticity) - INCONCLUSIVE @@ -792,9 +751,7 @@ def repro_hermetic_build_handler( }, ) - strong_signal, witness_result = _detect_strong_hermeticity_signal( - path, all_ci_files, ctx.dependency_results, ctx, config - ) + strong_signal = _detect_strong_hermeticity_signal(path, all_ci_files, ctx.dependency_results) if strong_signal: return HandlerResult( status=HandlerResultStatus.PASS, @@ -814,14 +771,6 @@ def repro_hermetic_build_handler( nondeterministic: list[str] = [] files_scanned: list[str] = [] - # A verified Witness attestation that positively recorded network activity - # is stronger evidence than the grep heuristic below — surface it as a - # violation on its own rather than waiting for a matching CI-text pattern. - if witness_result.verified and witness_result.network_clean is False: - violations.append( - f"witness attestation ({witness_result.evidence.get('artifact', '?')}): {witness_result.detail}" - ) - for f in all_files: is_dockerfile = f in container_file_set try: @@ -895,7 +844,7 @@ def repro_hermetic_build_handler( message=( f"No suspicious patterns found in {len(files_scanned)} scanned file(s) — " "grep absence alone cannot confirm hermeticity; " - "a strong signal (Witness, Nix, Bazel sandbox) or manual review is needed." + "a strong signal (Nix flake build, Bazel network sandbox) or manual review is needed." f"{nix_note}" ), confidence=0.4, @@ -903,6 +852,97 @@ def repro_hermetic_build_handler( ) +def repro_witness_attestation_handler( + config: dict[str, Any], + ctx: HandlerContext, +) -> HandlerResult: + """Check verified Witness / in-toto runtime-trace attestations for network access (#553). + + Fetches the attestation artifacts of the successful CI runs for the audited + commit, verifies each against this repository's GitHub Actions identity and + that commit (see witness_attestation.py), and reads only the runtime-trace + predicate's ``monitorLog.network`` and Witness command-run process command + lines. + + Registered with the ceiling ``{fail}``: a verified trace can show that the + build accessed the network, but an empty or absent network log is also what + a monitor that does not trace sockets records, so it cannot show the + opposite. + + ``verify_witness_attestations = false`` skips the network round-trip (air- + gapped audits, or no usable `gh` login). There is deliberately no "skip if + CI text doesn't mention witness" heuristic: a reusable or composite workflow + can produce a valid attestation without the calling repo's own CI files + ever spelling out "witness". + + Result semantics: + - FAIL: a verified attestation recorded network events, or a + verified command-run process matched an installer pattern + - INCONCLUSIVE: everything else, including a verified clean trace (evidence + only) and every missing prerequisite (disabled, no sigstore, + no repository identity or commit, no gh or auth, no run, + artifact, or verifiable bundle). Never ERROR: this step can + only add FAIL evidence, so a missing optional source must + not turn the control's WARN into ERROR. + """ + if not config.get("verify_witness_attestations", True): + return HandlerResult( + status=HandlerResultStatus.INCONCLUSIVE, + message="Witness attestation verification is disabled (verify_witness_attestations = false)", + confidence=0.0, + evidence={"witness_attestation": {"verified": False, "detail": "verification disabled via config"}}, + ) + + try: + result = check_witness_attestation(ctx) + except Exception as exc: # every known failure degrades inside the check; an unknown one must not fail the audit + logger.warning("witness attestation check failed unexpectedly: %s", exc) + result = WitnessCheckResult(attempted=True, detail=f"attestation check failed unexpectedly: {exc}") + + evidence = { + "witness_attestation": { + "verified": result.verified, + "network_recorded": result.network_recorded, + "detail": result.detail, + **result.evidence, + } + } + artifact = result.evidence.get("artifact", "?") + commit = str(result.evidence.get("commit", ""))[:12] or "?" + + if result.verified and result.network_recorded: + return HandlerResult( + status=HandlerResultStatus.FAIL, + message=( + f"Verified Witness attestation ({artifact}) shows network access during the build of " + f"commit {commit}: {result.detail}" + ), + confidence=0.9, + evidence=evidence, + ) + + if result.verified: + monitors = ", ".join(result.evidence.get("monitor_types") or []) or "unknown" + return HandlerResult( + status=HandlerResultStatus.INCONCLUSIVE, + message=( + f"Verified Witness attestation ({artifact}, monitor: {monitors}) for commit {commit}: " + f"{result.detail}. A clean runtime trace is recorded as evidence but cannot by itself " + "establish that the build had no network access: what a monitor records depends on its " + "type and trace policy." + ), + confidence=0.4, + evidence=evidence, + ) + + return HandlerResult( + status=HandlerResultStatus.INCONCLUSIVE, + message=f"No verified Witness attestation evidence: {result.detail}", + confidence=0.0, + evidence=evidence, + ) + + def repro_provenance_exists_handler( config: dict[str, Any], ctx: HandlerContext, diff --git a/packages/darnit-reproducibility/src/darnit_reproducibility/implementation.py b/packages/darnit-reproducibility/src/darnit_reproducibility/implementation.py index 717b9560..2a50a2af 100644 --- a/packages/darnit-reproducibility/src/darnit_reproducibility/implementation.py +++ b/packages/darnit-reproducibility/src/darnit_reproducibility/implementation.py @@ -113,7 +113,7 @@ def register_sieve_handlers(self) -> None: registry = get_sieve_handler_registry() registry.set_plugin_context(self.name) - # Feature 044 (FR-010): all five decide from text and file-presence + # Feature 044 (FR-010): these five decide from text and file-presence # signals, which can show a requirement is unmet but not that it is # met. They register {fail}: a PASS they report is evidence for the # control's later steps, and concludes only with a corpus-backed @@ -142,6 +142,21 @@ def register_sieve_handlers(self) -> None: handler_fn=handlers.repro_hermetic_build_handler, description="Scan CI workflows for live network fetches during build", ceiling={"fail"}, + settings=frozenset(), + ) + # #553: a verified runtime trace can show the build accessed the + # network, but an empty or absent network log is also what a monitor + # that does not trace sockets records, so it cannot show the opposite. + # {fail} until monitor types are allowlisted against real attestations. + registry.register( + "repro_witness_attestation", + phase="deterministic", + handler_fn=handlers.repro_witness_attestation_handler, + description=( + "Verify the audited commit's Witness / in-toto runtime-trace attestations " + "and fail on recorded network access" + ), + ceiling={"fail"}, settings={"verify_witness_attestations"}, ) registry.register( diff --git a/packages/darnit-reproducibility/src/darnit_reproducibility/reproducibility.toml b/packages/darnit-reproducibility/src/darnit_reproducibility/reproducibility.toml index a3fb0435..2b1b8ecf 100644 --- a/packages/darnit-reproducibility/src/darnit_reproducibility/reproducibility.toml +++ b/packages/darnit-reproducibility/src/darnit_reproducibility/reproducibility.toml @@ -92,9 +92,11 @@ steps = [ # Dockerfiles/Containerfiles. Strips comments before pattern matching to cut FPs. # System-package installs (apt-get, apk, yum) inside Dockerfiles are DEFERRED — # building the image environment is acceptable; app-dep fetches at build time are not. -# Strong signals (Witness runtime attestation, Nix flake CI, Bazel network sandbox) -# short-circuit the scan with a handler PASS. A clean grep alone returns -# INCONCLUSIVE — grep absence is not proof of hermeticity. Roadmap: +# Strong signals (Nix flake CI, Bazel network sandbox) short-circuit the scan +# with a handler PASS, which is evidence only (step ceiling {fail}). A clean +# grep alone returns INCONCLUSIVE — grep absence is not proof of hermeticity. +# Witness / in-toto runtime-trace attestations are a separate step that runs +# first and can only FAIL (#553). Roadmap: # https://github.com/kusari-oss/darnit/issues/227 [controls."RE-02.01"] name = "HermeticBuild" @@ -102,11 +104,19 @@ description = "Build does not fetch dependencies at build time" tags = { level = 2, domain = "RE" } depends_on = ["RE-01.02"] # Nix flake strong signal is gated on BuildEnvDeclared having confirmed it -# verify_witness_attestations (bool, default true): fetches and Sigstore-verifies -# Witness/runtime-trace attestation artifacts from the repo's latest CI run via -# `gh`. Set to false to skip that network round-trip entirely (air-gapped -# audits, or repos with no usable `gh` login) — falls back to the CI-text/Nix/ -# Bazel signals below it. See witness_attestation.py for details. +# repro_witness_attestation fetches the Witness / in-toto attestation artifacts +# of the successful CI runs for the audited commit via `gh`, Sigstore-verifies +# them against this repository's GitHub Actions identity and that commit, and +# FAILs when a verified runtime trace records network events (or a verified +# command-run process line matches an installer). A verified clean trace is +# evidence only: an empty network log is also what a monitor that does not +# trace sockets records (#553). +# verify_witness_attestations (bool, default true): set to false to skip the +# network round-trip entirely (air-gapped audits, or repos with no usable `gh` +# login). See witness_attestation.py for details. +[[controls."RE-02.01".passes]] +handler = "repro_witness_attestation" + [[controls."RE-02.01".passes]] handler = "repro_hermetic_build" @@ -117,7 +127,7 @@ steps = [ "Review Makefiles, build scripts (scripts/build*, scripts/install*), and Dockerfiles for the same", "Check for curl, wget, pip install (without --no-index), npm install (without --ci), brew install in build steps", "Verify all dependencies come from the lock file, not fetched live at build time", - "For a PASS without a strong signal, confirm via Witness attestation, Nix flake build, or Bazel network sandbox", + "For a PASS, confirm via a Nix flake build, a Bazel network sandbox, or a runtime trace whose monitor is known to record network access (an empty network log alone does not confirm it)", ] # ============================================================================= diff --git a/packages/darnit-reproducibility/src/darnit_reproducibility/witness_attestation.py b/packages/darnit-reproducibility/src/darnit_reproducibility/witness_attestation.py index 13016aef..d9d1797a 100644 --- a/packages/darnit-reproducibility/src/darnit_reproducibility/witness_attestation.py +++ b/packages/darnit-reproducibility/src/darnit_reproducibility/witness_attestation.py @@ -1,37 +1,36 @@ -"""Verified Witness/in-toto runtime attestation check for RE-02.01. +"""Verified Witness / in-toto runtime-trace attestation check (``repro_witness_attestation``). Unlike every other check in this plugin (pure local filesystem inspection), -this module reaches out to GitHub to fetch the latest CI run's attestation -artifacts and cryptographically verifies them before trusting anything they -claim. A JSON file that merely *says* "no network access" is not evidence — -only a Sigstore-verified DSSE envelope bound to the repo's GitHub Actions -OIDC identity is. - -Two predicate shapes are recognized: - -- Witness's own ``attestation-collection/v0.1``, whose nested ``command-run`` - attestation records ``processes[].cmdline``/``program`` (and opened file - digests) but has **no dedicated network field** — Witness's built-in tracer - does not observe sockets. For this shape we can only fall back to scanning - process command lines for the same suspicious substrings used by the grep - heuristic in ``handlers.py``, which is not an authoritative "no network - access" claim, only a negative-evidence hint. -- The newer, monitor-agnostic ``runtime-trace/v0.1`` predicate, which *does* - define a top-level ``network`` array. An empty array is treated as an - authoritative "no network access" claim; a non-empty one is authoritative - evidence of network access. (The spec still defers the internal shape of - each event to ``monitor.type``, so we only rely on emptiness, not content.) - -Every failure mode here (no ``gh`` CLI, no auth, no matching run/artifact, -``sigstore`` not installed, verification failure, unrecognized predicate) -degrades to "no attestation evidence" rather than raising — this must never -be able to fail an audit outright. +this module reaches out to GitHub for the attestation artifacts of the CI runs +that built the audited commit, and cryptographically verifies them before +reading anything they claim. A JSON file that merely *says* "no network +access" is not evidence -- only a Sigstore-verified DSSE envelope signed by a +GitHub Actions workflow of this repository, running on the audited commit, is. + +What is read from a verified statement (#553): + +- ``runtime-trace/v0.1``, as the statement's predicate type or as an entry of + a Witness ``attestation-collection/v0.1`` whose ``type`` is exactly that URI: + ``monitorLog.network`` only. A non-empty list shows network access. An empty + or absent list does NOT show its absence: under the in-toto parsing rules an + absent optional list equals an empty one, and what the log records depends + on ``monitor.type`` and its ``tracePolicy``, so a monitor that does not trace + sockets records the same empty log. +- Witness ``command-run``: process ``program``/``cmdline`` scanned for the + installer substrings also used by the CI-file scan in ``handlers.py``. A + match shows network access; no match shows nothing. + +So this check can produce FAIL evidence and nothing else. Every failure mode +(no ``gh`` CLI, no auth, no commit, no matching run or artifact, ``sigstore`` +not installed, verification failure, unrecognized predicate) degrades to "no +attestation evidence" with the specific reason rather than raising. """ from __future__ import annotations import base64 import json +import re import subprocess import tempfile from dataclasses import dataclass, field @@ -46,7 +45,7 @@ try: from sigstore.models import Bundle from sigstore.verify import Verifier - from sigstore.verify.policy import AllOf, GitHubWorkflowRepository, OIDCIssuer + from sigstore.verify.policy import AllOf, GitHubWorkflowRepository, GitHubWorkflowSHA, OIDCIssuer SIGSTORE_VERIFY_AVAILABLE = True except ImportError: @@ -71,8 +70,12 @@ ) _GH_TIMEOUT_SECONDS = 60 +_GIT_TIMEOUT_SECONDS = 15 +_MAX_RUNS = 5 _MAX_ARTIFACT_FILES = 20 +_COMMIT_SHA = re.compile(r"[0-9a-f]{40}(?:[0-9a-f]{24})?") + # Filenames ending in these suffixes are far more likely to be the actual # attestation bundle than an incidental *.json artifact (e.g. a build log # dumped as json) — check them first so the cap above can't skip past the @@ -94,11 +97,18 @@ @dataclass class WitnessCheckResult: - """Outcome of attempting to verify a Witness/runtime-trace attestation.""" + """Outcome of attempting to verify Witness/runtime-trace attestations. + + ``network_recorded`` is True only when a verified attestation recorded + network access during the build. False means a verified runtime trace + recorded no network events, which is not proof that none happened (see the + module docstring). None means no verified runtime trace and no suspicious + command line was found. + """ attempted: bool verified: bool = False - network_clean: bool | None = None # True/False = authoritative; None = no authoritative signal + network_recorded: bool | None = None detail: str = "" evidence: dict[str, Any] = field(default_factory=dict) @@ -133,42 +143,82 @@ def _run_gh(args: list[str]) -> _GhOutcome: return _GhOutcome(None, f"gh exited {proc.returncode}: {proc.stderr.strip()[:200]}") -def _latest_successful_run_id(owner: str, repo: str, branch: str) -> tuple[str | None, str | None]: - """Returns (run_id, failure_reason) — exactly one is None.""" +def _head_commit(local_path: str) -> tuple[str | None, str | None]: + """The audited commit: ``git rev-parse HEAD`` in the audited checkout. + + Returns (sha, failure_reason) -- exactly one is None. + """ + if not local_path or not Path(local_path).is_dir(): + return None, "audited path is not a directory, so there is no commit to bind attestations to" + try: + proc = subprocess.run( + ["git", "rev-parse", "HEAD"], + cwd=local_path, + capture_output=True, + text=True, + timeout=_GIT_TIMEOUT_SECONDS, + ) + except FileNotFoundError: + return None, "git not found in PATH, so the audited commit is unknown" + except subprocess.TimeoutExpired: + return None, f"git rev-parse HEAD timed out after {_GIT_TIMEOUT_SECONDS}s" + except OSError as exc: + return None, f"git rev-parse HEAD could not run: {exc}" + sha = proc.stdout.strip() + if proc.returncode != 0 or not _COMMIT_SHA.fullmatch(sha): + return ( + None, + "audited path has no commit (git rev-parse HEAD failed), so there is nothing to bind attestations to", + ) + return sha, None + + +def _successful_run_ids(owner: str, repo: str, sha: str) -> tuple[list[str], str | None]: + """Successful CI runs for the audited commit. + + Returns (run_ids, failure_reason) -- run_ids is empty iff failure_reason is set. + """ outcome = _run_gh( [ "run", "list", "--repo", f"{owner}/{repo}", - "--branch", - branch, + "--commit", + sha, "--status", "success", "--limit", - "1", + str(_MAX_RUNS), "--json", "databaseId", ] ) if outcome.proc is None: - return None, outcome.reason + return [], outcome.reason if not outcome.proc.stdout: - return None, "gh returned no output for the run list query" + return [], "gh returned no output for the run list query" try: rows = json.loads(outcome.proc.stdout) except json.JSONDecodeError: - return None, "gh returned unparseable output for the run list query" - if not rows: - return None, f"no successful CI run found on branch '{branch}'" - run_id = rows[0].get("databaseId") - if not run_id: - return None, "latest successful run has no databaseId" - return str(run_id), None + return [], "gh returned unparseable output for the run list query" + if not isinstance(rows, list) or not rows: + return [], f"no successful CI run found for commit {sha[:12]}" + run_ids = [str(row["databaseId"]) for row in rows if isinstance(row, dict) and row.get("databaseId")] + if not run_ids: + return [], f"successful runs for commit {sha[:12]} have no databaseId" + return run_ids, None + + +def _order_candidates(files: list[Path]) -> list[Path]: + priority = [f for f in files if f.name.endswith(_PRIORITY_ARTIFACT_SUFFIXES)] + rest = [f for f in files if f not in priority] + return (priority + rest)[:_MAX_ARTIFACT_FILES] def _download_candidate_artifacts(owner: str, repo: str, run_id: str, dest: Path) -> tuple[list[Path], str | None]: - """Returns (files, failure_reason) — files is empty iff failure_reason is set.""" + """Returns (files, failure_reason) -- files is empty iff failure_reason is set.""" + dest.mkdir(parents=True, exist_ok=True) outcome = _run_gh( [ "run", @@ -184,32 +234,47 @@ def _download_candidate_artifacts(owner: str, repo: str, run_id: str, dest: Path ) if outcome.proc is None: return [], outcome.reason - all_files = sorted(dest.rglob("*.json")) - priority = [f for f in all_files if f.name.endswith(_PRIORITY_ARTIFACT_SUFFIXES)] - rest = [f for f in all_files if f not in priority] - found = (priority + rest)[:_MAX_ARTIFACT_FILES] + found = _order_candidates(sorted(dest.rglob("*.json"))) if not found: return [], f"run {run_id} has no artifacts matching '*witness*'" return found, None -def _fetch_candidate_files(ctx: HandlerContext, scratch_dir: Path) -> tuple[list[Path], str | None]: - """Best-effort fetch of Witness attestation artifacts from the latest CI run. +def _fetch_candidate_files( + owner: str, repo: str, run_ids: list[str], scratch_dir: Path +) -> tuple[list[Path], str | None]: + """Attestation artifacts of the given runs, priority files first, capped. - Returns (files, failure_reason) — files is empty iff failure_reason is set. + Returns (files, failure_reason) -- files is empty iff failure_reason is set. """ - if not ctx.owner or not ctx.repo: - return [], "repository owner/name not available in this context" - run_id, reason = _latest_successful_run_id(ctx.owner, ctx.repo, ctx.default_branch) - if not run_id: - return [], reason - return _download_candidate_artifacts(ctx.owner, ctx.repo, run_id, scratch_dir) + files: list[Path] = [] + reasons: list[str] = [] + for run_id in run_ids: + found, reason = _download_candidate_artifacts(owner, repo, run_id, scratch_dir / run_id) + files.extend(found) + if reason and reason not in reasons: + reasons.append(reason) + candidates = _order_candidates(files) + if not candidates: + return [], "; ".join(reasons) or "no Witness attestation artifacts found" + return candidates, None + + +def _verification_policy(owner: str, repo: str, sha: str) -> Any: + """Signed by a GitHub Actions workflow of this repository, running on the audited commit.""" + return AllOf( + [ + OIDCIssuer(GITHUB_ACTIONS_OIDC_ISSUER), + GitHubWorkflowRepository(f"{owner}/{repo}"), + GitHubWorkflowSHA(sha), + ] + ) -def _verify_bundle(raw_bytes: bytes, owner: str, repo: str) -> dict[str, Any] | None: - """Verify a Sigstore-bundled DSSE envelope against the repo's GitHub - Actions OIDC identity. Returns the decoded in-toto statement on success, - or None if verification is unavailable or fails. +def _verify_bundle(raw_bytes: bytes, owner: str, repo: str, sha: str) -> dict[str, Any] | None: + """Verify a Sigstore-bundled DSSE envelope against the repository's GitHub + Actions OIDC identity for the audited commit. Returns the decoded in-toto + statement on success, or None if verification is unavailable or fails. """ if not SIGSTORE_VERIFY_AVAILABLE: return None @@ -219,15 +284,8 @@ def _verify_bundle(raw_bytes: bytes, owner: str, repo: str) -> dict[str, Any] | logger.debug("not a Sigstore bundle: %s", exc) return None - policy = AllOf( - [ - OIDCIssuer(GITHUB_ACTIONS_OIDC_ISSUER), - GitHubWorkflowRepository(f"{owner}/{repo}"), - ] - ) - try: - payload_type, payload_bytes = Verifier.production().verify_dsse(bundle, policy) + payload_type, payload_bytes = Verifier.production().verify_dsse(bundle, _verification_policy(owner, repo, sha)) except Exception as exc: logger.debug("Sigstore verification failed: %s", exc) return None @@ -235,9 +293,10 @@ def _verify_bundle(raw_bytes: bytes, owner: str, repo: str) -> dict[str, Any] | if "in-toto" not in payload_type: return None try: - return json.loads(payload_bytes) + statement = json.loads(payload_bytes) except json.JSONDecodeError: return None + return statement if isinstance(statement, dict) else None def _decode_raw_dsse(raw_bytes: bytes) -> dict[str, Any] | None: @@ -256,62 +315,115 @@ def _decode_raw_dsse(raw_bytes: bytes) -> dict[str, Any] | None: def _nested_attestations(statement: dict[str, Any]) -> list[dict[str, Any]]: - predicate = statement.get("predicate", {}) + predicate = statement.get("predicate") + if not isinstance(predicate, dict): + return [] if statement.get("predicateType") == _WITNESS_COLLECTION_TYPE: - return predicate.get("attestations", []) + entries = predicate.get("attestations") + return [e for e in entries if isinstance(e, dict)] if isinstance(entries, list) else [] return [{"type": statement.get("predicateType", ""), "attestation": predicate}] -def _check_network_cleanliness(statement: dict[str, Any]) -> tuple[bool | None, str]: - """Inspect a verified in-toto statement for network-access evidence. +def _monitor_type(trace: dict[str, Any]) -> str | None: + monitor = trace.get("monitor") + if isinstance(monitor, dict) and isinstance(monitor.get("type"), str): + return monitor["type"] + return None + + +def _check_network_evidence(statement: dict[str, Any]) -> tuple[bool | None, str, list[str]]: + """Inspect a verified in-toto statement for evidence of network access. + + Returns ``(network_recorded, detail, monitor_types)``: + - ``True`` -- a runtime-trace ``monitorLog.network`` list is non-empty, or + a command-run process command line matched a suspicious pattern. + - ``False`` -- a runtime trace was present and recorded no network events. + Not proof of no network access (see the module docstring). + - ``None`` -- no runtime trace and nothing suspicious. - Returns ``(network_clean, detail)``: - - ``(True, ...)`` — authoritative: a runtime-trace ``network`` array was - present and empty. - - ``(False, ...)`` — authoritative: a non-empty ``network`` array, or a - command-run process cmdline matched a suspicious pattern. - - ``(None, ...)`` — no authoritative signal (command-run only, nothing - suspicious found — absence of evidence, not evidence of absence). + Only a predicate whose type is exactly the runtime-trace URI is read for + network events; a ``network`` key under any other predicate type is ignored. """ + monitor_types: list[str] = [] + traced_without_network = False for entry in _nested_attestations(statement): entry_type = entry.get("type", "") - payload = entry.get("attestation", {}) - - if "runtime-trace" in entry_type or "network" in payload: - network_events = payload.get("monitorLog", {}).get("network", payload.get("network")) - if network_events is not None: - if len(network_events) == 0: - return True, "runtime-trace predicate recorded an empty network log" - return False, f"runtime-trace predicate recorded {len(network_events)} network event(s)" + payload = entry.get("attestation") + if not isinstance(entry_type, str) or not isinstance(payload, dict): + continue + + if entry_type == _RUNTIME_TRACE_TYPE: + monitor_type = _monitor_type(payload) + if monitor_type: + monitor_types.append(monitor_type) + monitor_log = payload.get("monitorLog") + network_events = monitor_log.get("network") if isinstance(monitor_log, dict) else None + if network_events is None or network_events == []: + traced_without_network = True + elif isinstance(network_events, list): + return ( + True, + f"runtime-trace predicate recorded {len(network_events)} network event(s) " + f"(monitor: {monitor_type or 'unknown'})", + monitor_types, + ) + continue if "command-run" in entry_type or "commandrun" in entry_type: - for proc in payload.get("processes", []) or []: + processes = payload.get("processes") + for proc in processes if isinstance(processes, list) else []: + if not isinstance(proc, dict): + continue haystack = f"{proc.get('program', '')} {proc.get('cmdline', '')}" for pattern in _SUSPICIOUS_CMDLINE_PATTERNS: if pattern in haystack: - return False, f"command-run process matched '{pattern.strip()}': {haystack.strip()[:120]}" + return ( + True, + f"command-run process matched '{pattern.strip()}': {haystack.strip()[:120]}", + monitor_types, + ) - return None, "no authoritative network signal in verified attestation" + if traced_without_network: + return False, "runtime-trace predicate recorded no network events", monitor_types + return None, "no runtime-trace network log or suspicious command line in verified attestation", monitor_types def check_witness_attestation(ctx: HandlerContext) -> WitnessCheckResult: - """Fetch, verify, and inspect the latest CI run's Witness attestation. + """Fetch, verify, and inspect the attestations of the CI runs that built the audited commit. - Returns a result with ``verified=False`` (and no PASS-worthy signal) for - any missing prerequisite. Never raises. + Returns a result with ``verified=False`` and the specific reason for any + missing prerequisite. Never raises for a missing prerequisite. """ if not SIGSTORE_VERIFY_AVAILABLE: return WitnessCheckResult( attempted=False, detail="sigstore not installed — install darnit-core[attestation] to enable", ) + if not ctx.owner or not ctx.repo: + return WitnessCheckResult(attempted=False, detail="repository owner/name not available in this context") + + sha, reason = _head_commit(ctx.local_path) + if sha is None: + return WitnessCheckResult(attempted=False, detail=reason or "audited commit unknown") + evidence: dict[str, Any] = {"commit": sha} + + run_ids, reason = _successful_run_ids(ctx.owner, ctx.repo, sha) + if not run_ids: + return WitnessCheckResult(attempted=True, detail=reason or "no successful CI run found", evidence=evidence) + evidence["runs"] = run_ids with tempfile.TemporaryDirectory(prefix="darnit-witness-") as tmp: - candidates, reason = _fetch_candidate_files(ctx, Path(tmp)) + candidates, reason = _fetch_candidate_files(ctx.owner, ctx.repo, run_ids, Path(tmp)) if not candidates: - return WitnessCheckResult(attempted=True, detail=reason or "no Witness attestation artifacts found") + return WitnessCheckResult( + attempted=True, detail=reason or "no Witness attestation artifacts found", evidence=evidence + ) checked_files: list[str] = [] + verified_artifacts: list[str] = [] + monitor_types: list[str] = [] + traced_without_network = False + evidence.update(checked_files=checked_files, verified_artifacts=verified_artifacts, monitor_types=monitor_types) for f in candidates: checked_files.append(f.name) try: @@ -320,21 +432,42 @@ def check_witness_attestation(ctx: HandlerContext) -> WitnessCheckResult: logger.debug("could not read %s: %s", f, exc) continue - statement = _verify_bundle(raw_bytes, ctx.owner, ctx.repo) + statement = _verify_bundle(raw_bytes, ctx.owner, ctx.repo, sha) if statement is None: continue # not verifiable — do not fall back to trusting unsigned content - network_clean, detail = _check_network_cleanliness(statement) + verified_artifacts.append(f.name) + network_recorded, detail, types = _check_network_evidence(statement) + monitor_types.extend(t for t in types if t not in monitor_types) + if network_recorded: + evidence["artifact"] = f.name + return WitnessCheckResult( + attempted=True, verified=True, network_recorded=True, detail=detail, evidence=evidence + ) + traced_without_network = traced_without_network or network_recorded is False + + if not verified_artifacts: return WitnessCheckResult( attempted=True, - verified=True, - network_clean=network_clean, - detail=detail, - evidence={"artifact": f.name, "checked_files": checked_files}, + detail=( + "attestation artifact(s) found but none verified against the repo's GitHub Actions " + f"identity for commit {sha[:12]}" + ), + evidence=evidence, ) + evidence["artifact"] = verified_artifacts[0] + if traced_without_network: + return WitnessCheckResult( + attempted=True, + verified=True, + network_recorded=False, + detail="runtime-trace predicate recorded no network events", + evidence=evidence, + ) return WitnessCheckResult( attempted=True, - detail="attestation artifact(s) found but none verified against the repo's GitHub Actions identity", - evidence={"checked_files": checked_files}, + verified=True, + detail="no runtime-trace network log or suspicious command line in verified attestation", + evidence=evidence, ) diff --git a/tests/darnit/sieve/test_handler_registry_metadata.py b/tests/darnit/sieve/test_handler_registry_metadata.py index a279a0f1..daed47a4 100644 --- a/tests/darnit/sieve/test_handler_registry_metadata.py +++ b/tests/darnit/sieve/test_handler_registry_metadata.py @@ -43,6 +43,7 @@ "repro_deps_pinned", "repro_build_env_declared", "repro_hermetic_build", + "repro_witness_attestation", "repro_provenance_exists", "repro_bit_for_bit", "csl_llm_if_present", diff --git a/tests/darnit/test_plugin_handler_registration.py b/tests/darnit/test_plugin_handler_registration.py index e91f4122..3d28482b 100644 --- a/tests/darnit/test_plugin_handler_registration.py +++ b/tests/darnit/test_plugin_handler_registration.py @@ -28,6 +28,7 @@ "repro_deps_pinned", "repro_build_env_declared", "repro_hermetic_build", + "repro_witness_attestation", "repro_provenance_exists", "repro_bit_for_bit", ) diff --git a/tests/darnit_reproducibility/conftest.py b/tests/darnit_reproducibility/conftest.py index d3ff9e5b..9a5308ba 100644 --- a/tests/darnit_reproducibility/conftest.py +++ b/tests/darnit_reproducibility/conftest.py @@ -78,10 +78,3 @@ def repro_ctx(repo: Path, dependency_results: dict[str, str] | None = None) -> H shared_cache={}, dependency_results=dependency_results or {}, ) - - -# Feature 038: keeps the corpus test off the network. Tests added to the classes -# in test_handlers.py do NOT need this -- that module has an autouse -# `_stub_witness_attestation` fixture (line 33) which already isolates them. -# Two mechanisms for one concern would be worse than one. -OFFLINE_CONFIG: dict[str, object] = {"verify_witness_attestations": False} diff --git a/tests/darnit_reproducibility/test_handlers.py b/tests/darnit_reproducibility/test_handlers.py index c919856d..89606844 100644 --- a/tests/darnit_reproducibility/test_handlers.py +++ b/tests/darnit_reproducibility/test_handlers.py @@ -15,7 +15,6 @@ _iter_container_files, _iter_other_ci_files, _iter_workflow_files, - _maybe_check_witness_attestation, _scan_line, _strip_comment, repro_bit_for_bit_handler, @@ -23,14 +22,14 @@ repro_deps_pinned_handler, repro_hermetic_build_handler, repro_provenance_exists_handler, + repro_witness_attestation_handler, ) from darnit_reproducibility.witness_attestation import WitnessCheckResult from darnit.sieve.handler_registry import HandlerContext, HandlerResultStatus -# _detect_strong_hermeticity_signal and repro_hermetic_build_handler both call -# check_witness_attestation(), which shells out to `gh` and the network. Tests -# that aren't specifically exercising that path stub it out to a no-op result; +# repro_witness_attestation_handler calls check_witness_attestation(), which +# shells out to `gh` and the network. It is stubbed to a no-op result here; # witness-specific tests override it again with monkeypatch.setattr. NO_WITNESS_EVIDENCE = WitnessCheckResult(attempted=False) @@ -245,72 +244,25 @@ def test_container_files_none_present(self, tmp_path: Path) -> None: class TestDetectStrongSignal: - """Unit tests for _detect_strong_hermeticity_signal. - - ``check_witness_attestation`` is stubbed to a no-op by the module-level - ``_stub_witness_attestation`` fixture unless a test overrides it below. - """ + """Unit tests for _detect_strong_hermeticity_signal.""" def test_returns_none_with_no_signals(self, tmp_path: Path) -> None: - signal, _ = _detect_strong_hermeticity_signal(tmp_path, [], {}, make_ctx(tmp_path), {}) + signal = _detect_strong_hermeticity_signal(tmp_path, [], {}) assert signal is None def test_witness_mention_alone_is_not_a_signal(self, tmp_path: Path) -> None: # Merely mentioning "witness run" in CI text proves the tool ran, not - # what it observed — only a verified attestation with a clean network - # log counts now (see test_verified_witness_attestation_is_a_signal). + # what it observed. Attestations are the repro_witness_attestation + # step's concern (#553), and even a verified clean one is not a signal. wf = tmp_path / "ci.yml" wf.write_text("- run: witness run -- make build\n") - signal, witness_result = _detect_strong_hermeticity_signal(tmp_path, [wf], {}, make_ctx(tmp_path), {}) - assert signal is None - assert witness_result.verified is False - - def test_verified_witness_attestation_is_a_signal(self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: - verified = WitnessCheckResult( - attempted=True, - verified=True, - network_clean=True, - detail="runtime-trace predicate recorded an empty network log", - ) - monkeypatch.setattr("darnit_reproducibility.handlers.check_witness_attestation", lambda ctx: verified) - signal, witness_result = _detect_strong_hermeticity_signal(tmp_path, [], {}, make_ctx(tmp_path), {}) - assert signal is not None - assert "Witness" in signal - assert witness_result is verified - - def test_verified_witness_attestation_with_network_activity_is_not_a_pass_signal( - self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch - ) -> None: - dirty = WitnessCheckResult( - attempted=True, - verified=True, - network_clean=False, - detail="runtime-trace predicate recorded 2 network event(s)", - ) - monkeypatch.setattr("darnit_reproducibility.handlers.check_witness_attestation", lambda ctx: dirty) - signal, witness_result = _detect_strong_hermeticity_signal(tmp_path, [], {}, make_ctx(tmp_path), {}) - assert signal is None - assert witness_result.network_clean is False - - def test_witness_check_disabled_via_config(self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: - verified = WitnessCheckResult(attempted=True, verified=True, network_clean=True, detail="clean") - monkeypatch.setattr("darnit_reproducibility.handlers.check_witness_attestation", lambda ctx: verified) - signal, witness_result = _detect_strong_hermeticity_signal( - tmp_path, [], {}, make_ctx(tmp_path), {"verify_witness_attestations": False} - ) - # The mocked check_witness_attestation returns a verified/clean result, - # but the config toggle must prevent it from ever being called. - assert signal is None - assert witness_result.attempted is False - assert "disabled via config" in witness_result.detail + assert _detect_strong_hermeticity_signal(tmp_path, [wf], {}) is None def test_nix_flake_with_nix_build_in_ci(self, tmp_path: Path) -> None: (tmp_path / "flake.nix").write_text("{ outputs = {}; }") wf = tmp_path / "ci.yml" wf.write_text("- run: nix build .#default\n") - signal, _ = _detect_strong_hermeticity_signal( - tmp_path, [wf], {"RE-01.02": "PASS"}, make_ctx(tmp_path, {"RE-01.02": "PASS"}), {} - ) + signal = _detect_strong_hermeticity_signal(tmp_path, [wf], {"RE-01.02": "PASS"}) assert signal is not None assert "Nix" in signal @@ -318,9 +270,7 @@ def test_nix_flake_present_but_no_ci_usage(self, tmp_path: Path) -> None: (tmp_path / "flake.nix").write_text("{ outputs = {}; }") wf = tmp_path / "ci.yml" wf.write_text("- run: uv sync\n") - signal, _ = _detect_strong_hermeticity_signal( - tmp_path, [wf], {"RE-01.02": "PASS"}, make_ctx(tmp_path, {"RE-01.02": "PASS"}), {} - ) + signal = _detect_strong_hermeticity_signal(tmp_path, [wf], {"RE-01.02": "PASS"}) assert signal is None def test_nix_flake_not_gated_without_build_env_declared_pass(self, tmp_path: Path) -> None: @@ -329,23 +279,21 @@ def test_nix_flake_not_gated_without_build_env_declared_pass(self, tmp_path: Pat (tmp_path / "flake.nix").write_text("{ outputs = {}; }") wf = tmp_path / "ci.yml" wf.write_text("- run: nix build .#default\n") - signal, _ = _detect_strong_hermeticity_signal(tmp_path, [wf], {}, make_ctx(tmp_path), {}) + signal = _detect_strong_hermeticity_signal(tmp_path, [wf], {}) assert signal is None def test_nix_flake_not_gated_when_build_env_declared_failed(self, tmp_path: Path) -> None: (tmp_path / "flake.nix").write_text("{ outputs = {}; }") wf = tmp_path / "ci.yml" wf.write_text("- run: nix build .#default\n") - signal, _ = _detect_strong_hermeticity_signal( - tmp_path, [wf], {"RE-01.02": "FAIL"}, make_ctx(tmp_path, {"RE-01.02": "FAIL"}), {} - ) + signal = _detect_strong_hermeticity_signal(tmp_path, [wf], {"RE-01.02": "FAIL"}) assert signal is None def test_bazel_with_network_sandbox_flag(self, tmp_path: Path) -> None: (tmp_path / "MODULE.bazel").write_text("module(name = 'myproject')") wf = tmp_path / "ci.yml" wf.write_text("- run: bazel build //... --sandbox_default_allow_network=false\n") - signal, _ = _detect_strong_hermeticity_signal(tmp_path, [wf], {}, make_ctx(tmp_path), {}) + signal = _detect_strong_hermeticity_signal(tmp_path, [wf], {}) assert signal is not None assert "Bazel" in signal @@ -354,32 +302,14 @@ def test_bazel_workspace_without_sandbox_flag(self, tmp_path: Path) -> None: (tmp_path / "WORKSPACE").write_text("") wf = tmp_path / "ci.yml" wf.write_text("- run: bazel build //...\n") - signal, _ = _detect_strong_hermeticity_signal(tmp_path, [wf], {}, make_ctx(tmp_path), {}) + signal = _detect_strong_hermeticity_signal(tmp_path, [wf], {}) assert signal is None - def test_witness_takes_priority_over_nix(self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: - # No CI file mentions "witness" at all here — the point of this test - # is that a verified attestation still wins even though there is no - # text-based hint that Witness is in use (e.g. it ran via a reusable - # workflow the caller's own CI files never name). - verified = WitnessCheckResult(attempted=True, verified=True, network_clean=True, detail="clean") - monkeypatch.setattr("darnit_reproducibility.handlers.check_witness_attestation", lambda ctx: verified) - (tmp_path / "flake.nix").write_text("{ outputs = {}; }") - wf = tmp_path / "ci.yml" - wf.write_text("- run: nix build .#default\n") - signal, _ = _detect_strong_hermeticity_signal( - tmp_path, [wf], {"RE-01.02": "PASS"}, make_ctx(tmp_path, {"RE-01.02": "PASS"}), {} - ) - assert signal is not None - assert "Witness" in signal - def test_commented_nix_reference_is_not_a_signal(self, tmp_path: Path) -> None: (tmp_path / "flake.nix").write_text("{ outputs = {}; }") wf = tmp_path / "ci.yml" wf.write_text("# TODO: nix build .#default someday\nsteps:\n - run: uv sync\n") - signal, _ = _detect_strong_hermeticity_signal( - tmp_path, [wf], {"RE-01.02": "PASS"}, make_ctx(tmp_path, {"RE-01.02": "PASS"}), {} - ) + signal = _detect_strong_hermeticity_signal(tmp_path, [wf], {"RE-01.02": "PASS"}) assert signal is None def test_commented_bazel_sandbox_flag_is_not_a_signal(self, tmp_path: Path) -> None: @@ -388,37 +318,99 @@ def test_commented_bazel_sandbox_flag_is_not_a_signal(self, tmp_path: Path) -> N wf.write_text( "steps:\n # TODO: bazel build //... --sandbox_default_allow_network=false\n - run: bazel build //...\n" ) - signal, _ = _detect_strong_hermeticity_signal(tmp_path, [wf], {}, make_ctx(tmp_path), {}) + signal = _detect_strong_hermeticity_signal(tmp_path, [wf], {}) assert signal is None -class TestMaybeCheckWitnessAttestation: - """Unit tests for the config-toggle wrapper around check_witness_attestation().""" +class TestReproWitnessAttestationHandler: + """The FAIL-only attestation step (#553): a verified record of network access FAILs; nothing PASSes.""" - def test_disabled_via_config_short_circuits(self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: - called = False + def _verified(self, network_recorded: bool | None, detail: str) -> WitnessCheckResult: + return WitnessCheckResult( + attempted=True, + verified=True, + network_recorded=network_recorded, + detail=detail, + evidence={ + "commit": "0123456789abcdef0123456789abcdef01234567", + "artifact": "build.att.json", + "monitor_types": ["https://tetragon.io/"], + }, + ) + + def test_recorded_network_events_fail(self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + dirty = self._verified(True, "runtime-trace predicate recorded 3 network event(s) (monitor: x)") + monkeypatch.setattr("darnit_reproducibility.handlers.check_witness_attestation", lambda ctx: dirty) + result = repro_witness_attestation_handler({}, make_ctx(tmp_path)) + assert result.status == HandlerResultStatus.FAIL + assert "build.att.json" in result.message + assert "3 network event(s)" in result.message + assert "0123456789ab" in result.message + assert result.evidence["witness_attestation"]["network_recorded"] is True + + def test_suspicious_command_line_fails(self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + dirty = self._verified(True, "command-run process matched 'curl': /usr/bin/curl curl https://x") + monkeypatch.setattr("darnit_reproducibility.handlers.check_witness_attestation", lambda ctx: dirty) + result = repro_witness_attestation_handler({}, make_ctx(tmp_path)) + assert result.status == HandlerResultStatus.FAIL + assert "command-run process matched 'curl'" in result.message + + @pytest.mark.parametrize("network_recorded", [False, None]) + def test_verified_clean_attestation_is_never_pass( + self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch, network_recorded: bool | None + ) -> None: + clean = self._verified(network_recorded, "runtime-trace predicate recorded no network events") + monkeypatch.setattr("darnit_reproducibility.handlers.check_witness_attestation", lambda ctx: clean) + result = repro_witness_attestation_handler({}, make_ctx(tmp_path)) + assert result.status == HandlerResultStatus.INCONCLUSIVE + assert "cannot by itself establish that the build had no network access" in result.message + assert "build.att.json" in result.message + assert "https://tetragon.io/" in result.message + assert result.evidence["witness_attestation"]["artifact"] == "build.att.json" + assert result.evidence["witness_attestation"]["monitor_types"] == ["https://tetragon.io/"] + + def test_missing_evidence_is_inconclusive_with_reason( + self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch + ) -> None: + missing = WitnessCheckResult(attempted=True, detail="gh is not authenticated for this repository") + monkeypatch.setattr("darnit_reproducibility.handlers.check_witness_attestation", lambda ctx: missing) + result = repro_witness_attestation_handler({}, make_ctx(tmp_path)) + assert result.status == HandlerResultStatus.INCONCLUSIVE + assert "gh is not authenticated" in result.message + + def test_disabled_via_config_makes_no_calls(self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.setattr( + "darnit_reproducibility.handlers.check_witness_attestation", + lambda ctx: pytest.fail("verification must not run when disabled"), + ) + monkeypatch.setattr( + "darnit_reproducibility.witness_attestation._run_gh", + lambda args: pytest.fail("gh must not run when disabled"), + ) + result = repro_witness_attestation_handler({"verify_witness_attestations": False}, make_ctx(tmp_path)) + assert result.status == HandlerResultStatus.INCONCLUSIVE + assert "disabled" in result.message + + @pytest.mark.parametrize("config", [{}, {"verify_witness_attestations": True}]) + def test_enabled_by_default(self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch, config: dict[str, bool]) -> None: + called: list[HandlerContext] = [] def spy(ctx: HandlerContext) -> WitnessCheckResult: - nonlocal called - called = True - return WitnessCheckResult(attempted=True, verified=True, network_clean=True) + called.append(ctx) + return NO_WITNESS_EVIDENCE monkeypatch.setattr("darnit_reproducibility.handlers.check_witness_attestation", spy) - result = _maybe_check_witness_attestation(make_ctx(tmp_path), {"verify_witness_attestations": False}) - assert called is False - assert result.attempted is False + repro_witness_attestation_handler(config, make_ctx(tmp_path)) + assert len(called) == 1 - def test_enabled_by_default(self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: - expected = WitnessCheckResult(attempted=True, verified=True, network_clean=True) - monkeypatch.setattr("darnit_reproducibility.handlers.check_witness_attestation", lambda ctx: expected) - result = _maybe_check_witness_attestation(make_ctx(tmp_path), {}) - assert result is expected + def test_never_raises(self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + def boom(ctx: HandlerContext) -> WitnessCheckResult: + raise RuntimeError("unexpected") - def test_explicitly_enabled_via_config(self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: - expected = WitnessCheckResult(attempted=True, verified=True, network_clean=True) - monkeypatch.setattr("darnit_reproducibility.handlers.check_witness_attestation", lambda ctx: expected) - result = _maybe_check_witness_attestation(make_ctx(tmp_path), {"verify_witness_attestations": True}) - assert result is expected + monkeypatch.setattr("darnit_reproducibility.handlers.check_witness_attestation", boom) + result = repro_witness_attestation_handler({}, make_ctx(tmp_path)) + assert result.status == HandlerResultStatus.INCONCLUSIVE + assert "unexpected" in result.message class TestRepoDepsPin: @@ -953,57 +945,28 @@ def test_inconclusive_inline_comment_with_violation(self, tmp_path: Path) -> Non # ------------------------------------------------------------------ def test_witness_mention_alone_does_not_pass(self, tmp_path: Path) -> None: - # Text-only mention of witness in CI is no longer sufficient for a - # PASS — see test_pass_verified_witness_attestation below. + # Text-only mention of witness in CI is not a strong signal. wf_dir = tmp_path / ".github" / "workflows" wf_dir.mkdir(parents=True) (wf_dir / "ci.yml").write_text("steps:\n - uses: testifysec/witness-run-action@v0.1\n") result = repro_hermetic_build_handler({}, make_ctx(tmp_path)) assert result.status == HandlerResultStatus.INCONCLUSIVE - def test_pass_verified_witness_attestation(self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: - verified = WitnessCheckResult( - attempted=True, - verified=True, - network_clean=True, - detail="runtime-trace predicate recorded an empty network log", - ) - monkeypatch.setattr("darnit_reproducibility.handlers.check_witness_attestation", lambda ctx: verified) - wf_dir = tmp_path / ".github" / "workflows" - wf_dir.mkdir(parents=True) - (wf_dir / "ci.yml").write_text("steps:\n - uses: testifysec/witness-run-action@v0.1\n") - result = repro_hermetic_build_handler({}, make_ctx(tmp_path)) - assert result.status == HandlerResultStatus.PASS - assert "Witness" in result.message + def test_does_not_consult_attestations(self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + # #553: attestations are the repro_witness_attestation step's; this + # step reads only the checkout, so it makes no gh (or other) calls. + def no_calls(*args: object, **kwargs: object) -> None: + pytest.fail("repro_hermetic_build must not consult attestations or run a subprocess") - def test_fail_verified_witness_attestation_with_network_activity( - self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch - ) -> None: - dirty = WitnessCheckResult( - attempted=True, - verified=True, - network_clean=False, - detail="runtime-trace predicate recorded 1 network event(s)", - evidence={"artifact": "witness-attestation.json"}, - ) - monkeypatch.setattr("darnit_reproducibility.handlers.check_witness_attestation", lambda ctx: dirty) + monkeypatch.setattr("darnit_reproducibility.handlers.check_witness_attestation", no_calls) + monkeypatch.setattr("darnit_reproducibility.witness_attestation._run_gh", no_calls) + monkeypatch.setattr("subprocess.run", no_calls) wf_dir = tmp_path / ".github" / "workflows" wf_dir.mkdir(parents=True) - (wf_dir / "ci.yml").write_text("steps:\n - run: uv sync\n") + (wf_dir / "ci.yml").write_text("steps:\n - uses: testifysec/witness-run-action@v0.1\n - run: make\n") result = repro_hermetic_build_handler({}, make_ctx(tmp_path)) - assert result.status == HandlerResultStatus.FAIL - assert any("witness attestation" in v for v in result.evidence["violations_found"]) - - def test_witness_verification_disabled_via_config(self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: - # Even a mocked verified/clean attestation must not produce a PASS - # when the pass config opts out of the network round-trip. - verified = WitnessCheckResult(attempted=True, verified=True, network_clean=True, detail="clean") - monkeypatch.setattr("darnit_reproducibility.handlers.check_witness_attestation", lambda ctx: verified) - wf_dir = tmp_path / ".github" / "workflows" - wf_dir.mkdir(parents=True) - (wf_dir / "ci.yml").write_text("steps:\n - uses: testifysec/witness-run-action@v0.1\n") - result = repro_hermetic_build_handler({"verify_witness_attestations": False}, make_ctx(tmp_path)) assert result.status == HandlerResultStatus.INCONCLUSIVE + assert "Witness" not in result.message def test_pass_nix_flake_build_in_ci(self, tmp_path: Path) -> None: (tmp_path / "flake.nix").write_text("{ outputs = {}; }") diff --git a/tests/darnit_reproducibility/test_implementation.py b/tests/darnit_reproducibility/test_implementation.py index 2278d311..65bda663 100644 --- a/tests/darnit_reproducibility/test_implementation.py +++ b/tests/darnit_reproducibility/test_implementation.py @@ -51,7 +51,7 @@ def test_check_handlers_covers_all_controls(self) -> None: expected = { "repro_deps_pinned", "repro_build_env_declared", "repro_hermetic_build", "repro_provenance_exists", - "repro_bit_for_bit", + "repro_bit_for_bit", "repro_witness_attestation", } for name in expected: assert registry.get(name) is not None, f"{name} not registered" @@ -67,7 +67,7 @@ def test_all_check_handlers_are_callable(self) -> None: for name in [ "repro_deps_pinned", "repro_build_env_declared", "repro_hermetic_build", "repro_provenance_exists", - "repro_bit_for_bit", + "repro_bit_for_bit", "repro_witness_attestation", ]: info = registry.get(name) assert info is not None and callable(info.fn), f"{name} not callable" diff --git a/tests/darnit_reproducibility/test_no_pass_from_signals.py b/tests/darnit_reproducibility/test_no_pass_from_signals.py index 1c54afba..9cb54aa5 100644 --- a/tests/darnit_reproducibility/test_no_pass_from_signals.py +++ b/tests/darnit_reproducibility/test_no_pass_from_signals.py @@ -12,14 +12,18 @@ from __future__ import annotations +import copy from pathlib import Path import pytest +from darnit_reproducibility.handlers import repro_hermetic_build_handler from darnit_reproducibility.implementation import ReproducibilityImplementation +from darnit_reproducibility.witness_attestation import WitnessCheckResult from darnit.config import load_controls_from_framework, load_framework_config from darnit.config.framework_schema import HandlerInvocation -from darnit.sieve.handler_registry import get_sieve_handler_registry +from darnit.core.errors import AuthorityViolation +from darnit.sieve.handler_registry import HandlerContext, get_sieve_handler_registry from darnit.sieve.models import CheckContext, ControlSpec, SieveResult from darnit.sieve.orchestrator import SieveOrchestrator @@ -29,6 +33,7 @@ "repro_bit_for_bit", "repro_deps_pinned", "repro_build_env_declared", + "repro_witness_attestation", ) # A release workflow that mentions signing only in a comment and produces no @@ -66,7 +71,7 @@ } # The only network call in the plugin; off so every verdict is a filesystem function. -OFFLINE = {"repro_hermetic_build": {"verify_witness_attestations": False}} +OFFLINE = {"repro_witness_attestation": {"verify_witness_attestations": False}} @pytest.fixture(autouse=True) @@ -82,12 +87,14 @@ def _build(root: Path, files: dict[str, str]) -> Path: return root -def _shipped_control(control_id: str) -> ControlSpec: +def _shipped_control(control_id: str, *, offline: bool = True) -> ControlSpec: path = ReproducibilityImplementation().get_framework_config_path() controls = {c.control_id: c for c in load_controls_from_framework(load_framework_config(path))} - spec = controls[control_id] + # A copy: the loaded configuration is cached and shared, so setting a step + # field on it would carry into every later load in the process. + spec = copy.deepcopy(controls[control_id]) for step in spec.metadata["handler_invocations"]: - for key, value in OFFLINE.get(step.handler, {}).items(): + for key, value in (OFFLINE.get(step.handler, {}) if offline else {}).items(): setattr(step, key, value) return spec @@ -112,6 +119,52 @@ def test_step_type_registers_fail_ceiling(step_type: str) -> None: assert info.ceiling == frozenset({"fail"}) +@pytest.mark.unit +def test_witness_setting_belongs_to_the_attestation_step() -> None: + """#553: `verify_witness_attestations` moved; on repro_hermetic_build it fails strict loading.""" + registry = get_sieve_handler_registry() + assert registry.get("repro_witness_attestation").settings == frozenset({"verify_witness_attestations"}) + assert registry.get("repro_hermetic_build").settings == frozenset() + + +_STRICT_FRAMEWORK = """\ +[metadata] +name = "repro-strict" +display_name = "Strict" +version = "0.0.1" +spec_version = "t" + +[controls."RE-X"] +name = "X" +description = "A control" +level = 1 +domain = "RE" + +[[controls."RE-X".passes]] +handler = "{handler}" +verify_witness_attestations = false +""" + + +@pytest.mark.unit +def test_witness_setting_on_the_scan_fails_loading(tmp_path: Path) -> None: + """#553: a configuration that set it on repro_hermetic_build is now refused, naming the key.""" + path = tmp_path / "fw.toml" + path.write_text(_STRICT_FRAMEWORK.format(handler="repro_hermetic_build"), encoding="utf-8") + with pytest.raises(AuthorityViolation, match="verify_witness_attestations"): + load_controls_from_framework(load_framework_config(path)) + + path.write_text(_STRICT_FRAMEWORK.format(handler="repro_witness_attestation"), encoding="utf-8") + assert load_controls_from_framework(load_framework_config(path)) + + +@pytest.mark.unit +def test_attestation_step_runs_before_the_scan() -> None: + """#553: a recorded network access concludes FAIL before the scan's evidence-only PASS.""" + handlers = [step.handler for step in _shipped_control("RE-02.01").metadata["handler_invocations"]] + assert handlers == ["repro_witness_attestation", "repro_hermetic_build", "manual"] + + @pytest.mark.unit def test_comment_mention_of_cosign_does_not_pass_provenance(tmp_path: Path) -> None: """SC-004, quickstart V4: RE-02.02 is not PASS and the mention is evidence.""" @@ -211,3 +264,56 @@ def test_partial_signals_reach_llm_eval(tmp_path: Path, case: str) -> None: assert result.status == "PENDING", result.message gathered = result.evidence["llm_consultation"]["gathered_evidence"] assert gathered.get(evidence_key), f"{evidence_key} missing from {sorted(gathered)}" + + +def _attestation(monkeypatch: pytest.MonkeyPatch, network_recorded: bool | None, detail: str) -> None: + """A verified attestation for the audited commit, without gh or sigstore (#553).""" + result = WitnessCheckResult( + attempted=True, + verified=True, + network_recorded=network_recorded, + detail=detail, + evidence={"commit": "a" * 40, "artifact": "build.att.json", "monitor_types": ["https://tetragon.io/"]}, + ) + monkeypatch.setattr("darnit_reproducibility.handlers.check_witness_attestation", lambda ctx: result) + + +@pytest.mark.unit +def test_recorded_network_access_fails_despite_a_strong_signal(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + """#553: a verified attestation recording network access concludes RE-02.01 FAIL. + + The repository also carries the Bazel network-sandbox strong signal, which + repro_hermetic_build reports as a PASS; it does not outweigh the record. + """ + files, _ = SIGNAL_REPOS["repro_hermetic_build"] + repo = _build(tmp_path, files) + assert repro_hermetic_build_handler({}, _handler_ctx(repo)).status.value == "pass" + _attestation(monkeypatch, True, "runtime-trace predicate recorded 2 network event(s) (monitor: x)") + + result = _verify(_shipped_control("RE-02.01", offline=False), repo) + + assert result.status == "FAIL", result.message + assert result.concluded_by == "repro_witness_attestation" + assert "2 network event(s)" in result.message + + +@pytest.mark.unit +@pytest.mark.parametrize("network_recorded", [False, None]) +def test_verified_clean_trace_does_not_pass( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch, network_recorded: bool | None +) -> None: + """#553: a verified clean trace, with every other signal present, is no RE-02.01 PASS.""" + files: dict[str, str] = {} + for repo_files, _ in SIGNAL_REPOS.values(): + files.update(repo_files) + repo = _build(tmp_path, files) + _attestation(monkeypatch, network_recorded, "runtime-trace predicate recorded no network events") + + result = _verify(_shipped_control("RE-02.01", offline=False), repo) + + assert result.status != "PASS", result.message + assert result.evidence["witness_attestation"]["verified"] is True + + +def _handler_ctx(repo: Path) -> HandlerContext: + return HandlerContext(local_path=str(repo), owner="org", repo="repo", default_branch="main") diff --git a/tests/darnit_reproducibility/test_repro_corpus.py b/tests/darnit_reproducibility/test_repro_corpus.py index fa882145..9eee7664 100644 --- a/tests/darnit_reproducibility/test_repro_corpus.py +++ b/tests/darnit_reproducibility/test_repro_corpus.py @@ -8,9 +8,10 @@ Feature 037 needed a captured baseline because its claim was the opposite -- "nothing changed" across 213 controls. It also learned that a stored golden of control statuses is not portable: statuses depend on whether `gh` is -authenticated. That is avoided here by disabling witness verification, which -removes the only network call in this plugin and makes every verdict a pure -filesystem function. +authenticated. The handlers measured here make no network call (#553 moved the +only one, Witness attestation verification, into its own step), so every +verdict is a pure filesystem function. The attestation step is measured below +against verified statements, with fetching and verification faked. """ from __future__ import annotations @@ -18,14 +19,16 @@ from pathlib import Path import pytest +from darnit_reproducibility import witness_attestation as wa from darnit_reproducibility.handlers import ( repro_bit_for_bit_handler, repro_build_env_declared_handler, repro_deps_pinned_handler, repro_hermetic_build_handler, + repro_witness_attestation_handler, ) -from .conftest import OFFLINE_CONFIG, repro_ctx +from .conftest import repro_ctx DIGEST = "sha256:" + "a" * 64 @@ -81,7 +84,7 @@ def test_corpus_matches_expected_verdicts(tmp_path: Path, fixture_name: str) -> repo = _build(tmp_path, CORPUS[fixture_name]) ctx = repro_ctx(repo) for control_id, expected in EXPECTED[fixture_name].items(): - result = HANDLERS[control_id](dict(OFFLINE_CONFIG), ctx) + result = HANDLERS[control_id]({}, ctx) assert result.status.value == expected, ( f"{fixture_name}: {control_id} expected {expected}, got {result.status.value} -- {result.message}" ) @@ -106,6 +109,74 @@ def test_deps_pinned_is_untouched_by_this_feature(tmp_path: Path) -> None: the later steps. """ repo = _build(tmp_path, {"requirements.txt": "numpy==1.26.4\n"}) - result = repro_deps_pinned_handler(dict(OFFLINE_CONFIG), repro_ctx(repo)) + result = repro_deps_pinned_handler({}, repro_ctx(repo)) assert result.status.value == "inconclusive" assert "transitive" in result.message + + +# #553: verified attestation statements -> the repro_witness_attestation outcome. +# Only recorded network access decides, and it decides FAIL. A clean or absent +# network log is what a monitor that does not trace sockets also records, so it +# is evidence and never "pass". +_RUNTIME_TRACE = "https://in-toto.io/attestation/runtime-trace/v0.1" +_COLLECTION = "https://witness.dev/attestation-collection/v0.1" +_COMMAND_RUN = "https://witness.dev/attestations/command-run/v0.1" + + +def _trace(monitor_log: dict) -> dict: + return {"monitor": {"type": "https://tetragon.io/", "tracePolicy": {}}, "monitorLog": monitor_log} + + +def _collection(entry_type: str, attestation: dict) -> dict: + return { + "predicateType": _COLLECTION, + "predicate": {"attestations": [{"type": entry_type, "attestation": attestation}]}, + } + + +ATTESTATION_CORPUS: dict[str, tuple[dict, str]] = { + "trace_with_network_events": ( + {"predicateType": _RUNTIME_TRACE, "predicate": _trace({"network": [{"connect": "203.0.113.7:443"}]})}, + "fail", + ), + "trace_with_empty_network_log": ( + {"predicateType": _RUNTIME_TRACE, "predicate": _trace({"network": []})}, + "inconclusive", + ), + # The runtime-trace spec's own Tetragon example: a `connect` policy, no `network` field. + "trace_without_network_field": ( + {"predicateType": _RUNTIME_TRACE, "predicate": _trace({"process": [{"exec": "make"}]})}, + "inconclusive", + ), + "collection_trace_with_network_events": (_collection(_RUNTIME_TRACE, _trace({"network": [{}, {}]})), "fail"), + "collection_command_run_installer": ( + _collection(_COMMAND_RUN, {"processes": [{"program": "/usr/bin/curl", "cmdline": "curl -O https://x"}]}), + "fail", + ), + "collection_command_run_clean": ( + _collection(_COMMAND_RUN, {"processes": [{"program": "/usr/bin/make", "cmdline": "make"}]}), + "inconclusive", + ), + "other_predicate_with_network_key": ( + {"predicateType": "https://slsa.dev/provenance/v1", "predicate": {"network": [{"host": "x"}]}}, + "inconclusive", + ), +} + + +@pytest.mark.unit +@pytest.mark.parametrize("case", sorted(ATTESTATION_CORPUS)) +def test_attestation_corpus(tmp_path: Path, monkeypatch: pytest.MonkeyPatch, case: str) -> None: + verified, expected = ATTESTATION_CORPUS[case] + artifact = tmp_path / "build.att.json" + artifact.write_text("{}", encoding="utf-8") + monkeypatch.setattr(wa, "SIGSTORE_VERIFY_AVAILABLE", True) + monkeypatch.setattr(wa, "_head_commit", lambda local_path: ("a" * 40, None)) + monkeypatch.setattr(wa, "_successful_run_ids", lambda owner, repo, sha: (["1"], None)) + monkeypatch.setattr(wa, "_fetch_candidate_files", lambda owner, repo, run_ids, scratch: ([artifact], None)) + monkeypatch.setattr(wa, "_verify_bundle", lambda raw, owner, repo, sha: verified) + + result = repro_witness_attestation_handler({}, repro_ctx(tmp_path)) + + assert result.status.value == expected, f"{case}: {result.message}" + assert result.evidence["witness_attestation"]["verified"] is True diff --git a/tests/darnit_reproducibility/test_witness_attestation.py b/tests/darnit_reproducibility/test_witness_attestation.py index 0fc4cfa3..9dec59b5 100644 --- a/tests/darnit_reproducibility/test_witness_attestation.py +++ b/tests/darnit_reproducibility/test_witness_attestation.py @@ -1,14 +1,17 @@ -"""Tests for the Witness/runtime-trace attestation verification helper. +"""Tests for the Witness / in-toto runtime-trace attestation check (#553). -Sigstore-dependent tests (``_verify_bundle`` success/failure paths) are -skipped when ``sigstore`` isn't installed — install the `attestation` extra -(``uv sync --extra attestation``) to run them. +``sigstore`` is an optional extra and is not installed in every environment, so +verification is exercised with fakes installed into the module (``raising=False`` +lets them stand in for names the failed import never bound). The one test that +needs the real library is skipped without it -- install the `attestation` extra +(``uv sync --extra attestation``) to run it. """ from __future__ import annotations import base64 import json +import os import subprocess from pathlib import Path from typing import Any @@ -23,15 +26,109 @@ reason="sigstore not installed — run `uv sync --extra attestation`", ) +SHA = "0123456789abcdef0123456789abcdef01234567" -def make_ctx(owner: str = "org", repo: str = "repo", branch: str = "main") -> HandlerContext: - return HandlerContext(local_path=".", owner=owner, repo=repo, default_branch=branch) + +def make_ctx(local_path: str = ".", owner: str = "org", repo: str = "repo") -> HandlerContext: + return HandlerContext(local_path=local_path, owner=owner, repo=repo, default_branch="main") def fake_proc(returncode: int = 0, stdout: str = "", stderr: str = "") -> subprocess.CompletedProcess[str]: return subprocess.CompletedProcess(args=["gh"], returncode=returncode, stdout=stdout, stderr=stderr) +def runtime_trace( + network: Any = None, *, monitor_type: str = "https://tetragon.io/", with_network: bool = True +) -> dict: + monitor_log: dict[str, Any] = {"process": [{"exec": "make"}]} + if with_network: + monitor_log["network"] = network + return { + "monitor": {"type": monitor_type, "tracePolicy": {}}, + "monitoredProcess": {"hostID": "runner"}, + "monitorLog": monitor_log, + } + + +def statement(predicate_type: str, predicate: dict) -> dict: + return {"_type": "https://in-toto.io/Statement/v1", "predicateType": predicate_type, "predicate": predicate} + + +def collection(*entries: tuple[str, dict]) -> dict: + return statement( + wa._WITNESS_COLLECTION_TYPE, + {"name": "build", "attestations": [{"type": t, "attestation": a} for t, a in entries]}, + ) + + +COMMAND_RUN = "https://witness.dev/attestations/command-run/v0.1" + + +def git_repo(path: Path) -> str: + """A real checkout with one commit; returns its HEAD.""" + env = {**os.environ, "GIT_CONFIG_GLOBAL": os.devnull, "GIT_CONFIG_NOSYSTEM": "1"} + subprocess.run(["git", "init", "-q", str(path)], check=True, env=env) + subprocess.run( + ["git", "-c", "user.name=t", "-c", "user.email=t@example.com", "commit", "-q", "--allow-empty", "-m", "x"], + cwd=path, + check=True, + env=env, + ) + return subprocess.run(["git", "rev-parse", "HEAD"], cwd=path, capture_output=True, text=True).stdout.strip() + + +class FakePolicy: + def __init__(self, *args: Any) -> None: + self.args = args + + def __repr__(self) -> str: + return f"{type(self).__name__}{self.args}" + + +class FakeAllOf(FakePolicy): + pass + + +class FakeOIDCIssuer(FakePolicy): + pass + + +class FakeRepository(FakePolicy): + pass + + +class FakeWorkflowSHA(FakePolicy): + pass + + +class FakeBundle: + @staticmethod + def from_json(raw: bytes) -> FakeBundle: + return FakeBundle() + + +@pytest.fixture +def fake_sigstore(monkeypatch: pytest.MonkeyPatch) -> dict[str, Any]: + """Stand-ins for the sigstore names; ``state["result"]`` is what verify_dsse returns.""" + state: dict[str, Any] = {"policies": [], "result": ("application/vnd.in-toto+json", b"{}")} + + class FakeVerifier: + def verify_dsse(self, bundle: Any, policy: Any) -> tuple[str, bytes]: + state["policies"].append(policy) + if isinstance(state["result"], Exception): + raise state["result"] + return state["result"] + + monkeypatch.setattr(wa, "SIGSTORE_VERIFY_AVAILABLE", True) + monkeypatch.setattr(wa, "Bundle", FakeBundle, raising=False) + monkeypatch.setattr(wa, "Verifier", type("V", (), {"production": staticmethod(FakeVerifier)}), raising=False) + monkeypatch.setattr(wa, "AllOf", FakeAllOf, raising=False) + monkeypatch.setattr(wa, "OIDCIssuer", FakeOIDCIssuer, raising=False) + monkeypatch.setattr(wa, "GitHubWorkflowRepository", FakeRepository, raising=False) + monkeypatch.setattr(wa, "GitHubWorkflowSHA", FakeWorkflowSHA, raising=False) + return state + + class TestRunGh: def test_returns_outcome_with_proc_on_success(self, monkeypatch: pytest.MonkeyPatch) -> None: monkeypatch.setattr(wa.subprocess, "run", lambda *a, **kw: fake_proc(stdout="ok")) @@ -78,12 +175,41 @@ def test_other_failure_includes_stderr(self, monkeypatch: pytest.MonkeyPatch) -> assert "repository not found" in outcome.reason -class TestLatestSuccessfulRunId: +class TestHeadCommit: + def test_returns_head_of_the_audited_checkout(self, tmp_path: Path) -> None: + head = git_repo(tmp_path) + assert wa._head_commit(str(tmp_path)) == (head, None) + + def test_checkout_without_commits_has_no_commit(self, tmp_path: Path) -> None: + subprocess.run(["git", "init", "-q", str(tmp_path)], check=True) + sha, reason = wa._head_commit(str(tmp_path)) + assert sha is None + assert "no commit" in reason + + def test_missing_directory_has_no_commit(self, tmp_path: Path) -> None: + sha, reason = wa._head_commit(str(tmp_path / "absent")) + assert sha is None + assert "not a directory" in reason + + def test_git_missing_sets_reason(self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + def raise_not_found(*args: Any, **kwargs: Any) -> None: + raise FileNotFoundError("git") + + monkeypatch.setattr(wa.subprocess, "run", raise_not_found) + sha, reason = wa._head_commit(str(tmp_path)) + assert sha is None + assert "git not found" in reason + + def test_output_that_is_not_a_commit_is_rejected(self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.setattr(wa.subprocess, "run", lambda *a, **kw: fake_proc(stdout="HEAD\n")) + sha, _ = wa._head_commit(str(tmp_path)) + assert sha is None + + +class TestSuccessfulRunIds: def test_gh_unavailable_propagates_reason(self, monkeypatch: pytest.MonkeyPatch) -> None: monkeypatch.setattr(wa, "_run_gh", lambda args: wa._GhOutcome(None, "gh CLI not found in PATH")) - run_id, reason = wa._latest_successful_run_id("org", "repo", "main") - assert run_id is None - assert reason == "gh CLI not found in PATH" + assert wa._successful_run_ids("org", "repo", SHA) == ([], "gh CLI not found in PATH") def test_auth_failure_propagates_reason(self, monkeypatch: pytest.MonkeyPatch) -> None: monkeypatch.setattr( @@ -91,37 +217,34 @@ def test_auth_failure_propagates_reason(self, monkeypatch: pytest.MonkeyPatch) - "_run_gh", lambda args: wa._GhOutcome(None, "gh is not authenticated for this repository (run `gh auth login`)"), ) - run_id, reason = wa._latest_successful_run_id("org", "repo", "main") - assert run_id is None + run_ids, reason = wa._successful_run_ids("org", "repo", SHA) + assert run_ids == [] assert "not authenticated" in reason def test_empty_stdout_returns_reason(self, monkeypatch: pytest.MonkeyPatch) -> None: monkeypatch.setattr(wa, "_run_gh", lambda args: wa._GhOutcome(fake_proc(stdout=""))) - run_id, reason = wa._latest_successful_run_id("org", "repo", "main") - assert run_id is None + run_ids, reason = wa._successful_run_ids("org", "repo", SHA) + assert run_ids == [] assert "no output" in reason def test_invalid_json_returns_reason(self, monkeypatch: pytest.MonkeyPatch) -> None: monkeypatch.setattr(wa, "_run_gh", lambda args: wa._GhOutcome(fake_proc(stdout="not json"))) - run_id, reason = wa._latest_successful_run_id("org", "repo", "main") - assert run_id is None + run_ids, reason = wa._successful_run_ids("org", "repo", SHA) + assert run_ids == [] assert "unparseable" in reason - def test_empty_list_returns_reason(self, monkeypatch: pytest.MonkeyPatch) -> None: + def test_no_runs_names_the_commit(self, monkeypatch: pytest.MonkeyPatch) -> None: monkeypatch.setattr(wa, "_run_gh", lambda args: wa._GhOutcome(fake_proc(stdout="[]"))) - run_id, reason = wa._latest_successful_run_id("org", "repo", "main") - assert run_id is None - assert "no successful CI run" in reason + run_ids, reason = wa._successful_run_ids("org", "repo", SHA) + assert run_ids == [] + assert f"no successful CI run found for commit {SHA[:12]}" == reason - def test_valid_run_returns_id_as_string(self, monkeypatch: pytest.MonkeyPatch) -> None: - monkeypatch.setattr( - wa, "_run_gh", lambda args: wa._GhOutcome(fake_proc(stdout=json.dumps([{"databaseId": 123456}]))) - ) - run_id, reason = wa._latest_successful_run_id("org", "repo", "main") - assert run_id == "123456" - assert reason is None + def test_returns_every_run_id_as_string(self, monkeypatch: pytest.MonkeyPatch) -> None: + rows = [{"databaseId": 1}, {"databaseId": 2}, {}] + monkeypatch.setattr(wa, "_run_gh", lambda args: wa._GhOutcome(fake_proc(stdout=json.dumps(rows)))) + assert wa._successful_run_ids("org", "repo", SHA) == (["1", "2"], None) - def test_requests_the_right_repo_and_branch(self, monkeypatch: pytest.MonkeyPatch) -> None: + def test_queries_successful_runs_of_the_audited_commit(self, monkeypatch: pytest.MonkeyPatch) -> None: captured: dict[str, list[str]] = {} def spy(args: list[str]) -> wa._GhOutcome: @@ -129,10 +252,13 @@ def spy(args: list[str]) -> wa._GhOutcome: return wa._GhOutcome(fake_proc(stdout=json.dumps([{"databaseId": 1}]))) monkeypatch.setattr(wa, "_run_gh", spy) - wa._latest_successful_run_id("kusari-oss", "darnit", "main") - assert "kusari-oss/darnit" in captured["args"] - assert "main" in captured["args"] - assert "success" in captured["args"] + wa._successful_run_ids("kusari-oss", "darnit", SHA) + args = captured["args"] + assert args[args.index("--repo") + 1] == "kusari-oss/darnit" + assert args[args.index("--commit") + 1] == SHA + assert args[args.index("--status") + 1] == "success" + assert args[args.index("--limit") + 1] == "5" + assert "--branch" not in args class TestDownloadCandidateArtifacts: @@ -170,138 +296,132 @@ def test_caps_at_max_artifact_files(self, tmp_path: Path, monkeypatch: pytest.Mo files, _ = wa._download_candidate_artifacts("org", "repo", "123", tmp_path) assert len(files) == wa._MAX_ARTIFACT_FILES + def test_priority_suffixes_come_first(self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + (tmp_path / "a-log.json").write_text("{}") + (tmp_path / "z.att.json").write_text("{}") + monkeypatch.setattr(wa, "_run_gh", lambda args: wa._GhOutcome(fake_proc(returncode=0))) -class TestFetchCandidateFiles: - def test_missing_owner_returns_reason(self, tmp_path: Path) -> None: - ctx = make_ctx(owner="") - files, reason = wa._fetch_candidate_files(ctx, tmp_path) - assert files == [] - assert "owner/name not available" in reason + files, _ = wa._download_candidate_artifacts("org", "repo", "123", tmp_path) + assert [f.name for f in files] == ["z.att.json", "a-log.json"] - def test_missing_repo_returns_reason(self, tmp_path: Path) -> None: - ctx = make_ctx(repo="") - files, reason = wa._fetch_candidate_files(ctx, tmp_path) - assert files == [] - assert "owner/name not available" in reason - def test_no_run_found_propagates_reason(self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: - monkeypatch.setattr(wa, "_latest_successful_run_id", lambda owner, repo, branch: (None, "no successful CI run found on branch 'main'")) - files, reason = wa._fetch_candidate_files(make_ctx(), tmp_path) - assert files == [] - assert "no successful CI run" in reason +class TestFetchCandidateFiles: + def test_downloads_each_run_into_its_own_directory(self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + calls: list[tuple[str, Path]] = [] + + def spy(owner: str, repo: str, run_id: str, dest: Path) -> tuple[list[Path], str | None]: + calls.append((run_id, dest)) + return [dest / f"{run_id}.json"], None - def test_delegates_to_download_with_run_id(self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: - monkeypatch.setattr(wa, "_latest_successful_run_id", lambda owner, repo, branch: ("999", None)) - captured: dict[str, Any] = {} + monkeypatch.setattr(wa, "_download_candidate_artifacts", spy) + files, reason = wa._fetch_candidate_files("org", "repo", ["1", "2"], tmp_path) + assert calls == [("1", tmp_path / "1"), ("2", tmp_path / "2")] + assert [f.name for f in files] == ["1.json", "2.json"] + assert reason is None + def test_a_run_without_artifacts_does_not_hide_another( + self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch + ) -> None: def spy(owner: str, repo: str, run_id: str, dest: Path) -> tuple[list[Path], str | None]: - captured.update(owner=owner, repo=repo, run_id=run_id) - return [dest / "attestation.json"], None + if run_id == "1": + return [], "run 1 has no artifacts matching '*witness*'" + return [dest / "a.json"], None monkeypatch.setattr(wa, "_download_candidate_artifacts", spy) - files, reason = wa._fetch_candidate_files(make_ctx(owner="org", repo="repo"), tmp_path) - assert captured == {"owner": "org", "repo": "repo", "run_id": "999"} - assert files == [tmp_path / "attestation.json"] + files, reason = wa._fetch_candidate_files("org", "repo", ["1", "2"], tmp_path) + assert [f.name for f in files] == ["a.json"] assert reason is None + def test_no_artifacts_in_any_run_returns_the_reasons(self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.setattr( + wa, "_download_candidate_artifacts", lambda o, r, run_id, d: ([], f"run {run_id} has no artifacts") + ) + files, reason = wa._fetch_candidate_files("org", "repo", ["1", "2"], tmp_path) + assert files == [] + assert reason == "run 1 has no artifacts; run 2 has no artifacts" + class TestNestedAttestations: def test_witness_collection_unwraps_attestations_array(self) -> None: - statement = { - "predicateType": wa._WITNESS_COLLECTION_TYPE, - "predicate": {"attestations": [{"type": "command-run", "attestation": {"processes": []}}]}, - } - result = wa._nested_attestations(statement) - assert result == [{"type": "command-run", "attestation": {"processes": []}}] + stmt = collection(("command-run", {"processes": []})) + assert wa._nested_attestations(stmt) == [{"type": "command-run", "attestation": {"processes": []}}] def test_non_collection_type_wraps_predicate_directly(self) -> None: - statement = { - "predicateType": wa._RUNTIME_TRACE_TYPE, - "predicate": {"network": []}, - } - result = wa._nested_attestations(statement) - assert result == [{"type": wa._RUNTIME_TRACE_TYPE, "attestation": {"network": []}}] - - -class TestCheckNetworkCleanliness: - def test_empty_runtime_trace_network_array_is_clean(self) -> None: - statement = {"predicateType": wa._RUNTIME_TRACE_TYPE, "predicate": {"network": []}} - clean, detail = wa._check_network_cleanliness(statement) - assert clean is True - assert "empty network log" in detail - - def test_nonempty_runtime_trace_network_array_is_dirty(self) -> None: - statement = { - "predicateType": wa._RUNTIME_TRACE_TYPE, - "predicate": {"network": [{"host": "evil.example.com"}]}, - } - clean, detail = wa._check_network_cleanliness(statement) - assert clean is False - assert "1 network event" in detail + stmt = statement(wa._RUNTIME_TRACE_TYPE, {"monitorLog": {}}) + assert wa._nested_attestations(stmt) == [{"type": wa._RUNTIME_TRACE_TYPE, "attestation": {"monitorLog": {}}}] - def test_network_under_monitor_log_is_recognized(self) -> None: - statement = { - "predicateType": wa._RUNTIME_TRACE_TYPE, - "predicate": {"monitorLog": {"network": []}}, - } - clean, _ = wa._check_network_cleanliness(statement) - assert clean is True - - def test_command_run_with_suspicious_cmdline_is_dirty(self) -> None: - statement = { - "predicateType": wa._WITNESS_COLLECTION_TYPE, - "predicate": { - "attestations": [ - { - "type": "https://witness.dev/attestations/command-run/v0.1", - "attestation": {"processes": [{"program": "/usr/bin/curl", "cmdline": "curl https://x"}]}, - } - ] - }, - } - clean, detail = wa._check_network_cleanliness(statement) - assert clean is False + def test_malformed_predicate_yields_nothing(self) -> None: + assert wa._nested_attestations({"predicateType": wa._WITNESS_COLLECTION_TYPE, "predicate": []}) == [] + assert wa._nested_attestations(statement(wa._WITNESS_COLLECTION_TYPE, {"attestations": "x"})) == [] + + +class TestCheckNetworkEvidence: + def test_nonempty_network_log_records_network_access(self) -> None: + stmt = statement(wa._RUNTIME_TRACE_TYPE, runtime_trace([{"connect": "203.0.113.7:443"}])) + recorded, detail, monitors = wa._check_network_evidence(stmt) + assert recorded is True + assert "1 network event" in detail + assert monitors == ["https://tetragon.io/"] + + def test_empty_network_log_is_not_a_claim_of_no_access(self) -> None: + recorded, detail, _ = wa._check_network_evidence(statement(wa._RUNTIME_TRACE_TYPE, runtime_trace([]))) + assert recorded is False + assert "no network events" in detail + + def test_absent_network_log_equals_empty(self) -> None: + # The runtime-trace spec's own Tetragon example has no `network` field. + stmt = statement(wa._RUNTIME_TRACE_TYPE, runtime_trace(with_network=False)) + assert wa._check_network_evidence(stmt)[0] is False + + def test_null_network_log_equals_empty(self) -> None: + stmt = statement(wa._RUNTIME_TRACE_TYPE, runtime_trace(None)) + assert wa._check_network_evidence(stmt)[0] is False + + def test_network_key_on_another_predicate_type_is_ignored(self) -> None: + stmt = statement("https://example.com/build-log/v1", {"network": [{"host": "evil.example.com"}]}) + assert wa._check_network_evidence(stmt)[0] is None + + def test_top_level_network_key_on_runtime_trace_is_not_read(self) -> None: + predicate = runtime_trace(with_network=False) + predicate["network"] = [{"host": "evil.example.com"}] + assert wa._check_network_evidence(statement(wa._RUNTIME_TRACE_TYPE, predicate))[0] is False + + def test_predicate_type_must_match_exactly(self) -> None: + stmt = statement("https://example.com/attestation/runtime-trace/v0.1", runtime_trace([{"x": 1}])) + assert wa._check_network_evidence(stmt)[0] is None + + def test_network_log_that_is_not_a_list_decides_nothing(self) -> None: + stmt = statement(wa._RUNTIME_TRACE_TYPE, runtime_trace({"events": 3})) + assert wa._check_network_evidence(stmt)[0] is None + + def test_nested_runtime_trace_entry_is_read(self) -> None: + stmt = collection((wa._RUNTIME_TRACE_TYPE, runtime_trace([{"x": 1}, {"y": 2}]))) + recorded, detail, _ = wa._check_network_evidence(stmt) + assert recorded is True + assert "2 network event" in detail + + def test_nested_entry_type_must_match_exactly(self) -> None: + stmt = collection(("https://witness.dev/attestations/runtime-trace/v0.1", runtime_trace([{"x": 1}]))) + assert wa._check_network_evidence(stmt)[0] is None + + def test_command_run_with_suspicious_cmdline_records_network_access(self) -> None: + stmt = collection((COMMAND_RUN, {"processes": [{"program": "/usr/bin/curl", "cmdline": "curl https://x"}]})) + recorded, detail, _ = wa._check_network_evidence(stmt) + assert recorded is True assert "curl" in detail - def test_command_run_with_clean_processes_has_no_authoritative_signal(self) -> None: - statement = { - "predicateType": wa._WITNESS_COLLECTION_TYPE, - "predicate": { - "attestations": [ - { - "type": "https://witness.dev/attestations/command-run/v0.1", - "attestation": {"processes": [{"program": "/usr/bin/make", "cmdline": "make build"}]}, - } - ] - }, - } - clean, detail = wa._check_network_cleanliness(statement) - assert clean is None - assert "no authoritative" in detail - - def test_no_recognized_attestations_has_no_authoritative_signal(self) -> None: - statement = {"predicateType": "https://example.com/something-else/v1", "predicate": {}} - clean, _ = wa._check_network_cleanliness(statement) - assert clean is None - - def test_witness_takes_runtime_trace_over_command_run_when_both_present(self) -> None: - # A collection could in principle carry both a command-run entry and a - # runtime-trace entry; the authoritative network signal must win even - # if it's not first in the list. - statement = { - "predicateType": wa._WITNESS_COLLECTION_TYPE, - "predicate": { - "attestations": [ - { - "type": "https://witness.dev/attestations/command-run/v0.1", - "attestation": {"processes": [{"program": "/usr/bin/make", "cmdline": "make build"}]}, - }, - {"type": wa._RUNTIME_TRACE_TYPE, "attestation": {"network": []}}, - ] - }, - } - clean, _ = wa._check_network_cleanliness(statement) - assert clean is True + def test_command_run_with_clean_processes_shows_nothing(self) -> None: + stmt = collection((COMMAND_RUN, {"processes": [{"program": "/usr/bin/make", "cmdline": "make build"}]})) + recorded, detail, _ = wa._check_network_evidence(stmt) + assert recorded is None + assert "no runtime-trace network log" in detail + + def test_recorded_access_wins_over_an_earlier_clean_entry(self) -> None: + stmt = collection( + (wa._RUNTIME_TRACE_TYPE, runtime_trace([])), + (COMMAND_RUN, {"processes": [{"program": "/usr/bin/wget", "cmdline": "wget https://x"}]}), + ) + assert wa._check_network_evidence(stmt)[0] is True class TestDecodeRawDsse: @@ -325,118 +445,195 @@ def test_invalid_base64_returns_none(self) -> None: class TestVerifyBundle: def test_sigstore_unavailable_returns_none(self, monkeypatch: pytest.MonkeyPatch) -> None: monkeypatch.setattr(wa, "SIGSTORE_VERIFY_AVAILABLE", False) - assert wa._verify_bundle(b"{}", "org", "repo") is None + assert wa._verify_bundle(b"{}", "org", "repo", SHA) is None @needs_sigstore def test_invalid_bundle_bytes_returns_none(self) -> None: - assert wa._verify_bundle(b"not a sigstore bundle", "org", "repo") is None - - @needs_sigstore - def test_verification_failure_returns_none(self, monkeypatch: pytest.MonkeyPatch) -> None: - class FakeBundle: - @staticmethod - def from_json(raw: bytes) -> FakeBundle: - return FakeBundle() - - class FakeVerifier: - def verify_dsse(self, bundle: Any, policy: Any) -> tuple[str, bytes]: - raise RuntimeError("boom") - - monkeypatch.setattr(wa, "Bundle", FakeBundle) - monkeypatch.setattr(wa, "Verifier", type("V", (), {"production": staticmethod(lambda: FakeVerifier())})) - assert wa._verify_bundle(b"{}", "org", "repo") is None + assert wa._verify_bundle(b"not a sigstore bundle", "org", "repo", SHA) is None - @needs_sigstore - def test_non_intoto_payload_type_returns_none(self, monkeypatch: pytest.MonkeyPatch) -> None: - class FakeBundle: - @staticmethod - def from_json(raw: bytes) -> FakeBundle: - return FakeBundle() + def test_policy_binds_issuer_repository_and_commit(self, fake_sigstore: dict[str, Any]) -> None: + wa._verify_bundle(b"{}", "kusari-oss", "darnit", SHA) + (policy,) = fake_sigstore["policies"] + assert isinstance(policy, FakeAllOf) + (members,) = policy.args + assert [(type(m), m.args) for m in members] == [ + (FakeOIDCIssuer, (wa.GITHUB_ACTIONS_OIDC_ISSUER,)), + (FakeRepository, ("kusari-oss/darnit",)), + (FakeWorkflowSHA, (SHA,)), + ] - class FakeVerifier: - def verify_dsse(self, bundle: Any, policy: Any) -> tuple[str, bytes]: - return "application/octet-stream", b"{}" + def test_verification_failure_returns_none(self, fake_sigstore: dict[str, Any]) -> None: + fake_sigstore["result"] = RuntimeError("certificate is for another commit") + assert wa._verify_bundle(b"{}", "org", "repo", SHA) is None - monkeypatch.setattr(wa, "Bundle", FakeBundle) - monkeypatch.setattr(wa, "Verifier", type("V", (), {"production": staticmethod(lambda: FakeVerifier())})) - assert wa._verify_bundle(b"{}", "org", "repo") is None + def test_non_intoto_payload_type_returns_none(self, fake_sigstore: dict[str, Any]) -> None: + fake_sigstore["result"] = ("application/octet-stream", b"{}") + assert wa._verify_bundle(b"{}", "org", "repo", SHA) is None - @needs_sigstore - def test_successful_verification_returns_statement(self, monkeypatch: pytest.MonkeyPatch) -> None: - inner_statement = {"predicateType": wa._RUNTIME_TRACE_TYPE, "predicate": {"network": []}} + def test_successful_verification_returns_statement(self, fake_sigstore: dict[str, Any]) -> None: + inner = statement(wa._RUNTIME_TRACE_TYPE, runtime_trace([])) + fake_sigstore["result"] = ("application/vnd.in-toto+json", json.dumps(inner).encode()) + assert wa._verify_bundle(b"{}", "org", "repo", SHA) == inner - class FakeBundle: - @staticmethod - def from_json(raw: bytes) -> FakeBundle: - return FakeBundle() - class FakeVerifier: - def verify_dsse(self, bundle: Any, policy: Any) -> tuple[str, bytes]: - return "application/vnd.in-toto+json", json.dumps(inner_statement).encode() +class TestCheckWitnessAttestation: + @pytest.fixture + def offline(self, monkeypatch: pytest.MonkeyPatch) -> None: + """Sigstore present, the audited commit known, one successful run.""" + monkeypatch.setattr(wa, "SIGSTORE_VERIFY_AVAILABLE", True) + monkeypatch.setattr(wa, "_head_commit", lambda local_path: (SHA, None)) + monkeypatch.setattr(wa, "_successful_run_ids", lambda owner, repo, sha: (["7"], None)) - monkeypatch.setattr(wa, "Bundle", FakeBundle) - monkeypatch.setattr(wa, "Verifier", type("V", (), {"production": staticmethod(lambda: FakeVerifier())})) - result = wa._verify_bundle(b"{}", "org", "repo") - assert result == inner_statement + def _candidates(self, monkeypatch: pytest.MonkeyPatch, tmp_path: Path, *names: str) -> list[Path]: + files = [] + for name in names: + f = tmp_path / name + f.write_text(name) + files.append(f) + monkeypatch.setattr(wa, "_fetch_candidate_files", lambda owner, repo, run_ids, scratch: (files, None)) + return files + def _statements(self, monkeypatch: pytest.MonkeyPatch, by_name: dict[str, dict | None]) -> None: + monkeypatch.setattr(wa, "_verify_bundle", lambda raw, owner, repo, sha: by_name[raw.decode()]) -class TestCheckWitnessAttestation: def test_sigstore_unavailable_short_circuits(self, monkeypatch: pytest.MonkeyPatch) -> None: monkeypatch.setattr(wa, "SIGSTORE_VERIFY_AVAILABLE", False) result = wa.check_witness_attestation(make_ctx()) assert result.attempted is False assert result.verified is False + assert "sigstore not installed" in result.detail + + def test_missing_repository_identity_makes_no_calls(self, monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.setattr(wa, "SIGSTORE_VERIFY_AVAILABLE", True) + monkeypatch.setattr(wa.subprocess, "run", lambda *a, **kw: pytest.fail("no subprocess expected")) + result = wa.check_witness_attestation(make_ctx(owner="")) + assert result.verified is False + assert "owner/name not available" in result.detail - def test_no_candidates_found_surfaces_specific_reason(self, monkeypatch: pytest.MonkeyPatch) -> None: + def test_no_commit_surfaces_reason(self, monkeypatch: pytest.MonkeyPatch) -> None: monkeypatch.setattr(wa, "SIGSTORE_VERIFY_AVAILABLE", True) + monkeypatch.setattr(wa, "_head_commit", lambda local_path: (None, "audited path has no commit")) + monkeypatch.setattr(wa, "_run_gh", lambda args: pytest.fail("gh must not run without a commit")) + result = wa.check_witness_attestation(make_ctx()) + assert result.verified is False + assert result.detail == "audited path has no commit" + + def test_no_runs_for_the_commit_surfaces_reason(self, monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.setattr(wa, "SIGSTORE_VERIFY_AVAILABLE", True) + monkeypatch.setattr(wa, "_head_commit", lambda local_path: (SHA, None)) monkeypatch.setattr( - wa, - "_fetch_candidate_files", - lambda ctx, scratch_dir: ([], "gh is not authenticated for this repository (run `gh auth login`)"), + wa, "_successful_run_ids", lambda o, r, sha: ([], f"no successful CI run found for commit {sha[:12]}") ) result = wa.check_witness_attestation(make_ctx()) - assert result.attempted is True assert result.verified is False - assert "not authenticated" in result.detail + assert SHA[:12] in result.detail + assert result.evidence["commit"] == SHA - def test_no_candidates_found_falls_back_to_generic_detail(self, monkeypatch: pytest.MonkeyPatch) -> None: - monkeypatch.setattr(wa, "SIGSTORE_VERIFY_AVAILABLE", True) - monkeypatch.setattr(wa, "_fetch_candidate_files", lambda ctx, scratch_dir: ([], None)) + def test_no_candidates_found_surfaces_specific_reason(self, offline: None, monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.setattr( + wa, + "_fetch_candidate_files", + lambda o, r, run_ids, scratch: ([], "gh is not authenticated for this repository (run `gh auth login`)"), + ) result = wa.check_witness_attestation(make_ctx()) assert result.attempted is True assert result.verified is False - assert "no Witness attestation artifacts" in result.detail + assert "not authenticated" in result.detail + assert result.evidence["runs"] == ["7"] - def test_candidates_found_but_none_verify(self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: - f = tmp_path / "attestation.json" - f.write_text("{}") - monkeypatch.setattr(wa, "SIGSTORE_VERIFY_AVAILABLE", True) - monkeypatch.setattr(wa, "_fetch_candidate_files", lambda ctx, scratch_dir: ([f], None)) - monkeypatch.setattr(wa, "_verify_bundle", lambda raw, owner, repo: None) + def test_candidates_found_but_none_verify( + self, offline: None, tmp_path: Path, monkeypatch: pytest.MonkeyPatch + ) -> None: + self._candidates(monkeypatch, tmp_path, "attestation.json") + self._statements(monkeypatch, {"attestation.json": None}) result = wa.check_witness_attestation(make_ctx()) - assert result.attempted is True assert result.verified is False + assert "none verified" in result.detail assert result.evidence["checked_files"] == ["attestation.json"] - def test_verified_clean_attestation_short_circuits_remaining_candidates( - self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch + def test_verification_is_bound_to_the_audited_commit( + self, offline: None, tmp_path: Path, monkeypatch: pytest.MonkeyPatch ) -> None: - first = tmp_path / "a.json" - first.write_text("{}") - second = tmp_path / "b.json" - second.write_text("{}") - monkeypatch.setattr(wa, "SIGSTORE_VERIFY_AVAILABLE", True) - monkeypatch.setattr(wa, "_fetch_candidate_files", lambda ctx, scratch_dir: ([first, second], None)) + self._candidates(monkeypatch, tmp_path, "a.json") + seen: list[tuple[str, str, str]] = [] - statement = {"predicateType": wa._RUNTIME_TRACE_TYPE, "predicate": {"network": []}} + def spy(raw: bytes, owner: str, repo: str, sha: str) -> None: + seen.append((owner, repo, sha)) - def fake_verify(raw: bytes, owner: str, repo: str) -> dict[str, Any]: - return statement + monkeypatch.setattr(wa, "_verify_bundle", spy) + wa.check_witness_attestation(make_ctx()) + assert seen == [("org", "repo", SHA)] - monkeypatch.setattr(wa, "_verify_bundle", fake_verify) + def test_verified_clean_trace_is_not_a_claim_of_no_access( + self, offline: None, tmp_path: Path, monkeypatch: pytest.MonkeyPatch + ) -> None: + self._candidates(monkeypatch, tmp_path, "a.json") + self._statements(monkeypatch, {"a.json": statement(wa._RUNTIME_TRACE_TYPE, runtime_trace([]))}) result = wa.check_witness_attestation(make_ctx()) assert result.verified is True - assert result.network_clean is True + assert result.network_recorded is False assert result.evidence["artifact"] == "a.json" - # only the first candidate's bytes should have been read/verified - assert result.evidence["checked_files"] == ["a.json"] + assert result.evidence["monitor_types"] == ["https://tetragon.io/"] + assert result.evidence["commit"] == SHA + + def test_a_later_file_recording_network_access_is_not_hidden( + self, offline: None, tmp_path: Path, monkeypatch: pytest.MonkeyPatch + ) -> None: + self._candidates(monkeypatch, tmp_path, "clean.json", "dirty.json") + self._statements( + monkeypatch, + { + "clean.json": statement(wa._RUNTIME_TRACE_TYPE, runtime_trace([])), + "dirty.json": statement(wa._RUNTIME_TRACE_TYPE, runtime_trace([{"connect": "x"}])), + }, + ) + result = wa.check_witness_attestation(make_ctx()) + assert result.network_recorded is True + assert result.evidence["artifact"] == "dirty.json" + assert result.evidence["verified_artifacts"] == ["clean.json", "dirty.json"] + + def test_verified_attestation_without_a_trace_has_no_network_signal( + self, offline: None, tmp_path: Path, monkeypatch: pytest.MonkeyPatch + ) -> None: + self._candidates(monkeypatch, tmp_path, "a.json") + self._statements(monkeypatch, {"a.json": statement("https://slsa.dev/provenance/v1", {"network": [1]})}) + result = wa.check_witness_attestation(make_ctx()) + assert result.verified is True + assert result.network_recorded is None + + +class TestCommitBindingEndToEnd: + """From the checkout's HEAD to the run query and the signing policy, with only gh and sigstore faked.""" + + def test_head_reaches_the_run_query_and_the_policy( + self, tmp_path: Path, fake_sigstore: dict[str, Any], monkeypatch: pytest.MonkeyPatch + ) -> None: + repo = tmp_path / "checkout" + repo.mkdir() + head = git_repo(repo) + fake_sigstore["result"] = ( + "application/vnd.in-toto+json", + json.dumps(statement(wa._RUNTIME_TRACE_TYPE, runtime_trace([{"connect": "x"}]))).encode(), + ) + gh_calls: list[list[str]] = [] + + def fake_gh(args: list[str]) -> wa._GhOutcome: + gh_calls.append(args) + if args[:2] == ["run", "list"]: + return wa._GhOutcome(fake_proc(stdout=json.dumps([{"databaseId": 42}]))) + dest = Path(args[args.index("--dir") + 1]) + (dest / "witness").mkdir(parents=True, exist_ok=True) + (dest / "witness" / "build.att.json").write_text("{}") + return wa._GhOutcome(fake_proc()) + + monkeypatch.setattr(wa, "_run_gh", fake_gh) + result = wa.check_witness_attestation(make_ctx(local_path=str(repo))) + + list_args, download_args = gh_calls + assert list_args[list_args.index("--commit") + 1] == head + assert download_args[:3] == ["run", "download", "42"] + (policy,) = fake_sigstore["policies"] + assert any(isinstance(m, FakeWorkflowSHA) and m.args == (head,) for m in policy.args[0]) + assert result.network_recorded is True + assert result.evidence["commit"] == head + assert result.evidence["artifact"] == "build.att.json"