fix(reproducibility): split Witness verification into a FAIL-only step bound to the audited commit (#553)

[mlieberman85]

Summary

RE-02.01's repro_hermetic_build also fetched and Sigstore-verified Witness / in-toto attestations. It treated a verified, empty network log as a strong PASS signal. #553 proposed splitting that out into its own step type that could conclude PASS. While writing it we found the premise doesn't hold.

A verified empty network log is not proof of no network access.

• Under the in-toto parsing rules, an absent or null optional list counts the same as an empty one.
• The runtime-trace v0.1 spec leaves what monitorLog.network records to monitor.type and its tracePolicy.
• A monitor that doesn't trace sockets produces the same empty log.
• The spec's own Tetragon example (with a connect policy) has no network field at all.

So this PR does the split, but the new step is FAIL-only:

• New step type repro_witness_attestation, ceiling {fail}, runs first in RE-02.01. It verifies attestations from the successful CI runs for the audited commit (git rev-parse HEAD). The signing policy is bound to this repository (GitHubWorkflowRepository) and that commit (GitHubWorkflowSHA). Before, the check took the latest successful run on the default branch.
• It reads network events only from runtime-trace statements. The predicate type must be exactly https://in-toto.io/attestation/runtime-trace/v0.1, at the top level or nested in a Witness collection, and only monitorLog.network is read. The old fallbacks accepted a network key on any predicate type. The existing Witness command-run installer scan stays as FAIL evidence.
• Outcomes:
  • A verified trace that records network events → FAIL concludes RE-02.01. This now happens even when the repo also has a Nix or Bazel signal.
  • A verified clean trace → evidence only; evaluation continues.
  • Missing prerequisites (no gh, no login, no run, no attestation, sigstore not installed) → INCONCLUSIVE with the reason, as today. It's not ERROR: the step can only add FAIL evidence, so a missing optional source shouldn't turn a control's WARN into ERROR.
  • Every verified artifact is checked, so a clean attestation listed first can't hide one that records network access.
• repro_hermetic_build no longer makes any gh calls. The verify_witness_attestations setting moves to the new step. Under strict step settings, an override that sets it on repro_hermetic_build now fails loading (CHANGELOG, marked BREAKING).

Spec changes come first:

• framework-design.md 3.0.1 and the section 12 table: the step type with its {fail} ceiling and the reason.
• A new scenario in 3.0.1, "A runtime trace proves only network access".
• Version 1.0.0-alpha.14.
• The contract is in specs/044-false-pass-paths/contracts/witness-step-addendum-553.md.

A PASS from runtime traces needs a built-in allowlist of monitor types whose trace policy records network access, backed by real attestations. That's left for a follow-up.

Refs #553

Type of Change

• Bug fix (non-breaking change fixing an issue)
• New feature (non-breaking change adding functionality)
• Breaking change (fix or feature causing existing functionality to change)
• Documentation update
• Refactoring (no functional changes)

Framework Changes Checklist

• Updated framework spec (docs/architecture/framework-design.md) if behavior changed
• Ran uv run python scripts/validate_sync.py --verbose and it passes

Testing

• Tests pass locally (uv run pytest tests/ -v): 5252 passed, 26 skipped
• Added tests for new functionality (if applicable)
• Linting passes (uv run ruff check .)

The reproducibility tests also pass with sigstore installed (338 passed), matching CI's --all-extras. New tests cover:

• commit binding: the run query uses --commit <sha> and the policy includes GitHubWorkflowSHA(sha); one test uses a real git repo;
• the exact predicate type, and that a network key on another predicate type is ignored;
• a non-empty network log → FAIL, and an empty or absent one → never PASS;
• the setting disabled → no gh calls;
• repro_hermetic_build making no gh calls;
• registration with ceiling {fail}, and the RE-02.01 step order;
• RE-02.01 concluding FAIL despite a Nix or Bazel signal;
• corpus cases showing a verified clean trace never yields a control PASS.

I also ran a real darnit audit . -f reproducibility with sigstore installed. The step queried runs for the local (unpushed) commit and stayed INCONCLUSIVE ("no successful CI run found for commit …").

AI assistance

• No AI assistance was used
• AI assistance was used

Claude (Claude Code, claude-opus-5-5) found the in-toto spec issue, and made this change: spec, code, and tests. This description was also drafted with Claude. Commits carry an Assisted-by: Claude:claude-opus-5-5 trailer.

Additional Notes

• Uncommitted changes aren't covered. The audited commit is HEAD, so an attestation can't speak to uncommitted changes in the checkout. That's acceptable for FAIL-only evidence.
• Test-only fix. test_no_pass_from_signals._shipped_control now deep-copies the cached framework config instead of mutating it. The mutation leaked step settings between tests.

🤖 Generated with Claude Code