From b134ceabd65e3ecb4d13e6fdf3e948344b8582cc Mon Sep 17 00:00:00 2001 From: Brian Deitte Date: Fri, 28 Aug 2026 15:03:06 -0500 Subject: [PATCH 001/106] docs: add design spec for control tagging, traceability export, part11 example Covers @control-* Gherkin tagging convention, a new `vip trace` command that joins tagged results.json output against a supplied control list, and a new opt-in examples/part11_validation extension as a worked starting point. PDF generation and cross-run history are explicitly out of scope. --- .../2026-08-28-part11-traceability-design.md | 174 ++++++++++++++++++ 1 file changed, 174 insertions(+) create mode 100644 docs/superpowers/specs/2026-08-28-part11-traceability-design.md diff --git a/docs/superpowers/specs/2026-08-28-part11-traceability-design.md b/docs/superpowers/specs/2026-08-28-part11-traceability-design.md new file mode 100644 index 00000000..a0d6183d --- /dev/null +++ b/docs/superpowers/specs/2026-08-28-part11-traceability-design.md @@ -0,0 +1,174 @@ +# Design: control tagging, traceability export, and a Part 11 example + +## Context + +Someone is looking for VIP to produce something close to an automated 21 CFR +Part 11 traceability matrix: a mapping from regulatory control to timestamped +test evidence. VIP already produces timestamped, versioned, machine-readable +per-check results (`report/results.json`, JUnit XML, SARIF), and its Gherkin +feature files are already written as plain-language requirement statements. +What's missing is a way to attach a control ID to a scenario, a way to turn +that into an actual matrix, and a worked example showing what real Part +11-flavored scenarios look like. + +PDF rendering of the final matrix is being handled by a separate team and is +explicitly out of scope here. A historical/longitudinal evidence store across +runs (needed for a true deviation log) was considered and explicitly dropped +for this round — VIP already emits one timestamped file per run; whoever owns +long-term archiving can accumulate those without any new VIP storage code. + +## Goals + +- Let a scenario declare which regulatory control(s) it satisfies, using a + mechanism that's already wired into VIP's reporting pipeline. +- Produce a control -> scenario -> status -> timestamp matrix from a + `results.json`, in a form a downstream PDF/report generator (or a + spreadsheet) can consume directly. +- Ship a worked, opt-in example of real Part 11-flavored scenarios so + customers have a concrete starting point rather than an abstract mechanism. +- Document all of the above so it doesn't rot as an undiscovered feature. + +## Non-goals + +- No PDF generation (a separate team owns that). +- No historical/deviation tracking across multiple runs. +- No VIP-shipped canonical CFR Part 11 control taxonomy. The control list is + supplied by whoever owns the regulatory mapping; VIP stays + regulation-agnostic the same way it doesn't hardcode what "GxP" means today. + +## Design + +### 1. Control tagging convention + +A scenario that satisfies a compliance control carries one or more Gherkin +tags of the form `@control-`, where `` is a free-form identifier +chosen by whoever owns the control list (e.g. `@control-cfr-11-10-e`, +`@control-audit-trail-publish`). This is a plain Gherkin tag, not a new VIP +mechanism: + +- A scenario can carry multiple `@control-*` tags (one test can satisfy + several controls). +- A control can be satisfied by multiple scenarios (tag all of them with the + same slug). +- pytest-bdd's default `pytest_bdd_apply_tag` hook turns any tag into a + pytest marker via `getattr(pytest.mark, tag)` — this works with hyphens + since it's dynamic attribute access, not literal Python attribute syntax. +- VIP's plugin already captures every marker on a test into + `TestResult.markers` (`plugin.py` around the `iter_markers()` call feeding + `pytest_runtest_logreport`), which already flows into `results.json` and + the SARIF `ruleId`/ `logicalLocations` output. No plugin or reporting + schema change is needed to carry the tag through. + +Data flow end to end: + +``` +.feature @control- tag + -> pytest-bdd marker (pytest_bdd_apply_tag) + -> TestResult.markers in results.json / SARIF + -> vip trace joins against a supplied control list + -> matrix (CSV/JSON) + -> consumed by the PDF pipeline (out of scope here) +``` + +### 2. Traceability export (`vip trace`) + +A new pure function in `src/vip/reporting.py`: + +```python +def build_traceability_matrix( + data: ReportData, + controls: dict[str, str], + tag_prefix: str = "control-", +) -> TraceabilityMatrix: + ... +``` + +For each `control_id` in the supplied `controls` dict (control ID -> +description), it scans `data.results` for any result whose `markers` +includes `f"{tag_prefix}{control_id}"` and produces one `ControlEntry`: + +- `control_id`, `description` (from the `controls` dict) +- the matching scenario title(s) +- each match's `status` (reusing `TestResult.status`, so an N/A-by-version + skip renders distinctly from an ordinary skip) +- the run's `generated_at` / `vip_version` for provenance + +A control with zero matching scenarios is reported as a coverage gap rather +than silently omitted. A `control-*` tag found in the results that is *not* +present in the supplied `controls` dict is reported as an "unrecognized +control tag" warning — this catches typos (`@control-cfr-11-10e` vs +`@control-cfr-11-10-e`) at export time instead of silently losing coverage. + +CLI surface (new subcommand in `src/vip/cli.py`): + +``` +vip trace --results report/results.json --controls path/to/controls.toml \ + [--tag-prefix control-] [--format csv|json] [--output path] +``` + +`controls.toml` is a simple `[controls]` table of `control_id = description` +pairs, loaded with `tomllib` (already imported in `reporting.py`). Output +defaults to stdout as CSV (the natural input to a downstream PDF/qualification +protocol generator or a spreadsheet); JSON is available for programmatic +consumption. + +### 3. New example: `examples/part11_validation/` + +Mirrors the existing `examples/cross_product_validation/` structure and the +four-layer architecture, registered in the template registry (added in #611) +so it's reachable via `vip scaffold --template part11-validation --output DIR`: + +- `test_part11_validation.feature` — a small, illustrative scenario set, each + tagged with both a product marker (`@connect`/`@workbench`) and a + `@control-*` tag: + - Audit trail on publish: Connect records actor + timestamp when content is + deployed (`@control-audit-trail-publish`), verified via the Connect API + client. + - Privileged-action access control: a non-admin user is denied a + privileged action, e.g. deleting another user's content + (`@control-access-control-privileged-action`). Illustrative here, not a + duplicate of `security/test_auth_policy.py` — the README points there + for the fuller reference implementation. + - Audit-log non-deletability: a non-admin cannot delete or alter an + existing audit-trail entry via the API (`@control-record-retention`). +- `test_part11_validation.py` — thin step definitions; logic pushed into + `clients/connect.py` (extended only if a needed method doesn't already + exist). +- `conftest.py` — override-fixture pattern matching the existing example + (e.g. which privileged action to exercise). +- `controls.toml` — a sample control list (the 3 entries above) demonstrating + the format `vip trace` expects, so a customer sees a working example of + both halves: tagged scenarios and the control list that names them. +- `README.md` — states plainly that this is a *template*, not a certified + Part 11 test set. Customers replace/extend `controls.toml` and add their + own scenarios for their actual regulatory mapping. Documents the + `@control-*` convention and points at `vip trace`. + +### 4. Testing + +- `selftests/` coverage for `build_traceability_matrix`: full coverage, a + coverage gap, an unrecognized-tag warning, one control satisfied by + multiple scenarios. +- A selftest exercising the `vip trace` CLI end-to-end against a fixture + `results.json` + `controls.toml`, checking both CSV and JSON output. +- `examples/part11_validation` collected via `--collect-only` in CI, the same + way `cross_product_validation` already is. +- `selftests/test_scaffold_agents_md.py` — verify the new example doesn't + need an inventory update (it reuses existing fixtures/markers); update it + if that assumption turns out wrong during implementation. + +### 5. Documentation + +- `docs/test-architecture.md` — new section documenting the `@control-*` + tagging convention and how it flows into `results.json`/SARIF. +- `vip trace --help` plus a short section in the CLI/reporting docs + (`docs/reporting.md`) covering the traceability export. +- The new example's own `README.md` (above), generated from + `examples/_shared/AGENTS.md` plus its own specifics, same as + `cross_product_validation`. + +## Open questions for implementation + +- Whether `clients/connect.py` already exposes what's needed to read an + audit-trail entry and attempt its deletion as a non-admin, or whether new + client methods are required. From 7f82f2899f2abdb9c40b143c53ffff1014e001d8 Mon Sep 17 00:00:00 2001 From: Brian Deitte Date: Fri, 28 Aug 2026 15:13:01 -0500 Subject: [PATCH 002/106] docs: revise part11 traceability design after verification and research Verified the spec's claims against the codebase and a live pytest probe, and researched real-world CSV/Part 11 artifacts (vendor compliance matrices, hapi-ds/ALC). Corrected: SARIF and JUnit do not carry markers (results.json only); unregistered @control-* marks emit PytestUnknownMarkWarning and are fatal under strict warning settings, so a small plugin change is required; gherkin.py's first-tag-wins heuristic can be hijacked by a control tag; results.json is not deterministically ordered. Added: three-valued coverage so 'not verifiable by automated test' is distinct from 'coverage gap'; a richer controls.toml carrying reference, risk, verification and responsibility; a provenance header and stable sort order in the export; and a scope section explaining that most of Part 11, including all of subpart C, is not evidenceable against Posit Team. --- .../2026-08-28-part11-traceability-design.md | 311 +++++++++++++++--- 1 file changed, 269 insertions(+), 42 deletions(-) diff --git a/docs/superpowers/specs/2026-08-28-part11-traceability-design.md b/docs/superpowers/specs/2026-08-28-part11-traceability-design.md index a0d6183d..bb08fa2e 100644 --- a/docs/superpowers/specs/2026-08-28-part11-traceability-design.md +++ b/docs/superpowers/specs/2026-08-28-part11-traceability-design.md @@ -1,5 +1,8 @@ # Design: control tagging, traceability export, and a Part 11 example +Status: reviewed 2026-08-28. Section 6 records the review findings; sections 1-5 +have been amended in place where the original claims did not survive verification. + ## Context Someone is looking for VIP to produce something close to an automated 21 CFR @@ -51,20 +54,58 @@ mechanism: - A control can be satisfied by multiple scenarios (tag all of them with the same slug). - pytest-bdd's default `pytest_bdd_apply_tag` hook turns any tag into a - pytest marker via `getattr(pytest.mark, tag)` — this works with hyphens - since it's dynamic attribute access, not literal Python attribute syntax. -- VIP's plugin already captures every marker on a test into - `TestResult.markers` (`plugin.py` around the `iter_markers()` call feeding - `pytest_runtest_logreport`), which already flows into `results.json` and - the SARIF `ruleId`/ `logicalLocations` output. No plugin or reporting - schema change is needed to carry the tag through. + pytest marker via `getattr(pytest.mark, tag)` + (`pytest_bdd/plugin.py:136-139`). This works with hyphens since it's dynamic + attribute access, and pytest-bdd 8.1.0 does no tag-name validation. Feature- + and rule-level tags are applied too, not just scenario-level ones + (`pytest_bdd/scenario.py:316-319`). +- VIP's plugin captures every marker on a test, unfiltered, in + `pytest_runtest_makereport` (`plugin.py:1073-1087`), stashes it on the report + for xdist transport, and reads it back in `pytest_runtest_logreport` + (`plugin.py:1211`) into `TestResult.markers`. Verified empirically: a + scenario tagged `@connect @control-cfr-11-10-e @control-audit-trail-publish` + yields `markers: ["connect", "control-cfr-11-10-e", + "control-audit-trail-publish", "xdist_group"]` in `results.json`. + +Three corrections to the original draft of this section: + +- SARIF and JUnit do not carry markers. `write_sarif` sets `ruleId` to the + nodeid and `logicalLocations[].name` to `" / "` + (`reporting.py:313,317`); `write_junit_xml` emits only `name`/`classname`/ + `time` (`reporting.py:227-274`). The string `markers` appears in neither. + Control tags therefore flow through `results.json` only, which is the input + `vip trace` reads — so this costs nothing here, but the claim that "no + reporting schema change is needed" is only true for the JSON path. Adding + control tags to SARIF (as `properties.tags`, which the format supports) is + listed as deferred work in section 6. +- Unregistered marks warn. Each distinct `@control-*` tag raises a + `PytestUnknownMarkWarning` (`_pytest/mark/structures.py:628`). Nothing in + VIP escalates it today, so a run merely gets a noisy warnings summary — but + under `-W error::pytest.PytestUnknownMarkWarning` it is a hard collection + error, which a regulated customer running strict CI is plausibly doing. + Dynamic slugs cannot be pre-registered by name, so `plugin.py::pytest_configure` + gains one `filterwarnings` line ignoring `PytestUnknownMarkWarning` for marks + matching the configured prefix, alongside the seven `ignore:` entries already + at `plugin.py:156-173`. This is a small plugin change, not zero. +- Tag ordering in a feature file matters. `gherkin.py:52-57` derives a + feature's `"marker"` from the first token of the first tag line in the file. + A `@control-*` tag written before `@connect` hijacks that value, which feeds + the report cards (`report_html.py:241`), `generate-test-catalog.py:46`, and + `generate-feature-matrix.py:142`. Fix `gherkin.py` to skip tags matching the + control prefix when deriving the marker, rather than relying on authors to + order tags correctly. + +Auto-skip is unaffected: `_should_deselect_for_product` and `_requires_auth` +(`plugin.py:687-755`) use exact `get_closest_marker` lookups, so an extra +unknown marker is inert. Marker selection also still works — +`pytest -m "control-cfr-11-10-e"` selects correctly despite the hyphens. Data flow end to end: ``` .feature @control- tag -> pytest-bdd marker (pytest_bdd_apply_tag) - -> TestResult.markers in results.json / SARIF + -> TestResult.markers in results.json -> vip trace joins against a supplied control list -> matrix (CSV/JSON) -> consumed by the PDF pipeline (out of scope here) @@ -77,50 +118,115 @@ A new pure function in `src/vip/reporting.py`: ```python def build_traceability_matrix( data: ReportData, - controls: dict[str, str], + controls: dict[str, ControlSpec], tag_prefix: str = "control-", ) -> TraceabilityMatrix: ... ``` -For each `control_id` in the supplied `controls` dict (control ID -> -description), it scans `data.results` for any result whose `markers` -includes `f"{tag_prefix}{control_id}"` and produces one `ControlEntry`: +For each control in the supplied `controls` mapping, it scans `data.results` +for any result whose `markers` includes `f"{tag_prefix}{control_id}"` and +produces one `ControlEntry`: -- `control_id`, `description` (from the `controls` dict) -- the matching scenario title(s) +- `control_id` plus every field carried on the `ControlSpec` (below) +- the matching scenario title(s) and nodeid(s) - each match's `status` (reusing `TestResult.status`, so an N/A-by-version skip renders distinctly from an ordinary skip) -- the run's `generated_at` / `vip_version` for provenance +- `concise_error` / `skip_reason` on non-passing matches, so the matrix + carries the actual evidence and not just a verdict + +Coverage outcomes are three-valued, not two. A control with zero matching +scenarios is a coverage gap; a control declared `verification: manual` or +`verification: procedural` in the control list is reported as "not verifiable +by automated test" and is not counted as a gap. Conflating the two is the +single most misleading thing this export could do — see section 6. + +A `control-*` tag found in the results that is not present in the supplied +control list is reported as an "unrecognized control tag" warning — this +catches typos (`@control-cfr-11-10e` vs `@control-cfr-11-10-e`) at export time +instead of silently losing coverage. + +#### Control list format + +`controls.toml` uses a table per control rather than flat `id = "description"` +string pairs, so it can carry the columns real qualification matrices use +without a breaking format change later: + +```toml +[controls.cfr-11-10-e] +description = "Secure, computer-generated, time-stamped audit trails" +reference = "21 CFR 11.10(e)" +risk = "high" +verification = "automated" # automated | manual | procedural +responsibility = "shared" # posit | customer | shared +notes = "Retention duration is a customer configuration decision." +``` + +`description` is the only required key; the rest are optional and pass through +to the output verbatim. VIP does not interpret `risk`, `reference`, or +`responsibility` — it stays regulation-agnostic and just carries the customer's +own taxonomy through to the matrix. Loaded with `tomllib`, already imported at +`reporting.py:12-15`. + +#### Provenance header + +The export carries a provenance block, because a matrix without one is not a +qualification artifact. Everything below already exists in `results.json` +(`plugin.py:1273-1310`) and just needs forwarding: + +- `generated_at`, `vip_version`, `deployment_name`, `exit_status` +- per-product `url` and detected `version` from the `products` table +- `basic_mode` — surfaced prominently, because a matrix built from a + `vip verify --basic` run silently omits every `@slow` scenario and would + otherwise assert coverage that was never exercised -A control with zero matching scenarios is reported as a coverage gap rather -than silently omitted. A `control-*` tag found in the results that is *not* -present in the supplied `controls` dict is reported as an "unrecognized -control tag" warning — this catches typos (`@control-cfr-11-10e` vs -`@control-cfr-11-10-e`) at export time instead of silently losing coverage. +Two provenance caveats to render honestly rather than paper over: +`python_version` and `platform` are the VIP runner's interpreter and OS, not +the system under test, and must be labelled that way. Hostname, git SHA of the +test suite, and CI run URL are not captured at all today — see section 6. -CLI surface (new subcommand in `src/vip/cli.py`): +#### Determinism + +`results.json` is not reproducible byte-for-byte: `results` is in xdist +arrival order (default `addopts = "-n auto --dist loadgroup"`), and +`generated_at`, `run_duration_seconds`, and per-test `duration` change every +run. The export therefore sorts deterministically — by `control_id`, then by +nodeid within a control — and omits durations from the matrix rows. Two runs +against the same deployment with the same results then produce diffable +output, which is what a downstream deterministic-PDF step needs. + +CLI surface (new subcommand in `src/vip/cli.py`, following the existing +argparse + `set_defaults(func=...)` pattern used by the nine current +subcommands, and added to the `subcommand_parsers` help map at `cli.py:1936`): ``` vip trace --results report/results.json --controls path/to/controls.toml \ [--tag-prefix control-] [--format csv|json] [--output path] ``` -`controls.toml` is a simple `[controls]` table of `control_id = description` -pairs, loaded with `tomllib` (already imported in `reporting.py`). Output -defaults to stdout as CSV (the natural input to a downstream PDF/qualification -protocol generator or a spreadsheet); JSON is available for programmatic -consumption. +Output defaults to stdout as CSV (the natural input to a downstream PDF/ +qualification protocol generator or a spreadsheet); JSON is available for +programmatic consumption and carries the full provenance block. ### 3. New example: `examples/part11_validation/` Mirrors the existing `examples/cross_product_validation/` structure and the -four-layer architecture, registered in the template registry (added in #611) -so it's reachable via `vip scaffold --template part11-validation --output DIR`: +four-layer architecture, registered in `_SCAFFOLD_TEMPLATES` +(`cli.py:1084-1097`, one dict entry mapping `part11-validation` -> +`examples/part11_validation`) so it's reachable via +`vip scaffold --template part11-validation --output DIR`. + +Note that `examples/cross_product_validation/` is already described as the GxP +example (`docs/test-architecture.md:333`) and its feature narrative already +says "So that GxP and other compliance requirements are continuously met". +Decide during implementation whether this is a third example or a control-tagged +extension of that one; if it stays separate, both READMEs must cross-link so +customers don't have to guess which is the GxP starting point. - `test_part11_validation.feature` — a small, illustrative scenario set, each tagged with both a product marker (`@connect`/`@workbench`) and a - `@control-*` tag: + `@control-*` tag, with the product tag written first (see the `gherkin.py` + footgun in section 1): - Audit trail on publish: Connect records actor + timestamp when content is deployed (`@control-audit-trail-publish`), verified via the Connect API client. @@ -133,24 +239,34 @@ so it's reachable via `vip scaffold --template part11-validation --output DIR`: existing audit-trail entry via the API (`@control-record-retention`). - `test_part11_validation.py` — thin step definitions; logic pushed into `clients/connect.py` (extended only if a needed method doesn't already - exist). + exist). Every `@scenario` function also carries a literal + `@pytest.mark.connect` / `@pytest.mark.workbench` decorator — per CLAUDE.md, + feature-level Gherkin tags alone do not drive auto-skip in extension + directories. - `conftest.py` — override-fixture pattern matching the existing example (e.g. which privileged action to exercise). -- `controls.toml` — a sample control list (the 3 entries above) demonstrating - the format `vip trace` expects, so a customer sees a working example of - both halves: tagged scenarios and the control list that names them. -- `README.md` — states plainly that this is a *template*, not a certified - Part 11 test set. Customers replace/extend `controls.toml` and add their - own scenarios for their actual regulatory mapping. Documents the - `@control-*` convention and points at `vip trace`. +- `controls.toml` — a sample control list demonstrating the format `vip trace` + expects. It includes at least one `verification = "procedural"` entry and one + `responsibility = "customer"` entry, so the worked example shows the + not-automatable path and not only the happy path. +- `README.md` — states plainly that this is a template, not a certified Part 11 + test set, and carries the scope disclaimer from section 6. Customers replace + and extend `controls.toml` and add their own scenarios for their actual + regulatory mapping. Documents the `@control-*` convention and points at + `vip trace`. ### 4. Testing - `selftests/` coverage for `build_traceability_matrix`: full coverage, a - coverage gap, an unrecognized-tag warning, one control satisfied by - multiple scenarios. + coverage gap, a not-automatable control, an unrecognized-tag warning, one + control satisfied by multiple scenarios, and stable sort order across two + differently-ordered input result lists. - A selftest exercising the `vip trace` CLI end-to-end against a fixture `results.json` + `controls.toml`, checking both CSV and JSON output. +- A selftest asserting a `@control-*` tag does not hijack + `gherkin.py`'s derived feature marker. +- A selftest asserting a `@control-*` tag raises no warning under the plugin's + filter — run it under `-W error::pytest.PytestUnknownMarkWarning`. - `examples/part11_validation` collected via `--collect-only` in CI, the same way `cross_product_validation` already is. - `selftests/test_scaffold_agents_md.py` — verify the new example doesn't @@ -160,15 +276,126 @@ so it's reachable via `vip scaffold --template part11-validation --output DIR`: ### 5. Documentation - `docs/test-architecture.md` — new section documenting the `@control-*` - tagging convention and how it flows into `results.json`/SARIF. -- `vip trace --help` plus a short section in the CLI/reporting docs - (`docs/reporting.md`) covering the traceability export. + tagging convention, the tag-ordering rule, and how it flows into + `results.json`. +- `docs/reporting.md` — currently documents none of `results.json`, `junit.xml`, + or `results.sarif`. Document the machine-readable outputs there first, then + add the traceability export on top; `vip trace --help` alone is not discovery. - The new example's own `README.md` (above), generated from `examples/_shared/AGENTS.md` plus its own specifics, same as `cross_product_validation`. +## 6. Review findings + +### 6.1 Scope: what an automated matrix can and cannot claim + +This is the most important finding and it changes what the export must be able +to express. + +Published vendor Part 11 matrices (Beckman QbD1200, Microtrac) use five +columns: CFR section, regulation text, compliance yes/no, vendor +implementation, and customer responsibilities. That last column exists because +most of Part 11 is a shared or wholly-customer obligation. + +Mapping the clauses against what VIP can assert about a Posit Team deployment: +roughly six are genuinely testable (11.10(a) validation, 11.10(d) access +limits, 11.10(e) audit trails, 11.10(g) authority checks, 11.30 open-system +transport controls, and partially 11.10(b) record copies); about five are +shared; and the remainder are procedural (11.10(i), 11.10(j)) or are properties +of the customer's application rather than of Posit Team. + +Critically, that remainder includes every clause the requester named first. +Posit Team does not implement electronic signatures, so 11.50 (signature +manifestations), 11.70 (signature/record linking), and all of subpart C +(11.100/11.200/11.300) cannot be evidenced by a test against Connect, +Workbench, or Package Manager. VIP's TOTP support proves an MFA login path, +which is not the same thing as a compliant signing ceremony and must not be +tagged as though it were. + +Consequences, all folded into sections 2 and 3 above: + +- The control list carries `verification` and `responsibility` fields. +- "Not verifiable by automated test" is a distinct outcome from "coverage gap". +- The example ships a procedural and a customer-responsibility control so the + distinction is visible in the worked example. +- The example README states that a fully green matrix is evidence for the + subset of controls a customer chose to automate, and is not a Part 11 + compliance attestation. + +### 6.2 Verified as stated + +Hyphenated control tags reach `results.json` unfiltered; auto-skip is +unaffected; `pytest -m` selection works; `reporting.py` already imports +`tomllib`; the `_SCAFFOLD_TEMPLATES` registry takes a new template in one dict +entry; `cli.py` uses plain argparse throughout. `TestResult` already carries +`scenario_title`, `feature_description`, `status`, `longrepr`, +`concise_error`, `skip_reason`, and `na_version` — enough evidence per row +without a schema change. + +### 6.3 Corrected in place + +SARIF/JUnit do not carry markers (section 1); unregistered marks warn and are +fatal under strict warning settings, so a small plugin change is required +(section 1); `gherkin.py` derives a feature's marker from the first tag and can +be hijacked (section 1); `results.json` is not deterministically ordered +(section 2); the captured `python_version`/`platform` describe the runner, not +the system under test (section 2). + +### 6.4 Deferred, considered and named + +Not scope creep — recorded so a later round doesn't rediscover them: + +- Attributability. `results.json` has no hostname, no git SHA for the test + suite, and no CI run URL. For evidence sourced from CI/CD — which is exactly + what was asked for — those are the fields that make a result attributable to + a specific pipeline execution. Adding them is a few lines in + `plugin.py:1298-1310` and is the highest-value follow-up. +- Per-test timestamps. Only run-level `generated_at` exists; individual results + carry `duration` but no start time. "Timestamped test outputs" is currently + true at run granularity only. +- Schema version. `results.json` has no version field, and a downstream PDF + generator consuming it will want one. +- Step-level evidence. Real RTMs cite a protocol step ("OQ, Test Case 3, + Step 52"). VIP captures nothing below the scenario: no + `pytest_bdd_after_step` / `pytest_bdd_step_error` hooks are implemented. + Gherkin Given/When/Then steps are the natural analogue, and pytest-bdd ships + a step-level emitter (`pytest_bdd/cucumber_json.py`) that VIP does not enable. +- Captured stdout/log as evidence. `longrepr` is nulled for skips and no + `capstdout`/`caplog` is retained, so a failure row carries a traceback but no + surrounding output. +- Deviation log. A structured failure record (expected vs actual, control + impacted, disposition) is what regulated customers mean by a deviation log. + This needs cross-run history, which stays a non-goal for now. +- Tamper-evidence. No checksum of `results.json` is emitted. A SHA-256 in the + matrix provenance block is cheap and directly serves the ALCOA+ "original" + and "accurate" attributes. +- SARIF `properties.tags` for control tags, if anything downstream wants to + read the mapping from SARIF rather than JSON. + +### 6.5 On the cited references + +The "GxP AI Validation framework" could not be located as a GitHub repository; +the search surfaces vendor and consultancy material rather than open source, +matching the requester's own inability to find an example. + +AlcoaBase resolves to `hapi-ds/ALC` — a single-star, early-stage project, so +useful as a structural reference rather than as an established standard. Two +things in it are worth borrowing and are reflected above: its +`Requirements/RiskBasedTesting.md` presents a final traceability matrix as +`URS ID | Risk | Test Case ID | Status`, which is why `risk` and `reference` +became control-list fields; and its `docs/test-protocols/*.md` show the +per-step protocol shape (action, command, expected result, actual result, +status, tester, timestamp) that motivates the deferred step-level item. + +It also validates the PDF non-goal: AlcoaBase is itself the deterministic-PDF +and document-management layer. The right division of labour is for VIP to +produce clean, deterministic, well-provenanced machine-readable input and for +that layer to render it. + ## Open questions for implementation - Whether `clients/connect.py` already exposes what's needed to read an audit-trail entry and attempt its deletion as a non-admin, or whether new client methods are required. +- Whether `examples/part11_validation` should be a third example or a + control-tagged extension of `examples/cross_product_validation` (section 3). From bf2963f707a7c2f074ec9ee801858db008b8fdbf Mon Sep 17 00:00:00 2001 From: Brian Deitte Date: Fri, 28 Aug 2026 15:28:03 -0500 Subject: [PATCH 003/106] docs: promote evidence-record provenance into the part11 design MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds a new section 2 covering four items previously listed as deferred: attributability (hostname, git SHA, CI run URL), per-test start/finish timestamps, a results.json schema_version with a compatibility policy, and a SHA-256 sidecar for tamper-evidence. Renumbers the following sections. Verified against the code first: report.start/stop exist and survive xdist worker-to-controller serialization (checked under -n 0 and -n 2), and the collection site fires on the call phase or a setup-phase skip, so started_at excludes fixture setup — documented rather than glossed. Two risks called out with the design: CI checkouts embed credentials in the git remote, so userinfo is redacted before recording (results.json is an uploaded artifact, and plugin.py already strips absolute paths for the same reason); and the checksum is tamper-evidence within a trusted pipeline, not tamper-proofing, since anyone who can edit the file can regenerate the sidecar. --- .../2026-08-28-part11-traceability-design.md | 323 ++++++++++++++---- 1 file changed, 254 insertions(+), 69 deletions(-) diff --git a/docs/superpowers/specs/2026-08-28-part11-traceability-design.md b/docs/superpowers/specs/2026-08-28-part11-traceability-design.md index bb08fa2e..546807b5 100644 --- a/docs/superpowers/specs/2026-08-28-part11-traceability-design.md +++ b/docs/superpowers/specs/2026-08-28-part11-traceability-design.md @@ -1,7 +1,9 @@ # Design: control tagging, traceability export, and a Part 11 example -Status: reviewed 2026-08-28. Section 6 records the review findings; sections 1-5 -have been amended in place where the original claims did not survive verification. +Status: reviewed and revised 2026-08-28. Section 7 records the review findings; +sections 1-6 have been amended in place where the original claims did not +survive verification. Section 2 (evidence record additions) was promoted out of +the deferred list into the main design on the second pass. ## Context @@ -11,8 +13,9 @@ test evidence. VIP already produces timestamped, versioned, machine-readable per-check results (`report/results.json`, JUnit XML, SARIF), and its Gherkin feature files are already written as plain-language requirement statements. What's missing is a way to attach a control ID to a scenario, a way to turn -that into an actual matrix, and a worked example showing what real Part -11-flavored scenarios look like. +that into an actual matrix, enough provenance on the evidence record to make a +result attributable to a specific pipeline execution, and a worked example +showing what real Part 11-flavored scenarios look like. PDF rendering of the final matrix is being handled by a separate team and is explicitly out of scope here. A historical/longitudinal evidence store across @@ -24,6 +27,9 @@ long-term archiving can accumulate those without any new VIP storage code. - Let a scenario declare which regulatory control(s) it satisfies, using a mechanism that's already wired into VIP's reporting pipeline. +- Make each result attributable to a specific pipeline execution and to a + specific point in time, and make the evidence file versioned and + tamper-evident. - Produce a control -> scenario -> status -> timestamp matrix from a `results.json`, in a form a downstream PDF/report generator (or a spreadsheet) can consume directly. @@ -38,6 +44,8 @@ long-term archiving can accumulate those without any new VIP storage code. - No VIP-shipped canonical CFR Part 11 control taxonomy. The control list is supplied by whoever owns the regulatory mapping; VIP stays regulation-agnostic the same way it doesn't hardcode what "GxP" means today. +- No cryptographic signing of the evidence record. See the honest limits of + the checksum in section 2.4. ## Design @@ -77,7 +85,7 @@ Three corrections to the original draft of this section: `vip trace` reads — so this costs nothing here, but the claim that "no reporting schema change is needed" is only true for the JSON path. Adding control tags to SARIF (as `properties.tags`, which the format supports) is - listed as deferred work in section 6. + listed as deferred work in section 7. - Unregistered marks warn. Each distinct `@control-*` tag raises a `PytestUnknownMarkWarning` (`_pytest/mark/structures.py:628`). Nothing in VIP escalates it today, so a run merely gets a noisy warnings summary — but @@ -97,8 +105,9 @@ Three corrections to the original draft of this section: Auto-skip is unaffected: `_should_deselect_for_product` and `_requires_auth` (`plugin.py:687-755`) use exact `get_closest_marker` lookups, so an extra -unknown marker is inert. Marker selection also still works — -`pytest -m "control-cfr-11-10-e"` selects correctly despite the hyphens. +unknown marker is inert. Marker selection also still works — verified with a +negative control: `pytest -m "control-cfr-11-10-e"` selects the tagged +scenario, `-m "control-nonexistent"` deselects it. Data flow end to end: @@ -111,7 +120,131 @@ Data flow end to end: -> consumed by the PDF pipeline (out of scope here) ``` -### 2. Traceability export (`vip trace`) +### 2. Evidence record: additions to `results.json` + +These are producer-side changes in `src/vip/plugin.py` and the `ReportData` +model in `src/vip/reporting.py`. They are not specific to Part 11 — they make +`results.json` a defensible evidence record for any audience — but they are +what turns "a test passed somewhere, sometime" into "this check ran at this +instant, on this host, from this commit, in this pipeline execution." + +All four are additive. Nothing existing is renamed or removed, so the HTML +report, JUnit and SARIF writers are unaffected. + +#### 2.1 Attributability + +A new nested `execution` block in the payload at `plugin.py:1299-1310`: + +```json +"execution": { + "hostname": "runner-04.example.com", + "git": {"commit": "b134ceab...", "branch": "main", "dirty": false, + "remote": "https://github.com/posit-dev/vip"}, + "ci": {"provider": "github", "run_id": "1234567890", + "run_attempt": "1", "run_url": "https://github.com/o/r/actions/runs/1234567890", + "job": "connect-smoke"} +} +``` + +Resolution rules, in order, each independently degrading to `null`: + +- `hostname` from `platform.node()`. This is the VIP runner's host, not the + system under test, and must be labelled that way wherever it is rendered — + the same trap `python_version`/`platform` already fall into (section 3). +- `git` from CI environment variables first (`GITHUB_SHA`, `GITHUB_REF_NAME`), + falling back to `git rev-parse HEAD` / `--abbrev-ref HEAD` / + `git status --porcelain` run in the current working directory with a short + timeout. The cwd is the right repo to interrogate: that is where `vip.toml` + and any `--vip-extensions` directories live. VIP core's own provenance is + already covered by `vip_version`. Document that limitation rather than + trying to resolve a SHA per extension directory. +- `ci` from environment variables only, never a subprocess. GitHub Actions + (`GITHUB_RUN_ID` + `GITHUB_SERVER_URL` + `GITHUB_REPOSITORY` composed into + `run_url`), GitLab (`CI_JOB_URL`), Jenkins (`BUILD_URL`). Unknown CI, or no + CI, yields `null` rather than a guess. + +Three constraints on the implementation: + +- Never fail a run. Every probe is wrapped; a missing `git` binary, a detached + worktree, a non-repo cwd, or a subprocess timeout produces `null`, never an + exception and never a warning that would pollute a clean run. +- Redact userinfo from the remote URL. CI checkouts routinely rewrite the + origin to embed a credential (`https://x-access-token:ghs_...@github.com/...`). + `results.json` is an uploaded CI artifact — `plugin.py:1196-1201` already + strips absolute paths for exactly this reason — so the remote is parsed and + any userinfo component dropped before it is recorded. A remote that cannot + be parsed is recorded as `null`, not passed through raw. +- Provide an opt-out. `--vip-no-attribution` omits the whole `execution` block. + Hostname and repository identity are modest but real infrastructure + disclosure, and some customers will not want them in an artifact that leaves + their network. Default is on, because for the use case driving this work + these fields are the point. + +#### 2.2 Per-test timestamps + +Each entry in `results` gains `started_at` and `finished_at`, UTC ISO 8601, +derived from `report.start` and `report.stop` at `plugin.py:1203-1216`. + +Verified empirically: `report.start`/`report.stop` are epoch floats present on +`TestReport`, and they survive xdist's worker-to-controller serialization — +confirmed under both `-n 0` and `-n 2`, which matters because the controller is +the only process that writes the report (`plugin.py:1173-1176`). + +One precision point to document rather than gloss: the collection site fires +for `report.when == "call"`, or for a setup-phase skip +(`plugin.py:1185`). So `started_at` is when the check itself began, excluding +fixture setup, except for setup-skips where it is the setup start. That is the +right semantic for "when was this control exercised", but it is not the same as +"when did this test item begin", and a qualification document that says the +latter would be wrong. Both fields fall back to `null` via `getattr` if a +future pytest drops the attributes. + +This is what makes the phrase "timestamped test evidence" true at per-check +granularity rather than only at run granularity. + +#### 2.3 Schema version + +A `schema_version` string at the top of the payload, introduced as `"1.0"` +together with these additions. + +Semantics: bump the minor for additive changes (a new field), bump the major +for a removal, a rename, or a change in the meaning of an existing field. A +consumer should accept an unknown minor and refuse an unknown major. The +historical unversioned shape is "pre-1.0" — a consumer seeing no +`schema_version` at all knows it predates this work. + +`vip trace` implements exactly that policy: unknown minor proceeds, unknown +major is a hard error naming both versions. A downstream PDF generator gets a +stable contract to code against instead of shape-sniffing. + +#### 2.4 Tamper-evidence + +The plugin writes a `results.json.sha256` sidecar immediately after +`p.write_text(...)` at `plugin.py:1315`, in the standard +` results.json` format so `shasum -c` verifies it directly. It is written +unconditionally, not gated on `--vip-format`, because the checksum is a +property of the evidence file rather than an output format. + +The digest covers the exact bytes written. Note that `results.json` is written +without a trailing newline while `failures.json` adds one +(`plugin.py:1315` vs `:1344`) — hash the bytes, not a re-serialization. + +`vip trace` then recomputes the digest of the file it reads, records it in the +matrix provenance block as `results_sha256`, and — if the sidecar is present — +compares. A mismatch is a hard error, not a warning. A checksum mismatch on a +compliance artifact is precisely the condition that must not be papered over. + +The honest limit, which belongs in the docs and not just here: this is +tamper-evidence within a trusted pipeline, not tamper-proofing. Anyone able to +edit `results.json` can also regenerate the sidecar. It detects accidental +corruption, truncated uploads, and casual editing; it does not resist a +motivated forger. Real integrity requires a signature under a key the editor +does not hold, or write-once storage — deliberately out of scope (see +Non-goals). Under ALCOA+ this supports "original" and "accurate" as a detection +aid; claiming more than that would be the same category of overreach section +7.1 warns about. + +### 3. Traceability export (`vip trace`) A new pure function in `src/vip/reporting.py`: @@ -132,6 +265,7 @@ produces one `ControlEntry`: - the matching scenario title(s) and nodeid(s) - each match's `status` (reusing `TestResult.status`, so an N/A-by-version skip renders distinctly from an ordinary skip) +- each match's `started_at` / `finished_at` (section 2.2) - `concise_error` / `skip_reason` on non-passing matches, so the matrix carries the actual evidence and not just a verdict @@ -139,7 +273,7 @@ Coverage outcomes are three-valued, not two. A control with zero matching scenarios is a coverage gap; a control declared `verification: manual` or `verification: procedural` in the control list is reported as "not verifiable by automated test" and is not counted as a gap. Conflating the two is the -single most misleading thing this export could do — see section 6. +single most misleading thing this export could do — see section 7. A `control-*` tag found in the results that is not present in the supplied control list is reported as an "unrecognized control tag" warning — this @@ -171,33 +305,43 @@ own taxonomy through to the matrix. Loaded with `tomllib`, already imported at #### Provenance header The export carries a provenance block, because a matrix without one is not a -qualification artifact. Everything below already exists in `results.json` -(`plugin.py:1273-1310`) and just needs forwarding: +qualification artifact. All of it comes from `results.json`: -- `generated_at`, `vip_version`, `deployment_name`, `exit_status` +- `generated_at`, `vip_version`, `deployment_name`, `exit_status`, + `schema_version` +- the full `execution` block from section 2.1 — hostname, git commit, CI run + URL +- `results_sha256` and the sidecar verification result from section 2.4 - per-product `url` and detected `version` from the `products` table - `basic_mode` — surfaced prominently, because a matrix built from a `vip verify --basic` run silently omits every `@slow` scenario and would otherwise assert coverage that was never exercised -Two provenance caveats to render honestly rather than paper over: -`python_version` and `platform` are the VIP runner's interpreter and OS, not -the system under test, and must be labelled that way. Hostname, git SHA of the -test suite, and CI run URL are not captured at all today — see section 6. +One provenance caveat to render honestly rather than paper over: +`python_version`, `platform`, and `execution.hostname` all describe the VIP +runner, not the system under test, and must be labelled that way. The system +under test is identified by the `products` table. #### Determinism -`results.json` is not reproducible byte-for-byte: `results` is in xdist -arrival order (default `addopts = "-n auto --dist loadgroup"`), and -`generated_at`, `run_duration_seconds`, and per-test `duration` change every -run. The export therefore sorts deterministically — by `control_id`, then by -nodeid within a control — and omits durations from the matrix rows. Two runs -against the same deployment with the same results then produce diffable -output, which is what a downstream deterministic-PDF step needs. +"Deterministic" here means the export is a pure function of its inputs: the +same `results.json` and `controls.toml` produce byte-identical output. It does +not mean two separate verification runs produce identical matrices — they +cannot, and should not, because timestamps and the run's identity are the +evidence. -CLI surface (new subcommand in `src/vip/cli.py`, following the existing -argparse + `set_defaults(func=...)` pattern used by the nine current -subcommands, and added to the `subcommand_parsers` help map at `cli.py:1936`): +Two things are needed for that. `results` in `results.json` is in xdist arrival +order (default `addopts = "-n auto --dist loadgroup"`), so the export sorts +deterministically: by `control_id`, then by nodeid within a control. And +per-test `duration` is omitted from matrix rows — it is performance noise that +varies run to run without carrying evidentiary value, unlike `started_at`, +which is retained precisely because it does. + +#### CLI surface + +New subcommand in `src/vip/cli.py`, following the existing argparse + +`set_defaults(func=...)` pattern used by the nine current subcommands, and +added to the `subcommand_parsers` help map at `cli.py:1936`: ``` vip trace --results report/results.json --controls path/to/controls.toml \ @@ -206,9 +350,11 @@ vip trace --results report/results.json --controls path/to/controls.toml \ Output defaults to stdout as CSV (the natural input to a downstream PDF/ qualification protocol generator or a spreadsheet); JSON is available for -programmatic consumption and carries the full provenance block. +programmatic consumption and carries the full provenance block. The matrix +output carries its own `schema_version`, versioned independently of +`results.json`. -### 3. New example: `examples/part11_validation/` +### 4. New example: `examples/part11_validation/` Mirrors the existing `examples/cross_product_validation/` structure and the four-layer architecture, registered in `_SCAFFOLD_TEMPLATES` @@ -250,12 +396,14 @@ customers don't have to guess which is the GxP starting point. `responsibility = "customer"` entry, so the worked example shows the not-automatable path and not only the happy path. - `README.md` — states plainly that this is a template, not a certified Part 11 - test set, and carries the scope disclaimer from section 6. Customers replace + test set, and carries the scope disclaimer from section 7. Customers replace and extend `controls.toml` and add their own scenarios for their actual regulatory mapping. Documents the `@control-*` convention and points at `vip trace`. -### 4. Testing +### 5. Testing + +For the export (section 3): - `selftests/` coverage for `build_traceability_matrix`: full coverage, a coverage gap, a not-automatable control, an unrecognized-tag warning, one @@ -263,31 +411,58 @@ customers don't have to guess which is the GxP starting point. differently-ordered input result lists. - A selftest exercising the `vip trace` CLI end-to-end against a fixture `results.json` + `controls.toml`, checking both CSV and JSON output. -- A selftest asserting a `@control-*` tag does not hijack - `gherkin.py`'s derived feature marker. +- Byte-identical output across two invocations on the same input, asserting + the determinism claim rather than assuming it. + +For the tagging convention (section 1): + +- A selftest asserting a `@control-*` tag does not hijack `gherkin.py`'s + derived feature marker. - A selftest asserting a `@control-*` tag raises no warning under the plugin's filter — run it under `-W error::pytest.PytestUnknownMarkWarning`. -- `examples/part11_validation` collected via `--collect-only` in CI, the same - way `cross_product_validation` already is. -- `selftests/test_scaffold_agents_md.py` — verify the new example doesn't - need an inventory update (it reuses existing fixtures/markers); update it - if that assumption turns out wrong during implementation. -### 5. Documentation +For the evidence record (section 2), all via the `pytester` fixture so a real +subprocess run produces a real `results.json`: + +- `started_at`/`finished_at` present, ISO 8601, UTC, and ordered + `started_at <= finished_at`; present for a setup-phase skip as well as a + passing call. +- The `execution` block degrades to `null` fields rather than raising when + `git` is unavailable, when cwd is not a repo, and when no CI env vars are + set — three separate cases. +- A remote URL carrying userinfo is redacted. This is a secret-leak + regression test, so assert the token string is absent from the whole file, + not merely that the remote field looks clean. +- `--vip-no-attribution` omits the block entirely. +- The `.sha256` sidecar matches the bytes actually on disk, and `vip trace` + raises on a deliberately corrupted `results.json`. +- `schema_version` is present; `vip trace` accepts an unknown minor and + refuses an unknown major. + +Plus: `examples/part11_validation` collected via `--collect-only` in CI, the +same way `cross_product_validation` already is; and +`selftests/test_scaffold_agents_md.py` verified for whether the new example +needs an inventory update (it should not — it reuses existing fixtures and +markers — but confirm rather than assume). + +### 6. Documentation - `docs/test-architecture.md` — new section documenting the `@control-*` tagging convention, the tag-ordering rule, and how it flows into `results.json`. - `docs/reporting.md` — currently documents none of `results.json`, `junit.xml`, - or `results.sarif`. Document the machine-readable outputs there first, then - add the traceability export on top; `vip trace --help` alone is not discovery. + or `results.sarif`. Document the machine-readable outputs there first, + including the section 2 additions and the `schema_version` compatibility + policy, then add the traceability export on top; `vip trace --help` alone is + not discovery. The tamper-evidence limitation from section 2.4 goes here in + plain language, not just in this spec. - The new example's own `README.md` (above), generated from `examples/_shared/AGENTS.md` plus its own specifics, same as `cross_product_validation`. -## 6. Review findings +## 7. Review findings -### 6.1 Scope: what an automated matrix can and cannot claim +### 7.1 Scope: what an automated matrix can and cannot claim This is the most important finding and it changes what the export must be able to express. @@ -312,7 +487,7 @@ Workbench, or Package Manager. VIP's TOTP support proves an MFA login path, which is not the same thing as a compliant signing ceremony and must not be tagged as though it were. -Consequences, all folded into sections 2 and 3 above: +Consequences, all folded into sections 3 and 4 above: - The control list carries `verification` and `responsibility` fields. - "Not verifiable by automated test" is a distinct outcome from "coverage gap". @@ -322,57 +497,62 @@ Consequences, all folded into sections 2 and 3 above: subset of controls a customer chose to automate, and is not a Part 11 compliance attestation. -### 6.2 Verified as stated +### 7.2 Verified as stated Hyphenated control tags reach `results.json` unfiltered; auto-skip is -unaffected; `pytest -m` selection works; `reporting.py` already imports -`tomllib`; the `_SCAFFOLD_TEMPLATES` registry takes a new template in one dict -entry; `cli.py` uses plain argparse throughout. `TestResult` already carries -`scenario_title`, `feature_description`, `status`, `longrepr`, -`concise_error`, `skip_reason`, and `na_version` — enough evidence per row -without a schema change. +unaffected; `pytest -m` selection works (confirmed with a negative control); +`reporting.py` already imports `tomllib`; the `_SCAFFOLD_TEMPLATES` registry +takes a new template in one dict entry; `cli.py` uses plain argparse +throughout. `TestResult` already carries `scenario_title`, +`feature_description`, `status`, `longrepr`, `concise_error`, `skip_reason`, +and `na_version` — enough evidence per row without a schema change. +`report.start`/`report.stop` exist and survive xdist serialization, which is +what makes section 2.2 cheap. -### 6.3 Corrected in place +### 7.3 Corrected in place SARIF/JUnit do not carry markers (section 1); unregistered marks warn and are fatal under strict warning settings, so a small plugin change is required (section 1); `gherkin.py` derives a feature's marker from the first tag and can be hijacked (section 1); `results.json` is not deterministically ordered -(section 2); the captured `python_version`/`platform` describe the runner, not -the system under test (section 2). +(section 3); the captured `python_version`/`platform` describe the runner, not +the system under test (section 3). + +### 7.4 Promoted into the design + +Four items from the first review's deferred list were promoted into section 2 +on the second pass: attributability, per-test timestamps, schema version, and +tamper-evidence. They share a rationale — the requester asked for evidence +sourced from CI/CD, and without them a result cannot be tied to a specific +pipeline execution, a specific instant, a stable contract, or a verifiable set +of bytes. -### 6.4 Deferred, considered and named +### 7.5 Still deferred, considered and named Not scope creep — recorded so a later round doesn't rediscover them: -- Attributability. `results.json` has no hostname, no git SHA for the test - suite, and no CI run URL. For evidence sourced from CI/CD — which is exactly - what was asked for — those are the fields that make a result attributable to - a specific pipeline execution. Adding them is a few lines in - `plugin.py:1298-1310` and is the highest-value follow-up. -- Per-test timestamps. Only run-level `generated_at` exists; individual results - carry `duration` but no start time. "Timestamped test outputs" is currently - true at run granularity only. -- Schema version. `results.json` has no version field, and a downstream PDF - generator consuming it will want one. - Step-level evidence. Real RTMs cite a protocol step ("OQ, Test Case 3, Step 52"). VIP captures nothing below the scenario: no `pytest_bdd_after_step` / `pytest_bdd_step_error` hooks are implemented. Gherkin Given/When/Then steps are the natural analogue, and pytest-bdd ships a step-level emitter (`pytest_bdd/cucumber_json.py`) that VIP does not enable. + This is the largest remaining gap against how qualification protocols are + actually written. - Captured stdout/log as evidence. `longrepr` is nulled for skips and no `capstdout`/`caplog` is retained, so a failure row carries a traceback but no surrounding output. - Deviation log. A structured failure record (expected vs actual, control impacted, disposition) is what regulated customers mean by a deviation log. This needs cross-run history, which stays a non-goal for now. -- Tamper-evidence. No checksum of `results.json` is emitted. A SHA-256 in the - matrix provenance block is cheap and directly serves the ALCOA+ "original" - and "accurate" attributes. +- Cryptographic signing of the evidence record, per section 2.4. +- Runtime versions on the system under test (R and Python interpreters + available on Workbench/Connect) as part of the provenance block. VIP already + has `expected_r_versions` / `expected_python_versions` config, so the + observed values are within reach. - SARIF `properties.tags` for control tags, if anything downstream wants to read the mapping from SARIF rather than JSON. -### 6.5 On the cited references +### 7.6 On the cited references The "GxP AI Validation framework" could not be located as a GitHub repository; the search surfaces vendor and consultancy material rather than open source, @@ -398,4 +578,9 @@ that layer to render it. audit-trail entry and attempt its deletion as a non-admin, or whether new client methods are required. - Whether `examples/part11_validation` should be a third example or a - control-tagged extension of `examples/cross_product_validation` (section 3). + control-tagged extension of `examples/cross_product_validation` (section 4). +- Whether the `execution` block should also be surfaced in the HTML report's + provenance table (`report_html.py:663-689`), which today shows six fields and + renders `None` as "not recorded". Cheap, and it would make the same + attribution visible to a human reader, but it widens the diff beyond the + machine-readable path this spec is scoped to. From b868294ed99a61fe3e123205fe5f049e8686000a Mon Sep 17 00:00:00 2001 From: Brian Deitte Date: Fri, 28 Aug 2026 15:29:30 -0500 Subject: [PATCH 004/106] docs: require backward-compatible load of pre-1.0 results.json The schema_version tests covered unknown-major and unknown-minor but not the case that actually happens: an archived results.json written before this work lands, with no schema_version, timestamps, or execution block. load_results already uses .get() with defaults throughout, so the new fields follow that pattern and take dataclass defaults. --- .../specs/2026-08-28-part11-traceability-design.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/docs/superpowers/specs/2026-08-28-part11-traceability-design.md b/docs/superpowers/specs/2026-08-28-part11-traceability-design.md index 546807b5..715311df 100644 --- a/docs/superpowers/specs/2026-08-28-part11-traceability-design.md +++ b/docs/superpowers/specs/2026-08-28-part11-traceability-design.md @@ -438,6 +438,14 @@ subprocess run produces a real `results.json`: raises on a deliberately corrupted `results.json`. - `schema_version` is present; `vip trace` accepts an unknown minor and refuses an unknown major. +- A pre-1.0 `results.json` — no `schema_version`, no `started_at`, no + `execution` block — still loads and traces, with the absent fields rendered + as null rather than raising. This is the common case in practice, not the + unknown-major case: anyone with an archived results file from before this + work lands hits it. `load_results` (`reporting.py:176-190`) already uses + `.get()` with defaults for every optional field, so the new fields must + follow that existing pattern and carry dataclass defaults rather than being + required constructor arguments. Plus: `examples/part11_validation` collected via `--collect-only` in CI, the same way `cross_product_validation` already is; and From 851fe07c9c3ec4e101e2ee23adabbf0ebda213f7 Mon Sep 17 00:00:00 2001 From: Brian Deitte Date: Fri, 28 Aug 2026 15:36:52 -0500 Subject: [PATCH 005/106] docs: register control marks instead of silencing the unknown-mark warning Review found the filterwarnings approach does not cover pytest's --strict-markers mode. Confirmed against the pinned pytest 9.1.1: strict markers calls fail() and aborts collection rather than warning, so no warning filter can reach it. Replaced with a pytest_configure pre-scan that registers each discovered control-* tag via addinivalue_line. Verified this collects and passes under --strict-markers and -W error together, removes the warnings entirely, and still delivers the tags to results.json -- one mechanism for both failure modes, so the filterwarnings entry is dropped. Test plan now covers both paths separately and combined. Also corrected the Context, which still described results.json as producing 'timestamped, versioned' per-check results. That is the post-section-2 state, not today's: only a run-level generated_at exists and there is no schema version, so a per-control row cannot currently claim its own timestamp. --- .../2026-08-28-part11-traceability-design.md | 63 ++++++++++++++----- 1 file changed, 49 insertions(+), 14 deletions(-) diff --git a/docs/superpowers/specs/2026-08-28-part11-traceability-design.md b/docs/superpowers/specs/2026-08-28-part11-traceability-design.md index 715311df..9be7ef4f 100644 --- a/docs/superpowers/specs/2026-08-28-part11-traceability-design.md +++ b/docs/superpowers/specs/2026-08-28-part11-traceability-design.md @@ -9,9 +9,18 @@ the deferred list into the main design on the second pass. Someone is looking for VIP to produce something close to an automated 21 CFR Part 11 traceability matrix: a mapping from regulatory control to timestamped -test evidence. VIP already produces timestamped, versioned, machine-readable -per-check results (`report/results.json`, JUnit XML, SARIF), and its Gherkin -feature files are already written as plain-language requirement statements. +test evidence. VIP already produces machine-readable per-check results +(`report/results.json`, JUnit XML, SARIF), and its Gherkin feature files are +already written as plain-language requirement statements. + +Be precise about what "timestamped, versioned" does and does not mean today, +because the downstream PDF pipeline will build on this wording. Before the +section 2 changes land, `results.json` carries a single run-level +`generated_at` and no schema version at all: individual results have a +`duration` but no start time. So the existing evidence is run-timestamped and +unversioned, and a per-control row could not honestly claim a timestamp of its +own. Section 2 is what makes per-check timestamping and a versioned schema +true rather than aspirational. What's missing is a way to attach a control ID to a scenario, a way to turn that into an actual matrix, enough provenance on the evidence record to make a result attributable to a specific pipeline execution, and a worked example @@ -86,15 +95,35 @@ Three corrections to the original draft of this section: reporting schema change is needed" is only true for the JSON path. Adding control tags to SARIF (as `properties.tags`, which the format supports) is listed as deferred work in section 7. -- Unregistered marks warn. Each distinct `@control-*` tag raises a - `PytestUnknownMarkWarning` (`_pytest/mark/structures.py:628`). Nothing in - VIP escalates it today, so a run merely gets a noisy warnings summary — but - under `-W error::pytest.PytestUnknownMarkWarning` it is a hard collection - error, which a regulated customer running strict CI is plausibly doing. - Dynamic slugs cannot be pre-registered by name, so `plugin.py::pytest_configure` - gains one `filterwarnings` line ignoring `PytestUnknownMarkWarning` for marks - matching the configured prefix, alongside the seven `ignore:` entries already - at `plugin.py:156-173`. This is a small plugin change, not zero. +- Unregistered marks must be registered, not merely silenced. Each distinct + `@control-*` tag is an unregistered mark, and pytest punishes that two + different ways. By default it raises `PytestUnknownMarkWarning` + (`_pytest/mark/structures.py:628`), which is a hard error under + `-W error::pytest.PytestUnknownMarkWarning`. Under `--strict-markers` / + `strict_markers` it does not warn at all — it calls `fail()` and aborts + collection with "`'control-audit-trail-publish' not found in markers + configuration option`". Verified against the pinned pytest 9.1.1. + + A `filterwarnings` ignore therefore does not solve this: it cannot touch the + strict-markers path, which is precisely the mode a regulated customer's CI + is most likely to enable. Instead, `plugin.py::pytest_configure` pre-scans + the feature files it is about to collect, extracts every tag matching the + control prefix, and registers each one via + `config.addinivalue_line("markers", f"{tag}: compliance control tag")`. + + Verified empirically: with pre-registration, a control-tagged scenario + collects and passes under `--strict-markers` and + `-W error::pytest.PytestUnknownMarkWarning` together, the warnings disappear + entirely, and the tags still arrive in `results.json`. One mechanism covers + both failure modes, so no `filterwarnings` entry is needed at all. + + Two implementation notes. The scan must cover the same roots pytest will + collect — `config.args` plus any `--vip-extensions` directories — rather than + blindly walking `rootpath`, which would be wasteful in a large monorepo and + would register controls from files that are not part of this run. And VIP + already has a Gherkin tag parser in `gherkin.py`; reuse it rather than adding + a second regex, keeping this consistent with the `gherkin.py` fix in the next + bullet. - Tag ordering in a feature file matters. `gherkin.py:52-57` derives a feature's `"marker"` from the first token of the first tag line in the file. A `@control-*` tag written before `@connect` hijacks that value, which feeds @@ -418,8 +447,14 @@ For the tagging convention (section 1): - A selftest asserting a `@control-*` tag does not hijack `gherkin.py`'s derived feature marker. -- A selftest asserting a `@control-*` tag raises no warning under the plugin's - filter — run it under `-W error::pytest.PytestUnknownMarkWarning`. +- A selftest asserting a `@control-*` tag collects cleanly under + `--strict-markers` and under `-W error::pytest.PytestUnknownMarkWarning`, as + separate cases and combined. Both must be covered: they are distinct code + paths in pytest (`fail()` vs `warnings.warn`), and a fix for one does not + imply a fix for the other. +- A selftest asserting the control tags still reach `results.json` after + pre-registration, so a future change to the registration mechanism cannot + silently drop the evidence it exists to preserve. For the evidence record (section 2), all via the `pytester` fixture so a real subprocess run produces a real `results.json`: From 51e4803bd13a0ea96c813f8bbdda8a7843774c37 Mon Sep 17 00:00:00 2001 From: Brian Deitte Date: Fri, 28 Aug 2026 15:47:36 -0500 Subject: [PATCH 006/106] docs: add implementation plan for part11 control traceability Twelve TDD tasks in three shipping phases: evidence-record provenance (schema version, per-test timestamps, execution attribution, sha256 sidecar), control tagging (gherkin marker derivation plus strict-marker-safe registration), then the traceability export, example and docs. Deviates from the spec in one place: build_traceability_matrix goes in a new src/vip/traceability.py rather than reporting.py, which already holds the data model and three writers. Follows the existing report_html.py split precedent. --- .../plans/2026-08-28-part11-traceability.md | 2578 +++++++++++++++++ 1 file changed, 2578 insertions(+) create mode 100644 docs/superpowers/plans/2026-08-28-part11-traceability.md diff --git a/docs/superpowers/plans/2026-08-28-part11-traceability.md b/docs/superpowers/plans/2026-08-28-part11-traceability.md new file mode 100644 index 00000000..82e356a3 --- /dev/null +++ b/docs/superpowers/plans/2026-08-28-part11-traceability.md @@ -0,0 +1,2578 @@ +# Part 11 Control Tagging and Traceability Export Implementation Plan + +> For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +Goal: Let a Gherkin scenario declare the compliance control it satisfies, harden `results.json` into an attributable evidence record, and add a `vip trace` command that joins the two into a traceability matrix. + +Architecture: Three layers, built bottom-up. First `results.json` gains provenance (schema version, per-test timestamps, execution attribution, a checksum sidecar) — this ships value on its own. Then `@control-*` Gherkin tags are registered as real pytest markers so they survive strict-marker CI and reach `results.json`. Finally a new `src/vip/traceability.py` joins a customer-supplied `controls.toml` against those markers and renders CSV/JSON for a downstream PDF pipeline. + +Tech Stack: Python 3.10+, pytest 9.1.1, pytest-bdd 8.1.0, pytest-xdist, argparse, tomllib, stdlib `csv`/`hashlib`/`subprocess`. No new dependencies. + +Spec: `docs/superpowers/specs/2026-08-28-part11-traceability-design.md` + +## Global Constraints + +- Run everything through `uv run`. Never bare `python` or `pip`. +- Ruff is linter and formatter, line length 100, rules `E`, `F`, `I`, `UP`. CI pins ruff 0.15.0. Lint paths must include `examples/`: `uv run ruff check src/ src/vip_tests/ selftests/ examples/`. If `uv run ruff` is unavailable locally, use `uvx ruff@0.15.0`. +- Selftests run randomized. Never pass `-p no:randomly` — CI runs them randomized and disabling the plugin hides the order-dependent failures it exists to catch. +- Never import a pytest-bdd step module (anything under `src/vip_tests/**` calling `@scenario`/`scenarios()`) from inside a selftest. `@scenario` inspects the caller's frame at import time and raises `IndexError`. Use `pytester` subprocess runs or `--collect-only` instead. +- Every new field added to `results.json` must be optional on the dataclass and read with `.get()` in `load_results`. `ReportData`'s existing docstring (`reporting.py:114-119`) states the rule: defaults are `None` rather than a concrete-looking value so an older `results.json` loads as "not recorded" instead of silently claiming a value that was never measured. +- Never fail or warn a verification run because provenance collection failed. Every probe degrades to `None`. +- Commit after every task. Conventional-commit titles, lowercase description, no trailing period, under 70 chars. +- No `**` bold in any markdown file this plan creates or edits. + +## Task ordering and shipping seams + +Tasks 1-4 (evidence record) are independent of 5-11 and ship value alone. Tasks 5-6 (tagging) are independent of 1-4. Task 7 onward consumes both. If this needs to be split across people, the seam is after Task 4 and after Task 6. + +--- + +### Task 1: `schema_version` on results.json + +Establishes the additive-field pattern every later task follows. + +Files: +- Modify: `src/vip/reporting.py` (add constant, `ReportData` field, `load_results` read) +- Modify: `src/vip/plugin.py:1299-1310` (payload) +- Test: `selftests/test_results_schema.py` (create) + +Interfaces: +- Consumes: nothing +- Produces: `reporting.RESULTS_SCHEMA_VERSION: str` (value `"1.0"`); `ReportData.schema_version: str | None` + +- [ ] Step 1: Write the failing test + +Create `selftests/test_results_schema.py`: + +```python +import json + +from vip.reporting import RESULTS_SCHEMA_VERSION, load_results + + +def test_schema_version_is_loaded(tmp_path): + p = tmp_path / "results.json" + p.write_text(json.dumps({"schema_version": "1.0", "results": []})) + assert load_results(p).schema_version == "1.0" + + +def test_pre_1_0_results_load_with_null_schema_version(tmp_path): + """An archived results.json written before versioning must still load.""" + p = tmp_path / "results.json" + p.write_text( + json.dumps( + { + "generated_at": "2026-01-01T00:00:00+00:00", + "results": [{"nodeid": "a.py::test_x", "outcome": "passed"}], + } + ) + ) + data = load_results(p) + assert data.schema_version is None + assert len(data.results) == 1 + + +def test_current_schema_version_constant(): + assert RESULTS_SCHEMA_VERSION == "1.0" +``` + +- [ ] Step 2: Run test to verify it fails + +Run: `uv run pytest selftests/test_results_schema.py -v` +Expected: FAIL with `ImportError: cannot import name 'RESULTS_SCHEMA_VERSION'` + +- [ ] Step 3: Write minimal implementation + +In `src/vip/reporting.py`, below `VALID_FORMATS`: + +```python +# results.json schema version. Bump the minor for additive changes (a new +# field); bump the major for a removal, a rename, or a change in the meaning +# of an existing field. Consumers accept an unknown minor and refuse an +# unknown major. A file with no schema_version at all predates versioning +# and is treated as "pre-1.0". +RESULTS_SCHEMA_VERSION = "1.0" +``` + +Add to `ReportData` (alongside the other provenance fields): + +```python + schema_version: str | None = None +``` + +In `load_results`, in the `ReportData(...)` construction, add: + +```python + schema_version=raw.get("schema_version"), +``` + +In `src/vip/plugin.py`, import the constant near the existing `from vip import __version__` usage and add to the payload dict as its first key: + +```python + "schema_version": RESULTS_SCHEMA_VERSION, +``` + +Add the import at the top of `plugin.py` with the other `vip.reporting` imports: + +```python +from vip.reporting import RESULTS_SCHEMA_VERSION +``` + +- [ ] Step 4: Run test to verify it passes + +Run: `uv run pytest selftests/test_results_schema.py -v` +Expected: 3 passed + +- [ ] Step 5: Run the full selftest suite for regressions + +Run: `uv run pytest selftests/ -q` +Expected: all pass + +- [ ] Step 6: Commit + +```bash +git add src/vip/reporting.py src/vip/plugin.py selftests/test_results_schema.py +git commit -m "feat(reporting): add schema_version to results.json" +``` + +--- + +### Task 2: Per-test timestamps + +Files: +- Modify: `src/vip/plugin.py` (helper + `pytest_runtest_logreport` results dict at `:1203-1216`) +- Modify: `src/vip/reporting.py` (`TestResult` fields, `load_results`) +- Test: `selftests/test_results_timestamps.py` (create) + +Interfaces: +- Consumes: Task 1's additive-field pattern +- Produces: `TestResult.started_at: str | None`, `TestResult.finished_at: str | None` (UTC ISO 8601) + +Background the implementer needs: `report.start` and `report.stop` are epoch floats on pytest's `TestReport`. They survive xdist worker-to-controller serialization (verified under `-n 0` and `-n 2`), which matters because only the controller writes the report (`plugin.py:1173-1176`). The collection site fires for `report.when == "call"` or a setup-phase skip (`plugin.py:1185`), so `started_at` is when the check itself began and excludes fixture setup. That is the intended semantic; do not try to widen it. + +- [ ] Step 1: Write the failing test + +Create `selftests/test_results_timestamps.py`: + +```python +import json +from datetime import datetime + + +def test_results_carry_per_test_timestamps(pytester): + pytester.makepyfile( + test_stamp=""" + def test_passes(): + assert True + + import pytest + + @pytest.mark.skip(reason="deliberate") + def test_skipped(): + pass + """ + ) + report = pytester.path / "results.json" + pytester.runpytest_subprocess("--vip-report", str(report), "-p", "no:cacheprovider") + + data = json.loads(report.read_text()) + assert data["results"], "expected at least one result" + for entry in data["results"]: + started, finished = entry["started_at"], entry["finished_at"] + assert started is not None and finished is not None + # Parses as ISO 8601 and is timezone-aware UTC. + start_dt = datetime.fromisoformat(started) + finish_dt = datetime.fromisoformat(finished) + assert start_dt.tzinfo is not None + assert start_dt.utcoffset().total_seconds() == 0 + assert start_dt <= finish_dt + + +def test_timestamps_absent_in_old_file_load_as_none(tmp_path): + from vip.reporting import load_results + + p = tmp_path / "results.json" + p.write_text(json.dumps({"results": [{"nodeid": "a.py::t", "outcome": "passed"}]})) + result = load_results(p).results[0] + assert result.started_at is None + assert result.finished_at is None +``` + +- [ ] Step 2: Run test to verify it fails + +Run: `uv run pytest selftests/test_results_timestamps.py -v` +Expected: FAIL with `KeyError: 'started_at'` + +- [ ] Step 3: Write minimal implementation + +In `src/vip/plugin.py`, add a module-level helper near the other `_extract_*` helpers: + +```python +def _epoch_to_iso(value: float | None) -> str | None: + """Convert a pytest report epoch float to a UTC ISO 8601 string. + + Returns None rather than raising for a missing or unrepresentable value: + a provenance field is never worth failing a verification run over. + """ + if value is None: + return None + try: + return datetime.fromtimestamp(value, timezone.utc).isoformat() + except (OSError, OverflowError, ValueError): + return None +``` + +In the `results.append({...})` dict, add: + +```python + "started_at": _epoch_to_iso(getattr(report, "start", None)), + "finished_at": _epoch_to_iso(getattr(report, "stop", None)), +``` + +In `src/vip/reporting.py`, add to `TestResult`: + +```python + # When this check began and ended, UTC ISO 8601, from pytest's report.start + # and report.stop. This is the call phase, so it excludes fixture setup + # (except for a setup-phase skip, where it is the setup start). None for a + # results.json written before these fields existed. + started_at: str | None = None + finished_at: str | None = None +``` + +In `load_results`, inside the `TestResult(...)` construction: + +```python + started_at=r.get("started_at"), + finished_at=r.get("finished_at"), +``` + +- [ ] Step 4: Run test to verify it passes + +Run: `uv run pytest selftests/test_results_timestamps.py -v` +Expected: 2 passed + +- [ ] Step 5: Verify timestamps survive xdist + +Run: `uv run pytest selftests/test_results_timestamps.py -v -n 2` +Expected: 2 passed (the pytester subprocess is independent, but this confirms no controller/worker regression) + +- [ ] Step 6: Commit + +```bash +git add src/vip/plugin.py src/vip/reporting.py selftests/test_results_timestamps.py +git commit -m "feat(reporting): record per-test start and finish timestamps" +``` + +--- + +### Task 3: Execution attribution block + +Files: +- Create: `src/vip/attribution.py` +- Modify: `src/vip/plugin.py` (`pytest_addoption`, payload) +- Modify: `src/vip/reporting.py` (`ReportData.execution`, `load_results`) +- Test: `selftests/test_attribution.py` (create) + +Interfaces: +- Consumes: nothing +- Produces: `attribution.collect_execution_metadata(*, cwd: Path | None = None, env: Mapping[str, str] | None = None) -> dict[str, Any]` returning keys `hostname`, `git`, `ci`; `attribution.redact_userinfo(url: str | None) -> str | None`; `ReportData.execution: dict | None` + +Why a new module: this is self-contained, needs no pytest, and is the only part of the provenance work with meaningful branching (three CI providers, two git sources, URL redaction). Keeping it out of `plugin.py` makes it unit-testable without a pytest run. + +Security requirement: CI checkouts routinely rewrite the origin remote to embed a credential (`https://x-access-token:ghs_...@github.com/org/repo`). `results.json` is an uploaded artifact — `plugin.py:1196-1201` already strips absolute paths for exactly this reason — so userinfo must be removed before recording. A remote that cannot be parsed is recorded as `None`, never passed through raw. + +- [ ] Step 1: Write the failing test + +Create `selftests/test_attribution.py`: + +```python +import subprocess + +import pytest + +from vip.attribution import collect_execution_metadata + + +def _init_repo(path): + def git(*args): + subprocess.run(["git", *args], cwd=path, check=True, capture_output=True) + + git("init") + git("config", "user.email", "t@example.com") + git("config", "user.name", "Test") + git("config", "commit.gpgsign", "false") + (path / "f.txt").write_text("hello") + git("add", "f.txt") + git("commit", "-m", "initial") + + +def test_git_metadata_from_a_real_repo(tmp_path): + _init_repo(tmp_path) + meta = collect_execution_metadata(cwd=tmp_path, env={}) + assert meta["git"]["commit"] is not None + assert len(meta["git"]["commit"]) == 40 + assert meta["git"]["dirty"] is False + + +def test_dirty_worktree_is_reported(tmp_path): + _init_repo(tmp_path) + (tmp_path / "f.txt").write_text("changed") + assert collect_execution_metadata(cwd=tmp_path, env={})["git"]["dirty"] is True + + +def test_non_repo_yields_null_git(tmp_path): + assert collect_execution_metadata(cwd=tmp_path, env={})["git"] is None + + +def test_no_ci_env_yields_null_ci(tmp_path): + assert collect_execution_metadata(cwd=tmp_path, env={})["ci"] is None + + +def test_github_actions_run_url_is_composed(tmp_path): + env = { + "GITHUB_ACTIONS": "true", + "GITHUB_RUN_ID": "42", + "GITHUB_RUN_ATTEMPT": "1", + "GITHUB_SERVER_URL": "https://github.com", + "GITHUB_REPOSITORY": "posit-dev/vip", + "GITHUB_JOB": "connect-smoke", + } + ci = collect_execution_metadata(cwd=tmp_path, env=env)["ci"] + assert ci["provider"] == "github" + assert ci["run_url"] == "https://github.com/posit-dev/vip/actions/runs/42" + assert ci["job"] == "connect-smoke" + + +def test_gitlab_and_jenkins_are_recognized(tmp_path): + gl = collect_execution_metadata( + cwd=tmp_path, env={"GITLAB_CI": "true", "CI_JOB_URL": "https://gl/x/-/jobs/9"} + )["ci"] + assert gl["provider"] == "gitlab" + assert gl["run_url"] == "https://gl/x/-/jobs/9" + + jk = collect_execution_metadata( + cwd=tmp_path, env={"JENKINS_URL": "https://j/", "BUILD_URL": "https://j/job/x/7/"} + )["ci"] + assert jk["provider"] == "jenkins" + assert jk["run_url"] == "https://j/job/x/7/" + + +def test_env_sha_takes_precedence_over_subprocess(tmp_path): + _init_repo(tmp_path) + env = {"GITHUB_SHA": "a" * 40, "GITHUB_REF_NAME": "feature/x"} + meta = collect_execution_metadata(cwd=tmp_path, env=env) + assert meta["git"]["commit"] == "a" * 40 + assert meta["git"]["branch"] == "feature/x" + + +@pytest.mark.parametrize( + ("raw", "expected"), + [ + ("https://x-access-token:ghs_SECRET@github.com/o/r", "https://github.com/o/r"), + ("https://user:pw@example.com:8443/o/r.git", "https://example.com:8443/o/r.git"), + ("https://github.com/o/r", "https://github.com/o/r"), + ("git@github.com:o/r.git", "github.com:o/r.git"), + ("", None), + (None, None), + ], +) +def test_userinfo_is_redacted(raw, expected): + from vip.attribution import redact_userinfo + + assert redact_userinfo(raw) == expected + + +def test_hostname_is_recorded(tmp_path): + assert collect_execution_metadata(cwd=tmp_path, env={})["hostname"] +``` + +- [ ] Step 2: Run test to verify it fails + +Run: `uv run pytest selftests/test_attribution.py -v` +Expected: FAIL with `ModuleNotFoundError: No module named 'vip.attribution'` + +- [ ] Step 3: Write minimal implementation + +Create `src/vip/attribution.py`: + +```python +"""Execution attribution for the results.json evidence record. + +Answers "which pipeline execution, on which host, from which commit produced +this evidence" — the fields that make an automated test result attributable. + +Every probe here degrades to None. A missing git binary, a detached worktree, +a non-repo working directory or an unrecognized CI system must never fail or +warn a verification run; provenance is not worth breaking a run over. +""" + +from __future__ import annotations + +import os +import platform +import subprocess +from collections.abc import Mapping +from pathlib import Path +from typing import Any +from urllib.parse import urlsplit, urlunsplit + +_GIT_TIMEOUT_SECONDS = 5 + + +def redact_userinfo(url: str | None) -> str | None: + """Strip any userinfo component from a remote URL. + + CI checkouts rewrite the origin to embed a credential + (``https://x-access-token:ghs_...@github.com/org/repo``). results.json is + an uploaded artifact, so the credential must never reach it. Userinfo is + never needed to identify a repository, so it is dropped unconditionally + rather than pattern-matched against known token shapes. + """ + if not url: + return None + if "://" not in url: + # scp-style (git@host:org/repo.git). No password component, but drop + # the user anyway so there is exactly one rule to reason about. + return url.split("@", 1)[-1] if "@" in url else url + try: + parts = urlsplit(url) + except ValueError: + return None + if not parts.hostname: + return None + netloc = parts.hostname + if parts.port: + netloc = f"{netloc}:{parts.port}" + return urlunsplit((parts.scheme, netloc, parts.path, "", "")) + + +def _git(args: list[str], cwd: Path) -> str | None: + """Run a git command, returning stripped stdout, or None if it failed. + + An empty string is a valid successful result (``status --porcelain`` on a + clean tree), so success-with-no-output must stay distinguishable from + failure. Callers rely on that difference. + """ + try: + proc = subprocess.run( + ["git", *args], + cwd=cwd, + capture_output=True, + text=True, + timeout=_GIT_TIMEOUT_SECONDS, + check=False, + ) + except (OSError, subprocess.SubprocessError): + return None + if proc.returncode != 0: + return None + return proc.stdout.strip() + + +def _git_metadata(cwd: Path, env: Mapping[str, str]) -> dict[str, Any] | None: + commit = env.get("GITHUB_SHA") or _git(["rev-parse", "HEAD"], cwd) + if not commit: + return None + status = _git(["status", "--porcelain"], cwd) + return { + "commit": commit, + "branch": env.get("GITHUB_REF_NAME") or _git(["rev-parse", "--abbrev-ref", "HEAD"], cwd), + "dirty": bool(status) if status is not None else None, + "remote": redact_userinfo(_git(["remote", "get-url", "origin"], cwd)), + } + + +def _ci_metadata(env: Mapping[str, str]) -> dict[str, Any] | None: + if env.get("GITHUB_ACTIONS") == "true": + run_id = env.get("GITHUB_RUN_ID") + repo = env.get("GITHUB_REPOSITORY") + server = env.get("GITHUB_SERVER_URL") or "https://github.com" + run_url = f"{server}/{repo}/actions/runs/{run_id}" if run_id and repo else None + return { + "provider": "github", + "run_id": run_id, + "run_attempt": env.get("GITHUB_RUN_ATTEMPT"), + "run_url": run_url, + "job": env.get("GITHUB_JOB"), + } + if env.get("GITLAB_CI") == "true": + return { + "provider": "gitlab", + "run_id": env.get("CI_PIPELINE_ID"), + "run_attempt": None, + "run_url": env.get("CI_JOB_URL"), + "job": env.get("CI_JOB_NAME"), + } + if env.get("JENKINS_URL"): + return { + "provider": "jenkins", + "run_id": env.get("BUILD_NUMBER"), + "run_attempt": None, + "run_url": env.get("BUILD_URL"), + "job": env.get("JOB_NAME"), + } + return None + + +def collect_execution_metadata( + *, cwd: Path | None = None, env: Mapping[str, str] | None = None +) -> dict[str, Any]: + """Collect host, git and CI attribution for the current run. + + ``hostname`` is the VIP runner's host, not the system under test. Anything + rendering it must label it that way. + """ + resolved_env = os.environ if env is None else env + resolved_cwd = Path.cwd() if cwd is None else cwd + return { + "hostname": platform.node() or None, + "git": _git_metadata(resolved_cwd, resolved_env), + "ci": _ci_metadata(resolved_env), + } +``` + +In `src/vip/plugin.py` `pytest_addoption`, add to the `vip` group: + +```python + group.addoption( + "--vip-no-attribution", + action="store_true", + default=False, + help="Omit host/git/CI attribution from results.json.", + ) +``` + +In the payload construction, add: + +```python + "execution": ( + None + if session.config.getoption("--vip-no-attribution", default=False) + else collect_execution_metadata() + ), +``` + +with the import at the top of `plugin.py`: + +```python +from vip.attribution import collect_execution_metadata +``` + +In `src/vip/reporting.py`, add to `ReportData`: + +```python + # Host / git / CI attribution; see vip.attribution. None when the run used + # --vip-no-attribution, or for a results.json predating the field. + execution: dict | None = None +``` + +and in `load_results`: + +```python + execution=raw.get("execution"), +``` + +- [ ] Step 4: Run test to verify it passes + +Run: `uv run pytest selftests/test_attribution.py -v` +Expected: 9 passed + +- [ ] Step 5: Write the opt-out and leak-regression test + +Append to `selftests/test_attribution.py`: + +```python +import json + + +def test_no_attribution_flag_omits_the_block(pytester): + pytester.makepyfile(test_x="def test_ok(): assert True") + report = pytester.path / "results.json" + pytester.runpytest_subprocess( + "--vip-report", str(report), "--vip-no-attribution", "-p", "no:cacheprovider" + ) + assert json.loads(report.read_text())["execution"] is None + + +def test_credential_in_remote_never_reaches_the_file(pytester, monkeypatch): + """Regression guard: assert the token is absent from the whole file.""" + secret = "ghs_THISMUSTNOTAPPEAR" + subprocess.run(["git", "init"], cwd=pytester.path, check=True, capture_output=True) + subprocess.run( + ["git", "remote", "add", "origin", f"https://x-access-token:{secret}@github.com/o/r"], + cwd=pytester.path, + check=True, + capture_output=True, + ) + for key, value in (("user.email", "t@example.com"), ("user.name", "T"), + ("commit.gpgsign", "false")): + subprocess.run(["git", "config", key, value], cwd=pytester.path, + check=True, capture_output=True) + pytester.makepyfile(test_x="def test_ok(): assert True") + subprocess.run(["git", "add", "-A"], cwd=pytester.path, check=True, capture_output=True) + subprocess.run(["git", "commit", "-m", "c"], cwd=pytester.path, + check=True, capture_output=True) + + report = pytester.path / "results.json" + pytester.runpytest_subprocess("--vip-report", str(report), "-p", "no:cacheprovider") + assert secret not in report.read_text() +``` + +- [ ] Step 6: Run the new tests + +Run: `uv run pytest selftests/test_attribution.py -v` +Expected: 11 passed + +- [ ] Step 7: Commit + +```bash +git add src/vip/attribution.py src/vip/plugin.py src/vip/reporting.py selftests/test_attribution.py +git commit -m "feat(reporting): attribute results to host, commit and CI run" +``` + +--- + +### Task 4: SHA-256 sidecar + +Files: +- Modify: `src/vip/plugin.py:1312-1318` (write path) +- Test: `selftests/test_results_checksum.py` (create) + +Interfaces: +- Consumes: nothing +- Produces: `report/results.json.sha256`, format ` results.json\n` (two spaces, `shasum -c` compatible) + +Note: `results.json` is currently written via `p.write_text(json.dumps(...))` with no trailing newline, while `failures.json` appends one. The digest must cover the exact bytes on disk, so switch to an explicit encode-then-`write_bytes` and hash that same buffer. Do not re-serialize to compute the hash. + +- [ ] Step 1: Write the failing test + +Create `selftests/test_results_checksum.py`: + +```python +import hashlib +import subprocess +import sys + + +def test_sidecar_matches_the_bytes_on_disk(pytester): + pytester.makepyfile(test_x="def test_ok(): assert True") + report = pytester.path / "results.json" + pytester.runpytest_subprocess("--vip-report", str(report), "-p", "no:cacheprovider") + + sidecar = report.parent / "results.json.sha256" + assert sidecar.exists() + + expected = hashlib.sha256(report.read_bytes()).hexdigest() + line = sidecar.read_text().strip() + digest, name = line.split() + assert digest == expected + assert name == "results.json" + + +def test_sidecar_is_written_even_for_json_only_format(pytester): + """The checksum is a property of the file, not an output format.""" + pytester.makepyfile(test_x="def test_ok(): assert True") + report = pytester.path / "results.json" + pytester.runpytest_subprocess( + "--vip-report", str(report), "--vip-format", "json", "-p", "no:cacheprovider" + ) + assert (report.parent / "results.json.sha256").exists() + + +def test_sidecar_verifies_with_shasum(pytester): + if sys.platform.startswith("win"): + return + pytester.makepyfile(test_x="def test_ok(): assert True") + report = pytester.path / "results.json" + pytester.runpytest_subprocess("--vip-report", str(report), "-p", "no:cacheprovider") + proc = subprocess.run( + ["shasum", "-a", "256", "-c", "results.json.sha256"], + cwd=report.parent, + capture_output=True, + text=True, + ) + assert proc.returncode == 0, proc.stdout + proc.stderr +``` + +- [ ] Step 2: Run test to verify it fails + +Run: `uv run pytest selftests/test_results_checksum.py -v` +Expected: FAIL with `assert sidecar.exists()` + +- [ ] Step 3: Write minimal implementation + +Add `import hashlib` at the top of `src/vip/plugin.py`. Replace the write block: + +```python + try: + p = Path(report_path) + p.parent.mkdir(parents=True, exist_ok=True) + p.write_text(json.dumps(payload, indent=2)) + except OSError as exc: + warnings.warn(f"VIP: could not write report to {report_path}: {exc}", stacklevel=1) + return +``` + +with: + +```python + try: + p = Path(report_path) + p.parent.mkdir(parents=True, exist_ok=True) + # Hash the exact bytes written, not a re-serialization: the sidecar is + # only useful if it verifies against the file actually on disk. + data = json.dumps(payload, indent=2).encode("utf-8") + p.write_bytes(data) + digest = hashlib.sha256(data).hexdigest() + # shasum -c compatible: " ". Written unconditionally + # rather than gated on --vip-format, because the checksum is a property + # of the evidence file rather than an output format. + p.with_name(f"{p.name}.sha256").write_text(f"{digest} {p.name}\n") + except OSError as exc: + warnings.warn(f"VIP: could not write report to {report_path}: {exc}", stacklevel=1) + return +``` + +- [ ] Step 4: Run test to verify it passes + +Run: `uv run pytest selftests/test_results_checksum.py -v` +Expected: 3 passed + +- [ ] Step 5: Run the full selftest suite + +Run: `uv run pytest selftests/ -q` +Expected: all pass + +- [ ] Step 6: Commit + +```bash +git add src/vip/plugin.py selftests/test_results_checksum.py +git commit -m "feat(reporting): write a sha256 sidecar next to results.json" +``` + +--- + +### Task 5: Gherkin parser collects all tags and ignores control tags for the marker + +Files: +- Modify: `src/vip/gherkin.py:37-58` and the return dict +- Test: `selftests/test_gherkin_control_tags.py` (create) + +Interfaces: +- Consumes: nothing +- Produces: `gherkin.CONTROL_TAG_PREFIX: str` (value `"control-"`); `parse_feature_file()` return dict gains a `tags: list[str]` key holding every tag in the file, `@` stripped + +Why this comes before the plugin change: `gherkin.py` currently derives a feature's `marker` from the first token of the first tag line (`:56-57`), so `@control-x @connect` sets the marker to `control-x`. That value feeds the HTML report cards (`report_html.py:241`), `scripts/generate-test-catalog.py:46` and `scripts/generate-feature-matrix.py:142`. Task 6 also needs a tag list this parser does not currently return. + +- [ ] Step 1: Write the failing test + +Create `selftests/test_gherkin_control_tags.py`: + +```python +from vip.gherkin import CONTROL_TAG_PREFIX, parse_feature_file + +FEATURE = """@control-cfr-11-10-e @connect +Feature: Audit trail + Scenario: Publish is recorded + Given Connect is reachable +""" + +PRODUCT_FIRST = """@connect @control-cfr-11-10-e +Feature: Audit trail + Scenario: Publish is recorded + Given Connect is reachable +""" + + +def test_control_tag_does_not_become_the_marker(tmp_path): + """Tag order inside a feature file must not change the derived marker.""" + f = tmp_path / "t.feature" + f.write_text(FEATURE) + assert parse_feature_file(f)["marker"] == "connect" + + +def test_product_first_ordering_is_unchanged(tmp_path): + f = tmp_path / "t.feature" + f.write_text(PRODUCT_FIRST) + assert parse_feature_file(f)["marker"] == "connect" + + +def test_all_tags_are_collected(tmp_path): + f = tmp_path / "t.feature" + f.write_text(FEATURE) + assert set(parse_feature_file(f)["tags"]) == {"control-cfr-11-10-e", "connect"} + + +def test_scenario_level_tags_are_collected(tmp_path): + f = tmp_path / "t.feature" + f.write_text( + "@connect\n" + "Feature: F\n" + " @control-access-control\n" + " Scenario: S\n" + " Given x\n" + ) + parsed = parse_feature_file(f) + assert parsed["marker"] == "connect" + assert "control-access-control" in parsed["tags"] + + +def test_prefix_constant(): + assert CONTROL_TAG_PREFIX == "control-" +``` + +- [ ] Step 2: Run test to verify it fails + +Run: `uv run pytest selftests/test_gherkin_control_tags.py -v` +Expected: FAIL with `ImportError: cannot import name 'CONTROL_TAG_PREFIX'` + +- [ ] Step 3: Write minimal implementation + +In `src/vip/gherkin.py`, add below `_STEP_PREFIXES`: + +```python +# Gherkin tags of the form @control- map a scenario to a compliance +# control. They are deliberately excluded from the derived feature marker: +# that value feeds the HTML report cards and the generated test catalog and +# feature matrix, so a control tag written before the product tag would +# otherwise silently mislabel the feature. +CONTROL_TAG_PREFIX = "control-" +``` + +Add `tags: list[str] = []` beside the other accumulators, and replace the tag-line branch: + +```python + # Tag line — first non-control tag becomes the marker. + if line.startswith("@"): + line_tags = [tok.lstrip("@") for tok in line.split() if tok.startswith("@")] + tags.extend(line_tags) + if not marker: + for tag in line_tags: + if not tag.startswith(CONTROL_TAG_PREFIX): + marker = tag + break + continue +``` + +Add `"tags": tags,` to the returned dict, and document the new key in the docstring's Returns section: + +``` + dict with keys: ``title``, ``description``, ``marker``, ``tags``, + ``file``, ``scenarios`` (list of dicts with ``title`` and ``steps``). +``` + +- [ ] Step 4: Run test to verify it passes + +Run: `uv run pytest selftests/test_gherkin_control_tags.py -v` +Expected: 5 passed + +- [ ] Step 5: Verify the catalog generators still work + +Run: `uv run python scripts/generate-test-catalog.py && uv run python scripts/generate-feature-matrix.py` +Expected: both exit 0. Per the project memory these outputs are gitignored and generated — do not commit them. + +- [ ] Step 6: Commit + +```bash +git add src/vip/gherkin.py selftests/test_gherkin_control_tags.py +git commit -m "fix(gherkin): keep control tags out of the derived feature marker" +``` + +--- + +### Task 6: Pre-register control markers so strict-marker CI passes + +Files: +- Modify: `src/vip/plugin.py::pytest_configure` (after the marker registrations at `:176-209`) +- Test: `selftests/test_control_marker_registration.py` (create) + +Interfaces: +- Consumes: `gherkin.CONTROL_TAG_PREFIX`, `gherkin.parse_feature_file` (Task 5) +- Produces: control tags registered as pytest markers before collection + +Background: pytest punishes unregistered marks two different ways. By default `getattr(pytest.mark, tag)` raises `PytestUnknownMarkWarning` (`_pytest/mark/structures.py:628`), fatal under `-W error::pytest.PytestUnknownMarkWarning`. Under `--strict-markers` it does not warn at all — it calls `fail()` and aborts collection with `'control-x' not found in `markers` configuration option`. Verified against pytest 9.1.1. A `filterwarnings` ignore cannot reach the strict path, which is why registration is the fix rather than suppression. Registration also removes the warning, so both failure modes are covered by one mechanism. + +- [ ] Step 1: Write the failing test + +Create `selftests/test_control_marker_registration.py`: + +```python +import json + +FEATURE = """@connect @control-cfr-11-10-e +Feature: Audit trail + Scenario: Publish is recorded + Given a thing +""" + +STEPS = """ +from pytest_bdd import given, scenario + + +@scenario("t.feature", "Publish is recorded") +def test_tagged(): + pass + + +@given("a thing") +def a_thing(): + pass +""" + + +def _write_suite(pytester): + (pytester.path / "t.feature").write_text(FEATURE) + pytester.makepyfile(test_t=STEPS) + (pytester.path / "vip.toml").write_text('[connect]\nurl = "https://c.example.com"\n') + + +def test_collects_under_strict_markers(pytester): + _write_suite(pytester) + result = pytester.runpytest_subprocess( + "--vip-config", "vip.toml", "--strict-markers", "-p", "no:cacheprovider" + ) + result.assert_outcomes(passed=1) + + +def test_collects_under_warnings_as_errors(pytester): + _write_suite(pytester) + result = pytester.runpytest_subprocess( + "--vip-config", + "vip.toml", + "-W", + "error::pytest.PytestUnknownMarkWarning", + "-p", + "no:cacheprovider", + ) + result.assert_outcomes(passed=1) + + +def test_collects_under_both_together(pytester): + _write_suite(pytester) + result = pytester.runpytest_subprocess( + "--vip-config", + "vip.toml", + "--strict-markers", + "-W", + "error::pytest.PytestUnknownMarkWarning", + "-p", + "no:cacheprovider", + ) + result.assert_outcomes(passed=1) + + +def test_control_tags_still_reach_results_json(pytester): + """Registration must not cost us the evidence it exists to preserve.""" + _write_suite(pytester) + report = pytester.path / "results.json" + pytester.runpytest_subprocess( + "--vip-config", "vip.toml", "--vip-report", str(report), "-p", "no:cacheprovider" + ) + markers = json.loads(report.read_text())["results"][0]["markers"] + assert "control-cfr-11-10-e" in markers + assert "connect" in markers +``` + +- [ ] Step 2: Run test to verify it fails + +Run: `uv run pytest selftests/test_control_marker_registration.py -v` +Expected: `test_collects_under_strict_markers` FAILS with `'control-cfr-11-10-e' not found in `markers` configuration option` + +- [ ] Step 3: Write minimal implementation + +In `src/vip/plugin.py`, add the import: + +```python +from vip.gherkin import CONTROL_TAG_PREFIX, parse_feature_file +``` + +Add these module-level helpers: + +```python +def _feature_roots(config: pytest.Config) -> list[Path]: + """Directories pytest is about to collect, plus any extension directories. + + Scanning these rather than walking rootpath keeps the pre-scan cheap in a + large monorepo and avoids registering controls from feature files that are + not part of this run. + """ + roots: list[Path] = [] + for arg in config.args: + candidate = Path(str(arg).split("::")[0]) + roots.append(candidate if candidate.is_absolute() else Path(config.rootpath) / candidate) + roots.extend(Path(d) for d in (config.getoption("--vip-extensions", default=[]) or [])) + return roots or [Path(config.rootpath)] + + +def _discover_control_tags(config: pytest.Config) -> set[str]: + """Collect every @control-* tag from the feature files about to be collected.""" + tags: set[str] = set() + for root in _feature_roots(config): + try: + if root.is_file(): + features = [root] if root.suffix == ".feature" else [] + else: + features = sorted(root.rglob("*.feature")) + except OSError: + continue + for feature in features: + try: + parsed = parse_feature_file(feature) + except (OSError, UnicodeDecodeError): + continue + tags.update(t for t in parsed["tags"] if t.startswith(CONTROL_TAG_PREFIX)) + return tags +``` + +In `pytest_configure`, immediately after the last `addinivalue_line("markers", ...)` call: + +```python + # Compliance control tags (@control-) become pytest markers via + # pytest-bdd's default pytest_bdd_apply_tag hook. Their slugs are chosen by + # the customer, so they cannot be registered by name ahead of time -- but an + # unregistered mark warns by default and aborts collection outright under + # --strict-markers, which regulated CI is likely to enable. Registering the + # tags we are about to collect satisfies both paths at once. + for tag in sorted(_discover_control_tags(config)): + config.addinivalue_line("markers", f"{tag}: compliance control tag") +``` + +- [ ] Step 4: Run test to verify it passes + +Run: `uv run pytest selftests/test_control_marker_registration.py -v` +Expected: 4 passed + +- [ ] Step 5: Confirm no regression in the existing suite + +Run: `uv run pytest selftests/ -q` +Expected: all pass + +- [ ] Step 6: Commit + +```bash +git add src/vip/plugin.py selftests/test_control_marker_registration.py +git commit -m "feat(plugin): register control tags as markers before collection" +``` + +--- + +### Task 7: Control list model and loader + +Files: +- Create: `src/vip/traceability.py` +- Test: `selftests/test_traceability_controls.py` (create) + +Interfaces: +- Consumes: nothing +- Produces: `ControlSpec` dataclass (fields `control_id`, `description`, `reference`, `risk`, `verification`, `responsibility`, `notes`); `load_controls(path) -> dict[str, ControlSpec]`; `ControlListError` + +Deviation from the spec, deliberate: the spec places `build_traceability_matrix` in `reporting.py`. This plan puts the whole traceability feature in a new `src/vip/traceability.py` instead. `reporting.py` already holds the data model plus the JSON, JUnit and SARIF writers, and this codebase's own precedent is to split — `report_html.py` exists as "reporting.py's testable rendering sibling" per CLAUDE.md. Same reasoning, same shape. + +- [ ] Step 1: Write the failing test + +Create `selftests/test_traceability_controls.py`: + +```python +import pytest + +from vip.traceability import ControlListError, load_controls + +FULL = """ +[controls.cfr-11-10-e] +description = "Secure, computer-generated, time-stamped audit trails" +reference = "21 CFR 11.10(e)" +risk = "high" +verification = "automated" +responsibility = "shared" +notes = "Retention duration is a customer configuration decision." + +[controls.training] +description = "Personnel training records" +verification = "procedural" +responsibility = "customer" +""" + + +def test_loads_all_fields(tmp_path): + p = tmp_path / "controls.toml" + p.write_text(FULL) + controls = load_controls(p) + + audit = controls["cfr-11-10-e"] + assert audit.description.startswith("Secure") + assert audit.reference == "21 CFR 11.10(e)" + assert audit.risk == "high" + assert audit.verification == "automated" + assert audit.responsibility == "shared" + assert audit.notes + + +def test_optional_fields_default(tmp_path): + p = tmp_path / "controls.toml" + p.write_text('[controls.x]\ndescription = "only required key"\n') + spec = load_controls(p)["x"] + assert spec.reference is None + assert spec.risk is None + assert spec.responsibility is None + assert spec.verification == "automated" + + +def test_missing_description_is_an_error(tmp_path): + p = tmp_path / "controls.toml" + p.write_text('[controls.x]\nrisk = "high"\n') + with pytest.raises(ControlListError, match="description"): + load_controls(p) + + +def test_unknown_verification_value_is_an_error(tmp_path): + p = tmp_path / "controls.toml" + p.write_text('[controls.x]\ndescription = "d"\nverification = "vibes"\n') + with pytest.raises(ControlListError, match="verification"): + load_controls(p) + + +def test_missing_file_is_an_error(tmp_path): + with pytest.raises(ControlListError, match="not found"): + load_controls(tmp_path / "nope.toml") + + +def test_missing_controls_table_is_an_error(tmp_path): + p = tmp_path / "controls.toml" + p.write_text('title = "wrong shape"\n') + with pytest.raises(ControlListError, match="controls"): + load_controls(p) +``` + +- [ ] Step 2: Run test to verify it fails + +Run: `uv run pytest selftests/test_traceability_controls.py -v` +Expected: FAIL with `ModuleNotFoundError: No module named 'vip.traceability'` + +- [ ] Step 3: Write minimal implementation + +Create `src/vip/traceability.py`: + +```python +"""Traceability matrix: join compliance controls against tagged test results. + +VIP stays regulation-agnostic. The control list is supplied by whoever owns the +regulatory mapping; nothing here interprets ``reference``, ``risk`` or +``responsibility`` beyond carrying them through to the output. +""" + +from __future__ import annotations + +import sys +from dataclasses import dataclass, field +from pathlib import Path + +if sys.version_info >= (3, 11): + import tomllib +else: + import tomli as tomllib + +# Matrix output schema, versioned independently of results.json. +MATRIX_SCHEMA_VERSION = "1.0" + +VERIFICATION_VALUES = frozenset({"automated", "manual", "procedural"}) + + +class ControlListError(Exception): + """Raised when a controls.toml file is missing or malformed.""" + + +@dataclass +class ControlSpec: + control_id: str + description: str + reference: str | None = None + risk: str | None = None + # "automated" controls are expected to be covered by a tagged scenario. + # "manual" and "procedural" ones are reported as not verifiable by + # automated test rather than as coverage gaps. + verification: str = "automated" + responsibility: str | None = None + notes: str | None = None + + +def load_controls(path: str | Path) -> dict[str, ControlSpec]: + """Load a controls.toml file into ControlSpec objects keyed by control id.""" + p = Path(path) + if not p.is_file(): + raise ControlListError(f"control list not found: {p}") + try: + raw = tomllib.loads(p.read_text(encoding="utf-8")) + except (tomllib.TOMLDecodeError, OSError, UnicodeDecodeError) as exc: + raise ControlListError(f"could not read control list {p}: {exc}") from exc + + table = raw.get("controls") + if not isinstance(table, dict) or not table: + raise ControlListError(f"{p} has no [controls] table") + + controls: dict[str, ControlSpec] = {} + for control_id, body in table.items(): + if not isinstance(body, dict): + raise ControlListError(f"[controls.{control_id}] must be a table") + description = body.get("description") + if not description: + raise ControlListError(f"[controls.{control_id}] is missing a description") + verification = body.get("verification", "automated") + if verification not in VERIFICATION_VALUES: + raise ControlListError( + f"[controls.{control_id}] has verification={verification!r};" + f" expected one of {sorted(VERIFICATION_VALUES)}" + ) + controls[control_id] = ControlSpec( + control_id=control_id, + description=description, + reference=body.get("reference"), + risk=body.get("risk"), + verification=verification, + responsibility=body.get("responsibility"), + notes=body.get("notes"), + ) + return controls +``` + +- [ ] Step 4: Run test to verify it passes + +Run: `uv run pytest selftests/test_traceability_controls.py -v` +Expected: 6 passed + +- [ ] Step 5: Commit + +```bash +git add src/vip/traceability.py selftests/test_traceability_controls.py +git commit -m "feat(traceability): add the control list model and loader" +``` + +--- + +### Task 8: Build the traceability matrix + +Files: +- Modify: `src/vip/traceability.py` +- Test: `selftests/test_traceability_matrix.py` (create) + +Interfaces: +- Consumes: `ControlSpec` (Task 7); `ReportData`/`TestResult` including `started_at`/`finished_at` (Task 2) and `execution`/`schema_version` (Tasks 1, 3) +- Produces: `ControlMatch`, `ControlEntry`, `TraceabilityMatrix` dataclasses; `build_traceability_matrix(data, controls, tag_prefix="control-") -> TraceabilityMatrix` + +Coverage is three-valued. `covered` when at least one scenario matched. `not_automatable` when no scenario matched and `verification` is `manual` or `procedural`. `gap` when no scenario matched and `verification` is `automated`. Conflating the last two is the single most misleading thing this export could do: a control nobody can automate is not the same as a control someone forgot to test. + +- [ ] Step 1: Write the failing test + +Create `selftests/test_traceability_matrix.py`: + +```python +from vip.reporting import ReportData, TestResult +from vip.traceability import ControlSpec, build_traceability_matrix + + +def _result(nodeid, markers, outcome="passed", **kw): + return TestResult( + nodeid=nodeid, + outcome=outcome, + markers=markers, + scenario_title=kw.pop("title", "A scenario"), + started_at="2026-08-28T12:00:00+00:00", + finished_at="2026-08-28T12:00:01+00:00", + **kw, + ) + + +def _controls(**kw): + return {cid: ControlSpec(control_id=cid, description=f"desc {cid}", **opts) + for cid, opts in kw.items()} + + +def test_covered_control_lists_its_scenarios(): + data = ReportData(results=[_result("t.py::a", ["connect", "control-x"], title="Scenario A")]) + matrix = build_traceability_matrix(data, _controls(x={})) + entry = matrix.entries[0] + assert entry.coverage == "covered" + assert entry.matches[0].scenario_title == "Scenario A" + assert entry.matches[0].status == "passed" + assert entry.matches[0].started_at == "2026-08-28T12:00:00+00:00" + + +def test_uncovered_automated_control_is_a_gap(): + matrix = build_traceability_matrix(ReportData(results=[]), _controls(x={})) + assert matrix.entries[0].coverage == "gap" + assert matrix.gap_count == 1 + + +def test_procedural_control_is_not_a_gap(): + controls = _controls(x={"verification": "procedural"}) + matrix = build_traceability_matrix(ReportData(results=[]), controls) + assert matrix.entries[0].coverage == "not_automatable" + assert matrix.gap_count == 0 + + +def test_one_control_satisfied_by_several_scenarios(): + data = ReportData( + results=[ + _result("t.py::a", ["control-x"], title="A"), + _result("t.py::b", ["control-x"], title="B"), + ] + ) + entry = build_traceability_matrix(data, _controls(x={})).entries[0] + assert [m.scenario_title for m in entry.matches] == ["A", "B"] + + +def test_one_scenario_satisfies_several_controls(): + data = ReportData(results=[_result("t.py::a", ["control-x", "control-y"])]) + matrix = build_traceability_matrix(data, _controls(x={}, y={})) + assert all(e.coverage == "covered" for e in matrix.entries) + + +def test_unrecognized_tag_is_reported(): + data = ReportData(results=[_result("t.py::a", ["control-typo"])]) + matrix = build_traceability_matrix(data, _controls(x={})) + assert matrix.unrecognized_tags == ["control-typo"] + + +def test_failure_detail_is_carried(): + data = ReportData( + results=[ + _result("t.py::a", ["control-x"], outcome="failed", concise_error="boom"), + ] + ) + match = build_traceability_matrix(data, _controls(x={})).entries[0].matches[0] + assert match.status == "failed" + assert match.detail == "boom" + + +def test_skip_reason_is_carried(): + data = ReportData( + results=[ + _result("t.py::a", ["control-x"], outcome="skipped", skip_reason="not configured"), + ] + ) + assert build_traceability_matrix(data, _controls(x={})).entries[0].matches[0].detail == ( + "not configured" + ) + + +def test_na_version_status_is_distinct(): + data = ReportData( + results=[_result("t.py::a", ["control-x"], outcome="skipped", na_version=True)] + ) + assert build_traceability_matrix(data, _controls(x={})).entries[0].matches[0].status == ( + "na_version" + ) + + +def test_entries_and_matches_are_sorted_deterministically(): + data = ReportData( + results=[ + _result("t.py::z", ["control-b"], title="Z"), + _result("t.py::a", ["control-b"], title="A"), + ] + ) + matrix = build_traceability_matrix(data, _controls(b={}, a={})) + assert [e.control.control_id for e in matrix.entries] == ["a", "b"] + b_entry = next(e for e in matrix.entries if e.control.control_id == "b") + assert [m.nodeid for m in b_entry.matches] == ["t.py::a", "t.py::z"] + + +def test_custom_tag_prefix(): + data = ReportData(results=[_result("t.py::a", ["req-x"])]) + matrix = build_traceability_matrix(data, _controls(x={}), tag_prefix="req-") + assert matrix.entries[0].coverage == "covered" + + +def test_provenance_is_carried_from_the_report(): + data = ReportData( + generated_at="2026-08-28T12:00:00+00:00", + vip_version="2026.8.3", + schema_version="1.0", + basic_mode=True, + execution={"hostname": "runner-1", "git": None, "ci": None}, + ) + prov = build_traceability_matrix(data, _controls(x={})).provenance + assert prov["vip_version"] == "2026.8.3" + assert prov["basic_mode"] is True + assert prov["execution"]["hostname"] == "runner-1" + assert prov["results_schema_version"] == "1.0" +``` + +- [ ] Step 2: Run test to verify it fails + +Run: `uv run pytest selftests/test_traceability_matrix.py -v` +Expected: FAIL with `ImportError: cannot import name 'build_traceability_matrix'` + +- [ ] Step 3: Write minimal implementation + +Add `from vip.reporting import ReportData` to the imports at the top of +`src/vip/traceability.py` (Task 10 extends this same line to also import +`RESULTS_SCHEMA_VERSION`), then append: + +```python +@dataclass +class ControlMatch: + """One scenario that carries a control's tag.""" + + nodeid: str + scenario_title: str | None + status: str + started_at: str | None + finished_at: str | None + detail: str | None + + +@dataclass +class ControlEntry: + control: ControlSpec + matches: list[ControlMatch] = field(default_factory=list) + # "covered" | "gap" | "not_automatable" + coverage: str = "gap" + + +@dataclass +class TraceabilityMatrix: + entries: list[ControlEntry] = field(default_factory=list) + unrecognized_tags: list[str] = field(default_factory=list) + provenance: dict = field(default_factory=dict) + schema_version: str = MATRIX_SCHEMA_VERSION + + @property + def gap_count(self) -> int: + return sum(1 for e in self.entries if e.coverage == "gap") + + @property + def covered_count(self) -> int: + return sum(1 for e in self.entries if e.coverage == "covered") + + +def _provenance(data: ReportData) -> dict: + products = { + p.name: {"url": p.url, "version": p.version, "configured": p.configured} + for p in data.products + } + return { + "generated_at": data.generated_at, + "deployment_name": data.deployment_name, + "vip_version": data.vip_version, + "results_schema_version": data.schema_version, + "exit_status": data.exit_status, + # basic_mode is surfaced deliberately: a matrix built from a + # `vip verify --basic` run omits every @slow scenario and would + # otherwise assert coverage that was never exercised. + "basic_mode": data.basic_mode, + "execution": data.execution, + # Describes the VIP runner, not the system under test. The products + # table below identifies the system under test. + "runner_python_version": data.python_version, + "runner_platform": data.platform, + "products": products, + } + + +def build_traceability_matrix( + data: ReportData, + controls: dict[str, ControlSpec], + tag_prefix: str = "control-", +) -> TraceabilityMatrix: + """Join control definitions against tagged test results. + + Sorted deterministically -- by control id, then by nodeid within a control + -- so the same results.json and control list always produce byte-identical + output for a downstream renderer to diff. + """ + by_tag: dict[str, list[ControlMatch]] = {} + seen_tags: set[str] = set() + for result in data.results: + for marker in result.markers: + if not marker.startswith(tag_prefix): + continue + seen_tags.add(marker) + by_tag.setdefault(marker, []).append( + ControlMatch( + nodeid=result.nodeid, + scenario_title=result.scenario_title, + status=result.status, + started_at=result.started_at, + finished_at=result.finished_at, + detail=result.concise_error or result.skip_reason, + ) + ) + + entries: list[ControlEntry] = [] + for control_id in sorted(controls): + control = controls[control_id] + matches = sorted(by_tag.get(f"{tag_prefix}{control_id}", []), key=lambda m: m.nodeid) + if matches: + coverage = "covered" + elif control.verification != "automated": + coverage = "not_automatable" + else: + coverage = "gap" + entries.append(ControlEntry(control=control, matches=matches, coverage=coverage)) + + known = {f"{tag_prefix}{cid}" for cid in controls} + return TraceabilityMatrix( + entries=entries, + unrecognized_tags=sorted(seen_tags - known), + provenance=_provenance(data), + ) +``` + +Note the `field` import: `ControlEntry` and `TraceabilityMatrix` use +`field(default_factory=...)`, which Task 7 already imported from `dataclasses`. + +- [ ] Step 4: Run test to verify it passes + +Run: `uv run pytest selftests/test_traceability_matrix.py -v` +Expected: 12 passed + +- [ ] Step 5: Commit + +```bash +git add src/vip/traceability.py selftests/test_traceability_matrix.py +git commit -m "feat(traceability): build the control-to-scenario matrix" +``` + +--- + +### Task 9: Render the matrix as CSV and JSON + +Files: +- Modify: `src/vip/traceability.py` +- Test: `selftests/test_traceability_render.py` (create) + +Interfaces: +- Consumes: `TraceabilityMatrix` (Task 8) +- Produces: `render_csv(matrix) -> str`, `render_json(matrix) -> str` + +CSV columns, in order: `control_id`, `description`, `reference`, `risk`, `verification`, `responsibility`, `coverage`, `scenario`, `nodeid`, `status`, `started_at`, `finished_at`, `detail`, `notes`. A control with several matches emits one row per match. A control with no matches emits a single row with empty scenario columns. `duration` is deliberately absent: it is performance noise that varies run to run without carrying evidentiary value, unlike `started_at`. + +- [ ] Step 1: Write the failing test + +Create `selftests/test_traceability_render.py`: + +```python +import csv +import io +import json + +from vip.reporting import ReportData, TestResult +from vip.traceability import ( + ControlSpec, + build_traceability_matrix, + render_csv, + render_json, +) + +CSV_COLUMNS = [ + "control_id", "description", "reference", "risk", "verification", + "responsibility", "coverage", "scenario", "nodeid", "status", + "started_at", "finished_at", "detail", "notes", +] + + +def _matrix(): + data = ReportData( + generated_at="2026-08-28T12:00:00+00:00", + vip_version="2026.8.3", + results=[ + TestResult( + nodeid="t.py::a", + outcome="passed", + markers=["control-x"], + scenario_title="Scenario A", + started_at="2026-08-28T12:00:00+00:00", + finished_at="2026-08-28T12:00:01+00:00", + ) + ], + ) + controls = { + "x": ControlSpec("x", "Audit trail", reference="21 CFR 11.10(e)", risk="high"), + "y": ControlSpec("y", "Training records", verification="procedural"), + } + return build_traceability_matrix(data, controls) + + +def test_csv_has_the_expected_header(): + reader = csv.reader(io.StringIO(render_csv(_matrix()))) + assert next(reader) == CSV_COLUMNS + + +def test_csv_emits_one_row_per_match_and_one_for_a_gap(): + rows = list(csv.DictReader(io.StringIO(render_csv(_matrix())))) + assert len(rows) == 2 + covered = next(r for r in rows if r["control_id"] == "x") + assert covered["scenario"] == "Scenario A" + assert covered["status"] == "passed" + assert covered["coverage"] == "covered" + + procedural = next(r for r in rows if r["control_id"] == "y") + assert procedural["coverage"] == "not_automatable" + assert procedural["scenario"] == "" + assert procedural["nodeid"] == "" + + +def test_csv_is_byte_identical_across_invocations(): + m = _matrix() + assert render_csv(m) == render_csv(m) + assert render_csv(_matrix()) == render_csv(_matrix()) + + +def test_json_carries_provenance_and_schema_version(): + payload = json.loads(render_json(_matrix())) + assert payload["schema_version"] == "1.0" + assert payload["provenance"]["vip_version"] == "2026.8.3" + assert payload["summary"]["gaps"] == 0 + assert payload["summary"]["covered"] == 1 + assert payload["summary"]["not_automatable"] == 1 + + +def test_json_is_byte_identical_across_invocations(): + assert render_json(_matrix()) == render_json(_matrix()) + + +def test_json_round_trips(): + payload = json.loads(render_json(_matrix())) + entry = next(e for e in payload["controls"] if e["control_id"] == "x") + assert entry["matches"][0]["nodeid"] == "t.py::a" + assert entry["reference"] == "21 CFR 11.10(e)" +``` + +- [ ] Step 2: Run test to verify it fails + +Run: `uv run pytest selftests/test_traceability_render.py -v` +Expected: FAIL with `ImportError: cannot import name 'render_csv'` + +- [ ] Step 3: Write minimal implementation + +Add `import csv`, `import io` and `import json` to the imports in `src/vip/traceability.py`, then append: + +```python +CSV_COLUMNS = [ + "control_id", + "description", + "reference", + "risk", + "verification", + "responsibility", + "coverage", + "scenario", + "nodeid", + "status", + "started_at", + "finished_at", + "detail", + "notes", +] + + +def _control_columns(entry: ControlEntry) -> dict: + c = entry.control + return { + "control_id": c.control_id, + "description": c.description, + "reference": c.reference or "", + "risk": c.risk or "", + "verification": c.verification, + "responsibility": c.responsibility or "", + "coverage": entry.coverage, + "notes": c.notes or "", + } + + +def render_csv(matrix: TraceabilityMatrix) -> str: + """Render the matrix as CSV: one row per control/scenario pair. + + A control with no matching scenario still gets a row, with the scenario + columns empty, so a coverage gap is visible rather than absent. + """ + buf = io.StringIO() + writer = csv.DictWriter(buf, fieldnames=CSV_COLUMNS, lineterminator="\n") + writer.writeheader() + for entry in matrix.entries: + base = _control_columns(entry) + if not entry.matches: + writer.writerow( + {**base, "scenario": "", "nodeid": "", "status": "", + "started_at": "", "finished_at": "", "detail": ""} + ) + continue + for match in entry.matches: + writer.writerow( + { + **base, + "scenario": match.scenario_title or "", + "nodeid": match.nodeid, + "status": match.status, + "started_at": match.started_at or "", + "finished_at": match.finished_at or "", + "detail": match.detail or "", + } + ) + return buf.getvalue() + + +def render_json(matrix: TraceabilityMatrix) -> str: + """Render the matrix as JSON, carrying the full provenance block.""" + payload = { + "schema_version": matrix.schema_version, + "provenance": matrix.provenance, + "summary": { + "total": len(matrix.entries), + "covered": matrix.covered_count, + "gaps": matrix.gap_count, + "not_automatable": sum( + 1 for e in matrix.entries if e.coverage == "not_automatable" + ), + }, + "unrecognized_tags": matrix.unrecognized_tags, + "controls": [ + { + **_control_columns(entry), + "matches": [ + { + "nodeid": m.nodeid, + "scenario": m.scenario_title, + "status": m.status, + "started_at": m.started_at, + "finished_at": m.finished_at, + "detail": m.detail, + } + for m in entry.matches + ], + } + for entry in matrix.entries + ], + } + return json.dumps(payload, indent=2, sort_keys=False) + "\n" +``` + +- [ ] Step 4: Run test to verify it passes + +Run: `uv run pytest selftests/test_traceability_render.py -v` +Expected: 6 passed + +- [ ] Step 5: Commit + +```bash +git add src/vip/traceability.py selftests/test_traceability_render.py +git commit -m "feat(traceability): render the matrix as csv and json" +``` + +--- + +### Task 10: `vip trace` CLI + +Files: +- Modify: `src/vip/cli.py` (add `run_trace`, the subparser near the `scaffold` block at `:1892-1934`, and the `subcommand_parsers` map at `:1936-1946`) +- Modify: `src/vip/traceability.py` (checksum verification, schema gate) +- Test: `selftests/test_trace_cli.py` (create) + +Interfaces: +- Consumes: `load_controls`, `build_traceability_matrix`, `render_csv`, `render_json` (Tasks 7-9); `load_results` (existing) +- Produces: `cli.run_trace(args: argparse.Namespace) -> None`; `traceability.verify_results_checksum(path: str | Path) -> str | None`; `traceability.check_results_schema(schema_version: str | None) -> None`; `traceability.ResultsIntegrityError` + +Behaviour: +- Unknown schema major is a hard error naming both versions. Unknown minor proceeds. A missing `schema_version` (pre-1.0) proceeds. +- A `.sha256` sidecar next to the results file is verified. A mismatch is a hard error, not a warning: a checksum mismatch on a compliance artifact is precisely the condition that must not be papered over. A missing sidecar is fine — older runs have none. +- Unrecognized control tags print to stderr as a warning and do not fail the command; they catch typos without blocking an otherwise valid export. + +- [ ] Step 1: Write the failing test + +Create `selftests/test_trace_cli.py`: + +```python +import hashlib +import json +import subprocess +import sys + +CONTROLS = """ +[controls.x] +description = "Audit trail" +reference = "21 CFR 11.10(e)" + +[controls.y] +description = "Training records" +verification = "procedural" +""" + + +def _results(tmp_path, schema_version="1.0", write_sidecar=True): + payload = { + "schema_version": schema_version, + "generated_at": "2026-08-28T12:00:00+00:00", + "vip_version": "2026.8.3", + "results": [ + { + "nodeid": "t.py::a", + "outcome": "passed", + "markers": ["connect", "control-x"], + "scenario_title": "Scenario A", + "started_at": "2026-08-28T12:00:00+00:00", + "finished_at": "2026-08-28T12:00:01+00:00", + } + ], + } + if schema_version is None: + payload.pop("schema_version") + p = tmp_path / "results.json" + data = json.dumps(payload, indent=2).encode() + p.write_bytes(data) + if write_sidecar: + p.with_name("results.json.sha256").write_text( + f"{hashlib.sha256(data).hexdigest()} results.json\n" + ) + controls = tmp_path / "controls.toml" + controls.write_text(CONTROLS) + return p, controls + + +def _run(*args, cwd=None): + return subprocess.run( + [sys.executable, "-m", "vip.cli", "trace", *args], + capture_output=True, + text=True, + cwd=cwd, + ) + + +def test_csv_to_stdout(tmp_path): + results, controls = _results(tmp_path) + proc = _run("--results", str(results), "--controls", str(controls)) + assert proc.returncode == 0, proc.stderr + assert proc.stdout.startswith("control_id,description,") + assert "Scenario A" in proc.stdout + + +def test_json_format(tmp_path): + results, controls = _results(tmp_path) + proc = _run("--results", str(results), "--controls", str(controls), "--format", "json") + assert proc.returncode == 0, proc.stderr + payload = json.loads(proc.stdout) + assert payload["summary"]["covered"] == 1 + assert payload["summary"]["not_automatable"] == 1 + + +def test_output_to_file(tmp_path): + results, controls = _results(tmp_path) + out = tmp_path / "matrix.csv" + proc = _run("--results", str(results), "--controls", str(controls), "--output", str(out)) + assert proc.returncode == 0, proc.stderr + assert out.read_text().startswith("control_id,") + + +def test_tampered_results_file_is_rejected(tmp_path): + results, controls = _results(tmp_path) + results.write_text(results.read_text().replace("passed", "failed")) + proc = _run("--results", str(results), "--controls", str(controls)) + assert proc.returncode != 0 + assert "checksum" in proc.stderr.lower() + + +def test_missing_sidecar_is_allowed(tmp_path): + results, controls = _results(tmp_path, write_sidecar=False) + assert _run("--results", str(results), "--controls", str(controls)).returncode == 0 + + +def test_pre_1_0_results_are_accepted(tmp_path): + results, controls = _results(tmp_path, schema_version=None) + assert _run("--results", str(results), "--controls", str(controls)).returncode == 0 + + +def test_unknown_major_schema_is_rejected(tmp_path): + results, controls = _results(tmp_path, schema_version="2.0") + proc = _run("--results", str(results), "--controls", str(controls)) + assert proc.returncode != 0 + assert "2.0" in proc.stderr + + +def test_unknown_minor_schema_is_accepted(tmp_path): + results, controls = _results(tmp_path, schema_version="1.7") + assert _run("--results", str(results), "--controls", str(controls)).returncode == 0 + + +def test_unrecognized_tag_warns_without_failing(tmp_path): + results, controls = _results(tmp_path) + payload = json.loads(results.read_text()) + payload["results"][0]["markers"].append("control-typo") + data = json.dumps(payload, indent=2).encode() + results.write_bytes(data) + results.with_name("results.json.sha256").write_text( + f"{hashlib.sha256(data).hexdigest()} results.json\n" + ) + proc = _run("--results", str(results), "--controls", str(controls)) + assert proc.returncode == 0 + assert "control-typo" in proc.stderr + + +def test_missing_control_file_errors_clearly(tmp_path): + results, _ = _results(tmp_path) + proc = _run("--results", str(results), "--controls", str(tmp_path / "nope.toml")) + assert proc.returncode != 0 + assert "not found" in proc.stderr +``` + +- [ ] Step 2: Run test to verify it fails + +Run: `uv run pytest selftests/test_trace_cli.py -v` +Expected: FAIL — `invalid choice: 'trace'` + +- [ ] Step 3: Add checksum and schema helpers + +Append to `src/vip/traceability.py` (add `import hashlib` to the imports): + +```python +class ResultsIntegrityError(Exception): + """Raised when a results file fails checksum or schema validation.""" + + +def verify_results_checksum(path: str | Path) -> str | None: + """Verify a results file against its .sha256 sidecar. + + Returns the digest of the file. Raises if a sidecar exists and disagrees. + A missing sidecar is not an error: results files written before the + sidecar existed have none. + + This is tamper-evidence within a trusted pipeline, not tamper-proofing -- + anyone who can edit the results file can regenerate the sidecar. It catches + corruption, truncated uploads and casual editing. + """ + p = Path(path) + digest = hashlib.sha256(p.read_bytes()).hexdigest() + sidecar = p.with_name(f"{p.name}.sha256") + if not sidecar.is_file(): + return digest + recorded = sidecar.read_text().split() + if recorded and recorded[0] != digest: + raise ResultsIntegrityError( + f"checksum mismatch for {p}: sidecar records {recorded[0]}, file hashes to {digest}" + ) + return digest + + +def check_results_schema(schema_version: str | None) -> None: + """Refuse an unknown major schema version; accept an unknown minor. + + A file with no schema_version predates versioning and is accepted. + """ + if not schema_version: + return + theirs = schema_version.split(".", 1)[0] + ours = RESULTS_SCHEMA_VERSION.split(".", 1)[0] + if theirs != ours: + raise ResultsIntegrityError( + f"results.json schema version {schema_version} is not supported by this " + f"vip (understands {RESULTS_SCHEMA_VERSION}); upgrade vip or regenerate the results" + ) +``` + +Add the import of the results schema constant at the top of `traceability.py`: + +```python +from vip.reporting import RESULTS_SCHEMA_VERSION, ReportData +``` + +- [ ] Step 4: Add the CLI subcommand + +In `src/vip/cli.py`, add the handler near the other `run_*` functions: + +```python +def run_trace(args: argparse.Namespace) -> None: + """Join a results.json against a control list and emit a traceability matrix.""" + from vip.reporting import load_results + from vip.traceability import ( + ControlListError, + ResultsIntegrityError, + build_traceability_matrix, + check_results_schema, + load_controls, + render_csv, + render_json, + verify_results_checksum, + ) + + results_path = Path(args.results) + if not results_path.is_file(): + print(f"Error: results file not found: {results_path}", file=sys.stderr) + sys.exit(1) + + try: + verify_results_checksum(results_path) + data = load_results(results_path) + check_results_schema(data.schema_version) + controls = load_controls(args.controls) + except (ResultsIntegrityError, ControlListError) as exc: + print(f"Error: {exc}", file=sys.stderr) + sys.exit(1) + + matrix = build_traceability_matrix(data, controls, tag_prefix=args.tag_prefix) + + if matrix.unrecognized_tags: + joined = ", ".join(matrix.unrecognized_tags) + print( + f"Warning: control tags present in results but absent from the control list: {joined}", + file=sys.stderr, + ) + + rendered = render_json(matrix) if args.format == "json" else render_csv(matrix) + if args.output: + out = Path(args.output) + out.parent.mkdir(parents=True, exist_ok=True) + out.write_text(rendered) + print(f"Wrote {out} ({len(matrix.entries)} controls, {matrix.gap_count} gaps)") + else: + sys.stdout.write(rendered) +``` + +Add the subparser after the `scaffold_parser.set_defaults(func=run_scaffold)` line: + +```python + trace_parser = subparsers.add_parser( + "trace", + help="Generate a compliance traceability matrix from test results", + description=( + "Join a results.json against a control list (controls.toml) and emit a " + "control-to-scenario traceability matrix as CSV or JSON.\n\n" + "Scenarios declare the control they satisfy with an @control- " + "Gherkin tag. Controls with no matching scenario are reported as coverage " + "gaps, except those marked verification = \"manual\" or \"procedural\", " + "which are reported as not verifiable by automated test." + ), + formatter_class=argparse.RawDescriptionHelpFormatter, + ) + trace_parser.add_argument( + "--results", + default="report/results.json", + help="Path to results.json (default: report/results.json)", + ) + trace_parser.add_argument( + "--controls", required=True, help="Path to the controls.toml control list" + ) + trace_parser.add_argument( + "--tag-prefix", + default="control-", + help="Gherkin tag prefix identifying control tags (default: control-)", + ) + trace_parser.add_argument( + "--format", choices=("csv", "json"), default="csv", help="Output format (default: csv)" + ) + trace_parser.add_argument( + "--output", default=None, help="Write to this path instead of stdout" + ) + trace_parser.set_defaults(func=run_trace) +``` + +Add `"trace": trace_parser,` to the `subcommand_parsers` dict. + +- [ ] Step 5: Run test to verify it passes + +Run: `uv run pytest selftests/test_trace_cli.py -v` +Expected: 10 passed + +- [ ] Step 6: Check the help renders + +Run: `uv run vip trace --help` +Expected: usage text listing all five options + +- [ ] Step 7: Commit + +```bash +git add src/vip/cli.py src/vip/traceability.py selftests/test_trace_cli.py +git commit -m "feat(cli): add vip trace for compliance traceability matrices" +``` + +--- + +### Task 11: `examples/part11_validation` scaffold template + +Files: +- Create: `examples/part11_validation/README.md`, `test_part11_validation.feature`, `test_part11_validation.py`, `conftest.py`, `controls.toml` +- Modify: `src/vip/cli.py:1087-1096` (`_SCAFFOLD_TEMPLATES`), `src/vip/cli.py:1136-1138` (`_scaffold_next_steps`) +- Modify: `pyproject.toml:157-163` (`force-include`) +- Test: `selftests/test_part11_example.py` (create) + +Interfaces: +- Consumes: the `@control-*` convention (Tasks 5-6), `vip trace` (Task 10) +- Produces: `vip scaffold --template part11-validation --output DIR` + +Two things that are easy to miss. The wheel embeds scaffold sources via `[tool.hatch.build.targets.wheel.force-include]` in `pyproject.toml` — without a new entry there, the template works from a source checkout and breaks from an installed wheel. And per CLAUDE.md every `@scenario` function needs a literal `@pytest.mark.connect` decorator: feature-level Gherkin tags alone do not drive auto-skip in extension directories. + +Scope honesty requirement: the README must state that this is a template, not a certified Part 11 test set, and that a fully green matrix is evidence for the subset of controls a customer chose to automate — not a Part 11 compliance attestation. Posit Team does not implement electronic signatures, so 11.50, 11.70 and all of subpart C cannot be evidenced by a test against these products. + +- [ ] Step 1: Write the failing test + +Create `selftests/test_part11_example.py`: + +```python +import subprocess +import sys +import tomllib +from pathlib import Path + +REPO = Path(__file__).resolve().parent.parent +EXAMPLE = REPO / "examples" / "part11_validation" + + +def test_example_directory_exists(): + assert (EXAMPLE / "test_part11_validation.feature").is_file() + assert (EXAMPLE / "test_part11_validation.py").is_file() + assert (EXAMPLE / "controls.toml").is_file() + assert (EXAMPLE / "README.md").is_file() + + +def test_template_is_registered(): + from vip.cli import _SCAFFOLD_TEMPLATES + + assert "part11-validation" in _SCAFFOLD_TEMPLATES + assert _SCAFFOLD_TEMPLATES["part11-validation"][0] == "part11_validation" + + +def test_template_is_bundled_into_the_wheel(): + """A template missing from force-include works in-repo and breaks when installed.""" + config = tomllib.loads((REPO / "pyproject.toml").read_text()) + includes = config["tool"]["hatch"]["build"]["targets"]["wheel"]["force-include"] + assert includes["examples/part11_validation"] == "vip/_scaffold/part11_validation" + + +def test_every_control_tag_is_defined_in_controls_toml(): + feature = (EXAMPLE / "test_part11_validation.feature").read_text() + tags = { + tok.lstrip("@") + for line in feature.splitlines() + for tok in line.split() + if tok.startswith("@control-") + } + controls = tomllib.loads((EXAMPLE / "controls.toml").read_text())["controls"] + for tag in tags: + assert tag.removeprefix("control-") in controls, f"{tag} missing from controls.toml" + + +def test_controls_toml_shows_the_not_automatable_path(): + """The worked example must demonstrate more than the happy path.""" + controls = tomllib.loads((EXAMPLE / "controls.toml").read_text())["controls"] + verifications = {c.get("verification", "automated") for c in controls.values()} + responsibilities = {c.get("responsibility") for c in controls.values()} + assert verifications & {"manual", "procedural"} + assert "customer" in responsibilities + + +def test_readme_states_it_is_not_an_attestation(): + text = (EXAMPLE / "README.md").read_text().lower() + assert "not a certified" in text or "not an attestation" in text + assert "electronic signature" in text + + +def test_scenarios_carry_literal_product_markers(): + """Feature-level Gherkin tags alone do not drive auto-skip in extensions.""" + steps = (EXAMPLE / "test_part11_validation.py").read_text() + assert steps.count("@pytest.mark.connect") + steps.count("@pytest.mark.workbench") >= 3 + + +def test_example_collects(): + proc = subprocess.run( + [sys.executable, "-m", "pytest", str(EXAMPLE), "--collect-only", "-q"], + capture_output=True, + text=True, + cwd=REPO, + ) + assert proc.returncode == 0, proc.stdout + proc.stderr +``` + +- [ ] Step 2: Run test to verify it fails + +Run: `uv run pytest selftests/test_part11_example.py -v` +Expected: FAIL on `test_example_directory_exists` + +- [ ] Step 3: Create the example + +`examples/part11_validation/controls.toml`: + +```toml +# Sample control list for `vip trace`. This is a TEMPLATE: replace these +# entries with your organisation's own regulatory mapping. VIP does not +# interpret `reference`, `risk` or `responsibility` -- they are carried +# through to the matrix verbatim. + +[controls.audit-trail-publish] +description = "Deployment of content is recorded with actor and timestamp" +reference = "21 CFR 11.10(e)" +risk = "high" +verification = "automated" +responsibility = "shared" +notes = "Retention duration of the audit log is a customer configuration decision." + +[controls.access-control-privileged-action] +description = "Only authorised individuals may perform privileged actions" +reference = "21 CFR 11.10(g)" +risk = "high" +verification = "automated" +responsibility = "shared" + +[controls.record-retention] +description = "Audit trail entries cannot be altered or deleted by ordinary users" +reference = "21 CFR 11.10(e)" +risk = "high" +verification = "automated" +responsibility = "shared" + +[controls.personnel-training] +description = "Personnel have the education, training and experience to perform their tasks" +reference = "21 CFR 11.10(i)" +risk = "medium" +verification = "procedural" +responsibility = "customer" +notes = "Evidenced by training records in your QMS. No automated test can establish this." + +[controls.signature-manifestation] +description = "Signed records display the signer's printed name, date/time and meaning" +reference = "21 CFR 11.50" +risk = "high" +verification = "manual" +responsibility = "customer" +notes = """ +Posit Team does not implement electronic signatures. This control is satisfied +by the application you build on top of Posit Team, not by the platform, and is +listed here to show how a non-automatable control appears in the matrix. +""" +``` + +`examples/part11_validation/test_part11_validation.feature`: + +```gherkin +@connect +Feature: Part 11 flavoured controls + As a validation lead in a regulated environment + I want automated evidence for the controls that can be automated + So that my traceability matrix is generated rather than hand-maintained + + @control-audit-trail-publish + Scenario: Publishing content is recorded with an actor and a timestamp + Given Connect is accessible at the configured URL + When I list recent audit log entries + Then each entry records an actor and a timestamp + + @control-access-control-privileged-action + Scenario: A privileged action requires authorisation + Given Connect is accessible at the configured URL + When I request a privileged administrative endpoint without credentials + Then the request is refused + + @control-record-retention + Scenario: Audit log entries cannot be deleted through the API + Given Connect is accessible at the configured URL + When I attempt to delete an audit log entry + Then the deletion is refused +``` + +`examples/part11_validation/conftest.py`: + +```python +"""Override points for the Part 11 example. + +Redefine these fixtures in your own conftest.py to point the scenarios at the +endpoints your deployment exposes. +""" + +import pytest + + +@pytest.fixture +def privileged_endpoint() -> str: + """An administrative endpoint that must refuse an unauthenticated caller.""" + return "/__api__/v1/users" + + +@pytest.fixture +def audit_log_endpoint() -> str: + """The audit log collection endpoint.""" + return "/__api__/v1/audit_logs" +``` + +`examples/part11_validation/test_part11_validation.py`: + +```python +"""Step definitions for the Part 11 example. + +Every @scenario function carries a literal @pytest.mark.connect decorator: +feature-level Gherkin tags alone do not drive VIP's auto-skip in extension +directories. +""" + +import pytest +from pytest_bdd import given, scenario, then, when + + +@pytest.mark.connect +@scenario("test_part11_validation.feature", "Publishing content is recorded with an actor and a timestamp") +def test_audit_trail_publish(): + pass + + +@pytest.mark.connect +@scenario("test_part11_validation.feature", "A privileged action requires authorisation") +def test_privileged_action_denied(): + pass + + +@pytest.mark.connect +@scenario("test_part11_validation.feature", "Audit log entries cannot be deleted through the API") +def test_audit_log_not_deletable(): + pass + + +@given("Connect is accessible at the configured URL") +def connect_accessible(connect_client): + if connect_client is None: + pytest.skip("Connect is not configured") + return connect_client + + +@when("I list recent audit log entries", target_fixture="audit_entries") +def list_audit_entries(connect_client, audit_log_endpoint): + response = connect_client.get(audit_log_endpoint) + if response.status_code == 404: + pytest.skip("this deployment does not expose an audit log endpoint") + payload = response.json() + return payload.get("results", payload) if isinstance(payload, dict) else payload + + +@then("each entry records an actor and a timestamp") +def entries_have_actor_and_timestamp(audit_entries): + if not audit_entries: + pytest.skip("no audit entries to inspect") + for entry in audit_entries: + assert entry.get("user_id") or entry.get("user_description"), ( + f"audit entry has no actor: {entry}" + ) + assert entry.get("time") or entry.get("timestamp"), ( + f"audit entry has no timestamp: {entry}" + ) + + +@when("I request a privileged administrative endpoint without credentials", + target_fixture="unauthenticated_response") +def request_privileged_endpoint(connect_client, privileged_endpoint): + return connect_client.get_unauthenticated(privileged_endpoint) + + +@then("the request is refused") +def request_refused(unauthenticated_response): + assert unauthenticated_response.status_code in (401, 403), ( + f"expected 401/403, got {unauthenticated_response.status_code}" + ) + + +@when("I attempt to delete an audit log entry", target_fixture="delete_response") +def delete_audit_entry(connect_client, audit_log_endpoint): + return connect_client.delete(f"{audit_log_endpoint}/1") + + +@then("the deletion is refused") +def deletion_refused(delete_response): + assert delete_response.status_code in (401, 403, 404, 405), ( + f"audit log deletion was not refused: {delete_response.status_code}" + ) +``` + +Note for the implementer: `connect_client.get_unauthenticated` and `.delete` may not exist on `src/vip/clients/connect.py`. This is the spec's stated open question. Check first; if either is missing, add it there following the existing method style (raw httpx, returns the response, no product SDK) as part of this task, with a selftest for the new client method. Do not inline httpx calls in the step file — that violates the four-layer architecture. + +`examples/part11_validation/README.md`: + +```markdown +# Part 11 validation example + +A worked example of mapping regulatory controls to automated tests, and +exporting the result as a traceability matrix with `vip trace`. + +## What this is not + +This is a template, not a certified Part 11 test set. A fully green matrix is +evidence for the subset of controls you chose to automate. It is not an +attestation of 21 CFR Part 11 compliance, and nobody should present it as one. + +Most of Part 11 cannot be evidenced by an automated test against Posit Team. +Roughly six clauses are genuinely testable against a deployment — 11.10(a), +11.10(d), 11.10(e), 11.10(g), 11.30, and partly 11.10(b). Several more are +shared with your own procedures. The rest are either procedural (11.10(i), +11.10(j)) or properties of the application you build on top of Posit Team. + +In particular, Posit Team does not implement electronic signatures. Clause +11.50 (signature manifestations), 11.70 (signature/record linking) and all of +subpart C (11.100, 11.200, 11.300) are satisfied by your application and your +SOPs, not by Connect, Workbench or Package Manager. `controls.toml` includes +two such controls so you can see how a non-automatable control appears in the +matrix: as "not verifiable by automated test", which is deliberately distinct +from "coverage gap". + +## How it works + +A scenario declares the control it satisfies with a Gherkin tag: + +```gherkin +@control-audit-trail-publish +Scenario: Publishing content is recorded with an actor and a timestamp +``` + +Write the product tag (`@connect`, `@workbench`) first — VIP derives the +feature's marker from the first non-control tag, and that value feeds the HTML +report and the generated test catalog. + +`controls.toml` names each control and carries whatever metadata your +regulatory mapping uses. Only `description` is required. + +## Running it + +```bash +vip verify --config vip.toml --vip-extensions ./part11_validation +vip trace --results report/results.json --controls ./part11_validation/controls.toml +``` + +Add `--format json` for a machine-readable matrix carrying the full provenance +block, or `--output matrix.csv` to write to a file. + +## Extending it + +Replace `controls.toml` with your own mapping and tag your own scenarios. See +`security/test_auth_policy.py` in the VIP source for a fuller reference +implementation of access-control testing, and +`examples/cross_product_validation/` for the broader GxP starting point. +``` + +- [ ] Step 4: Register the template + +In `src/vip/cli.py`, add to `_SCAFFOLD_TEMPLATES`: + +```python + "part11-validation": ( + "part11_validation", + "Compliance control tagging plus a controls.toml for `vip trace`", + ), +``` + +Add a branch to `_scaffold_next_steps`: + +```python + if template == "part11-validation": + return ( + f"\nNext steps:\n" + f" 1. Replace {dest / 'controls.toml'} with your own control list.\n" + f" 2. Tag your scenarios with @control- matching those ids.\n" + f" 3. Run: vip verify --vip-extensions {dest}\n" + f" 4. Run: vip trace --controls {dest / 'controls.toml'}\n" + ) +``` + +In `pyproject.toml`, add to `[tool.hatch.build.targets.wheel.force-include]`: + +```toml +"examples/part11_validation" = "vip/_scaffold/part11_validation" +``` + +- [ ] Step 5: Run test to verify it passes + +Run: `uv run pytest selftests/test_part11_example.py -v` +Expected: 8 passed + +- [ ] Step 6: Verify scaffolding end to end + +```bash +uv run vip scaffold --list +uv run vip scaffold --template part11-validation --output /tmp/p11-check +uv run vip trace --results report/results.json --controls /tmp/p11-check/controls.toml || true +``` +Expected: the template is listed, the directory is created with an `AGENTS.md`, and `vip trace` either produces a matrix or reports a missing results file cleanly. + +- [ ] Step 7: Lint and commit + +```bash +uv run ruff check src/ src/vip_tests/ selftests/ examples/ +uv run ruff format src/ src/vip_tests/ selftests/ examples/ +git add examples/part11_validation src/vip/cli.py pyproject.toml selftests/test_part11_example.py +git commit -m "feat(examples): add a part11 validation scaffold template" +``` + +--- + +### Task 12: Documentation + +Files: +- Modify: `docs/reporting.md`, `docs/test-architecture.md` +- Modify: `CLAUDE.md` (key source files table) +- Test: `selftests/test_scaffold_agents_md.py` (verify, update only if it fails) + +Interfaces: +- Consumes: everything above +- Produces: no code + +`docs/reporting.md` currently mentions none of `results.json`, `junit.xml` or `results.sarif`. Document the machine-readable outputs first, then layer the traceability export on top — `vip trace --help` alone is not discovery. + +- [ ] Step 1: Confirm the scaffold inventory test still passes + +Run: `uv run pytest selftests/test_scaffold_agents_md.py -v` +Expected: pass. The new example reuses existing fixtures and markers, so no inventory update should be needed. If it fails, update `examples/_shared/AGENTS.md` to match the real inventory. + +- [ ] Step 2: Document the machine-readable outputs in `docs/reporting.md` + +Add a section covering: the three output formats and how `--vip-format` selects them; the `results.json` field inventory including `schema_version`, `started_at`/`finished_at`, and the `execution` block; the schema compatibility policy (unknown minor accepted, unknown major refused); the `.sha256` sidecar and how to verify it with `shasum -a 256 -c results.json.sha256`. + +State the two honesty points in plain language, not just in the spec: +- `python_version`, `platform` and `execution.hostname` describe the VIP runner, not the system under test. The products table identifies the system under test. +- The checksum is tamper-evidence within a trusted pipeline, not tamper-proofing: anyone who can edit `results.json` can regenerate the sidecar. It catches corruption, truncated uploads and casual editing. + +Then add a traceability section covering the `controls.toml` format, the three coverage outcomes, and a worked `vip trace` invocation. + +- [ ] Step 3: Document the tagging convention in `docs/test-architecture.md` + +Add a section covering the `@control-` convention, the rule that the product tag goes first (and why: `gherkin.py` derives the feature marker from the first non-control tag), that tags become registered pytest markers so `--strict-markers` runs work, and that they flow into `results.json` markers but not into SARIF or JUnit. + +- [ ] Step 4: Update the key source files table in `CLAUDE.md` + +Add rows for `src/vip/attribution.py` and `src/vip/traceability.py`, and extend the `src/vip/cli.py` row to mention `trace`. + +- [ ] Step 5: Full verification + +```bash +uv run ruff check src/ src/vip_tests/ selftests/ examples/ +uv run ruff format --check src/ src/vip_tests/ selftests/ examples/ +uv run pytest selftests/ -q +uv run vip verify --config vip.toml --collect-only 2>/dev/null || uv run pytest src/vip_tests/ --collect-only -q +``` +Expected: lint clean, all selftests pass, product tests still collect. + +- [ ] Step 6: Commit + +```bash +git add docs/reporting.md docs/test-architecture.md CLAUDE.md +git commit -m "docs: document control tagging and the traceability export" +``` + +--- + +## Post-implementation cleanup + +Per CLAUDE.md, at the end of the plan remove the plan and the spec: + +```bash +git rm docs/superpowers/plans/2026-08-28-part11-traceability.md +git rm docs/superpowers/specs/2026-08-28-part11-traceability-design.md +git commit -m "chore: remove completed part11 traceability plan and spec" +``` + +Do this only after Task 12 is merged and verified. + +## Deliberately not in this plan + +Named so a later round does not rediscover them, and so nobody mistakes their +absence for an oversight. All are recorded in section 7.5 of the spec. + +- Step-level Given/When/Then evidence. Real qualification protocols cite a + protocol step; VIP captures nothing below the scenario. This is the largest + remaining gap against how these documents are actually written. +- Captured stdout/log as evidence on a failure row. +- A deviation log, which needs cross-run history. +- Cryptographic signing of the evidence record. +- Runtime versions on the system under test in the provenance block. +- SARIF `properties.tags` carrying control tags. +- PDF rendering, owned by a separate team. From 1e8f58a90c9f0715e756494069fd6e92c56fa779 Mon Sep 17 00:00:00 2001 From: Brian Deitte Date: Fri, 28 Aug 2026 16:17:38 -0500 Subject: [PATCH 007/106] fix(plan): make the audit-log immutability check non-destructive Review found the part11 example issued DELETE /v1/audit_logs/1 against the configured deployment. In a regulated environment the audit trail is the evidence, so deleting a real record to prove records cannot be deleted does the exact harm the control exists to prevent, and breaks VIP's non-destructive test contract. Now reads the Allow header via OPTIONS instead. Pushed into ConnectClient as domain methods rather than generic HTTP verbs, so the destructive shape is not expressible at the step layer; includes a regression test asserting only OPTIONS is ever issued. Also: ConnectClient has no generic get/delete/options at all, so all three methods the example calls are new -- the plan now specifies them with tests instead of a check-first note. And redact_userinfo now guards .port, which urlsplit parses lazily and which raises ValueError on 'https://host:bad/x', escaping the never-fail contract during report writing. --- .../plans/2026-08-28-part11-traceability.md | 198 +++++++++++++++--- .../2026-08-28-part11-traceability-design.md | 21 +- 2 files changed, 181 insertions(+), 38 deletions(-) diff --git a/docs/superpowers/plans/2026-08-28-part11-traceability.md b/docs/superpowers/plans/2026-08-28-part11-traceability.md index 82e356a3..6cd0b54d 100644 --- a/docs/superpowers/plans/2026-08-28-part11-traceability.md +++ b/docs/superpowers/plans/2026-08-28-part11-traceability.md @@ -373,6 +373,8 @@ def test_env_sha_takes_precedence_over_subprocess(tmp_path): ("https://user:pw@example.com:8443/o/r.git", "https://example.com:8443/o/r.git"), ("https://github.com/o/r", "https://github.com/o/r"), ("git@github.com:o/r.git", "github.com:o/r.git"), + # urlsplit accepts this and only raises when .port is read. + ("https://example.com:bad/repo", None), ("", None), (None, None), ], @@ -437,13 +439,17 @@ def redact_userinfo(url: str | None) -> str | None: return url.split("@", 1)[-1] if "@" in url else url try: parts = urlsplit(url) + hostname = parts.hostname + # urlsplit is lazy: it accepts "https://host:bad/x" and only raises + # when .port parses. Both accesses must sit inside the guard, or a + # malformed remote escapes the never-fail contract and takes down + # report writing at the end of an otherwise good run. + port = parts.port except ValueError: return None - if not parts.hostname: + if not hostname: return None - netloc = parts.hostname - if parts.port: - netloc = f"{netloc}:{parts.port}" + netloc = f"{hostname}:{port}" if port else hostname return urlunsplit((parts.scheme, netloc, parts.path, "", "")) @@ -2090,12 +2096,13 @@ git commit -m "feat(cli): add vip trace for compliance traceability matrices" Files: - Create: `examples/part11_validation/README.md`, `test_part11_validation.feature`, `test_part11_validation.py`, `conftest.py`, `controls.toml` - Modify: `src/vip/cli.py:1087-1096` (`_SCAFFOLD_TEMPLATES`), `src/vip/cli.py:1136-1138` (`_scaffold_next_steps`) +- Modify: `src/vip/clients/connect.py` (three new audit-log/authz methods — none exist today) - Modify: `pyproject.toml:157-163` (`force-include`) -- Test: `selftests/test_part11_example.py` (create) +- Test: `selftests/test_part11_example.py` (create), `selftests/test_connect_audit_client.py` (create) Interfaces: - Consumes: the `@control-*` convention (Tasks 5-6), `vip trace` (Task 10) -- Produces: `vip scaffold --template part11-validation --output DIR` +- Produces: `vip scaffold --template part11-validation --output DIR`; `ConnectClient.list_audit_logs(*, limit: int = 20) -> list[dict] | None`, `ConnectClient.audit_log_allowed_methods() -> set[str] | None`, `ConnectClient.unauthenticated_status(path: str) -> int` Two things that are easy to miss. The wheel embeds scaffold sources via `[tool.hatch.build.targets.wheel.force-include]` in `pyproject.toml` — without a new entry there, the template works from a source checkout and breaks from an installed wheel. And per CLAUDE.md every `@scenario` function needs a literal `@pytest.mark.connect` decorator: feature-level Gherkin tags alone do not drive auto-skip in extension directories. @@ -2260,12 +2267,20 @@ Feature: Part 11 flavoured controls Then the request is refused @control-record-retention - Scenario: Audit log entries cannot be deleted through the API + Scenario: The audit log does not offer a deletion method Given Connect is accessible at the configured URL - When I attempt to delete an audit log entry - Then the deletion is refused + When I ask which methods the audit log endpoint allows + Then deletion is not among them ``` +Note on that third scenario. An earlier draft issued a real +`DELETE /v1/audit_logs/1`. That is exactly backwards: in a regulated +deployment the audit trail is the evidence, so a test that destroys an entry +to prove entries cannot be destroyed has done the precise harm the control +exists to prevent — and it would have done it against a hard-coded, real ID. +VIP tests are non-destructive (CLAUDE.md), so the scenario reads the +advertised method set instead and never issues a mutating request. + `examples/part11_validation/conftest.py`: ```python @@ -2282,14 +2297,13 @@ import pytest def privileged_endpoint() -> str: """An administrative endpoint that must refuse an unauthenticated caller.""" return "/__api__/v1/users" - - -@pytest.fixture -def audit_log_endpoint() -> str: - """The audit log collection endpoint.""" - return "/__api__/v1/audit_logs" ``` +The audit log path is deliberately not a fixture here: it lives in +`ConnectClient` so the step layer never names a URL. Only the privileged +endpoint is overridable, because which action counts as privileged genuinely +varies by deployment. + `examples/part11_validation/test_part11_validation.py`: ```python @@ -2330,12 +2344,11 @@ def connect_accessible(connect_client): @when("I list recent audit log entries", target_fixture="audit_entries") -def list_audit_entries(connect_client, audit_log_endpoint): - response = connect_client.get(audit_log_endpoint) - if response.status_code == 404: +def list_audit_entries(connect_client): + entries = connect_client.list_audit_logs() + if entries is None: pytest.skip("this deployment does not expose an audit log endpoint") - payload = response.json() - return payload.get("results", payload) if isinstance(payload, dict) else payload + return entries @then("each entry records an actor and a timestamp") @@ -2352,31 +2365,150 @@ def entries_have_actor_and_timestamp(audit_entries): @when("I request a privileged administrative endpoint without credentials", - target_fixture="unauthenticated_response") + target_fixture="unauthenticated_status") def request_privileged_endpoint(connect_client, privileged_endpoint): - return connect_client.get_unauthenticated(privileged_endpoint) + return connect_client.unauthenticated_status(privileged_endpoint) @then("the request is refused") -def request_refused(unauthenticated_response): - assert unauthenticated_response.status_code in (401, 403), ( - f"expected 401/403, got {unauthenticated_response.status_code}" +def request_refused(unauthenticated_status): + assert unauthenticated_status in (401, 403), ( + f"expected 401/403, got {unauthenticated_status}" ) -@when("I attempt to delete an audit log entry", target_fixture="delete_response") -def delete_audit_entry(connect_client, audit_log_endpoint): - return connect_client.delete(f"{audit_log_endpoint}/1") +@when("I ask which methods the audit log endpoint allows", target_fixture="allowed_methods") +def audit_log_allowed_methods(connect_client): + """Read the advertised method set. Never issue a mutating request. + This scenario must not DELETE a real audit record to prove records cannot + be deleted -- in a regulated deployment that record is the evidence, and + destroying it is the exact harm this control exists to prevent. + """ + methods = connect_client.audit_log_allowed_methods() + if methods is None: + pytest.skip("this deployment does not advertise allowed methods for the audit log") + return methods -@then("the deletion is refused") -def deletion_refused(delete_response): - assert delete_response.status_code in (401, 403, 404, 405), ( - f"audit log deletion was not refused: {delete_response.status_code}" + +@then("deletion is not among them") +def deletion_not_offered(allowed_methods): + assert "DELETE" not in allowed_methods, ( + f"audit log endpoint advertises DELETE; allowed methods: {sorted(allowed_methods)}" ) ``` -Note for the implementer: `connect_client.get_unauthenticated` and `.delete` may not exist on `src/vip/clients/connect.py`. This is the spec's stated open question. Check first; if either is missing, add it there following the existing method style (raw httpx, returns the response, no product SDK) as part of this task, with a selftest for the new client method. Do not inline httpx calls in the step file — that violates the four-layer architecture. +This resolves the spec's open question about the Connect client. Verified against `src/vip/clients/connect.py` on main: it exposes only domain methods (`server_settings`, `list_users`, `delete_content`, `get_content`, ...) and no generic `get`, `delete`, `options`, or `get_unauthenticated`. All three methods the steps above call are new and must be added in this task. + +They are deliberately domain methods rather than generic HTTP verbs. Adding a public `get(path)` to `ConnectClient` would let any future step file drive raw HTTP from the test layer, which is what the four-layer architecture exists to prevent. `audit_log_allowed_methods()` returning a set is also what keeps the destructive-DELETE mistake from being expressible at the step layer at all. + +Add to `src/vip/clients/connect.py`, following the surrounding style (`self._client`, `raise_for_status()`, return dicts): + +```python + # -- Audit log ---------------------------------------------------------- + + def list_audit_logs(self, *, limit: int = 20) -> list[dict[str, Any]] | None: + """Return recent audit log entries, or None if unavailable. + + None (rather than an exception) for 404/403 so a caller can skip on a + deployment that does not expose the endpoint or a key that cannot read + it, without conflating that with an empty log. + """ + resp = self._client.get("/v1/audit_logs", params={"limit": limit}) + if resp.status_code in (403, 404): + return None + resp.raise_for_status() + payload = resp.json() + return payload.get("results", []) if isinstance(payload, dict) else payload + + def audit_log_allowed_methods(self) -> set[str] | None: + """Return the HTTP methods the audit log endpoint advertises, or None. + + Reads the Allow header from an OPTIONS request. Deliberately + read-only: proving the audit trail is immutable must never involve + deleting an audit record, which in a regulated deployment destroys the + very evidence the control protects. + """ + resp = self._client.request("OPTIONS", "/v1/audit_logs") + if resp.status_code in (401, 403, 404, 405, 501): + return None + allow = resp.headers.get("allow", "") + if not allow: + return None + return {m.strip().upper() for m in allow.split(",") if m.strip()} + + def unauthenticated_status(self, path: str) -> int: + """Return the status code for `path` requested with no credentials. + + Uses a separate short-lived client so the configured API key and + cookies are not sent. Routes through the same proxy map as every other + egress path (see the proxy notes in CLAUDE.md); never relies on httpx's + ambient env pickup. + """ + url = f"{self.base_url.rstrip('/')}{path}" + with httpx.Client( + verify=self.verify, + proxy=proxy_for_url(url, self.proxy_map), + trust_env=False, + timeout=30.0, + ) as client: + return client.get(url).status_code +``` + +Add `from vip.proxy import proxy_for_url` to the imports if it is not already there. + +Add `selftests/test_connect_audit_client.py` covering all three against a stubbed transport: + +```python +import httpx +import pytest + +from vip.clients.connect import ConnectClient + + +def _client(handler): + c = ConnectClient(base_url="https://connect.example.com", api_key="k") + c._client = httpx.Client( + transport=httpx.MockTransport(handler), base_url="https://connect.example.com" + ) + return c + + +def test_list_audit_logs_returns_results(): + def handler(request): + assert request.url.path == "/v1/audit_logs" + return httpx.Response(200, json={"results": [{"user_id": 1, "time": "t"}]}) + + assert _client(handler).list_audit_logs() == [{"user_id": 1, "time": "t"}] + + +@pytest.mark.parametrize("status", [403, 404]) +def test_list_audit_logs_returns_none_when_unavailable(status): + assert _client(lambda r: httpx.Response(status)).list_audit_logs() is None + + +def test_allowed_methods_parses_the_allow_header(): + handler = lambda r: httpx.Response(200, headers={"Allow": "GET, HEAD, OPTIONS"}) # noqa: E731 + assert _client(handler).audit_log_allowed_methods() == {"GET", "HEAD", "OPTIONS"} + + +def test_allowed_methods_returns_none_without_an_allow_header(): + assert _client(lambda r: httpx.Response(200)).audit_log_allowed_methods() is None + + +def test_allowed_methods_never_issues_a_mutating_request(): + """Regression guard for the non-destructive contract.""" + seen = [] + + def handler(request): + seen.append(request.method) + return httpx.Response(200, headers={"Allow": "GET"}) + + _client(handler).audit_log_allowed_methods() + assert seen == ["OPTIONS"] +``` + +If `/v1/audit_logs` turns out not to be the endpoint this deployment exposes, adjust the path in the client — not in the step file. `examples/part11_validation/README.md`: diff --git a/docs/superpowers/specs/2026-08-28-part11-traceability-design.md b/docs/superpowers/specs/2026-08-28-part11-traceability-design.md index 9be7ef4f..be2ff48a 100644 --- a/docs/superpowers/specs/2026-08-28-part11-traceability-design.md +++ b/docs/superpowers/specs/2026-08-28-part11-traceability-design.md @@ -410,8 +410,14 @@ customers don't have to guess which is the GxP starting point. (`@control-access-control-privileged-action`). Illustrative here, not a duplicate of `security/test_auth_policy.py` — the README points there for the fuller reference implementation. - - Audit-log non-deletability: a non-admin cannot delete or alter an - existing audit-trail entry via the API (`@control-record-retention`). + - Audit-log non-deletability: the audit log endpoint does not advertise a + deletion method (`@control-record-retention`). This is read-only by + construction — it reads the `Allow` header rather than attempting a + delete. Proving the audit trail is immutable must never destroy an audit + record: in a regulated deployment that record is the evidence, so the + obvious "try to delete one and assert it fails" shape would do the exact + harm the control exists to prevent, and would violate VIP's + non-destructive test contract. - `test_part11_validation.py` — thin step definitions; logic pushed into `clients/connect.py` (extended only if a needed method doesn't already exist). Every `@scenario` function also carries a literal @@ -617,9 +623,14 @@ that layer to render it. ## Open questions for implementation -- Whether `clients/connect.py` already exposes what's needed to read an - audit-trail entry and attempt its deletion as a non-admin, or whether new - client methods are required. +- Resolved. `clients/connect.py` exposes only domain methods and no generic + `get`/`delete`/`options`, so the example needs three new ones: + `list_audit_logs`, `audit_log_allowed_methods`, and + `unauthenticated_status`. They are domain methods rather than generic HTTP + verbs on purpose — a public `get(path)` would let any future step file drive + raw HTTP from the test layer, which is what the four-layer architecture + exists to prevent, and returning a method set rather than a response object + is what makes the destructive-delete shape inexpressible at the step layer. - Whether `examples/part11_validation` should be a third example or a control-tagged extension of `examples/cross_product_validation` (section 4). - Whether the `execution` block should also be surfaced in the HTML report's From 220b2339f107559f2159112746642986477ebaed Mon Sep 17 00:00:00 2001 From: Brian Deitte Date: Fri, 28 Aug 2026 16:21:03 -0500 Subject: [PATCH 008/106] docs: reconcile part11 spec and plan with the PDF report PR PR #618 adds a native Quarto/Typst PDF edition and restructures the report layer. Verified against its branch: it does not touch reporting.py, plugin.py or gherkin.py, so the evidence-record and tagging work (plan tasks 1-6) cannot conflict with it. It does collide with cli.py, the pyproject force-include block and AGENTS.md, which tasks 10-12 all touch. Retired the PDF non-goal, which assumed an external team owned rendering -- VIP now renders its own. Whether the matrix becomes a PDF section is recorded as an open decision rather than silently assumed either way; recommendation is to ship the CSV/JSON export first and not render an unsettled data model into two backends. Updated the moved references: the Gherkin step lookup went to report_content.py:261, and provenance rendering split into provenance_rows plus one renderer per backend. --- .../plans/2026-08-28-part11-traceability.md | 29 +++- .../2026-08-28-part11-traceability-design.md | 130 +++++++++++++++--- 2 files changed, 138 insertions(+), 21 deletions(-) diff --git a/docs/superpowers/plans/2026-08-28-part11-traceability.md b/docs/superpowers/plans/2026-08-28-part11-traceability.md index 6cd0b54d..0f4ba043 100644 --- a/docs/superpowers/plans/2026-08-28-part11-traceability.md +++ b/docs/superpowers/plans/2026-08-28-part11-traceability.md @@ -21,6 +21,29 @@ Spec: `docs/superpowers/specs/2026-08-28-part11-traceability-design.md` - Commit after every task. Conventional-commit titles, lowercase description, no trailing period, under 70 chars. - No `**` bold in any markdown file this plan creates or edits. +## Relationship to PR #618 (the PDF report) + +This plan is written against `main`. PR #618 (`feat/report-pdf`) is open and +restructures the report layer. Verified against its branch: + +- It does not touch `src/vip/reporting.py`, `src/vip/plugin.py` or + `src/vip/gherkin.py`. Tasks 1 through 6 — the entire evidence record and the + tagging work — are free of textual conflict with it. +- It does collide with Tasks 10, 11 and 12: `src/vip/cli.py`, + `pyproject.toml`'s `force-include` block, and `AGENTS.md`. + +Line numbers below are given for `main`, with the #618 value alongside where +they differ. If #618 merges before you reach Task 10, rebase and re-derive the +three `cli.py` insertion points rather than trusting any number here: +`_SCAFFOLD_TEMPLATES` moves 1087 -> 1139, `_scaffold_next_steps` 1136 -> 1188, +and the `subcommand_parsers` map 1936 -> 1989. + +Rendering the traceability matrix into the PDF is deliberately not in this +plan. See section 8.4 of the spec for why, and for the three constraints +(#618's shared-content-layer rule, `_lit` escaping of customer-supplied control +text, and `render_document`'s lack of any control-list input) that a follow-up +would have to satisfy. + ## Task ordering and shipping seams Tasks 1-4 (evidence record) are independent of 5-11 and ship value alone. Tasks 5-6 (tagging) are independent of 1-4. Task 7 onward consumes both. If this needs to be split across people, the seam is after Task 4 and after Task 6. @@ -769,7 +792,7 @@ Interfaces: - Consumes: nothing - Produces: `gherkin.CONTROL_TAG_PREFIX: str` (value `"control-"`); `parse_feature_file()` return dict gains a `tags: list[str]` key holding every tag in the file, `@` stripped -Why this comes before the plugin change: `gherkin.py` currently derives a feature's `marker` from the first token of the first tag line (`:56-57`), so `@control-x @connect` sets the marker to `control-x`. That value feeds the HTML report cards (`report_html.py:241`), `scripts/generate-test-catalog.py:46` and `scripts/generate-feature-matrix.py:142`. Task 6 also needs a tag list this parser does not currently return. +Why this comes before the plugin change: `gherkin.py` currently derives a feature's `marker` from the first token of the first tag line (`:56-57`), so `@control-x @connect` sets the marker to `control-x`. That value feeds the report's Gherkin step lookup (`report_content.py:261` on pr-618, `report_html.py:241` on main), `scripts/generate-test-catalog.py:46` and `scripts/generate-feature-matrix.py:142`. Task 6 also needs a tag list this parser does not currently return. - [ ] Step 1: Write the failing test @@ -2095,9 +2118,9 @@ git commit -m "feat(cli): add vip trace for compliance traceability matrices" Files: - Create: `examples/part11_validation/README.md`, `test_part11_validation.feature`, `test_part11_validation.py`, `conftest.py`, `controls.toml` -- Modify: `src/vip/cli.py:1087-1096` (`_SCAFFOLD_TEMPLATES`), `src/vip/cli.py:1136-1138` (`_scaffold_next_steps`) +- Modify: `src/vip/cli.py` `_SCAFFOLD_TEMPLATES` (`:1087` on main, `:1139` on pr-618) and `_scaffold_next_steps` (`:1136` on main, `:1188` on pr-618) - Modify: `src/vip/clients/connect.py` (three new audit-log/authz methods — none exist today) -- Modify: `pyproject.toml:157-163` (`force-include`) +- Modify: `pyproject.toml` `[tool.hatch.build.targets.wheel.force-include]` (`:157-163` on main, `:157-181` on pr-618, which adds the PDF template and vendored fonts) - Test: `selftests/test_part11_example.py` (create), `selftests/test_connect_audit_client.py` (create) Interfaces: diff --git a/docs/superpowers/specs/2026-08-28-part11-traceability-design.md b/docs/superpowers/specs/2026-08-28-part11-traceability-design.md index be2ff48a..5258a343 100644 --- a/docs/superpowers/specs/2026-08-28-part11-traceability-design.md +++ b/docs/superpowers/specs/2026-08-28-part11-traceability-design.md @@ -13,24 +13,40 @@ test evidence. VIP already produces machine-readable per-check results (`report/results.json`, JUnit XML, SARIF), and its Gherkin feature files are already written as plain-language requirement statements. -Be precise about what "timestamped, versioned" does and does not mean today, -because the downstream PDF pipeline will build on this wording. Before the -section 2 changes land, `results.json` carries a single run-level -`generated_at` and no schema version at all: individual results have a -`duration` but no start time. So the existing evidence is run-timestamped and -unversioned, and a per-control row could not honestly claim a timestamp of its -own. Section 2 is what makes per-check timestamping and a versioned schema -true rather than aspirational. What's missing is a way to attach a control ID to a scenario, a way to turn that into an actual matrix, enough provenance on the evidence record to make a result attributable to a specific pipeline execution, and a worked example showing what real Part 11-flavored scenarios look like. -PDF rendering of the final matrix is being handled by a separate team and is -explicitly out of scope here. A historical/longitudinal evidence store across -runs (needed for a true deviation log) was considered and explicitly dropped -for this round — VIP already emits one timestamped file per run; whoever owns -long-term archiving can accumulate those without any new VIP storage code. +Be precise about what "timestamped, versioned" does and does not mean today, +because a downstream renderer will build on this wording. Before the section 2 +changes land, `results.json` carries a single run-level `generated_at` and no +schema version at all: individual results have a `duration` but no start time. +So the existing evidence is run-timestamped and unversioned, and a per-control +row could not honestly claim a timestamp of its own. Section 2 is what makes +per-check timestamping and a versioned schema true rather than aspirational. + +A historical/longitudinal evidence store across runs (needed for a true +deviation log) was considered and explicitly dropped for this round — VIP +already emits one timestamped file per run; whoever owns long-term archiving +can accumulate those without any new VIP storage code. + +### Superseded: the PDF non-goal + +The first two drafts of this spec said PDF rendering was owned by a separate +team and out of scope, and pointed at AlcoaBase as the model for VIP producing +machine-readable input that someone else renders. PR #618 changes that premise: +VIP now renders its own PDF edition of the report, natively via Quarto/Typst +(`src/vip/report_typst.py`, `report/vip-report.qmd`), and every +`quarto render` produces `_output/vip-report.pdf`. + +So "a downstream PDF pipeline" is no longer hypothetical or external — it is in +this repo. That does not by itself mean the traceability matrix belongs in the +PDF, but it does mean the choice is now a live design decision rather than +something ruled out by ownership. Section 8 records it. The CSV/JSON export +remains the primary deliverable either way: a spreadsheet and an external +qualification-protocol generator are both real consumers, and neither wants a +PDF. ## Goals @@ -48,7 +64,8 @@ long-term archiving can accumulate those without any new VIP storage code. ## Non-goals -- No PDF generation (a separate team owns that). +- No new PDF *engine*. PR #618 already added one; whether the matrix becomes a + section inside it is an open decision (section 8), not a goal assumed here. - No historical/deviation tracking across multiple runs. - No VIP-shipped canonical CFR Part 11 control taxonomy. The control list is supplied by whoever owns the regulatory mapping; VIP stays @@ -127,7 +144,8 @@ Three corrections to the original draft of this section: - Tag ordering in a feature file matters. `gherkin.py:52-57` derives a feature's `"marker"` from the first token of the first tag line in the file. A `@control-*` tag written before `@connect` hijacks that value, which feeds - the report cards (`report_html.py:241`), `generate-test-catalog.py:46`, and + the report's Gherkin step lookup (`report_content.py:261` after #618, + `report_html.py:241` before it), `generate-test-catalog.py:46`, and `generate-feature-matrix.py:142`. Fix `gherkin.py` to skip tags matching the control prefix when deriving the marker, rather than relying on authors to order tags correctly. @@ -370,7 +388,8 @@ which is retained precisely because it does. New subcommand in `src/vip/cli.py`, following the existing argparse + `set_defaults(func=...)` pattern used by the nine current subcommands, and -added to the `subcommand_parsers` help map at `cli.py:1936`: +added to the `subcommand_parsers` help map (`cli.py:1936` on main, +`cli.py:1989-1999` after #618): ``` vip trace --results report/results.json --controls path/to/controls.toml \ @@ -621,6 +640,78 @@ and document-management layer. The right division of labour is for VIP to produce clean, deterministic, well-provenanced machine-readable input and for that layer to render it. +## 8. Interaction with PR #618 (the PDF report) + +PR #618 (`feat/report-pdf`) adds a native Quarto/Typst PDF edition of the +report. Verified against the PR branch, here is exactly how it touches this +work. + +### 8.1 What it does not touch + +`src/vip/reporting.py`, `src/vip/plugin.py` and `src/vip/gherkin.py` do not +appear in its diff at all. Every one of section 2's evidence-record changes and +section 1's tagging changes is therefore free of textual conflict with it. #618 +*consumes* those modules (`report_content.py` imports `parse_feature_file`, +`ReportData`, `TestResult`; `vip-report.qmd` imports `load_results`), so adding +fields or dict keys is a semantic change it inherits, not a collision. + +### 8.2 What it moves + +The report layer was restructured: content decisions extracted into a new +`src/vip/report_content.py`, with `report_html.py` reduced to markup only and a +new `report_typst.py` beside it. Two references in this spec moved: + +- the Gherkin step lookup that consumes a feature's derived marker: + `report_html.py:241` -> `report_content.py:261` +- provenance rendering: the old single `report_html.py:663-689` is now + `provenance_rows` in `report_content.py:367-388` (data) plus a renderer in + each backend (`report_html.py:401-413`, `report_typst.py:423-437`) + +### 8.3 Where it collides + +Textual conflicts to expect when this work is implemented on top of #618: +`src/vip/cli.py` (both add subcommand plumbing and constants near +`_REPORT_TEMPLATE_FILES`), `pyproject.toml` (both extend the +`force-include` block), and `AGENTS.md`. Section 4's scaffold registry moved +from `cli.py:1087` to `cli.py:1139`, and the `subcommand_parsers` help map from +`cli.py:1936` to `cli.py:1989`. + +One trap worth naming: `selftests/test_cli_report.py` has a test keeping the +force-include block in sync with `_REPORT_TEMPLATE_FILES`, but it filters on +`vip/_report/`. A scaffold template maps to `vip/_scaffold/` and is silently +outside that filter, so it provides no coverage for section 4's new entry — +which is why the plan asserts the scaffold entry directly instead of assuming +the existing guard catches it. + +### 8.4 The open decision: matrix as a PDF section + +#618 makes it possible to render the traceability matrix into the archivable +PDF alongside the summary and per-check listing. That is attractive for this +audience — a validation lead wants one signed, archivable artifact, not a CSV +they must paste into a document — but it is a scope increase and is not +required by anything in sections 1-6. + +If it is taken up, three constraints apply, all from #618 itself: + +- AGENTS.md on that branch states that visual changes must land in + `report_content` and `styles.css` in the same commit so the HTML and PDF + editions stay identical. So the matrix cannot be a Typst-only section: it + needs a shared content layer plus both backends. +- Every dynamic value must pass through `report_typst._lit`. A control + description containing `#`, `*`, `_` or `$` is live Typst markup otherwise, + and control descriptions are customer-supplied free text — this is an + injection surface, not a cosmetic concern. +- `render_document(data, hints)` currently takes only what `results.json` + provides. A matrix additionally needs a `controls.toml`, which the report + pipeline has no notion of today, so either `vip-report.qmd` grows a control- + list load with a sensible "no control list configured, skip the section" + path, or `render_document` grows a parameter. The first is less invasive. + +Recommended: ship sections 1-6 first (CSV/JSON export, which serves the +spreadsheet and external-generator consumers), and treat the PDF section as a +follow-up once the matrix data model has settled. Rendering an unstable data +model into two backends doubles the cost of every change to it. + ## Open questions for implementation - Resolved. `clients/connect.py` exposes only domain methods and no generic @@ -634,7 +725,10 @@ that layer to render it. - Whether `examples/part11_validation` should be a third example or a control-tagged extension of `examples/cross_product_validation` (section 4). - Whether the `execution` block should also be surfaced in the HTML report's - provenance table (`report_html.py:663-689`), which today shows six fields and - renders `None` as "not recorded". Cheap, and it would make the same + provenance rows, which today show six fields and render `None` as "not + recorded". After #618 this is no longer one function: the row data lives in + `report_content.py:367-388` (`provenance_rows`) and each backend renders it + separately (`report_html.py:401-413`, `report_typst.py:423-437`), so adding a + row means touching the shared layer and both backends together. Cheap, and it would make the same attribution visible to a human reader, but it widens the diff beyond the machine-readable path this spec is scoped to. From 15ac0996122feca577ef151899873658ccd7fb30 Mon Sep 17 00:00:00 2001 From: Brian Deitte Date: Fri, 28 Aug 2026 16:32:58 -0500 Subject: [PATCH 009/106] feat(plan): add tasks 13-14 rendering the matrix in both report editions Task 13 builds the format-neutral rows in report_content.py; task 14 renders them in report_html and report_typst together, which #618's AGENTS.md requires so the two editions cannot drift. Gated on #618 merging: task 13 modifies report_content.py, which does not exist on main. Placed last so tasks 1-12 still ship if that PR slips. Two hazards handled explicitly. Control descriptions are customer-authored, so a bare # is live Typst markup -- every cell goes through _lit, with a test asserting #panic() in a description renders inert. And render_document grows an optional third parameter rather than a required one, so a deployment with no control list gets byte-identical output to today. Also fixes two review findings on the connect client: the ad-hoc unauthenticated probe now uses verify_with_env_ca (trust_env=False disables SSL_CERT_FILE too, so without it the probe fails TLS against a corporate CA the pooled client accepts), and unauthenticated_status gets real tests asserting it sends no credentials. --- .../plans/2026-08-28-part11-traceability.md | 569 +++++++++++++++++- 1 file changed, 556 insertions(+), 13 deletions(-) diff --git a/docs/superpowers/plans/2026-08-28-part11-traceability.md b/docs/superpowers/plans/2026-08-28-part11-traceability.md index 0f4ba043..a872412a 100644 --- a/docs/superpowers/plans/2026-08-28-part11-traceability.md +++ b/docs/superpowers/plans/2026-08-28-part11-traceability.md @@ -2,7 +2,7 @@ > For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. -Goal: Let a Gherkin scenario declare the compliance control it satisfies, harden `results.json` into an attributable evidence record, and add a `vip trace` command that joins the two into a traceability matrix. +Goal: Let a Gherkin scenario declare the compliance control it satisfies, harden `results.json` into an attributable evidence record, add a `vip trace` command that joins the two into a traceability matrix, and render that matrix into the HTML and PDF report editions. Architecture: Three layers, built bottom-up. First `results.json` gains provenance (schema version, per-test timestamps, execution attribution, a checksum sidecar) — this ships value on its own. Then `@control-*` Gherkin tags are registered as real pytest markers so they survive strict-marker CI and reach `results.json`. Finally a new `src/vip/traceability.py` joins a customer-supplied `controls.toml` against those markers and renders CSV/JSON for a downstream PDF pipeline. @@ -38,16 +38,20 @@ three `cli.py` insertion points rather than trusting any number here: `_SCAFFOLD_TEMPLATES` moves 1087 -> 1139, `_scaffold_next_steps` 1136 -> 1188, and the `subcommand_parsers` map 1936 -> 1989. -Rendering the traceability matrix into the PDF is deliberately not in this -plan. See section 8.4 of the spec for why, and for the three constraints -(#618's shared-content-layer rule, `_lit` escaping of customer-supplied control -text, and `render_document`'s lack of any control-list input) that a follow-up -would have to satisfy. +Tasks 13 and 14 render the traceability matrix into both report editions and +therefore depend on #618 being merged. They are the last two tasks on purpose: +everything through Task 12 stands alone, so if #618 slips or changes shape, the +CSV/JSON export still ships. Do not start Task 13 before #618 lands -- it +modifies `report_content.py`, which does not exist on main. ## Task ordering and shipping seams Tasks 1-4 (evidence record) are independent of 5-11 and ship value alone. Tasks 5-6 (tagging) are independent of 1-4. Task 7 onward consumes both. If this needs to be split across people, the seam is after Task 4 and after Task 6. +Tasks 13-14 are a separate phase gated on PR #618. They render the matrix into +the HTML and PDF report editions. Task 12 is the natural release point; 13-14 +are additive on top of a working feature, not a prerequisite for it. + --- ### Task 1: `schema_version` on results.json @@ -2464,21 +2468,32 @@ Add to `src/vip/clients/connect.py`, following the surrounding style (`self._cli """Return the status code for `path` requested with no credentials. Uses a separate short-lived client so the configured API key and - cookies are not sent. Routes through the same proxy map as every other - egress path (see the proxy notes in CLAUDE.md); never relies on httpx's - ambient env pickup. + cookies are not sent -- sending them would make the scenario assert + nothing at all, since an authorised caller is *supposed* to get 200. + + Mirrors ``fetch_content``'s ad-hoc-request contract: route through the + proxy the pooled client already resolved and pin ``trust_env=False`` + so that decision is authoritative, then fold the CA env overrides back + in by hand. ``trust_env=False`` disables httpx's own reading of + ``SSL_CERT_FILE``/``SSL_CERT_DIR`` along with the proxy vars, so + without ``verify_with_env_ca`` this probe would fail TLS against a + corporate CA that the pooled client verifies fine -- surfacing as a + Part 11 scenario erroring on a deployment that is actually healthy. """ + from vip.proxy import proxy_for_url, verify_with_env_ca + url = f"{self.base_url.rstrip('/')}{path}" with httpx.Client( - verify=self.verify, - proxy=proxy_for_url(url, self.proxy_map), + verify=verify_with_env_ca(self._verify), + proxy=proxy_for_url(url, self._proxy_map), trust_env=False, timeout=30.0, ) as client: return client.get(url).status_code ``` -Add `from vip.proxy import proxy_for_url` to the imports if it is not already there. +The deferred import matches `fetch_content` (`connect.py:362`), which is the +established pattern for ad-hoc requests in this client. Add `selftests/test_connect_audit_client.py` covering all three against a stubbed transport: @@ -2529,8 +2544,79 @@ def test_allowed_methods_never_issues_a_mutating_request(): _client(handler).audit_log_allowed_methods() assert seen == ["OPTIONS"] + + +class _RecordingClient: + """Stands in for httpx.Client so we can inspect how it was constructed. + + unauthenticated_status builds its own client rather than using + self._client, so MockTransport on the pooled client cannot see it. + """ + + instances: list["_RecordingClient"] = [] + + def __init__(self, **kwargs): + self.kwargs = kwargs + self.requested = None + _RecordingClient.instances.append(self) + + def __enter__(self): + return self + + def __exit__(self, *exc): + return False + + def get(self, url): + self.requested = url + return httpx.Response(401, request=httpx.Request("GET", url)) + + +@pytest.fixture +def recording_client(monkeypatch): + _RecordingClient.instances = [] + monkeypatch.setattr(httpx, "Client", _RecordingClient) + return _RecordingClient + + +def test_unauthenticated_status_returns_the_status(recording_client): + c = ConnectClient(base_url="https://connect.example.com", api_key="k") + assert c.unauthenticated_status("/__api__/v1/users") == 401 + assert recording_client.instances[0].requested == ( + "https://connect.example.com/__api__/v1/users" + ) + + +def test_unauthenticated_status_sends_no_credentials(recording_client): + """The whole point of the method: an authorised caller would get 200.""" + c = ConnectClient(base_url="https://connect.example.com", api_key="SECRET_KEY") + c.unauthenticated_status("/__api__/v1/users") + + kwargs = recording_client.instances[0].kwargs + # No auth, no cookies, and no headers carrying the key were configured. + assert "auth" not in kwargs or kwargs["auth"] is None + assert not kwargs.get("cookies") + assert "SECRET_KEY" not in repr(kwargs.get("headers", {})) + + +def test_unauthenticated_status_pins_trust_env_and_keeps_env_ca(recording_client): + """trust_env=False also disables SSL_CERT_FILE; verify_with_env_ca restores it.""" + from vip.proxy import verify_with_env_ca + + c = ConnectClient(base_url="https://connect.example.com", api_key="k") + c.unauthenticated_status("/__api__/v1/users") + + kwargs = recording_client.instances[0].kwargs + assert kwargs["trust_env"] is False + assert kwargs["verify"] == verify_with_env_ca(c._verify) + assert "proxy" in kwargs ``` +Note the last test compares against `verify_with_env_ca(c._verify)` rather than +asserting a literal: the point is that the ad-hoc client's trust store matches +what the pooled client would use, not that it equals any particular value. If +`verify_with_env_ca` returns a fresh `SSLContext` per call, compare +`type(...)` and the CA env vars instead of using `==`. + If `/v1/audit_logs` turns out not to be the endpoint this deployment exposes, adjust the path in the client — not in the step file. `examples/part11_validation/README.md`: @@ -2705,6 +2791,460 @@ git commit -m "docs: document control tagging and the traceability export" --- +### Task 13: Traceability section in the shared content layer + +Files: +- Modify: `src/vip/report_content.py` (new content function + constants) +- Modify: `src/vip/reporting.py` (control-list discovery for the report pipeline) +- Test: `selftests/test_report_content_traceability.py` (create) + +Interfaces: +- Consumes: `traceability.build_traceability_matrix`, `load_controls`, `TraceabilityMatrix` (Tasks 7-9) +- Produces: `report_content.traceability_rows(matrix) -> list[list[str]]`; `report_content.TRACEABILITY_HEADERS: list[str]`; `report_content.COVERAGE_LABELS: dict[str, str]`; `reporting.controls_path() -> Path | None` + +Why the shared layer first, in its own task: AGENTS.md on #618 requires that +visual changes land in `report_content` with both backends updated in the same +commit, so the HTML and PDF editions cannot drift. Task 13 builds the +format-neutral half; Task 14 renders it in both backends together. Splitting +that way keeps each task independently testable without ever committing a +state where the two editions disagree. + +The report pipeline has no notion of a control list today. `vip-report.qmd` +loads `results.json` from the working report directory; the control list is +resolved the same way, by convention, so no config schema changes. A report +directory with no `controls.toml` simply has no traceability section — this +must be the silent, ordinary case, since every existing user has no control +list and their report must not sprout an error. + +- [ ] Step 1: Write the failing test + +Create `selftests/test_report_content_traceability.py`: + +```python +from vip.report_content import COVERAGE_LABELS, TRACEABILITY_HEADERS, traceability_rows +from vip.reporting import ReportData, TestResult +from vip.traceability import ControlSpec, build_traceability_matrix + + +def _matrix(): + data = ReportData( + results=[ + TestResult( + nodeid="t.py::a", + outcome="passed", + markers=["control-x"], + scenario_title="Publishing is recorded", + started_at="2026-08-28T12:00:00+00:00", + ) + ] + ) + controls = { + "x": ControlSpec("x", "Audit trail", reference="21 CFR 11.10(e)", risk="high"), + "y": ControlSpec("y", "Training", verification="procedural"), + "z": ControlSpec("z", "Untested control"), + } + return build_traceability_matrix(data, controls) + + +def test_headers_are_stable(): + assert TRACEABILITY_HEADERS == ["Control", "Requirement", "Coverage", "Evidence"] + + +def test_one_row_per_control_not_per_match(): + """The PDF is page-constrained; collapse matches into one cell.""" + rows = traceability_rows(_matrix()) + assert len(rows) == 3 + assert [r[0] for r in rows] == ["x", "y", "z"] + + +def test_covered_row_names_the_scenario_and_status(): + row = traceability_rows(_matrix())[0] + assert "Publishing is recorded" in row[3] + assert "passed" in row[3] + + +def test_reference_is_folded_into_the_requirement_cell(): + assert "21 CFR 11.10(e)" in traceability_rows(_matrix())[0][1] + + +def test_gap_and_not_automatable_read_differently(): + rows = {r[0]: r for r in traceability_rows(_matrix())} + assert rows["y"][2] == COVERAGE_LABELS["not_automatable"] + assert rows["z"][2] == COVERAGE_LABELS["gap"] + assert rows["y"][2] != rows["z"][2] + + +def test_uncovered_evidence_cell_is_not_blank(): + """A blank cell reads as a rendering bug, not as an absence of evidence.""" + rows = {r[0]: r for r in traceability_rows(_matrix())} + assert rows["z"][3].strip() + assert rows["y"][3].strip() + + +def test_every_cell_is_a_string(): + """Both backends escape strings; a None would crash or print 'None'.""" + for row in traceability_rows(_matrix()): + assert len(row) == len(TRACEABILITY_HEADERS) + assert all(isinstance(cell, str) for cell in row) +``` + +- [ ] Step 2: Run test to verify it fails + +Run: `uv run pytest selftests/test_report_content_traceability.py -v` +Expected: FAIL with `ImportError: cannot import name 'traceability_rows'` + +- [ ] Step 3: Write minimal implementation + +Append to `src/vip/report_content.py`: + +```python +# Coverage wording. "not automatable" must never read as a gap: a control +# nobody can automate is a statement about the control, not a hole in the +# test suite, and an auditor reading the two as the same thing is exactly +# the misreading this whole feature exists to prevent. +COVERAGE_LABELS = { + "covered": "covered", + "gap": "no automated test", + "not_automatable": "not verifiable by automated test", +} + +TRACEABILITY_HEADERS = ["Control", "Requirement", "Coverage", "Evidence"] + +# Shown in the Evidence column when there is nothing to cite. +_NO_EVIDENCE = { + "gap": "none — no scenario carries this control's tag", + "not_automatable": "outside the automated suite by design", +} + + +def traceability_rows(matrix) -> list[list[str]]: + """Format-neutral rows for the traceability matrix table. + + One row per control rather than per match: the PDF is page-constrained and + a control satisfied by six scenarios would otherwise push everything else + off the page. The CSV export (`vip trace --format csv`) is the per-match + view; this is the at-a-glance one. + """ + rows: list[list[str]] = [] + for entry in matrix.entries: + control = entry.control + requirement = control.description + if control.reference: + requirement = f"{requirement} ({control.reference})" + if entry.matches: + evidence = "; ".join( + f"{m.scenario_title or m.nodeid} — {m.status}" for m in entry.matches + ) + else: + evidence = _NO_EVIDENCE.get(entry.coverage, "none") + rows.append( + [ + control.control_id, + requirement, + COVERAGE_LABELS.get(entry.coverage, entry.coverage), + evidence, + ] + ) + return rows +``` + +Add to `src/vip/reporting.py`, beside `troubleshooting_path`: + +```python +def controls_path(report_dir: str | Path | None = None) -> Path | None: + """Locate a controls.toml in the working report directory, or None. + + Resolved by convention rather than configuration: `vip report --controls` + copies the file here, the same way results.json arrives. Returning None is + the ordinary case -- every deployment without a compliance mapping has no + control list, and their report must simply omit the section rather than + warn about it. + """ + base = Path(report_dir) if report_dir is not None else Path.cwd() + candidate = base / "controls.toml" + return candidate if candidate.is_file() else None +``` + +- [ ] Step 4: Run test to verify it passes + +Run: `uv run pytest selftests/test_report_content_traceability.py -v` +Expected: 7 passed + +- [ ] Step 5: Commit + +```bash +git add src/vip/report_content.py src/vip/reporting.py selftests/test_report_content_traceability.py +git commit -m "feat(report): add traceability rows to the shared content layer" +``` + +--- + +### Task 14: Render the traceability section in both report editions + +Files: +- Modify: `src/vip/report_typst.py` (table renderer + `render_document`) +- Modify: `src/vip/report_html.py` (table renderer) +- Modify: `report/vip-report.qmd`, `report/index.qmd` +- Modify: `src/vip/cli.py` (`run_report` gains `--controls`) +- Test: `selftests/test_report_typst_traceability.py` (create), extend `selftests/test_report_html.py` + +Interfaces: +- Consumes: `report_content.traceability_rows`, `TRACEABILITY_HEADERS` (Task 13) +- Produces: `report_typst.render_traceability_table(matrix) -> str`; `report_html.render_traceability_table(matrix) -> str`; `report_typst.render_document(data, hints, matrix=None)` (new optional third parameter) + +Both backends in one commit, per the #618 rule. Two hazards specific to this +section: + +Control descriptions and notes are customer-authored free text, and in Typst a +bare `#`, `*`, `_` or `$` is live markup — `#panic()` in a description would +abort the render. Every dynamic value must go through `_lit`. The existing +`_text` and `_call` helpers already do this; never interpolate a value into +Typst source by hand. + +`render_document` grows an optional third parameter rather than a required +one, so the existing call in `vip-report.qmd` and every existing test keeps +working and a report with no control list is unchanged. + +- [ ] Step 1: Write the failing test + +Create `selftests/test_report_typst_traceability.py`: + +```python +from vip.report_typst import render_document, render_traceability_table +from vip.reporting import ReportData, TestResult +from vip.traceability import ControlSpec, build_traceability_matrix + + +def _matrix(description="Audit trail"): + data = ReportData( + results=[ + TestResult( + nodeid="t.py::a", + outcome="passed", + markers=["control-x"], + scenario_title="Publishing is recorded", + ) + ] + ) + return build_traceability_matrix(data, {"x": ControlSpec("x", description)}) + + +def test_table_renders_a_vip_table_call(): + out = render_traceability_table(_matrix()) + assert out.startswith("#vip-table(") + assert "Publishing is recorded" in out + + +def test_typst_markup_in_a_control_description_is_inert(): + """A control list is customer-authored; a bare # is live Typst markup.""" + out = render_traceability_table(_matrix(description='#panic("boom") *bold* $x$')) + # The dangerous characters survive only inside an escaped string literal, + # never as a bare call at the start of a token. + assert '#panic("boom")' not in out.replace('\\"', '"').replace('"', "") + assert "panic" in out + + +def test_quotes_and_backslashes_are_escaped(): + out = render_traceability_table(_matrix(description='a "quoted" c:\\path')) + assert '\\"quoted\\"' in out + assert "c:\\\\path" in out + + +def test_document_omits_the_section_without_a_matrix(): + data = ReportData(results=[TestResult(nodeid="t.py::a", outcome="passed")]) + assert "Traceability" not in render_document(data, {}) + + +def test_document_includes_the_section_with_a_matrix(): + data = ReportData(results=[TestResult(nodeid="t.py::a", outcome="passed")]) + out = render_document(data, {}, matrix=_matrix()) + assert "Traceability Matrix" in out + + +def test_render_document_still_accepts_two_positional_arguments(): + """The qmd and every existing test call it with two.""" + data = ReportData(results=[TestResult(nodeid="t.py::a", outcome="passed")]) + assert render_document(data, {}) +``` + +- [ ] Step 2: Run test to verify it fails + +Run: `uv run pytest selftests/test_report_typst_traceability.py -v` +Expected: FAIL with `ImportError: cannot import name 'render_traceability_table'` + +- [ ] Step 3: Implement the Typst backend + +In `src/vip/report_typst.py`, import the new content helpers alongside the +existing `report_content` imports, then add after `render_provenance_table` +(around line 437): + +```python +def render_traceability_table(matrix) -> str: + """Control-to-scenario coverage, for an archivable evidence document. + + Every cell goes through ``_text``/``_lit``: control descriptions are + customer-authored free text, so an unescaped ``#`` would be a live Typst + call rather than a character. + """ + rows = [ + [ + _call("vip-mono", "8.5pt", _lit("#374151"), _lit(row[0])), + _text(row[1], size="9pt"), + _text(row[2], size="9pt"), + _text(row[3], size="9pt"), + ] + for row in traceability_rows(matrix) + ] + if not rows: + return _paragraph("No controls defined.", italic=True) + return _table("(auto, 1.4fr, auto, 1.6fr)", TRACEABILITY_HEADERS, rows) +``` + +Change `render_document`'s signature and append the section: + +```python +def render_document(data: ReportData, hints: dict[str, dict], matrix=None) -> str: + """The whole PDF body, preamble included, ready to emit as a ``{=typst}`` block. + + ``matrix`` is optional: a deployment with no control list gets exactly the + document it got before this section existed. + """ +``` + +Immediately before the `"#pagebreak()\n"` that precedes Detailed Results, add: + +```python + if matrix is not None: + parts.extend( + [ + "#pagebreak()\n", + _heading("Traceability Matrix", 2), + _labelled_line( + "Coverage", + f"{matrix.covered_count} covered, {matrix.gap_count} without an " + f"automated test, of {len(matrix.entries)} controls", + ), + render_traceability_table(matrix), + ] + ) +``` + +Note `parts` is currently a single list literal; convert it to a list built +before the `if`, then `return "".join(parts)` unchanged. + +- [ ] Step 4: Run the Typst tests + +Run: `uv run pytest selftests/test_report_typst_traceability.py -v` +Expected: 6 passed + +- [ ] Step 5: Implement the HTML backend and test it + +In `src/vip/report_html.py`, after `render_provenance_table`: + +```python +def render_traceability_table(matrix) -> str: + """Control-to-scenario coverage, matching the PDF edition's section.""" + rows = traceability_rows(matrix) + if not rows: + return "

No controls defined.

" + head = "".join(f"{_esc(h)}" for h in TRACEABILITY_HEADERS) + body = "".join( + "" + "".join(f"{_esc(cell)}" for cell in row) + "" for row in rows + ) + return f"{head}{body}
" +``` + +Append to `selftests/test_report_html.py`: + +```python +def test_traceability_table_escapes_control_text(): + from vip.report_html import render_traceability_table + from vip.reporting import ReportData, TestResult + from vip.traceability import ControlSpec, build_traceability_matrix + + data = ReportData(results=[TestResult(nodeid="t.py::a", outcome="passed")]) + matrix = build_traceability_matrix( + data, {"x": ControlSpec("x", "")} + ) + out = render_traceability_table(matrix) + assert "")} - ) - out = render_traceability_table(matrix) - assert "")} + ) + out = render_traceability_table(matrix) + assert "")} - ) - out = render_traceability_table(matrix) - assert " & co")} + ) + html = report_html.render_traceability(matrix) + assert "" + + def test_hostile_error_text_is_escaped(self): + html = report_html.render_traceability_error(ValueError(self.HOSTILE)) + assert "" not in html + assert "<img src=x onerror=alert(1)>" in html + assert "<script>alert(2)</script>" in html + + def test_the_failure_is_still_visible(self): + html = report_html.render_traceability_error(ValueError("boom")) + assert "Could not render the traceability section" in html + assert "boom" in html + + def test_index_qmd_routes_the_failure_through_the_escaping_helper(self): + """Drift guard on the cell itself, where the vulnerability lived. + + The helper cannot escape a value the .qmd never hands it, so assert + the except branch calls it through HTML() and no longer formats the + message into Markdown. + """ + qmd = (Path(__file__).parent.parent / "report" / "index.qmd").read_text() + assert "display(HTML(report_html.render_traceability_error(exc)))" in qmd + assert "TRACEABILITY_RENDER_FAILURE" not in qmd, ( + "the .qmd must not format the failure message itself; " + "report_html.render_traceability_error owns the escaping" + ) + + class TestTypstBackend: def test_renders_every_state(self): typ = report_typst.render_traceability(_matrix()) diff --git a/src/vip/report_html.py b/src/vip/report_html.py index 15cb1b88..4979bfd2 100644 --- a/src/vip/report_html.py +++ b/src/vip/report_html.py @@ -30,6 +30,7 @@ OUTCOME_LABELS, OUTCOME_ORDER, TRACEABILITY_CAVEAT, + TRACEABILITY_RENDER_FAILURE, Badge, FeatureStepIndex, category_label, @@ -419,6 +420,27 @@ def render_provenance_table(data: ReportData) -> str: return f"{body}
" +def render_traceability_error(error: object) -> str: + """The visible marker shown when the traceability section could not be built. + + ``error`` is the exception that stopped it -- a missing or malformed + control list, a checksum that failed verification. Its message carries + the ``VIP_CONTROLS`` path and control ids read straight out of a + customer-authored ``controls.toml``, which makes it the same fully + untrusted text ``render_traceability`` describes, and it goes through + ``_esc`` for the same reason. ``IPython.display.Markdown`` passes raw + HTML through in Quarto, so rendering this message as Markdown put + customer-controlled markup into a publicly published page; the Typst + edition already routes the identical value through ``_lit``. + + A dropped section is invisible to a regulated reader, so the failure is + a visible marker in both editions rather than a silent skip -- the + caller still emits the heading alongside this fragment. + """ + message = TRACEABILITY_RENDER_FAILURE.format(error=_esc(str(error))) + return f"

{message}

" + + def render_traceability(matrix) -> str: # noqa: ANN001 - vip.traceability.TraceabilityMatrix """The compliance traceability section: summary counts, then one row per control. From a029484d08fbc1656c234b38dc00bfc271033bcc Mon Sep 17 00:00:00 2001 From: Brian Deitte Date: Sat, 29 Aug 2026 20:03:01 -0500 Subject: [PATCH 085/106] fix(traceability): refuse a sidecar that names one file twice Two sites compounded into a false attestation. verify_results_checksum's basename fallback collected every matching entry and verified if any digest agreed, so a sidecar with several path-qualified results.json lines could attest to a file it does not describe. _rehome_sidecar then rewrote every basename match to the destination name, turning one wrong exact entry plus one right path-qualified entry into two results.json lines -- which the exact-match branch happily accepted, defeating the precedence that branch exists to enforce. verify_results_checksum now raises when the selected entries carry more than one distinct digest, on both selection paths, and says how to proceed. Entries that agree (case and path qualification aside) still verify. _rehome_sidecar keeps exact-name precedence: it rewrites exact entries when there are any, falls back to the basename only when exactly one candidate exists, and otherwise leaves the lines alone rather than inventing an authority the source never had. --- selftests/test_results_checksum.py | 107 +++++++++++++++++++++++++++++ src/vip/cli.py | 31 +++++++-- src/vip/traceability.py | 24 +++++++ 3 files changed, 157 insertions(+), 5 deletions(-) diff --git a/selftests/test_results_checksum.py b/selftests/test_results_checksum.py index c31b8bbc..4b1f0f14 100644 --- a/selftests/test_results_checksum.py +++ b/selftests/test_results_checksum.py @@ -157,6 +157,68 @@ def test_exact_match_still_wins_over_a_basename_collision(self, tmp_path): verify_results_checksum(p) +class TestSidecarAmbiguity: + """A sidecar must never say two different things about one file. + + Both selection paths -- the exact recorded name and the basename fallback + -- can collect several entries. Accepting the file because *any* of them + agrees means the attestation may describe a different artifact entirely, + which is the false attestation the recorded-name match exists to prevent. + """ + + def _results(self, tmp_path): + p = tmp_path / "results.json" + p.write_text('{"schema_version": "1.0", "results": []}', encoding="utf-8") + return p, hashlib.sha256(p.read_bytes()).hexdigest() + + def test_two_path_qualified_entries_disagreeing_are_refused(self, tmp_path): + """The basename fallback must not verify against whichever line agrees.""" + p, digest = self._results(tmp_path) + p.with_name("results.json.sha256").write_text( + f"{digest} archive/results.json\n{'0' * 64} nightly/results.json\n" + ) + with pytest.raises(ResultsIntegrityError, match="different digests"): + verify_results_checksum(p) + + def test_two_exact_entries_disagreeing_are_refused(self, tmp_path): + """A rehomed sidecar that grew a second same-named line is caught here.""" + p, digest = self._results(tmp_path) + p.with_name("results.json.sha256").write_text( + f"{'0' * 64} results.json\n{digest} results.json\n" + ) + with pytest.raises(ResultsIntegrityError, match="different digests"): + verify_results_checksum(p) + + def test_the_message_says_how_to_proceed(self, tmp_path): + p, digest = self._results(tmp_path) + p.with_name("results.json.sha256").write_text( + f"{digest} archive/results.json\n{'0' * 64} nightly/results.json\n" + ) + with pytest.raises(ResultsIntegrityError) as exc: + verify_results_checksum(p) + assert "Regenerate it" in str(exc.value) + assert "delete it" in str(exc.value) + + def test_entries_that_agree_are_not_ambiguous(self, tmp_path): + """Saying the same thing twice is not a disagreement. + + Case and path qualification both vary, because hex case is not a + mismatch and a path-qualified line describes the same file. + """ + p, digest = self._results(tmp_path) + p.with_name("results.json.sha256").write_text( + f"{digest} archive/results.json\n{digest.upper()} nightly/results.json\n" + ) + assert verify_results_checksum(p) == (digest, True) + + def test_agreeing_exact_entries_still_verify(self, tmp_path): + p, digest = self._results(tmp_path) + p.with_name("results.json.sha256").write_text( + f"{digest} results.json\n{digest.upper()} results.json\n" + ) + assert verify_results_checksum(p) == (digest, True) + + class TestStaleSidecarInvalidation: def test_writing_results_removes_a_stale_sidecar_first(self, tmp_path, pytester): """A sidecar write that fails must not leave the previous run's digest.""" @@ -252,6 +314,51 @@ def test_whitespace_only_source_removes_the_destination(self, tmp_path): _, present = verify_results_checksum(dest) assert present is False + def test_a_basename_collision_does_not_produce_two_destination_entries(self, tmp_path): + """Exact-name precedence has to survive the rehome, not just verification. + + Rewriting every basename match turned one wrong exact entry plus one + right path-qualified entry into two `results.json` lines, and + verification's exact-match branch then found both and accepted the + agreeable one -- defeating the precedence + test_exact_match_still_wins_over_a_basename_collision protects. + """ + src = tmp_path / "results.json" + src.write_text('{"results": []}', encoding="utf-8") + digest = hashlib.sha256(src.read_bytes()).hexdigest() + self._sidecar_for(src).write_text( + f"{'0' * 64} results.json\n{digest} archive/results.json\n", encoding="utf-8" + ) + + dest = tmp_path / "out" / "results.json" + dest.parent.mkdir() + shutil.copy2(src, dest) + _rehome_sidecar(self._sidecar_for(src), self._sidecar_for(dest), src.name, dest.name) + + rehomed = self._sidecar_for(dest).read_text().splitlines() + named = [line for line in rehomed if line.split(None, 1)[1].strip() == "results.json"] + assert len(named) == 1, f"the rehome invented a second results.json entry: {rehomed}" + with pytest.raises(ResultsIntegrityError, match="checksum mismatch"): + verify_results_checksum(dest) + + def test_several_basename_matches_with_no_exact_entry_are_left_alone(self, tmp_path): + """The source never said which one describes the destination.""" + src = tmp_path / "run-42.json" + src.write_text('{"results": []}', encoding="utf-8") + digest = hashlib.sha256(src.read_bytes()).hexdigest() + self._sidecar_for(src).write_text( + f"{digest} archive/run-42.json\n{'0' * 64} nightly/run-42.json\n", encoding="utf-8" + ) + + dest = tmp_path / "out" / "results.json" + dest.parent.mkdir() + shutil.copy2(src, dest) + _rehome_sidecar(self._sidecar_for(src), self._sidecar_for(dest), src.name, dest.name) + + assert "results.json" not in self._sidecar_for(dest).read_text() + with pytest.raises(ResultsIntegrityError, match="does not record an entry"): + verify_results_checksum(dest) + def test_undecodable_source_raises_unicode_error_for_the_caller(self, tmp_path): """The call site catches this; it must not escape as a bare traceback.""" src = tmp_path / "results.json" diff --git a/src/vip/cli.py b/src/vip/cli.py index aacb4b40..2fd11d75 100644 --- a/src/vip/cli.py +++ b/src/vip/cli.py @@ -1662,27 +1662,48 @@ def _rehome_sidecar(src: Path, dest: Path, src_name: str, dest_name: str) -> Non No source sidecar means the stale destination one is removed rather than left behind: no sidecar is a documented benign state, a wrong one is a false tamper alarm. + + Exact-name precedence is preserved across the rehome, because the + destination sidecar must never record two different digests under the + destination name. A source that names both ``results.json`` and + ``archive/results.json`` used to rewrite *both* lines, so the copy said + two things about one file and ``verify_results_checksum`` picked whichever + one agreed. Rewrite the exact matches when there are any; fall back to the + basename only when there is exactly one candidate to be unambiguous about. + Several basename matches with no exact entry are left alone -- the source + never had the authority to say which one describes the destination, so the + copy does not invent it, and verification reports that rather than + guessing. """ from vip.traceability import sidecar_basename if not src.is_file(): dest.unlink(missing_ok=True) return - lines = [] + parsed: list[tuple[str, str, str | None]] = [] for line in src.read_text(encoding="utf-8-sig").splitlines(): parts = line.split(None, 1) if not parts: continue recorded = parts[1].strip().lstrip("*") if len(parts) > 1 else None + parsed.append((line, parts[0], recorded)) + + # A bare digest counts as an exact entry: it names no other file, so it + # can only be describing the one being copied. + rewrite = {i for i, (_, _, r) in enumerate(parsed) if r is None or r == src_name} + if not rewrite: # Compare basenames, not the raw recorded name. A sidecar generated # from a parent directory records a path, and copying that line # through verbatim produces a rehomed sidecar that then fails # verification at the destination -- the false tamper alarm this # function exists to prevent. - if recorded is None or sidecar_basename(recorded) == sidecar_basename(src_name): - lines.append(f"{parts[0]} {dest_name}") - else: - lines.append(line) + src_base = sidecar_basename(src_name) + matches = [i for i, (_, _, r) in enumerate(parsed) if r and sidecar_basename(r) == src_base] + rewrite = set(matches) if len(matches) == 1 else set() + lines = [ + f"{digest} {dest_name}" if i in rewrite else raw + for i, (raw, digest, _) in enumerate(parsed) + ] if not lines: # A source that parses to zero entries (whitespace-only, truncated) # would otherwise produce an empty destination sidecar, which diff --git a/src/vip/traceability.py b/src/vip/traceability.py index 40fb526c..1385b641 100644 --- a/src/vip/traceability.py +++ b/src/vip/traceability.py @@ -490,6 +490,11 @@ def verify_results_checksum(path: str | Path) -> tuple[str, bool]: ``sidecar_present`` to distinguish "verified" from "nothing to verify" rather than treating both as the same success. + Also raises when the entries selected for this file disagree with each + other: a sidecar that records two different digests under the same name + cannot attest to anything, so accepting the file because one of them + happens to match would be a false attestation. + This is tamper-evidence within a trusted pipeline, not tamper-proofing -- anyone who can edit the results file can regenerate the sidecar. It catches corruption, truncated uploads and casual editing. @@ -543,6 +548,25 @@ def verify_results_checksum(path: str | Path) -> tuple[str, bool]: "without verification." ) + # A sidecar must never say two different things about one file. Several + # selected entries carrying *different* digests means one of them is + # describing some other artifact, and accepting the file because *any* of + # them agrees turns the sidecar into an attestation about a file it does + # not describe -- the exact false attestation the recorded-name match + # above exists to prevent, reintroduced through the basename fallback (or + # through a rehomed sidecar that ended up with two same-named lines). + # Several entries agreeing on one digest is not ambiguous and still + # verifies: `shasum` run twice, or a rehomed line beside its original, + # says the same thing twice. + distinct = {d.lower() for d in named} + if len(distinct) > 1: + listed = ", ".join(sorted(distinct)) + raise ResultsIntegrityError( + f"checksum sidecar {sidecar} records {len(distinct)} different digests for " + f"{p.name} ({listed}); it cannot say which one describes this file. " + "Regenerate it, or delete it to proceed without verification." + ) + # Case-insensitive: hex is hex. PowerShell's Get-FileHash and 7-Zip emit # uppercase, and rejecting those as a mismatch reads to an operator as # "this evidence file was tampered with" over nothing but letter case. From fb63cfdb6c3f257e7822d14502dd4421d31b8015 Mon Sep 17 00:00:00 2001 From: Brian Deitte Date: Sat, 29 Aug 2026 20:05:58 -0500 Subject: [PATCH 086/106] fix(report): refuse an invalid source sidecar on a compliance render `vip report --controls --results /external/results.json` with an empty or unreadable /external/results.json.sha256 rendered happily. The chain: _rehome_sidecar correctly declines to manufacture a destination sidecar from a source that parses to zero entries, and a missing destination sidecar is legal and benign, so the --controls gate -- which only ever looked at the destination -- saw nothing to refuse. `vip trace` rejects that same input as a truncated attestation. Verify the source against its own sidecar before the copy whenever --controls is set, so the compliance render is never more permissive than `vip trace` on identical bytes. Plain `vip report` stays lenient, and a source with genuinely no sidecar stays benign on both paths. Also chdir the four TestReportControls cases that still rendered into the repo's own report/ directory, where they raced each other under xdist over a shared results.json. --- selftests/test_cli_report.py | 85 ++++++++++++++++++++++++++++++++++++ src/vip/cli.py | 20 +++++++++ 2 files changed, 105 insertions(+) diff --git a/selftests/test_cli_report.py b/selftests/test_cli_report.py index ca985ae4..c5dc2b7c 100644 --- a/selftests/test_cli_report.py +++ b/selftests/test_cli_report.py @@ -435,6 +435,7 @@ def _args(self, tmp_path, controls=None): def test_malformed_control_list_fails_before_quarto_starts(self, cli, tmp_path, monkeypatch): """A notebook cell can only degrade to a warning, so validate out here.""" + monkeypatch.chdir(tmp_path) bad = tmp_path / "c.toml" bad.write_text("[controls]\n", encoding="utf-8") @@ -446,6 +447,7 @@ def test_malformed_control_list_fails_before_quarto_starts(self, cli, tmp_path, assert called == [] def test_missing_control_list_fails_before_quarto_starts(self, cli, tmp_path, monkeypatch): + monkeypatch.chdir(tmp_path) called = [] monkeypatch.setattr(cli, "_quarto_render", lambda *a, **k: called.append(a) or 0) with pytest.raises(SystemExit): @@ -454,6 +456,7 @@ def test_missing_control_list_fails_before_quarto_starts(self, cli, tmp_path, mo def test_controls_reach_both_renders_through_the_environment(self, cli, tmp_path, monkeypatch): """The HTML pages and the PDF are separate quarto invocations.""" + monkeypatch.chdir(tmp_path) controls = tmp_path / "c.toml" controls.write_text('[controls.x]\ndescription = "d"\n', encoding="utf-8") @@ -560,6 +563,7 @@ def test_without_controls_the_variable_is_absent_and_nothing_warns( self, cli, tmp_path, monkeypatch, capsys ): """The overwhelmingly common path: no control list, no section, no noise.""" + monkeypatch.chdir(tmp_path) envs = [] def fake_render(document, report_dir, env): @@ -604,6 +608,87 @@ def test_report_with_controls_refuses_a_mismatched_sidecar( assert called == [] assert "checksum mismatch" in capsys.readouterr().err + def test_report_with_controls_refuses_an_empty_source_sidecar( + self, cli, tmp_path, monkeypatch, capsys + ): + """An invalid attestation must not launder itself into a benign absence. + + `--results /external/path/results.json` copies the file into the + report directory, and _rehome_sidecar correctly refuses to + manufacture a destination sidecar out of an empty source one. A + *missing* destination sidecar is legal and benign, so a gate that + only ever looked at the destination let the compliance render + proceed on input `vip trace` refuses as a truncated attestation. + The compliance path must never be more permissive than `vip trace`. + """ + monkeypatch.chdir(tmp_path) + external = tmp_path / "external" + external.mkdir() + results = external / "results.json" + results.write_text('{"schema_version": "1.0", "results": []}', encoding="utf-8") + results.with_name("results.json.sha256").write_text(" \n\n", encoding="utf-8") + + controls = tmp_path / "c.toml" + controls.write_text('[controls.x]\ndescription = "d"\n', encoding="utf-8") + + called = [] + monkeypatch.setattr(cli, "_quarto_render", lambda *a, **k: called.append(a) or 0) + args = argparse.Namespace( + results=str(results), controls=str(controls), open=False, output=None + ) + with pytest.raises(SystemExit) as exc: + cli.run_report(args) + + assert exc.value.code == 1 + assert called == [] + assert "is empty" in capsys.readouterr().err + + def test_the_same_empty_sidecar_still_renders_without_controls( + self, cli, tmp_path, monkeypatch + ): + """Plain `vip report` stays lenient; only the compliance path is strict.""" + monkeypatch.chdir(tmp_path) + external = tmp_path / "external" + external.mkdir() + results = external / "results.json" + results.write_text('{"schema_version": "1.0", "results": []}', encoding="utf-8") + results.with_name("results.json.sha256").write_text(" \n\n", encoding="utf-8") + + envs = [] + monkeypatch.setattr(cli, "_quarto_render", self._fake_render(envs)) + args = argparse.Namespace(results=str(results), controls=None, open=False, output=None) + cli.run_report(args) + + assert [document for document, _ in envs] == ["index.qmd", "details.qmd", "vip-report.qmd"] + assert not (tmp_path / "report" / "results.json.sha256").exists() + + def test_a_source_with_no_sidecar_at_all_is_still_benign_under_controls( + self, cli, tmp_path, monkeypatch + ): + """No sidecar is a documented benign state on both paths. + + Results files written before the sidecar existed have none, and + `vip trace` accepts them; the compliance render must match rather + than exceed that strictness. + """ + monkeypatch.chdir(tmp_path) + external = tmp_path / "external" + external.mkdir() + results = external / "results.json" + results.write_text('{"schema_version": "1.0", "results": []}', encoding="utf-8") + + controls = tmp_path / "c.toml" + controls.write_text('[controls.x]\ndescription = "d"\n', encoding="utf-8") + + envs = [] + monkeypatch.setattr(cli, "_quarto_render", self._fake_render(envs)) + args = argparse.Namespace( + results=str(results), controls=str(controls), open=False, output=None + ) + cli.run_report(args) + + assert [document for document, _ in envs] == ["index.qmd", "details.qmd", "vip-report.qmd"] + def test_report_without_controls_ignores_a_mismatched_sidecar(self, cli, tmp_path, monkeypatch): """Plain vip report stays lenient: a report must render regardless. diff --git a/src/vip/cli.py b/src/vip/cli.py index 2fd11d75..18dc95a1 100644 --- a/src/vip/cli.py +++ b/src/vip/cli.py @@ -765,6 +765,26 @@ def run_report(args: argparse.Namespace) -> None: if not results_src.exists(): print(f"Error: results file not found: {results_src}", file=sys.stderr) sys.exit(1) + if getattr(args, "controls", None): + # Verify the SOURCE before the copy, not only the destination + # after it. _rehome_sidecar is right to discard an empty or + # unreadable source sidecar rather than manufacture one at the + # destination -- but a missing destination sidecar is legal and + # benign, so the gate below would then wave through the very + # input `vip trace` refuses as a truncated attestation. The + # compliance render must never be more permissive than + # `vip trace` on identical bytes. A source with genuinely no + # sidecar stays benign here, exactly as it is for `vip trace`. + from vip.traceability import ResultsIntegrityError, verify_results_checksum + + try: + verify_results_checksum(results_src) + except ResultsIntegrityError as exc: + print(f"Error: {exc}", file=sys.stderr) + sys.exit(1) + except (OSError, UnicodeDecodeError) as exc: + print(f"Error: could not read results file {results_src}: {exc}", file=sys.stderr) + sys.exit(1) shutil.copy2(results_src, results_dest) # Keep the checksum sidecar with the results it describes. Copying a # results.json from a CI artifact over the local one leaves the From 6154a81dda06a206dcc2bca0db46145159bc833e Mon Sep 17 00:00:00 2001 From: Brian Deitte Date: Sat, 29 Aug 2026 20:16:08 -0500 Subject: [PATCH 087/106] fix(cli): narrow the optional output path for mypy Make the mypy error in _resolve_trace_format explicit: the condition now checks `out is not None` before accessing `out.suffix` in the warning message, so the type narrowing is unambiguous to mypy. This error is pre-existing and unrelated to the traceability review fixes on this branch. The function is byte-identical at commit 5fbd4d08 and does not exist on main at all; it was carried through the initial traceability review PR without running full mypy checks. Runtime behavior is unchanged: the warning still prints only when inferred format disagrees with explicit format. --- src/vip/cli.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/vip/cli.py b/src/vip/cli.py index 18dc95a1..b045529a 100644 --- a/src/vip/cli.py +++ b/src/vip/cli.py @@ -1662,7 +1662,7 @@ def _resolve_trace_format(explicit: str | None, out: Path | None) -> str: inferred = {".json": "json", ".csv": "csv"}.get(out.suffix.lower()) if out else None if explicit is None: return inferred or "csv" - if inferred and inferred != explicit: + if out is not None and inferred and inferred != explicit: print( f"Warning: --format {explicit} does not match the {out.suffix} extension of " f"{out}; writing {explicit}.", From 2b99d55803b14e9f4deb19424b989292cacc7b9e Mon Sep 17 00:00:00 2001 From: Brian Deitte Date: Sat, 29 Aug 2026 20:22:05 -0500 Subject: [PATCH 088/106] fix(cli): apply distinct-digest rule to sidecar rehome basename fallback _rehome_sidecar's basename fallback refused to rewrite whenever more than one entry shared a basename, even when they carried identical digests -- diverging from verify_results_checksum's distinct-digest rule and silently dropping checksum verification across a `vip report --results` copy. Compare matched digests case-insensitively and rewrite them together when they agree, matching the verifier's rule exactly. --- selftests/test_results_checksum.py | 43 ++++++++++++++++++++++++++++++ src/vip/cli.py | 15 ++++++----- 2 files changed, 52 insertions(+), 6 deletions(-) diff --git a/selftests/test_results_checksum.py b/selftests/test_results_checksum.py index 4b1f0f14..6c4fbbf0 100644 --- a/selftests/test_results_checksum.py +++ b/selftests/test_results_checksum.py @@ -359,6 +359,49 @@ def test_several_basename_matches_with_no_exact_entry_are_left_alone(self, tmp_p with pytest.raises(ResultsIntegrityError, match="does not record an entry"): verify_results_checksum(dest) + def test_several_basename_matches_that_agree_are_rehomed(self, tmp_path): + """Several basename matches with the *same* digest are unambiguous. + + This mirrors verify_results_checksum's distinct-digest rule: several + entries agreeing on one digest is not ambiguous, even though the + basename fallback found more than one match. + """ + src = tmp_path / "run-42.json" + src.write_text('{"results": []}', encoding="utf-8") + digest = hashlib.sha256(src.read_bytes()).hexdigest() + self._sidecar_for(src).write_text( + f"{digest} archive/run-42.json\n{digest} nightly/run-42.json\n", encoding="utf-8" + ) + + dest = tmp_path / "out" / "results.json" + dest.parent.mkdir() + shutil.copy2(src, dest) + _rehome_sidecar(self._sidecar_for(src), self._sidecar_for(dest), src.name, dest.name) + + assert verify_results_checksum(dest) == (digest, True) + + def test_several_basename_matches_that_agree_case_insensitively_are_rehomed(self, tmp_path): + """Digest comparison is case-insensitive, matching verify_results_checksum. + + PowerShell's Get-FileHash and 7-Zip emit uppercase hex; a sidecar + mixing an uppercase and a lowercase rendering of the same digest must + not be treated as disagreement. + """ + src = tmp_path / "run-42.json" + src.write_text('{"results": []}', encoding="utf-8") + digest = hashlib.sha256(src.read_bytes()).hexdigest() + self._sidecar_for(src).write_text( + f"{digest.upper()} archive/run-42.json\n{digest} nightly/run-42.json\n", + encoding="utf-8", + ) + + dest = tmp_path / "out" / "results.json" + dest.parent.mkdir() + shutil.copy2(src, dest) + _rehome_sidecar(self._sidecar_for(src), self._sidecar_for(dest), src.name, dest.name) + + assert verify_results_checksum(dest) == (digest, True) + def test_undecodable_source_raises_unicode_error_for_the_caller(self, tmp_path): """The call site catches this; it must not escape as a bare traceback.""" src = tmp_path / "results.json" diff --git a/src/vip/cli.py b/src/vip/cli.py index b045529a..f02fabf9 100644 --- a/src/vip/cli.py +++ b/src/vip/cli.py @@ -1689,11 +1689,13 @@ def _rehome_sidecar(src: Path, dest: Path, src_name: str, dest_name: str) -> Non ``archive/results.json`` used to rewrite *both* lines, so the copy said two things about one file and ``verify_results_checksum`` picked whichever one agreed. Rewrite the exact matches when there are any; fall back to the - basename only when there is exactly one candidate to be unambiguous about. - Several basename matches with no exact entry are left alone -- the source - never had the authority to say which one describes the destination, so the - copy does not invent it, and verification reports that rather than - guessing. + basename otherwise, using the same distinct-digest rule + ``verify_results_checksum`` applies: several basename matches that all + carry the same digest (compared case-insensitively) are unambiguous and + are rewritten together, same as a single match. Basename matches that + disagree on the digest are left alone -- the source never had the + authority to say which one describes the destination, so the copy does + not invent it, and verification reports that rather than guessing. """ from vip.traceability import sidecar_basename @@ -1719,7 +1721,8 @@ def _rehome_sidecar(src: Path, dest: Path, src_name: str, dest_name: str) -> Non # function exists to prevent. src_base = sidecar_basename(src_name) matches = [i for i, (_, _, r) in enumerate(parsed) if r and sidecar_basename(r) == src_base] - rewrite = set(matches) if len(matches) == 1 else set() + distinct = {parsed[i][1].lower() for i in matches} + rewrite = set(matches) if len(distinct) == 1 else set() lines = [ f"{digest} {dest_name}" if i in rewrite else raw for i, (raw, digest, _) in enumerate(parsed) From 623dfa85020366cb2c1174a708c2b1942154741e Mon Sep 17 00:00:00 2001 From: Brian Deitte Date: Sat, 29 Aug 2026 20:23:35 -0500 Subject: [PATCH 089/106] chore: remove the completed traceability-review-fixes plan and spec --- .../2026-08-29-traceability-review-fixes.md | 1352 ----------------- ...-08-29-traceability-review-fixes-design.md | 268 ---- 2 files changed, 1620 deletions(-) delete mode 100644 docs/superpowers/plans/2026-08-29-traceability-review-fixes.md delete mode 100644 docs/superpowers/specs/2026-08-29-traceability-review-fixes-design.md diff --git a/docs/superpowers/plans/2026-08-29-traceability-review-fixes.md b/docs/superpowers/plans/2026-08-29-traceability-review-fixes.md deleted file mode 100644 index 12674711..00000000 --- a/docs/superpowers/plans/2026-08-29-traceability-review-fixes.md +++ /dev/null @@ -1,1352 +0,0 @@ -# Traceability Review Fixes Implementation Plan - -> For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -Goal: Fix eight verified defects in the `feat/part11-traceability` branch so VIP stops refusing untampered evidence and stops rendering a failed control as green. - -Architecture: Four independent groups. Sidecar matching gains an exact-then-basename key. The matrix model gains a `failing` fact alongside the existing `executed` fact, which the display layer flattens into a new red badge. The two report backends are brought back into visual and behavioural parity. One cosmetic comparison bug is closed. - -Tech Stack: Python 3.10+, pytest, pytest-bdd, httpx, Quarto/Typst. All commands run through `uv run`. Ruff 0.15.0 is the pinned linter and formatter. - -Spec: `docs/superpowers/specs/2026-08-29-traceability-review-fixes-design.md` - -## Global Constraints - -- Run every command through `uv run`. Never bare `python` or `pip`. -- Ruff rules `E`, `F`, `I`, `UP`. Line length 100. Check and format - `src/ src/vip_tests/ selftests/ examples/`. CI pins ruff to 0.15.0. -- Never run selftests with `-p no:randomly`. CI runs them randomized. -- `ControlEntry.coverage` keeps exactly three values: `covered`, `gap`, - `not_automatable`. No task adds a fourth. -- `MATRIX_SCHEMA_VERSION` stays `"1.0"`. -- Every dynamic value reaching `src/vip/report_typst.py` passes through `_lit`. -- Visual changes land in `report_content.py` and `report/styles.css` in the same - commit, so the HTML and PDF editions stay identical. -- Commit after every task. PR-title-style conventional commit messages. -- Never use bold (`**`) when writing markdown files in this repo. - ---- - -### Task 1: Accept a path-qualified checksum sidecar - -Files: -- Modify: `src/vip/traceability.py:490` (`verify_results_checksum`) -- Test: `selftests/test_results_checksum.py` (class `TestSidecarParsing`) - -Interfaces: -- Consumes: nothing from earlier tasks. -- Produces: `vip.traceability.sidecar_basename(name: str) -> str`, used by Task 2. - -- [ ] Step 1: Write the failing tests - -Add to `class TestSidecarParsing` in `selftests/test_results_checksum.py`: - -```python - def test_path_qualified_sidecar_verifies(self, tmp_path): - """`shasum -a 256 report/results.json` from a parent directory.""" - p, digest = self._results(tmp_path) - p.with_name("results.json.sha256").write_text(f"{digest} report/results.json\n") - assert verify_results_checksum(p) == (digest, True) - - def test_windows_path_qualified_sidecar_verifies(self, tmp_path): - p, digest = self._results(tmp_path) - p.with_name("results.json.sha256").write_text(f"{digest} report\\results.json\n") - assert verify_results_checksum(p) == (digest, True) - - def test_exact_match_still_wins_over_a_basename_collision(self, tmp_path): - """A sidecar naming this file exactly never reaches the fallback.""" - p, digest = self._results(tmp_path) - p.with_name("results.json.sha256").write_text( - f"{'0' * 64} archive/results.json\n{digest} results.json\n" - ) - assert verify_results_checksum(p) == (digest, True) -``` - -- [ ] Step 2: Run the tests to verify they fail - -Run: `uv run pytest selftests/test_results_checksum.py::TestSidecarParsing -v` -Expected: the first two FAIL with `ResultsIntegrityError: ... does not record an entry for results.json`. The third PASSES already. - -- [ ] Step 3: Add the basename helper - -Add near `_parse_sidecar` in `src/vip/traceability.py`: - -```python -def sidecar_basename(name: str) -> str: - """The bare filename from a sidecar's recorded name. - - Backslashes are normalized first: shasum under Git Bash or MSYS can record - a Windows-style path, and PurePosixPath would treat the whole thing as one - filename. - """ - return PurePosixPath(name.replace("\\", "/")).name -``` - -Add `PurePosixPath` to the existing `from pathlib import Path` line: - -```python -from pathlib import Path, PurePosixPath -``` - -- [ ] Step 4: Add the fallback to the match - -In `verify_results_checksum`, immediately after the existing -`named = [d for d, name in entries if name == p.name]` line and before the -existing `if not named:` block, insert: - -```python - if not named: - # Fall back to comparing basenames. `shasum -a 256 report/results.json` - # run from a directory above the file records the path it was given - # rather than the bare name, and refusing that sidecar reads to an - # operator as a tamper alarm on a file nobody touched. Exact match - # stays the primary key, so a multi-file sidecar that already names - # this file exactly never reaches here and keeps its strict behaviour. - named = [d for d, name in entries if name and sidecar_basename(name) == p.name] -``` - -- [ ] Step 5: Run the tests to verify they pass - -Run: `uv run pytest selftests/test_results_checksum.py -v` -Expected: PASS, including every pre-existing test in the file. - -- [ ] Step 6: Reproduce the original failure end to end - -Run: - -```bash -cd "$(mktemp -d)" && mkdir report -printf '{"schema_version": "1.0", "results": []}' > report/results.json -shasum -a 256 report/results.json > report/results.json.sha256 -uv run --project "$OLDPWD" vip trace --results report/results.json \ - --controls "$OLDPWD/examples/21CFR_part11_validation/controls.toml" --format json > /dev/null -echo "exit=$?" -``` - -Expected: `exit=0`. Before this task it printed a `does not record an entry` error and exited 1. - -- [ ] Step 7: Lint and commit - -```bash -uvx ruff@0.15.0 check src/ src/vip_tests/ selftests/ examples/ -uvx ruff@0.15.0 format src/ src/vip_tests/ selftests/ examples/ -git add src/vip/traceability.py selftests/test_results_checksum.py -git commit -m "fix(traceability): accept a sidecar that records a path, not a bare name" -``` - ---- - -### Task 2: Stop the rehomed sidecar manufacturing false alarms - -Files: -- Modify: `src/vip/cli.py:1645` (`_rehome_sidecar`) -- Modify: `src/vip/cli.py:780` (the `except OSError` at the call site) -- Test: `selftests/test_results_checksum.py` (the rehome test class) - -Interfaces: -- Consumes: `vip.traceability.sidecar_basename` from Task 1. -- Produces: nothing later tasks rely on. - -- [ ] Step 1: Write the failing tests - -Add to the rehome test class in `selftests/test_results_checksum.py` (the one -whose docstring begins "`vip report --results` copies a results file"): - -```python - def test_path_qualified_source_line_is_rehomed(self, tmp_path): - """A recorded path must be rewritten, not copied through verbatim.""" - src = tmp_path / "results.json" - src.write_text('{"results": []}', encoding="utf-8") - digest = hashlib.sha256(src.read_bytes()).hexdigest() - self._sidecar_for(src).write_text(f"{digest} report/results.json\n", encoding="utf-8") - - dest = tmp_path / "out" / "results.json" - dest.parent.mkdir() - shutil.copy2(src, dest) - _rehome_sidecar(self._sidecar_for(src), self._sidecar_for(dest), src.name, dest.name) - - assert self._sidecar_for(dest).read_text().split()[1] == "results.json" - assert verify_results_checksum(dest) == (digest, True) - - def test_whitespace_only_source_removes_the_destination(self, tmp_path): - """An empty sidecar is the truncated-upload case; do not manufacture one.""" - src = tmp_path / "results.json" - src.write_text('{"results": []}', encoding="utf-8") - self._sidecar_for(src).write_text(" \n\n", encoding="utf-8") - - dest = tmp_path / "out" / "results.json" - dest.parent.mkdir() - shutil.copy2(src, dest) - self._sidecar_for(dest).write_text(f"{'0' * 64} results.json\n", encoding="utf-8") - _rehome_sidecar(self._sidecar_for(src), self._sidecar_for(dest), src.name, dest.name) - - assert not self._sidecar_for(dest).exists() - _, present = verify_results_checksum(dest) - assert present is False - - def test_undecodable_source_raises_unicode_error_for_the_caller(self, tmp_path): - """The call site catches this; it must not escape as a bare traceback.""" - src = tmp_path / "results.json" - src.write_text('{"results": []}', encoding="utf-8") - self._sidecar_for(src).write_bytes(b"\xff\xfe\x00\x00 not utf-8") - - dest = tmp_path / "out" / "results.json" - dest.parent.mkdir() - with pytest.raises(UnicodeDecodeError): - _rehome_sidecar(self._sidecar_for(src), self._sidecar_for(dest), src.name, dest.name) -``` - -- [ ] Step 2: Run the tests to verify they fail - -Run: `uv run pytest selftests/test_results_checksum.py -v -k "rehomed or whitespace_only or undecodable"` -Expected: the first two FAIL. The third PASSES already (the raise happens, it is -just uncaught at the call site, which Step 5 fixes). - -- [ ] Step 3: Rewrite the body of `_rehome_sidecar` - -Replace everything in `src/vip/cli.py` from `if not src.is_file():` through the -final `dest.write_text(...)` line of `_rehome_sidecar` with: - -```python - if not src.is_file(): - dest.unlink(missing_ok=True) - return - lines = [] - for line in src.read_text(encoding="utf-8-sig").splitlines(): - parts = line.split(None, 1) - if not parts: - continue - recorded = parts[1].strip().lstrip("*") if len(parts) > 1 else None - # Compare basenames, not the raw recorded name. A sidecar generated - # from a parent directory records a path, and copying that line - # through verbatim produces a rehomed sidecar that then fails - # verification at the destination -- the false tamper alarm this - # function exists to prevent. - if recorded is None or sidecar_basename(recorded) == sidecar_basename(src_name): - lines.append(f"{parts[0]} {dest_name}") - else: - lines.append(line) - if not lines: - # A source that parses to zero entries (whitespace-only, truncated) - # would otherwise produce an empty destination sidecar, which - # verify_results_checksum refuses as the truncated-upload case. No - # sidecar is the documented benign state, so produce that instead. - dest.unlink(missing_ok=True) - return - dest.write_text("\n".join(lines) + "\n", encoding="utf-8") -``` - -- [ ] Step 4: Add the import - -`_rehome_sidecar` is module-level in `cli.py`. Add the import inside the -function body, at its top, matching this module's existing lazy-import style: - -```python - from vip.traceability import sidecar_basename -``` - -- [ ] Step 5: Widen the call site's exception handling - -At `src/vip/cli.py:780`, change: - -```python - except OSError as exc: -``` - -to: - -```python - # UnicodeDecodeError as well as OSError: _rehome_sidecar reads with - # encoding="utf-8-sig" and a corrupt sidecar would otherwise reach the - # user as a traceback. verify_results_checksum already catches both on - # the identical read, and the two paths should agree. - except (OSError, UnicodeDecodeError) as exc: -``` - -- [ ] Step 6: Run the tests to verify they pass - -Run: `uv run pytest selftests/test_results_checksum.py -v` -Expected: PASS, all tests in the file. - -- [ ] Step 7: Lint and commit - -```bash -uvx ruff@0.15.0 check src/ src/vip_tests/ selftests/ examples/ -uvx ruff@0.15.0 format src/ src/vip_tests/ selftests/ examples/ -git add src/vip/cli.py selftests/test_results_checksum.py -git commit -m "fix(cli): stop a rehomed sidecar raising a false tamper alarm" -``` - ---- - -### Task 3: Separate "a scenario ran" from "a scenario passed" - -Files: -- Modify: `src/vip/traceability.py` (`ControlEntry`, `TraceabilityMatrix`) -- Test: `selftests/test_traceability_matrix.py` - -Interfaces: -- Consumes: nothing from earlier tasks. -- Produces: `ControlEntry.failing -> bool` and - `TraceabilityMatrix.covered_with_failure -> list[str]`, used by Tasks 4 and 5. - -- [ ] Step 1: Add the shared matrix helper - -Tasks 3, 5, 6 and 7 all need to build a matrix from a list of scenario statuses. -Add this once to `selftests/conftest.py`, as a plain module-level function (not -a fixture), so every test file can import it: - -```python -def matrix_from_statuses(statuses: dict[str, list[str]]): - """Build a TraceabilityMatrix from {control_id: [scenario status, ...]}. - - One TestResult per status, tagged `control-`. A status of "na_version" - is written as a version-gated skip, which is how the plugin records it. - """ - from vip.reporting import ReportData, TestResult - from vip.traceability import ControlSpec, build_traceability_matrix - - results = [] - for control_id, control_statuses in statuses.items(): - for i, status in enumerate(control_statuses): - results.append( - TestResult( - nodeid=f"test_{control_id}.py::test_{i}", - outcome="skipped" if status == "na_version" else status, - na_version=status == "na_version", - markers=[f"control-{control_id}"], - ) - ) - controls = { - cid: ControlSpec(control_id=cid, description=f"control {cid}", verification="automated") - for cid in statuses - } - return build_traceability_matrix(ReportData(results=results), controls) -``` - -Verify it works before writing assertions against it: - -Run: `uv run python -c "import sys; sys.path.insert(0, 'selftests'); from conftest import matrix_from_statuses; m = matrix_from_statuses({'c1': ['passed']}); print(m.entries[0].coverage, m.entries[0].executed)"` -Expected: `covered True` - -- [ ] Step 3: Write the failing tests - -Add to `selftests/test_traceability_matrix.py`, importing the helper with -`from conftest import matrix_from_statuses`. The assertions: - -```python -class TestFailingControls: - """A control whose scenarios ran and did not pass is not evidence.""" - - def test_a_failed_scenario_marks_the_control_failing(self): - matrix = matrix_from_statuses(statuses={"c1": ["failed"]}) - entry = matrix.entries[0] - assert entry.coverage == "covered" - assert entry.executed is True - assert entry.failing is True - assert matrix.covered_with_failure == ["c1"] - - def test_an_errored_scenario_marks_the_control_failing(self): - """`error` is a reachable outcome; enumerating only "failed" would miss it.""" - matrix = matrix_from_statuses(statuses={"c1": ["error"]}) - assert matrix.entries[0].failing is True - - def test_a_mixed_pass_and_failure_marks_the_control_failing(self): - matrix = matrix_from_statuses(statuses={"c1": ["passed", "failed"]}) - assert matrix.entries[0].failing is True - assert matrix.covered_with_failure == ["c1"] - - def test_a_pass_beside_a_skip_is_not_failing(self): - matrix = matrix_from_statuses(statuses={"c1": ["passed", "skipped"]}) - assert matrix.entries[0].failing is False - assert matrix.covered_with_failure == [] - - def test_an_all_skipped_control_is_not_failing(self): - """Not executed and not failing are different states.""" - matrix = matrix_from_statuses(statuses={"c1": ["skipped"]}) - entry = matrix.entries[0] - assert entry.executed is False - assert entry.failing is False - - def test_a_version_gated_control_is_not_failing(self): - matrix = matrix_from_statuses(statuses={"c1": ["na_version"]}) - assert matrix.entries[0].failing is False -``` - - -- [ ] Step 3: Run the tests to verify they fail - -Run: `uv run pytest selftests/test_traceability_matrix.py::TestFailingControls -v` -Expected: FAIL with `AttributeError: 'ControlEntry' object has no attribute 'failing'`. - -- [ ] Step 4: Add the `failing` property - -In `src/vip/traceability.py`, immediately after `ControlEntry.executed`: - -```python - @property - def failing(self) -> bool: - """Whether any tagged scenario produced a result that was not a pass. - - The third fact about a control, after "a scenario is tagged" - (``coverage``) and "a scenario ran" (``executed``). Without it a - control whose only scenario failed reports as covered and executed, - which is true and reads as evidence. - - Defined by exclusion rather than by enumerating failure statuses: - ``error`` is a reachable outcome alongside ``failed``, and an - enumerated list would let an errored control read as evidenced. - Non-executing statuses are excluded via ``NON_EXECUTING_STATUSES``, so - a skip alongside a pass never counts against a control. - """ - return any( - m.status not in NON_EXECUTING_STATUSES and m.status != "passed" for m in self.matches - ) -``` - -- [ ] Step 5: Add the matrix-level list - -In `TraceabilityMatrix`, immediately after `covered_without_execution`: - -```python - @property - def covered_with_failure(self) -> list[str]: - """Control ids that are covered but whose scenarios did not all pass. - - The mirror of ``covered_without_execution``. That one catches a matrix - that is green because nothing ran; this one catches a matrix that is - green because a run that did happen was not a success. Any failing - scenario qualifies the control, not only an all-failed one: a green - badge above a visible failing scenario row is the misreading this - exists to prevent. - """ - return [ - e.control.control_id for e in self.entries if e.coverage == "covered" and e.failing - ] -``` - -- [ ] Step 6: Run the tests to verify they pass - -Run: `uv run pytest selftests/test_traceability_matrix.py -v` -Expected: PASS. - -- [ ] Step 7: Lint and commit - -```bash -uvx ruff@0.15.0 check src/ src/vip_tests/ selftests/ examples/ -uvx ruff@0.15.0 format src/ src/vip_tests/ selftests/ examples/ -git add src/vip/traceability.py selftests/test_traceability_matrix.py -git commit -m "feat(traceability): record whether a covered control's scenarios passed" -``` - ---- - -### Task 4: Render a failed control red, not green - -Files: -- Modify: `src/vip/report_content.py:407` (`COVERAGE_STYLE_KEY`, `COVERAGE_LABELS`, - `display_coverage`, `traceability_summary_rows`) -- Test: `selftests/test_report_content.py` - -Interfaces: -- Consumes: `ControlEntry.failing` from Task 3. -- Produces: the display value `"covered_failed"` and its entries in - `COVERAGE_STYLE_KEY` and `COVERAGE_LABELS`, used by Task 6. - -- [ ] Step 1: Write the failing tests - -Add to `selftests/test_report_content.py`: - -```python -class TestFailedControlDisplay: - def test_a_failing_control_displays_as_covered_failed(self): - entry = SimpleNamespace(coverage="covered", executed=True, failing=True) - assert report_content.display_coverage(entry) == "covered_failed" - - def test_a_failing_control_uses_the_failed_style(self): - """Reuses the outcome palette so the styles.css drift guard still holds.""" - assert report_content.COVERAGE_STYLE_KEY["covered_failed"] == "failed" - - def test_a_failing_control_is_labelled_failed(self): - assert report_content.COVERAGE_LABELS["covered_failed"] == "FAILED" - - def test_a_passing_control_still_displays_as_covered(self): - entry = SimpleNamespace(coverage="covered", executed=True, failing=False) - assert report_content.display_coverage(entry) == "covered" - - def test_an_all_skipped_control_still_displays_as_not_executed(self): - entry = SimpleNamespace(coverage="covered", executed=False, failing=False) - assert report_content.display_coverage(entry) == "covered_not_executed" - - def test_a_gap_is_unaffected(self): - entry = SimpleNamespace(coverage="gap", executed=False, failing=False) - assert report_content.display_coverage(entry) == "gap" - - def test_every_coverage_value_has_a_style_and_a_label(self): - assert set(report_content.COVERAGE_STYLE_KEY) == set(report_content.COVERAGE_LABELS) -``` - -Import `from types import SimpleNamespace` at the top of the file if it is not -already imported. - -- [ ] Step 2: Run the tests to verify they fail - -Run: `uv run pytest selftests/test_report_content.py::TestFailedControlDisplay -v` -Expected: FAIL with `KeyError: 'covered_failed'`. - -- [ ] Step 3: Add the style and label entries - -In `src/vip/report_content.py`, replace the two dicts with: - -```python -COVERAGE_STYLE_KEY = { - "covered": "passed", - "covered_not_executed": "na_version", - "covered_failed": "failed", - "gap": "failed", - "not_automatable": "skipped", -} - -COVERAGE_LABELS = { - "covered": "COVERED", - "covered_not_executed": "NOT RUN", - "covered_failed": "FAILED", - "gap": "GAP", - "not_automatable": "N/A (manual)", -} -``` - -Extend the comment above `COVERAGE_STYLE_KEY` with a final sentence: - -``` -# and a covered control whose scenarios ran without passing like a failure -- -# the same red as a gap, because both mean the control is not evidenced. -``` - -- [ ] Step 4: Add the branch to `display_coverage` - -Replace the body of `display_coverage`: - -```python -def display_coverage(entry) -> str: # noqa: ANN001 - vip.traceability.ControlEntry - """Flatten coverage, execution and outcome into the one value the report shows.""" - if entry.coverage == "covered" and entry.failing: - return "covered_failed" - if entry.coverage == "covered" and not entry.executed: - return "covered_not_executed" - return entry.coverage -``` - -The two new branches cannot both apply: `failing` is false whenever nothing -executed. - -- [ ] Step 5: Add the summary row - -In `traceability_summary_rows`, insert one row between "Covered, not executed" -and "Gaps": - -```python - ("Covered, failing", str(counts.get("covered_failed", 0))), -``` - -- [ ] Step 6: Extend the caveat - -`TRACEABILITY_CAVEAT` currently says a control shown as NOT RUN skipped itself. -Append one sentence before the final two sentences: - -``` -"A control shown as FAILED has a tagged scenario that ran and did not pass, " -"so the control is not evidenced by this run. " -``` - -- [ ] Step 7: Run the tests to verify they pass - -Run: `uv run pytest selftests/test_report_content.py -v` -Expected: PASS, including the pre-existing styles.css color drift guard. - -- [ ] Step 8: Lint and commit - -```bash -uvx ruff@0.15.0 check src/ src/vip_tests/ selftests/ examples/ -uvx ruff@0.15.0 format src/ src/vip_tests/ selftests/ examples/ -git add src/vip/report_content.py selftests/test_report_content.py -git commit -m "fix(report): render a control whose scenarios failed as failed" -``` - ---- - -### Task 5: Warn about failing controls in `vip trace` and the report - -Files: -- Modify: `src/vip/report_content.py:500` (`traceability_warning` becomes - `traceability_warnings`) -- Modify: `src/vip/report_html.py:437` and `src/vip/report_typst.py:566` - (both call sites) -- Modify: `src/vip/traceability.py:407` (`render_json` summary block) -- Modify: `src/vip/cli.py:1774` (`run_trace` warnings and the closing line) -- Test: `selftests/test_report_content.py`, `selftests/test_traceability_render.py`, - `selftests/test_trace_cli.py` - -Interfaces: -- Consumes: `TraceabilityMatrix.covered_with_failure` from Task 3. -- Produces: `report_content.traceability_warnings(matrix) -> list[str]`, - replacing the singular `traceability_warning`. Task 6 does not use it, but - both report backends now iterate the list. - -- [ ] Step 1: Write the failing tests - -In `selftests/test_report_content.py`: - -```python -class TestTraceabilityWarnings: - def test_a_failing_control_produces_a_warning(self): - matrix = SimpleNamespace(covered_without_execution=[], covered_with_failure=["c1"]) - warnings_out = report_content.traceability_warnings(matrix) - assert any("did not pass" in w and "c1" in w for w in warnings_out) - - def test_both_conditions_produce_two_warnings(self): - matrix = SimpleNamespace(covered_without_execution=["c2"], covered_with_failure=["c1"]) - assert len(report_content.traceability_warnings(matrix)) == 2 - - def test_a_clean_matrix_produces_none(self): - matrix = SimpleNamespace(covered_without_execution=[], covered_with_failure=[]) - assert report_content.traceability_warnings(matrix) == [] -``` - -In `selftests/test_traceability_render.py`, add a test asserting the JSON -summary carries the new key: - -```python -def test_json_summary_counts_failing_controls(): - matrix = matrix_from_statuses(statuses={"c1": ["failed"], "c2": ["passed"]}) - summary = json.loads(render_json(matrix))["summary"] - assert summary["covered_failed"] == 1 - assert summary["covered_and_executed"] == 2 -``` - -Import the helper with `from conftest import matrix_from_statuses` (added in -Task 3). - -In `selftests/test_trace_cli.py`, following whatever pattern that file uses to -invoke `run_trace` and capture stderr: - -```python -def test_trace_warns_when_a_covered_control_failed(tmp_path, capsys): - """The mirror of the covered-not-executed warning.""" - results = tmp_path / "results.json" - results.write_text( - json.dumps( - { - "schema_version": "1.0", - "results": [ - { - "nodeid": "t.py::test_a", - "outcome": "failed", - "markers": ["control-c1"], - } - ], - } - ), - encoding="utf-8", - ) - controls = tmp_path / "controls.toml" - controls.write_text( - '[controls.c1]\ndescription = "a control"\nverification = "automated"\n', - encoding="utf-8", - ) - run_trace( - argparse.Namespace( - results=str(results), controls=str(controls), format="json", output=None - ) - ) - captured = capsys.readouterr() - assert "did not pass" in captured.err - assert "c1" in captured.err -``` - -Match the `argparse.Namespace` fields to whatever `run_trace` actually reads in -this branch's `src/vip/cli.py`; add any attribute it accesses that is missing -here rather than guessing a default. - -- [ ] Step 2: Run the tests to verify they fail - -Run: `uv run pytest selftests/test_report_content.py::TestTraceabilityWarnings selftests/test_traceability_render.py selftests/test_trace_cli.py -v` -Expected: FAIL with `AttributeError: module 'vip.report_content' has no attribute 'traceability_warnings'` and `KeyError: 'covered_failed'`. - -- [ ] Step 3: Replace `traceability_warning` with the plural form - -In `src/vip/report_content.py`, replace the whole function: - -```python -def traceability_warnings(matrix) -> list[str]: # noqa: ANN001 - """Lines naming controls that look covered but are not evidence. - - Two independent conditions, so two lines rather than one combined - sentence: a control can be counted as covered because nothing ran, or - because what ran did not pass, and a reader needs to know which. - """ - lines = [] - failing = matrix.covered_with_failure - if failing: - lines.append( - f"{pluralize(len(failing), 'control')} counted as covered but had a " - f"scenario that did not pass: {', '.join(failing)}." - ) - unexecuted = matrix.covered_without_execution - if unexecuted: - lines.append( - f"{pluralize(len(unexecuted), 'control')} counted as covered but had no " - f"scenario that ran: {', '.join(unexecuted)}." - ) - return lines -``` - -- [ ] Step 4: Update both backends to iterate - -In `src/vip/report_html.py`, replace: - -```python - warning = traceability_warning(matrix) - if warning: - parts.append(f"

{_esc(warning)}

") -``` - -with: - -```python - for warning in traceability_warnings(matrix): - parts.append(f"

{_esc(warning)}

") -``` - -In `src/vip/report_typst.py`, replace: - -```python - warning = traceability_warning(matrix) - if warning: - parts.append(_paragraph(warning)) -``` - -with: - -```python - for warning in traceability_warnings(matrix): - parts.append(_paragraph(warning)) -``` - -Update the `traceability_warning` import to `traceability_warnings` in both -files (`src/vip/report_html.py:32` and `src/vip/report_typst.py:48` name the -import block). - -- [ ] Step 5: Add the JSON summary key - -In `src/vip/traceability.py`'s `render_json` summary dict, after -`"covered_not_executed"`: - -```python - # Covered and executed, but not a success. The third way a green - # matrix can mislead, after "nothing is tagged" and "nothing ran". - "covered_failed": len(matrix.covered_with_failure), -``` - -`MATRIX_SCHEMA_VERSION` stays `"1.0"`: the key is additive and the `coverage` -field's value set is unchanged. - -- [ ] Step 6: Add the CLI warning - -In `src/vip/cli.py`'s `run_trace`, immediately before the existing -`unexecuted = matrix.covered_without_execution` block: - -```python - # A covered control whose scenarios ran and failed also counts toward - # "0 gaps". Coverage records that a scenario ran, not that it passed, and - # a compliance matrix that stays silent here is the more expensive of the - # two ways this tool can mislead. - failing = matrix.covered_with_failure - if failing: - print( - f"Warning: {len(failing)} covered control(s) had a scenario that did not " - f"pass: {', '.join(failing)}. Coverage records that a scenario ran, not " - "that it passed.", - file=sys.stderr, - ) -``` - -- [ ] Step 7: Report failures in the closing line - -Replace the final line of `run_trace`: - -```python - print(f"Wrote {out} ({len(matrix.entries)} controls, {matrix.gap_count} gaps)") -``` - -with: - -```python - print( - f"Wrote {out} ({len(matrix.entries)} controls, {matrix.gap_count} gaps, " - f"{len(matrix.covered_with_failure)} failing)" - ) -``` - -- [ ] Step 8: Run the tests to verify they pass - -Run: `uv run pytest selftests/ -q` -Expected: PASS. Any pre-existing test asserting on the old -`Wrote ... gaps)` string or importing `traceability_warning` needs updating in -this same task. - -- [ ] Step 9: Lint and commit - -```bash -uvx ruff@0.15.0 check src/ src/vip_tests/ selftests/ examples/ -uvx ruff@0.15.0 format src/ src/vip_tests/ selftests/ examples/ -git add src/vip/report_content.py src/vip/report_html.py src/vip/report_typst.py \ - src/vip/traceability.py src/vip/cli.py selftests/ -git commit -m "feat(trace): warn when a covered control's scenarios did not pass" -``` - ---- - -### Task 6: Make the coverage badge identical in both editions - -Files: -- Modify: `src/vip/report_typst.py:583` (the `vip-pill` call) -- Modify: `src/vip/report_html.py:445` (the badge class) -- Modify: `report/styles.css` (add `.trace-caveat` and `.trace-warning`) -- Test: `selftests/test_report_typst.py`, `selftests/test_report_traceability.py` - -Interfaces: -- Consumes: `COVERAGE_STYLE_KEY` and `COVERAGE_LABELS` from Task 4. -- Produces: nothing later tasks rely on. - -- [ ] Step 1: Write the failing tests - -In `selftests/test_report_typst.py`, importing the helper with -`from conftest import matrix_from_statuses` (added in Task 3): - -```python -def test_coverage_badge_uses_the_same_chip_as_an_outcome(): - """vip-pill is a saturated fill with white text; the HTML edition is a chip.""" - matrix = matrix_from_statuses(statuses={"c1": ["passed"]}) - out = report_typst.render_traceability(matrix) - assert "vip-chip" in out - assert "vip-pill" not in out -``` - -In `selftests/test_report_traceability.py`, with the same import: - -```python -def test_html_coverage_badge_uses_a_class_that_exists_in_styles_css(): - css = (Path(__file__).parent.parent / "report" / "styles.css").read_text() - matrix = matrix_from_statuses(statuses={"c1": ["passed"]}) - html = report_html.render_traceability(matrix) - for cls in ("vip-badge", "trace-caveat", "trace-warning"): - assert f".{cls}" in css, f"{cls} is referenced by the renderer but absent from styles.css" - assert "class='badge'" not in html -``` - -The CSS assertions read `styles.css` directly rather than going through the -rendered HTML, because `.trace-warning` only appears in the output when the -matrix has a warning to show. Checking the stylesheet covers both classes -whatever the matrix contains. - -- [ ] Step 2: Run the tests to verify they fail - -Run: `uv run pytest selftests/test_report_typst.py selftests/test_report_traceability.py -v` -Expected: FAIL. `vip-pill` is present in the Typst output, and `.vip-badge` is -in `styles.css` but `.trace-caveat` and `.trace-warning` are not. - -- [ ] Step 3: Switch Typst to the chip - -In `src/vip/report_typst.py`, replace: - -```python - _call("vip-pill", _lit(COVERAGE_LABELS[row.coverage]), _lit(style.color)), -``` - -with: - -```python - # vip-chip, not vip-pill: the HTML edition renders dark text on - # a pale fill (outcome_badge_html), and vip-pill is a saturated - # fill with white text. The two editions must match. - _call( - "vip-chip", - _lit(COVERAGE_LABELS[row.coverage]), - _lit(style.color), - _lit(style.background), - ), -``` - -- [ ] Step 4: Point the HTML badge at the real class - -In `src/vip/report_html.py`, replace: - -```python - f" str: # noqa: ANN001 -``` - -Where the function currently decides whether to append the traceability -section, add the error branch. `trace_error` renders through `_paragraph`, -which routes the text through `_lit`, so a `#`, `*` or `$` in an exception -message is escaped rather than executed: - -```python - if trace_error: - parts.append(_heading("Compliance Traceability")) - parts.append(_paragraph(TRACEABILITY_RENDER_FAILURE.format(error=trace_error))) - elif matrix is not None: - parts.append(_heading("Compliance Traceability")) - parts.append(render_traceability(matrix)) -``` - -Read the existing `if matrix is not None:` block in `render_document` first and -keep its exact heading call and append order. The two branches must emit the -same heading, so a reader of the PDF sees the section start either way. Use the -module's own heading helper rather than inventing one; if it is not named -`_heading`, use the real name in both branches. - -- [ ] Step 5: Verify the checksum in `report/index.qmd` - -Replace the body of the `if _controls:` block: - -```python -if _controls: - _trace_error = None - try: - from vip.traceability import ( - build_traceability_matrix, - load_controls, - verify_results_checksum, - ) - - _results_path = Path("results.json") - # Verify rather than only digest. --controls makes this a compliance - # artifact, and the sidecar attestation in the provenance block is - # worthless if nothing checked it. verify_results_checksum raises, and - # the except below is now a visible marker rather than a dropped - # section, so raising here is safe. - _digest, _verified = verify_results_checksum(_results_path) - _matrix = build_traceability_matrix( - data, - load_controls(_controls), - results_sha256=_digest, - results_sha256_sidecar_verified=_verified or None, - ) - display(Markdown("## Compliance Traceability")) - display(HTML(report_html.render_traceability(_matrix))) - except Exception as exc: # noqa: BLE001 - a report must render regardless - display(Markdown("## Compliance Traceability")) - display(Markdown(f"> {report_content.TRACEABILITY_RENDER_FAILURE.format(error=exc)}")) -``` - -Add `from vip import report_content` to that cell's imports if the file does not -already import it. Remove the now-unused `import hashlib` if nothing else in the -file uses it. - -- [ ] Step 6: Do the same in `report/vip-report.qmd` - -```python -matrix = None -trace_error = None -_controls = os.environ.get("VIP_CONTROLS") -if _controls: - try: - from vip.traceability import ( - build_traceability_matrix, - load_controls, - verify_results_checksum, - ) - - _digest, _verified = verify_results_checksum(Path("results.json")) - matrix = build_traceability_matrix( - data, - load_controls(_controls), - results_sha256=_digest, - results_sha256_sidecar_verified=_verified or None, - ) - except Exception as exc: # noqa: BLE001 - a report must render regardless - matrix = None - trace_error = str(exc) - -print("```{=typst}") -print(report_typst.render_document(data, hints, matrix, trace_error)) -print("```") -``` - -Remove the now-unused `import hashlib` if nothing else in the file uses it. - -- [ ] Step 7: Correct the stale workflow comment - -In `.github/workflows/example-report.yml`, replace the comment above the -"Render the compliance report" step: - -```yaml - # VIP_CONTROLS must be absolute. Quarto renders with report/ as the - # working directory, so a relative path resolves against report/ and the - # control list is not found. Both documents now print a visible "could - # not render the traceability section" marker instead of dropping it, so - # this fails loudly rather than going green with the section missing -- - # but an absolute path is still what makes it work. -``` - -- [ ] Step 8: Run the tests to verify they pass - -Run: `uv run pytest selftests/ -q` -Expected: PASS. - -- [ ] Step 9: Render both editions for real - -Run: - -```bash -uv run vip verify --categories prerequisites -- -q || true -VIP_CONTROLS="$PWD/examples/21CFR_part11_validation/controls.toml" \ - uv run vip report -``` - -Expected: `_output/index.html` and `_output/vip-report.pdf` both produced, both -carrying a Compliance Traceability section. If Quarto is not installed locally, -say so and skip this step rather than reporting it as passed. - -- [ ] Step 10: Lint and commit - -```bash -uvx ruff@0.15.0 check src/ src/vip_tests/ selftests/ examples/ -uvx ruff@0.15.0 format src/ src/vip_tests/ selftests/ examples/ -git add src/vip/report_content.py src/vip/report_typst.py report/index.qmd \ - report/vip-report.qmd .github/workflows/example-report.yml selftests/ -git commit -m "fix(report): show a traceability render failure in both editions" -``` - ---- - -### Task 8: Gate `vip report --controls` on the checksum - -Files: -- Modify: `src/vip/cli.py:813-845` (the `--controls` validation block) -- Test: `selftests/test_cli_report.py` - -Interfaces: -- Consumes: nothing from earlier tasks. -- Produces: nothing later tasks rely on. - -- [ ] Step 1: Write the failing test - -In `selftests/test_cli_report.py`, following that file's existing pattern for -invoking `run_report` with a temp report directory: - -```python -def test_report_with_controls_refuses_a_mismatched_sidecar(tmp_path, capsys): - """--controls makes this a compliance artifact; it inherits trace's strictness.""" - results = tmp_path / "results.json" - results.write_text('{"schema_version": "1.0", "results": []}', encoding="utf-8") - results.with_name("results.json.sha256").write_text(f"{'a' * 64} results.json\n") - controls = tmp_path / "controls.toml" - controls.write_text( - '[[control]]\nid = "c1"\ndescription = "d"\nverification = "automated"\n', - encoding="utf-8", - ) - with pytest.raises(SystemExit) as exc: - run_report(_report_args(results=results, controls=controls)) - assert exc.value.code == 1 - assert "checksum mismatch" in capsys.readouterr().err - - -def test_report_without_controls_ignores_a_mismatched_sidecar(tmp_path, monkeypatch): - """Plain vip report stays lenient: a report must render regardless.""" - results = tmp_path / "results.json" - results.write_text('{"schema_version": "1.0", "results": []}', encoding="utf-8") - results.with_name("results.json.sha256").write_text(f"{'a' * 64} results.json\n") - # Stub the Quarto invocation so the test asserts on the gate, not on a - # local Quarto install; follow whatever this file already does for that. - monkeypatch.setattr(cli, "_quarto_render", lambda *a, **k: 0) - run_report(_report_args(results=results, controls=None)) - # No SystemExit: the mismatched sidecar is not consulted without --controls. -``` - -Match the `controls.toml` shape to whatever `load_controls` actually expects; -copy the smallest valid example out of `examples/21CFR_part11_validation/controls.toml`. - -- [ ] Step 2: Run the test to verify it fails - -Run: `uv run pytest selftests/test_cli_report.py -v -k sidecar` -Expected: FAIL. No `SystemExit` is raised, because nothing verifies the sidecar. - -- [ ] Step 3: Add the verification to the gate - -In `src/vip/cli.py`, add `verify_results_checksum` to the import list in the -`if getattr(args, "controls", None):` block, then add the call inside the -existing `try`, after `check_results_rows(results_dest)`: - -```python - # The sidecar too, not only the schema and the rows. A compliance - # render is an evidence artifact, so it inherits `vip trace`'s - # strictness in full rather than in part. - verify_results_checksum(results_dest) -``` - -The existing `except (ResultsIntegrityError, ControlListError)` clause already -catches what this raises, so no new handler is needed. - -- [ ] Step 4: Run the tests to verify they pass - -Run: `uv run pytest selftests/test_cli_report.py -v` -Expected: PASS. - -- [ ] Step 5: Lint and commit - -```bash -uvx ruff@0.15.0 check src/ src/vip_tests/ selftests/ examples/ -uvx ruff@0.15.0 format src/ src/vip_tests/ selftests/ examples/ -git add src/vip/cli.py selftests/test_cli_report.py -git commit -m "fix(cli): verify the checksum sidecar on a compliance render" -``` - ---- - -### Task 9: Compare schema majors as numbers - -Files: -- Modify: `src/vip/reporting.py:213` -- Test: `selftests/test_results_schema.py` - -Interfaces: -- Consumes: nothing. Produces: nothing. - -- [ ] Step 1: Write the failing test - -In `selftests/test_results_schema.py`: - -```python -def test_direction_is_numeric_not_lexicographic(monkeypatch, tmp_path): - """"9" > "10" as strings; a results file at major 9 is older, not newer.""" - monkeypatch.setattr(reporting, "RESULTS_SCHEMA_VERSION", "10.0") - p = tmp_path / "results.json" - p.write_text('{"schema_version": "9.0", "results": []}', encoding="utf-8") - with pytest.warns(UserWarning, match="older than"): - reporting.load_results(p) -``` - -- [ ] Step 2: Run the test to verify it fails - -Run: `uv run pytest selftests/test_results_schema.py -v -k lexicographic` -Expected: FAIL. The warning says "newer than". - -- [ ] Step 3: Compare as integers - -In `src/vip/reporting.py`, replace: - -```python - direction = "newer than" if theirs > ours else "older than" -``` - -with: - -```python - # int, not string: "9" > "10" lexicographically, so a string - # compare misreports the direction once either major reaches two - # digits. A non-numeric major is possible in a hand-edited file, - # so fall back to the string compare rather than raising inside a - # loader whose contract is to warn and carry on. - try: - newer = int(theirs) > int(ours) - except ValueError: - newer = theirs > ours - direction = "newer than" if newer else "older than" -``` - -- [ ] Step 4: Run the tests to verify they pass - -Run: `uv run pytest selftests/test_results_schema.py -v` -Expected: PASS. - -- [ ] Step 5: Lint and commit - -```bash -uvx ruff@0.15.0 check src/ src/vip_tests/ selftests/ examples/ -uvx ruff@0.15.0 format src/ src/vip_tests/ selftests/ examples/ -git add src/vip/reporting.py selftests/test_results_schema.py -git commit -m "fix(reporting): compare schema majors numerically" -``` - ---- - -### Task 10: Update the docs the behaviour changes touch - -Files: -- Modify: `docs/reporting.md` (the `.sha256` sidecar section, around line 100) -- Modify: `AGENTS.md` (the `src/vip/traceability.py` row of the key-source-files table) -- Test: none. Documentation only. - -Interfaces: consumes nothing, produces nothing. - -- [ ] Step 1: Document the sidecar matching rule - -In `docs/reporting.md`, after the `shasum -a 256 -c results.json.sha256` block, -add: - -```markdown -The recorded filename is matched on its exact value first, then on its basename. -A sidecar generated from a directory above the results file records the path it -was given (` report/results.json`) and still verifies. A multi-file -sidecar that names the file exactly keeps the stricter exact match, so it cannot -be satisfied by a same-named file in another directory. -``` - -- [ ] Step 2: Document the failing-control state - -In `docs/reporting.md`, wherever the traceability coverage values are described, -add `FAILED` alongside `COVERED`, `NOT RUN`, `GAP` and `N/A (manual)`, described -as: a covered control with at least one tagged scenario that ran and did not -pass. If that list does not exist in the file yet, add it near the traceability -section rather than inventing a new one elsewhere. - -- [ ] Step 3: Update the AGENTS.md traceability row - -The row currently explains the `executed` / `covered_without_execution` split. -Add one sentence: `ControlEntry.failing` and -`TraceabilityMatrix.covered_with_failure` are the third fact, separating "a -scenario ran" from "a scenario passed", and any failing scenario demotes the -control's badge rather than only an all-failed one. - -- [ ] Step 4: Run the docs drift guard - -Run: `uv run pytest selftests/test_scaffold_agents_md.py -v` -Expected: PASS. This parses the real source and fails if the inventory drifts. - -- [ ] Step 5: Commit - -```bash -git add docs/reporting.md AGENTS.md -git commit -m "docs: describe the sidecar match rule and the failing-control state" -``` - ---- - -## Final verification - -- [ ] Run the full selftest suite: `uv run pytest selftests/ -q`. Expected: every - test passes. Report the actual count. -- [ ] Run the linter and formatter over all four directories. -- [ ] Collect the product tests as a dry run: `uv run pytest src/vip_tests/ --collect-only -q`. -- [ ] Collect the example: `uv run pytest examples/21CFR_part11_validation/ --collect-only -q`. -- [ ] Confirm `MATRIX_SCHEMA_VERSION` is still `"1.0"` and `ControlEntry.coverage` - still has exactly three values. diff --git a/docs/superpowers/specs/2026-08-29-traceability-review-fixes-design.md b/docs/superpowers/specs/2026-08-29-traceability-review-fixes-design.md deleted file mode 100644 index cead6638..00000000 --- a/docs/superpowers/specs/2026-08-29-traceability-review-fixes-design.md +++ /dev/null @@ -1,268 +0,0 @@ -# Traceability review fixes: design - -Date: 2026-08-29 -Branch: feat/part11-traceability - -## Why - -A review of `feat/part11-traceability` found eight defects, all verified against -the working tree. Two of them make VIP refuse evidence that was never tampered -with. Two more let a compliance artifact render a failing control as green. The -rest are divergences between the HTML and PDF report editions, plus one cosmetic -comparison bug. - -The branch is a compliance feature. Both failure directions are expensive here: -refusing good evidence blocks a validation run, and passing bad evidence is worse -than not having the tool. - -## Decisions taken - -These were settled during brainstorming and are the constraints the design works -within. - -1. A failed control gets a new value at the display layer only. - `ControlEntry.coverage` keeps its three values (`covered`, `gap`, - `not_automatable`), so the `coverage` column in the CSV and JSON matrix is - unchanged and downstream consumers see nothing new there. -2. Sidecar filename matching tries the exact recorded name first and falls back - to a basename comparison only when no entry matched exactly. -3. `vip report --controls` verifies the sha256 sidecar under the same gate that - already runs the schema and row checks. -4. Any failing scenario demotes a control's badge to red, not only an - all-failed control. A green badge above a visible failing scenario row is the - misreading this change exists to prevent. The cost is accepted: a control - checked by three scenarios goes red when one of them flakes. - -## Section 1: Sidecar trust - -### The defect - -`verify_results_checksum` (`src/vip/traceability.py:490`) selects sidecar entries -with `name == p.name`, comparing against the bare filename. A sidecar generated -from a directory above the results file records a path, so nothing matches and -the function raises `ResultsIntegrityError`. - -Reproduced: - -``` -$ shasum -a 256 report/results.json > report/results.json.sha256 -$ vip trace --results report/results.json --controls controls.toml -Error: checksum sidecar report/results.json.sha256 does not record an entry for -results.json; it names report/results.json. Regenerate it, or delete it to -proceed without verification. -$ echo $? -1 -``` - -`shasum -a 256 report/results.json` is the natural command for an operator who is -not standing in `report/`, and it is what a CI job re-hashing an archived -artifact from the workspace root produces. `docs/reporting.md` documents the bare -name form, so the docs are not wrong, but they do not cover the case that breaks. - -`_rehome_sidecar` (`src/vip/cli.py:1660`) has the mirror of the same defect and -two more edges. - -### The fix - -In `verify_results_checksum`, keep the exact comparison as the primary key. When -`named` comes back empty and before the unnamed-entry fallback, retry comparing -`PurePosixPath(name).name` against `p.name`. A single-file sidecar written from -any directory verifies. A multi-file sidecar that already records the exact name -takes the first branch and keeps today's strict behaviour, so the disambiguation -the existing comment describes is preserved. - -In `_rehome_sidecar`, three changes: - -- The `recorded in (None, src_name)` test gains the same basename fallback, so a - path-qualified line is rewritten to the destination name rather than copied - through verbatim. Copying it through produces a rehomed sidecar that then fails - verification at the destination, which is the false tamper alarm the function - exists to prevent. -- Catch `UnicodeDecodeError` from `src.read_text(encoding="utf-8-sig")`. The call - site at `src/vip/cli.py:779` catches only `OSError`, so a corrupt sidecar - currently reaches the user as a traceback. `verify_results_checksum` already - catches `UnicodeDecodeError` on the identical read, and the two paths should - agree. -- When the source sidecar exists but parses to zero entries, unlink the - destination instead of writing an empty file. `verify_results_checksum` - deliberately refuses an empty sidecar as the truncated-upload case, so writing - one manufactures the very state the reader is told to distrust. No sidecar is a - documented benign state. - -## Section 2: Coverage honesty - -### The defect - -`build_traceability_matrix` sets `coverage = "covered"` whenever a control has any -tagged scenario. `ControlEntry.executed` separates "a scenario is tagged" from "a -scenario ran", and `display_coverage` (`src/vip/report_content.py:443`) demotes -an all-skipped control to amber. Nothing separates "a scenario ran" from "a -scenario passed". - -Reproduced with a results file whose only `control-record-retention` row has -outcome `failed`: the JSON matrix reports `coverage: "covered"`, -`covered_and_executed: 2`, exit status 0, and no warning. The report renders a -green COVERED badge, because `COVERAGE_STYLE_KEY` maps `covered` onto the -`passed` style. - -### The fix - -`ControlEntry` gains a `failing` property: - -```python -@property -def failing(self) -> bool: - """Whether any tagged scenario produced a result that was not a pass.""" - return any( - m.status not in NON_EXECUTING_STATUSES and m.status != "passed" - for m in self.matches - ) -``` - -Named `failing`, not `verified` or `passed`. `ControlSpec.verification` already -means automated/manual/procedural and `results_sha256_sidecar_verified` already -means checksum-checked, both in this file, so a third meaning on that word would -be a real ambiguity. `passed` reads as a question about the whole control, which -under decision 4 is not the question being asked. - -Defined by exclusion rather than by enumerating failure statuses. `error` is a -reachable outcome alongside `failed` (`src/vip/plugin.py:1411`), and an -enumerated list would let an errored control read green. Non-executing statuses -are excluded by reusing `NON_EXECUTING_STATUSES`, so a skip never counts against -a control. - -`display_coverage` gains one branch: - -```python -def display_coverage(entry): - if entry.coverage == "covered" and entry.failing: - return "covered_failed" - if entry.coverage == "covered" and not entry.executed: - return "covered_not_executed" - return entry.coverage -``` - -The two new branches cannot both apply, because `failing` is false whenever -nothing executed. The resulting truth table over a control's match statuses: - -| Statuses | Display | -|---|---| -| all passed | covered (green) | -| passed + skipped | covered (green) | -| passed + failed | covered_failed (red) | -| failed only | covered_failed (red) | -| error only | covered_failed (red) | -| skipped only | covered_not_executed (amber) | - -Row three is decision 4. A control checked by several scenarios goes red when one -of them fails, and the per-scenario evidence column shows which. - -`TraceabilityMatrix` gains `covered_with_failure`, mirroring -`covered_without_execution` in both shape and docstring intent. It feeds: - -- A new warning in `run_trace` (`src/vip/cli.py`), printed alongside the existing - unrecognized-tag and covered-not-executed warnings. -- The closing `Wrote ... (N controls, M gaps)` line, which gains a failure count. -- A `covered_failed` key in `render_json`'s summary block. -- `traceability_warning` (`src/vip/report_content.py:500`), so the rendered - report states it too. - -`MATRIX_SCHEMA_VERSION` stays at `1.0`. The `coverage` field's value set is -unchanged under decision 1, and a new summary key is additive. - -## Section 3: Report parity, and a real attestation - -### Three defects, one change - -The coverage badge renders differently in the two editions. `report_html` -(`src/vip/report_html.py:445`) emits an inline `color` and `background` pair, -dark text on a pale fill, matching `outcome_badge_html`. `report_typst` -(`src/vip/report_typst.py:583`) calls `vip-pill(label, style.color)`, a saturated -fill with white text. The Typst equivalent of the HTML treatment is -`vip-chip(label, fg, bg)`, which `outcome_chip` already uses at -`src/vip/report_typst.py:210`. - -The HTML classes `badge`, `trace-caveat` and `trace-warning` do not exist in -`report/styles.css`. The styled class is `.vip-badge` (`report/styles.css:133`), -so the coverage badge currently renders as unpadded, non-uppercased inline text. - -On an error, `report/index.qmd:90` displays `Could not render the traceability -section: ` while `report/vip-report.qmd` sets `matrix = None` and renders -nothing at all. CLAUDE.md requires the two editions stay in step. - -### The fix - -`covered_failed` maps onto the existing `failed` outcome style in -`COVERAGE_STYLE_KEY`, reusing the outcome palette exactly as the comment above -that dict describes, so the drift guard in `selftests/test_report_content.py` -keeps working unchanged. `COVERAGE_LABELS` gains `covered_failed: "FAILED"`. Two -red labels result, GAP and FAILED, which is intended: both mean the control is -not evidenced. - -Typst switches to `vip-chip(label, style.color, style.background)`. The HTML -badge points at `.vip-badge`. `styles.css` gains real `.trace-caveat` and `.trace-warning` rules rather than -those paragraphs dropping their classes, because the Typst edition renders the -caveat italic (`report_typst.render_traceability` passes `italic=True`) and the -two editions have to match. - -The render-failure sentence moves into `report_content` so both editions word it -identically, and `vip-report.qmd` displays it instead of silently rendering -nothing. On the Typst side the exception text passes through `_lit`, because an -exception message is exactly the dynamic value that rule exists for. - -### Why decision 3 needs the qmd cells - -`.github/workflows/example-report.yml:456` renders the compliance report with a -raw `quarto render`, so it never enters `run_report` and never sees a CLI gate. -The qmd cells rebuild the matrix themselves and pass no sidecar argument, so -`results_sha256_sidecar_verified` stays `None` in the rendered provenance no -matter what the CLI checked. - -Adding `verify_results_checksum` to `run_report`'s existing try block is still -correct and still goes in. But for the attestation to mean anything in the report -a regulated reader actually opens, both qmd cells must attempt verification -inside their own try block and pass the result into -`build_traceability_matrix`. `index.qmd`'s comment currently explains that it -computes the digest directly because `verify_results_checksum` raises by design. -That reasoning holds only while there is nowhere to show the failure. With the -shared render-failure marker from the previous subsection there is, so a -`ResultsIntegrityError` becomes a visible line in both editions rather than a -missing section. - -The half-stale comment at `.github/workflows/example-report.yml:452` is updated -in the same change. It claims both documents swallow the error silently, which -stopped being true of `index.qmd`. - -## Section 4: Loose end - -`src/vip/reporting.py:213` computes the schema-version warning direction with -`theirs > ours` on strings. With `RESULTS_SCHEMA_VERSION = "1.0"` every reachable -input compares correctly today, and it only misreports once VIP's own major -reaches double digits. A guarded `int()` conversion costs one line, so it goes in -rather than being left as a trap. - -## Testing - -Every fix gets a selftest. New or extended files: - -- `selftests/test_results_checksum.py`: a path-qualified single-file sidecar - verifies; a multi-file sidecar recording the exact name keeps its current - strict behaviour. -- A rehome test module: a path-qualified line is rewritten to the destination - name and verifies at the destination; a whitespace-only source unlinks the - destination rather than writing an empty file; a sidecar with undecodable bytes - produces a warning, not a traceback. -- `selftests/test_traceability_matrix.py`: a control whose only scenario failed; - a control mixing a pass and a failure; a control whose only scenario errored; a - control mixing a pass and a skip stays green. -- `selftests/test_trace_cli.py`: the failure warning fires, and the closing line - reports the failure count. -- `selftests/test_report_content.py`: both backends render the same coverage - label set, alongside the existing color drift guard. -- `selftests/test_cli_report.py`: `vip report --controls` refuses a results file - whose sidecar disagrees. - -## Out of scope - -No unrelated refactoring of `src/vip/traceability.py`. No change to the -`coverage` field's three values, and no `MATRIX_SCHEMA_VERSION` bump. From c513b8ce8c4908c7652e5d9a08b1995e16ea7b83 Mon Sep 17 00:00:00 2001 From: Brian Deitte Date: Sat, 29 Aug 2026 20:25:35 -0500 Subject: [PATCH 090/106] docs(examples): say 21 CFR Part 11 rather than bare Part 11 --- examples/21CFR_part11_validation/README.md | 22 ++++++++++++---------- 1 file changed, 12 insertions(+), 10 deletions(-) diff --git a/examples/21CFR_part11_validation/README.md b/examples/21CFR_part11_validation/README.md index cea77f76..1a1974fc 100644 --- a/examples/21CFR_part11_validation/README.md +++ b/examples/21CFR_part11_validation/README.md @@ -1,4 +1,4 @@ -# Part 11 validation example +# 21 CFR Part 11 validation example A worked example of mapping regulatory controls to automated tests, and exporting the result as a traceability matrix with `vip trace`. @@ -10,15 +10,17 @@ is worth in your process. ## What this is not -This is a template, not a certified Part 11 test set. A fully green matrix is -evidence for the subset of controls you chose to automate. It is not an -attestation of 21 CFR Part 11 compliance, and nobody should present it as one. - -Most of Part 11 cannot be evidenced by an automated test against Posit Team. -Roughly six clauses are genuinely testable against a deployment -- 11.10(a), -11.10(d), 11.10(e), 11.10(g), 11.30, and partly 11.10(b). Several more are -shared with your own procedures. The rest are either procedural (11.10(i), -11.10(j)) or properties of the application you build on top of Posit Team. +This is a template, not a certified 21 CFR Part 11 test set. A fully green +matrix is evidence for the subset of controls you chose to automate. It is not +an attestation of 21 CFR Part 11 compliance, and nobody should present it as +one. + +Most of 21 CFR Part 11 cannot be evidenced by an automated test against Posit +Team. Roughly six clauses are genuinely testable against a deployment -- +11.10(a), 11.10(d), 11.10(e), 11.10(g), 11.30, and partly 11.10(b). Several +more are shared with your own procedures. The rest are either procedural +(11.10(i), 11.10(j)) or properties of the application you build on top of +Posit Team. In particular, Posit Team does not implement electronic signatures. Clause 11.50 (signature manifestations), 11.70 (signature/record linking) and all of From b37698047a678a7ff005a3f8f3764d9c322b177e Mon Sep 17 00:00:00 2001 From: Brian Deitte Date: Sat, 29 Aug 2026 20:28:06 -0500 Subject: [PATCH 091/106] docs(examples): say 21 CFR Part 11 in feature names and docstrings --- docs/test-architecture.md | 2 +- examples/21CFR_part11_validation/conftest.py | 2 +- .../21CFR_part11_validation/test_21CFR_part11_connect.feature | 2 +- examples/21CFR_part11_validation/test_21CFR_part11_connect.py | 2 +- .../test_21CFR_part11_packagemanager.feature | 2 +- .../test_21CFR_part11_packagemanager.py | 3 ++- .../test_21CFR_part11_workbench.feature | 2 +- .../21CFR_part11_validation/test_21CFR_part11_workbench.py | 2 +- website/src/pages/getting-started.astro | 2 +- 9 files changed, 10 insertions(+), 9 deletions(-) diff --git a/docs/test-architecture.md b/docs/test-architecture.md index 3ccf0533..94fd63ad 100644 --- a/docs/test-architecture.md +++ b/docs/test-architecture.md @@ -87,7 +87,7 @@ A scenario can declare which regulatory or compliance control it verifies with a ```gherkin @connect -Feature: Part 11 flavoured controls +Feature: 21 CFR Part 11 flavoured controls @control-audit-trail-publish Scenario: Publishing content is recorded with an actor and a timestamp diff --git a/examples/21CFR_part11_validation/conftest.py b/examples/21CFR_part11_validation/conftest.py index e27b32fa..23f5135a 100644 --- a/examples/21CFR_part11_validation/conftest.py +++ b/examples/21CFR_part11_validation/conftest.py @@ -1,4 +1,4 @@ -"""Override points for the Part 11 example. +"""Override points for the 21 CFR Part 11 example. Redefine these fixtures in your own conftest.py to point the scenarios at the endpoints, repositories and snapshot dates your deployment exposes. diff --git a/examples/21CFR_part11_validation/test_21CFR_part11_connect.feature b/examples/21CFR_part11_validation/test_21CFR_part11_connect.feature index 8a25cc8f..5f968014 100644 --- a/examples/21CFR_part11_validation/test_21CFR_part11_connect.feature +++ b/examples/21CFR_part11_validation/test_21CFR_part11_connect.feature @@ -1,5 +1,5 @@ @connect -Feature: Part 11 flavoured controls +Feature: 21 CFR Part 11 flavoured controls As a validation lead in a regulated environment I want automated evidence for the controls that can be automated So that my traceability matrix is generated rather than hand-maintained diff --git a/examples/21CFR_part11_validation/test_21CFR_part11_connect.py b/examples/21CFR_part11_validation/test_21CFR_part11_connect.py index f93408d8..548f5556 100644 --- a/examples/21CFR_part11_validation/test_21CFR_part11_connect.py +++ b/examples/21CFR_part11_validation/test_21CFR_part11_connect.py @@ -1,4 +1,4 @@ -"""Step definitions for the Part 11 example's Connect scenarios. +"""Step definitions for the 21 CFR Part 11 example's Connect scenarios. Every @scenario function carries a literal @pytest.mark.connect decorator: feature-level Gherkin tags alone do not drive VIP's auto-skip in extension diff --git a/examples/21CFR_part11_validation/test_21CFR_part11_packagemanager.feature b/examples/21CFR_part11_validation/test_21CFR_part11_packagemanager.feature index 63897dc1..be35b625 100644 --- a/examples/21CFR_part11_validation/test_21CFR_part11_packagemanager.feature +++ b/examples/21CFR_part11_validation/test_21CFR_part11_packagemanager.feature @@ -1,5 +1,5 @@ @package_manager -Feature: Part 11 flavoured controls for Package Manager +Feature: 21 CFR Part 11 flavoured controls for Package Manager As a validation lead in a regulated environment I want evidence that analyses draw on a controlled, reconstructable package set So that a record can be traced back to the software that produced it diff --git a/examples/21CFR_part11_validation/test_21CFR_part11_packagemanager.py b/examples/21CFR_part11_validation/test_21CFR_part11_packagemanager.py index a268c9e3..6c7a6522 100644 --- a/examples/21CFR_part11_validation/test_21CFR_part11_packagemanager.py +++ b/examples/21CFR_part11_validation/test_21CFR_part11_packagemanager.py @@ -1,4 +1,5 @@ -"""Step definitions for the Part 11 example's Package Manager scenarios. +"""Step definitions for the 21 CFR Part 11 example's Package Manager +scenarios. Every @scenario function carries a literal @pytest.mark.package_manager decorator: feature-level Gherkin tags alone do not drive VIP's auto-skip in diff --git a/examples/21CFR_part11_validation/test_21CFR_part11_workbench.feature b/examples/21CFR_part11_validation/test_21CFR_part11_workbench.feature index 9a2cb64d..e36fd859 100644 --- a/examples/21CFR_part11_validation/test_21CFR_part11_workbench.feature +++ b/examples/21CFR_part11_validation/test_21CFR_part11_workbench.feature @@ -1,5 +1,5 @@ @workbench -Feature: Part 11 flavoured controls for Workbench +Feature: 21 CFR Part 11 flavoured controls for Workbench As a validation lead in a regulated environment I want evidence that interactive analysis is limited to authorised individuals So that the work behind a record can be attributed to an account diff --git a/examples/21CFR_part11_validation/test_21CFR_part11_workbench.py b/examples/21CFR_part11_validation/test_21CFR_part11_workbench.py index ce90002d..75f13c5e 100644 --- a/examples/21CFR_part11_validation/test_21CFR_part11_workbench.py +++ b/examples/21CFR_part11_validation/test_21CFR_part11_workbench.py @@ -1,4 +1,4 @@ -"""Step definitions for the Part 11 example's Workbench scenarios. +"""Step definitions for the 21 CFR Part 11 example's Workbench scenarios. The two scenarios are one control read from both sides. Refusing an unauthenticated caller is not on its own evidence that access is limited to diff --git a/website/src/pages/getting-started.astro b/website/src/pages/getting-started.astro index 54591d20..820b5cf9 100644 --- a/website/src/pages/getting-started.astro +++ b/website/src/pages/getting-started.astro @@ -478,7 +478,7 @@ quarto publish connect --server https://connect.example.com @control-<slug> Gherkin tag: 

@connect
-Feature: Part 11 flavoured controls
+Feature: 21 CFR Part 11 flavoured controls
 
   @control-audit-trail-publish
   Scenario: Publishing content is recorded with an actor and a timestamp

From 513d776e81a5f2d9d3b17f70ef721401c28862cd Mon Sep 17 00:00:00 2001
From: Brian Deitte 
Date: Sat, 29 Aug 2026 20:38:49 -0500
Subject: [PATCH 092/106] feat(report): record who performed a run and show
 provenance and risk

Three gaps found reviewing this branch against FDA's Computer Software
Assurance guidance, which lists what the record of an assurance activity
should contain.

Nothing recorded who performed the testing. attribution.py identified a
machine and a commit, which answers "which execution" but not "which person
is accountable". VIP_PERFORMED_BY names an operator; without it VIP falls
back to the CI actor, then the local login, and tags the value with which
source it came from so an auditor reading a service-account name knows a
human did not type it. --vip-no-attribution remains the opt-out.

The execution block never reached either report edition. It has been written
into results.json since attribution landed, but only vip trace --format json
rendered it, so the PDF a customer archives and hands to an auditor was
anonymous. Extending report_content.provenance_rows covers both backends at
once. An absent block omits the rows rather than showing five "not recorded"
lines, and a dirty tree is flagged next to the commit it cannot be
reproduced from.

The customer's risk rating reached the CSV and JSON exports but not the
rendered matrix, so the artifact an auditor sees read as a flat checklist
under a framework that is explicitly risk-based. It renders as a subline
under the control id, like reference, rather than as a fifth column the
Typst table has no width for.

VALIDATION-PACKAGE.md now states where VIP sits against the CSA record
checklist -- four of five items, with review and approval the customer's by
design -- and points at actions/attest-build-provenance as the honest
upgrade path from the sha256 sidecar.
---
 AGENTS.md                                     |  2 +-
 .../VALIDATION-PACKAGE.md                     | 54 +++++++++++++-
 selftests/test_attribution.py                 | 49 +++++++++++-
 selftests/test_report_content.py              | 74 +++++++++++++++++++
 selftests/test_report_traceability.py         | 48 ++++++++++++
 src/vip/attribution.py                        | 42 +++++++++++
 src/vip/report_content.py                     | 55 +++++++++++++-
 src/vip/report_html.py                        | 10 ++-
 src/vip/report_typst.py                       |  2 +
 9 files changed, 328 insertions(+), 8 deletions(-)

diff --git a/AGENTS.md b/AGENTS.md
index b8014bf4..8322eb7f 100644
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -158,7 +158,7 @@ Key principles:
 | `src/vip/report_content.py` | Format-neutral report content shared by both rendering backends: titles, outcome/badge styling (colors drift-guarded against `styles.css` by `selftests/test_report_content.py`), grouping, skip wording, provenance rows |
 | `src/vip/report_html.py` | HTML backend: renders `report_content` into the fragments `index.qmd`/`details.qmd` display |
 | `src/vip/report_typst.py` | Typst backend: renders the same content as Typst markup for `report/vip-report.qmd` → `_output/vip-report.pdf`; every dynamic value passes through `_lit` (Typst-injection escaping) |
-| `src/vip/attribution.py` | Collects the `execution` block written into `results.json` — hostname, git (commit/branch/dirty/remote, with any userinfo redacted from the remote URL), and CI (provider/run_id/run_url/job for GitHub Actions, GitLab CI, Jenkins). Every probe degrades to `None` rather than failing or warning; omitted entirely with `--vip-no-attribution` |
+| `src/vip/attribution.py` | Collects the `execution` block written into `results.json` — hostname, git (commit/branch/dirty/remote, with any userinfo redacted from the remote URL), CI (provider/run_id/run_url/job for GitHub Actions, GitLab CI, Jenkins), and `performed_by` (`VIP_PERFORMED_BY` → CI actor → local login, each tagged with its `source`; FDA's Computer Software Assurance guidance asks the record to say who performed the testing). Every probe degrades to `None` rather than failing or warning; omitted entirely with `--vip-no-attribution`, which is also the opt-out for writing an operator identity into an archived artifact. `report_content.provenance_rows` renders the whole block into both report editions — before that it reached only `vip trace --format json` |
 | `src/vip/traceability.py` | Control list model (`ControlSpec`, loaded from `controls.toml`) and `build_traceability_matrix`, which joins controls against `@control-`-tagged scenarios in a loaded `results.json`; CSV (`render_csv`, apostrophe-neutralizes formula-leading cells) and JSON (`render_json`) renderers; `verify_results_checksum` (per-line sidecar parse, case-insensitive digest, matched by recorded filename) and the schema-version gate (`check_results_schema` hard-errors on an unknown major, unlike `reporting.load_results`, which only warns). `ControlEntry.executed` / `TraceabilityMatrix.covered_without_execution` separate "a scenario is tagged" from "a scenario ran" — a scenario that runs and skips itself (absent endpoint, no data, version gate) still carries its control tag, so coverage alone would report a green matrix for a run that verified nothing. `ControlEntry.failing` and `TraceabilityMatrix.covered_with_failure` are the third fact, separating "a scenario ran" from "a scenario passed", and any failing scenario demotes the control's badge rather than only an all-failed one. An *unconfigured* product is the opposite case and is often misdescribed: `plugin.pytest_collection_modifyitems` deselects those scenarios rather than skipping them, so they never reach `results.json` and a control tagged only by them reports as a `gap`, not as covered-not-executed. Powers `vip trace` |
 | `src/vip/clients/connect.py` | httpx client for Connect API |
 | `src/vip/clients/workbench.py` | httpx client for Workbench API; `quit_vip_sessions` warns loudly (not silently) when a VIP session persists after all retries. `session_owner` / `is_vip_session_for_owner` decide whether a VIP session belongs to the sweeping worker — see "Session ownership" below |
diff --git a/examples/21CFR_part11_validation/VALIDATION-PACKAGE.md b/examples/21CFR_part11_validation/VALIDATION-PACKAGE.md
index a618d4a6..db9ee0ca 100644
--- a/examples/21CFR_part11_validation/VALIDATION-PACKAGE.md
+++ b/examples/21CFR_part11_validation/VALIDATION-PACKAGE.md
@@ -34,10 +34,17 @@ and the archivable PDF. The join is the deliverable: it is what lets a reviewer
 go from a control to the check that evidences it without a spreadsheet
 maintained by hand.
 
-Execution provenance. Each results file records which host ran the tests, which
-git commit and branch they came from, whether that tree was dirty, and which CI
-job produced them. This is what makes a result attributable to a pipeline
-execution rather than to an anonymous green tick.
+Execution provenance. Each results file records who ran the tests, which host
+they ran on, which git commit and branch they came from, whether that tree was
+dirty, and which CI job produced them. The same block is rendered into the HTML
+report and the PDF, so the archived artifact carries it rather than only the
+machine-readable output. This is what makes a result attributable to a named
+operator and a pipeline execution rather than to an anonymous green tick.
+
+Set `VIP_PERFORMED_BY` to record the person accountable for the run. Without
+it, VIP falls back to the CI system's actor, then to the local login, and
+labels which one it used. `--vip-no-attribution` omits the whole block for
+anyone who does not want an operator identity written into an archived file.
 
 Tamper-evidence. A `results.json.sha256` sidecar detects corruption in transit,
 a truncated upload, or a file edited after the fact and not re-checksummed.
@@ -48,6 +55,37 @@ results file can regenerate the sidecar to match. It catches accidents, which
 is the failure that actually happens to archived CI artifacts. It does not
 resist a motivated forger, and it must never be presented as though it does.
 
+## Where this sits in FDA's current thinking
+
+FDA finalised [Computer Software Assurance for Production and Quality
+Management System
+Software](https://www.fda.gov/regulatory-information/search-fda-guidance-documents/computer-software-assurance-production-and-quality-management-system-software)
+in September 2025 and updated it in February 2026. Two of its positions matter
+for anyone deciding what VIP is worth.
+
+The guidance recommends "incorporating the use of digital records, such as
+system logs, audit trails, and other data generated and maintained by the
+software, as opposed to paper documentation, screenshots, or duplicating
+results already digitally retained." A machine-generated results file with
+per-scenario timestamps and execution provenance is the artifact that sentence
+describes. Screenshots pasted into a Word protocol are what it discourages.
+
+The guidance also lists what the record of an assurance activity should
+contain. VIP supplies four of the five: a description of the testing and its
+results, the record of who performed it and when, the result of your
+risk-based analysis (carried through from `controls.toml` and rendered in the
+matrix), and the intended use of what was tested, insofar as your control
+descriptions state it. The fifth is the established review and approval, which
+is yours by design and covered under "What you author" below. Two further
+items are only partly covered: VIP records issues found as failed and skipped
+scenarios, but keeps no resolution or disposition against them, and it writes
+no conclusion statement declaring acceptability. Declaring acceptability is a
+judgement, not a test result.
+
+None of this makes a VIP run a computer software assurance activity on its
+own. The guidance is risk-based, and the risk analysis that decides how much
+assurance a function needs is yours.
+
 ## What you author
 
 These are yours. No test tool produces them, because each one describes your
@@ -112,6 +150,14 @@ Two smaller absences worth stating: there is no cross-run deviation log, which
 would need history VIP does not keep, and there is no cryptographic signing of
 results.
 
+If you need results to resist a motivated forger rather than only to detect
+corruption, sign them in your pipeline rather than waiting for VIP to grow its
+own crypto. On GitHub Actions, `actions/attest-build-provenance` produces a
+Sigstore-backed attestation over `results.json` that is recorded in a public
+transparency log, and `gh attestation verify` checks it later. That gives you
+what the sha256 sidecar deliberately does not claim: a signature the producer
+cannot quietly regenerate.
+
 ## Reading a green report
 
 A fully green matrix evidences the controls you chose to automate, on the
diff --git a/selftests/test_attribution.py b/selftests/test_attribution.py
index 00d1605f..9c43a3ad 100644
--- a/selftests/test_attribution.py
+++ b/selftests/test_attribution.py
@@ -5,7 +5,12 @@
 
 import pytest
 
-from vip.attribution import _git, collect_execution_metadata, redact_userinfo
+from vip.attribution import (
+    _git,
+    _performed_by,
+    collect_execution_metadata,
+    redact_userinfo,
+)
 
 
 def _init_repo(path):
@@ -202,3 +207,45 @@ def test_credentialless_url_is_returned_unchanged(self):
         assert redact_userinfo("https://github.com/org/repo.git") == (
             "https://github.com/org/repo.git"
         )
+
+
+class TestPerformedBy:
+    """Who ran the verification.
+
+    FDA's Computer Software Assurance guidance asks the record of an assurance
+    activity to carry who performed the testing alongside the date. Every other
+    field this module collects identifies a machine or a commit.
+    """
+
+    def test_explicit_override_wins_over_every_ci_actor(self):
+        env = {"VIP_PERFORMED_BY": "QA Lead", "GITHUB_ACTOR": "octocat"}
+        assert _performed_by(env) == {"identity": "QA Lead", "source": "explicit"}
+
+    def test_whitespace_only_override_falls_through(self):
+        """An empty variable exported by a shell must not shadow the real actor."""
+        env = {"VIP_PERFORMED_BY": "   ", "GITHUB_ACTOR": "octocat"}
+        assert _performed_by(env) == {"identity": "octocat", "source": "github"}
+
+    @pytest.mark.parametrize(
+        ("var", "source"),
+        [("GITHUB_ACTOR", "github"), ("GITLAB_USER_LOGIN", "gitlab"), ("BUILD_USER_ID", "jenkins")],
+    )
+    def test_each_ci_actor_is_recognized_with_its_source(self, var, source):
+        assert _performed_by({var: "runner"}) == {"identity": "runner", "source": source}
+
+    def test_local_login_is_the_last_resort_and_says_so(self, monkeypatch):
+        monkeypatch.setattr("vip.attribution.getpass.getuser", lambda: "bdeitte")
+        assert _performed_by({}) == {"identity": "bdeitte", "source": "login"}
+
+    def test_an_unresolvable_login_degrades_to_none(self, monkeypatch):
+        """No passwd entry and no LOGNAME/USER set -- a bare container."""
+
+        def boom():
+            raise KeyError("uid not found")
+
+        monkeypatch.setattr("vip.attribution.getpass.getuser", boom)
+        assert _performed_by({}) is None
+
+    def test_collect_execution_metadata_carries_the_performer(self, tmp_path):
+        meta = collect_execution_metadata(cwd=tmp_path, env={"VIP_PERFORMED_BY": "QA Lead"})
+        assert meta["performed_by"] == {"identity": "QA Lead", "source": "explicit"}
diff --git a/selftests/test_report_content.py b/selftests/test_report_content.py
index e9202a6b..6ca3d950 100644
--- a/selftests/test_report_content.py
+++ b/selftests/test_report_content.py
@@ -407,3 +407,77 @@ def test_render_failure_message_names_the_error(self):
         msg = report_content.TRACEABILITY_RENDER_FAILURE.format(error="boom")
         assert "boom" in msg
         assert "traceability" in msg.lower()
+
+
+class TestExecutionProvenanceRows:
+    """Attribution reaches the artifact the customer archives.
+
+    ``results.json`` has recorded the execution block since attribution
+    landed, but only ``vip trace --format json`` rendered it. A result that is
+    attributable in the machine-readable output and anonymous in the PDF is
+    attributable in the wrong place, because the PDF is what goes to an
+    auditor.
+    """
+
+    EXECUTION = {
+        "hostname": "runner-07",
+        "git": {
+            "commit": "a1b2c3d4e5f6",
+            "branch": "main",
+            "dirty": False,
+            "remote": "https://github.com/posit-dev/vip.git",
+        },
+        "ci": {"provider": "github", "run_url": "https://github.com/o/r/actions/runs/5"},
+        "performed_by": {"identity": "octocat", "source": "github"},
+    }
+
+    @staticmethod
+    def _rows(execution):
+        from vip.reporting import ReportData
+
+        return dict(report_content.provenance_rows(ReportData(results=[], execution=execution)))
+
+    def test_every_execution_field_reaches_the_report(self):
+        rows = self._rows(self.EXECUTION)
+        assert rows["Performed by"] == "octocat"
+        assert rows["Run host"] == "runner-07"
+        assert rows["Commit"] == "a1b2c3d4e5f6"
+        assert rows["Branch"] == "main"
+        assert rows["CI run"] == "https://github.com/o/r/actions/runs/5"
+
+    def test_an_absent_execution_block_omits_the_rows_entirely(self):
+        """``--vip-no-attribution`` asked for this; five "not recorded" rows
+        would read as a broken run rather than a deliberate one."""
+        rows = self._rows(None)
+        for label in ("Performed by", "Run host", "Commit", "Branch", "CI run"):
+            assert label not in rows
+
+    def test_the_pre_attribution_rows_still_render_without_an_execution_block(self):
+        assert "Exit status" in self._rows(None)
+
+    def test_a_dirty_tree_is_flagged_next_to_the_commit(self):
+        """Evidence from an uncommitted tree cannot be reproduced from the
+        commit alone, so the caveat belongs in the same cell."""
+        execution = {**self.EXECUTION, "git": {**self.EXECUTION["git"], "dirty": True}}
+        assert self._rows(execution)["Commit"] == "a1b2c3d4e5f6 (uncommitted changes present)"
+
+    def test_a_local_login_is_labelled_as_weaker_than_a_named_operator(self):
+        execution = {**self.EXECUTION, "performed_by": {"identity": "bd", "source": "login"}}
+        assert self._rows(execution)["Performed by"] == "bd (local login)"
+
+    def test_an_explicit_operator_is_not_labelled(self):
+        performer = {"identity": "QA Lead", "source": "explicit"}
+        execution = {**self.EXECUTION, "performed_by": performer}
+        assert self._rows(execution)["Performed by"] == "QA Lead"
+
+    def test_a_missing_field_inside_a_present_block_follows_the_none_contract(self):
+        """Present-but-partial is different from absent: the row stays, and the
+        backend renders NOT_RECORDED rather than a fabricated value."""
+        rows = self._rows({"hostname": "runner-07"})
+        assert rows["Run host"] == "runner-07"
+        assert rows["Performed by"] is None
+        assert rows["Commit"] is None
+
+    def test_a_ci_run_without_a_url_falls_back_to_the_run_id(self):
+        execution = {**self.EXECUTION, "ci": {"provider": "gitlab", "run_id": "4412"}}
+        assert self._rows(execution)["CI run"] == "4412"
diff --git a/selftests/test_report_traceability.py b/selftests/test_report_traceability.py
index 969cedfc..998793ba 100644
--- a/selftests/test_report_traceability.py
+++ b/selftests/test_report_traceability.py
@@ -198,3 +198,51 @@ def test_html_coverage_badge_uses_a_class_that_exists_in_styles_css(self):
                 f"{cls} is referenced by the renderer but absent from styles.css"
             )
         assert "class='badge'" not in html
+
+
+class TestRiskIsVisibleInBothEditions:
+    """The customer's risk rating, rendered rather than left in the CSV export.
+
+    `risk` reached `vip trace --format csv/json` from the day the control
+    loader carried it, but neither report edition showed it -- and the report
+    is what an auditor is handed. CSA is a risk-based framework, so a matrix
+    with no risk column reads as a flat checklist.
+    """
+
+    @staticmethod
+    def _matrix():
+        data = ReportData(results=[_result("t.py::ok", "ok", title="Audit trail is written")])
+        controls = {
+            "ok": ControlSpec(
+                "ok", "Audit trail recorded", reference="21 CFR 11.10(e)", risk="high"
+            ),
+            "unrated": ControlSpec("unrated", "No risk assigned"),
+        }
+        return build_traceability_matrix(data, controls)
+
+    def test_control_row_carries_the_risk_verbatim(self):
+        by_id = {r.control_id: r.risk for r in control_rows(self._matrix())}
+        assert by_id["ok"] == "high"
+        assert by_id["unrated"] == ""
+
+    def test_html_edition_shows_the_risk(self):
+        html = report_html.render_traceability(self._matrix())
+        assert "risk: high" in html
+
+    def test_typst_edition_shows_the_risk(self):
+        typst = report_typst.render_traceability(self._matrix())
+        assert "risk: high" in typst
+
+    @pytest.mark.parametrize("backend", [report_html, report_typst])
+    def test_an_unrated_control_renders_no_empty_risk_line(self, backend):
+        """An absent rating must not become a dangling "risk: " subline."""
+        rendered = backend.render_traceability(self._matrix())
+        assert rendered.count("risk: ") == 1
+
+    def test_a_risk_value_vip_does_not_recognize_is_still_rendered(self):
+        """VIP carries the rating through uninterpreted; it does not rank it."""
+        data = ReportData(results=[_result("t.py::ok", "ok")])
+        controls = {"ok": ControlSpec("ok", "Some control", risk="Class II / banana")}
+        matrix = build_traceability_matrix(data, controls)
+        assert "Class II / banana" in report_html.render_traceability(matrix)
+        assert "Class II / banana" in report_typst.render_traceability(matrix)
diff --git a/src/vip/attribution.py b/src/vip/attribution.py
index 23b970d5..72ae8f29 100644
--- a/src/vip/attribution.py
+++ b/src/vip/attribution.py
@@ -10,6 +10,7 @@
 
 from __future__ import annotations
 
+import getpass
 import os
 import platform
 import subprocess
@@ -150,6 +151,42 @@ def _ci_metadata(env: Mapping[str, str]) -> dict[str, Any] | None:
     return None
 
 
+def _performed_by(env: Mapping[str, str]) -> dict[str, Any] | None:
+    """Who ran this verification.
+
+    FDA's Computer Software Assurance guidance asks for a record of who
+    performed the testing alongside the date. Every other field in this module
+    identifies a *machine* or a *commit*, which answers "which execution" but
+    not "which person is accountable for it".
+
+    Resolution order, most deliberate first: ``VIP_PERFORMED_BY`` names the
+    person on whose behalf the run happens, which is what a QA engineer
+    kicking off a scheduled pipeline needs to record; then the CI system's own
+    actor; then the local login. ``source`` travels with the value because an
+    auditor reading "svc-vip-runner" needs to know whether a human typed that
+    or a service account inherited it.
+    """
+    explicit = (env.get("VIP_PERFORMED_BY") or "").strip()
+    if explicit:
+        return {"identity": explicit, "source": "explicit"}
+    for var, source in (
+        ("GITHUB_ACTOR", "github"),
+        ("GITLAB_USER_LOGIN", "gitlab"),
+        ("BUILD_USER_ID", "jenkins"),
+    ):
+        value = (env.get(var) or "").strip()
+        if value:
+            return {"identity": value, "source": source}
+    try:
+        login = getpass.getuser().strip()
+    except Exception:
+        # getpass.getuser() raises on a container with no passwd entry and no
+        # LOGNAME/USER/LNAME/USERNAME set. Same never-fail contract as the rest
+        # of this module.
+        return None
+    return {"identity": login, "source": "login"} if login else None
+
+
 def collect_execution_metadata(
     *, cwd: Path | None = None, env: Mapping[str, str] | None = None
 ) -> dict[str, Any]:
@@ -157,6 +194,10 @@ def collect_execution_metadata(
 
     ``hostname`` is the VIP runner's host, not the system under test. Anything
     rendering it must label it that way.
+
+    ``performed_by`` records an operator identity, so the whole block is what
+    ``--vip-no-attribution`` exists to omit for anyone who does not want that
+    written into an archived artifact.
     """
     resolved_env = os.environ if env is None else env
     if cwd is None:
@@ -176,4 +217,5 @@ def collect_execution_metadata(
         # git rather than hand subprocess a None it would reject.
         "git": _git_metadata(resolved_cwd, resolved_env) if resolved_cwd is not None else None,
         "ci": _ci_metadata(resolved_env),
+        "performed_by": _performed_by(resolved_env),
     }
diff --git a/src/vip/report_content.py b/src/vip/report_content.py
index fac19b1e..79c934fb 100644
--- a/src/vip/report_content.py
+++ b/src/vip/report_content.py
@@ -364,8 +364,51 @@ def skip_reason_parts(item: TestResult) -> tuple[str, str]:
 NOT_RECORDED = "not recorded"
 
 
+def _execution_rows(execution: dict | None) -> list[tuple[str, str | None]]:
+    """Who ran this, on which host, from which commit, under which CI job.
+
+    ``results.json`` has recorded this block since attribution landed, but
+    until now only ``vip trace --format json`` rendered it. The report is the
+    artifact a customer archives and hands to an auditor, so a result that is
+    attributable in the machine-readable output and anonymous in the PDF is
+    attributable in the wrong place.
+
+    The whole block is omitted rather than shown as five ``NOT_RECORDED`` rows
+    when ``execution`` is absent: that is what ``--vip-no-attribution`` asked
+    for, and repeating "not recorded" five times reads as a broken run rather
+    than a deliberate one. Within a present block, an individual missing field
+    still follows the ``None`` contract above.
+    """
+    if not execution:
+        return []
+    git = execution.get("git") or {}
+    ci = execution.get("ci") or {}
+    performer = execution.get("performed_by") or {}
+
+    commit = git.get("commit")
+    if commit and git.get("dirty"):
+        # An uncommitted tree means the evidence cannot be reproduced from the
+        # commit alone. That belongs next to the commit, not in a footnote.
+        commit = f"{commit} (uncommitted changes present)"
+
+    identity = performer.get("identity")
+    if identity and performer.get("source") == "login":
+        # A local login is who was at the keyboard, which is weaker than a
+        # named operator or a CI actor. Say which one the reader is looking at.
+        identity = f"{identity} (local login)"
+
+    return [
+        ("Performed by", identity),
+        ("Run host", execution.get("hostname")),
+        ("Commit", commit),
+        ("Branch", git.get("branch")),
+        ("CI run", ci.get("run_url") or ci.get("run_id")),
+    ]
+
+
 def provenance_rows(data: ReportData) -> list[tuple[str, str | None]]:
-    """VIP version, run duration, interpreter/platform, mode, exit status (F9).
+    """VIP version, run duration, interpreter/platform, mode, exit status (F9),
+    then the execution attribution block when the run recorded one.
 
     A ``None`` value means the field is absent from this ``results.json`` and
     the backend must render ``NOT_RECORDED`` rather than a fabricated value.
@@ -385,6 +428,7 @@ def provenance_rows(data: ReportData) -> list[tuple[str, str | None]]:
         ("Platform", data.platform),
         ("Mode", mode),
         ("Exit status", f"{data.exit_status} ({exit_label})"),
+        *_execution_rows(data.execution),
     ]
 
 
@@ -429,6 +473,14 @@ class ControlRow:
     control_id: str
     description: str
     reference: str
+    risk: str
+    """The customer's own risk rating, carried through uninterpreted.
+
+    Rendered because FDA's Computer Software Assurance guidance asks the
+    record to carry the result of the risk-based analysis, and because a
+    reviewer triaging a matrix reads the high-risk gaps first. VIP does not
+    rank or validate the value -- ``risk = "banana"`` renders as "banana".
+    """
     coverage: str
     """"covered" | "covered_not_executed" | "covered_failed" | "gap" |
     "not_automatable".
@@ -469,6 +521,7 @@ def control_rows(matrix) -> list[ControlRow]:  # noqa: ANN001 - TraceabilityMatr
                 control_id=entry.control.control_id,
                 description=entry.control.description,
                 reference=entry.control.reference or "",
+                risk=entry.control.risk or "",
                 coverage=display_coverage(entry),
                 scenarios=scenarios,
             )
diff --git a/src/vip/report_html.py b/src/vip/report_html.py
index 4979bfd2..9ef240a7 100644
--- a/src/vip/report_html.py
+++ b/src/vip/report_html.py
@@ -473,7 +473,15 @@ def render_traceability(matrix) -> str:  # noqa: ANN001 - vip.traceability.Trace
             )
         else:
             evidence = "no tagged scenario"
-        reference = f"
{_esc(row.reference)}" if row.reference else ""
+        # Reference and risk sit under the control id as sublines rather than
+        # as their own columns: the Typst edition's table is already at its
+        # width budget with four columns, and the two editions must match.
+        sublines = "".join(
+            f"
{_esc(text)}"
+            for text in (row.reference, f"risk: {row.risk}" if row.risk else "")
+            if text
+        )
+        reference = sublines
         rows.append(
             f"{_esc(row.control_id)}{reference}"
             f"{_esc(row.description)}{badge}{evidence}"
diff --git a/src/vip/report_typst.py b/src/vip/report_typst.py
index fd495c03..0741d5b4 100644
--- a/src/vip/report_typst.py
+++ b/src/vip/report_typst.py
@@ -580,6 +580,8 @@ def render_traceability(matrix) -> str:  # noqa: ANN001 - TraceabilityMatrix
         control_parts = [_text(row.control_id, size="9pt")]
         if row.reference:
             control_parts.append(_text(row.reference, size="8pt", fill='rgb("#6b7280")'))
+        if row.risk:
+            control_parts.append(_text(f"risk: {row.risk}", size="8pt", fill='rgb("#6b7280")'))
         if row.scenarios:
             evidence = _stacked(
                 [_text(f"{t} - {s} at {w}", size="8.5pt") for t, s, w in row.scenarios]

From 008ff8f461a92b303e29239e338380481d879587 Mon Sep 17 00:00:00 2001
From: Brian Deitte 
Date: Sat, 29 Aug 2026 20:41:56 -0500
Subject: [PATCH 093/106] docs(examples): quote the CSA record requirements
 from the guidance itself

---
 .../VALIDATION-PACKAGE.md                     | 53 ++++++++++++-------
 1 file changed, 33 insertions(+), 20 deletions(-)

diff --git a/examples/21CFR_part11_validation/VALIDATION-PACKAGE.md b/examples/21CFR_part11_validation/VALIDATION-PACKAGE.md
index db9ee0ca..b96519c4 100644
--- a/examples/21CFR_part11_validation/VALIDATION-PACKAGE.md
+++ b/examples/21CFR_part11_validation/VALIDATION-PACKAGE.md
@@ -60,30 +60,43 @@ resist a motivated forger, and it must never be presented as though it does.
 FDA finalised [Computer Software Assurance for Production and Quality
 Management System
 Software](https://www.fda.gov/regulatory-information/search-fda-guidance-documents/computer-software-assurance-production-and-quality-management-system-software)
-in September 2025 and updated it in February 2026. Two of its positions matter
-for anyone deciding what VIP is worth.
-
-The guidance recommends "incorporating the use of digital records, such as
+in September 2025 and updated it in February 2026. Two passages in section
+V.A.6, "Establishing the Appropriate Record", matter for anyone deciding what
+VIP is worth.
+
+The first describes what VIP produces almost by name. Advances in digital
+technology, the guidance says, "may allow for manufacturers to leverage
+digital retention of results, automated traceability, automated testing, and
+electronic capture of work performed as objective evidence, reducing the need
+for manual or paper-based documentation." It then recommends, as the
+least-burdensome approach, "incorporating the use of digital records, such as
 system logs, audit trails, and other data generated and maintained by the
 software, as opposed to paper documentation, screenshots, or duplicating
-results already digitally retained." A machine-generated results file with
-per-scenario timestamps and execution provenance is the artifact that sentence
-describes. Screenshots pasted into a Word protocol are what it discourages.
-
-The guidance also lists what the record of an assurance activity should
-contain. VIP supplies four of the five: a description of the testing and its
-results, the record of who performed it and when, the result of your
-risk-based analysis (carried through from `controls.toml` and rendered in the
-matrix), and the intended use of what was tested, insofar as your control
-descriptions state it. The fifth is the established review and approval, which
-is yours by design and covered under "What you author" below. Two further
-items are only partly covered: VIP records issues found as failed and skipped
-scenarios, but keeps no resolution or disposition against them, and it writes
-no conclusion statement declaring acceptability. Declaring acceptability is a
-judgement, not a test result.
+results already digitally retained by the software when establishing the
+record associated with the assurance activities." A machine-generated results
+file with per-scenario timestamps and execution provenance is the artifact
+those sentences describe. Screenshots pasted into a Word protocol are what
+they discourage.
+
+The second is the list of what the record should include. VIP covers, in whole
+or in part:
+
+- the intended use of the feature or function, insofar as your control
+  descriptions state it
+- the result of your risk-based analysis, carried through from `controls.toml`
+  and rendered in the matrix
+- a description of the testing conducted, as the tagged scenario and its steps
+- issues found during testing, as failed and skipped scenarios
+- who performed the testing and the date it was performed
+
+Two items are yours. VIP writes no conclusion statement declaring
+acceptability, and it carries no resolution against the issues it records:
+both are judgements, not test results. Neither is there any review and
+approval signature, which the guidance asks for "when appropriate" and which
+belongs in your quality system.
 
 None of this makes a VIP run a computer software assurance activity on its
-own. The guidance is risk-based, and the risk analysis that decides how much
+own. The approach is risk-based, and the risk analysis that decides how much
 assurance a function needs is yours.
 
 ## What you author

From c06a78f92df9241b087e2338256697c5d2cd2152 Mon Sep 17 00:00:00 2001
From: Brian Deitte 
Date: Sat, 29 Aug 2026 20:42:39 -0500
Subject: [PATCH 094/106] docs: document performed_by and VIP_PERFORMED_BY
 where the other fields live

---
 docs/reporting.md                       | 27 +++++++++++++++++++------
 website/src/pages/getting-started.astro |  4 ++++
 2 files changed, 25 insertions(+), 6 deletions(-)

diff --git a/docs/reporting.md b/docs/reporting.md
index 0497742c..e0ddac4a 100644
--- a/docs/reporting.md
+++ b/docs/reporting.md
@@ -55,7 +55,8 @@ preset that turns on `json,junit,sarif` together with concise tracebacks.
   "execution": {
     "hostname": "ci-runner-01",
     "git": { "commit": "...", "branch": "...", "dirty": false, "remote": "https://github.com/org/repo" },
-    "ci": { "provider": "github", "run_id": "...", "run_url": "...", "job": "verify" }
+    "ci": { "provider": "github", "run_id": "...", "run_url": "...", "job": "verify" },
+    "performed_by": { "identity": "octocat", "source": "github" }
   }
 }
 ```
@@ -69,11 +70,25 @@ preset that turns on `json,junit,sarif` together with concise tracebacks.
   start). `None` for a `results.json` written before these fields existed.
 - `execution` -- attribution for the run that produced this evidence: which host ran
   it, which git commit/branch it ran from (dirty flag, remote with any credential
-  stripped out of the URL), and which CI job (GitHub Actions, GitLab CI, or Jenkins)
-  ran it, if any. To omit this block entirely, pass the pytest-level option after
-  `--`: `vip verify --config vip.toml -- --vip-no-attribution`. There is no
-  `vip verify` flag of its own for this. Useful if a deployment's policy is not to
-  record hostnames or CI identifiers in an archived artifact.
+  stripped out of the URL), which CI job (GitHub Actions, GitLab CI, or Jenkins)
+  ran it, if any, and who performed it. To omit this block entirely, pass the
+  pytest-level option after `--`: `vip verify --config vip.toml --
+  --vip-no-attribution`. There is no `vip verify` flag of its own for this. Useful
+  if a deployment's policy is not to record hostnames, CI identifiers or an
+  operator identity in an archived artifact.
+- `execution.performed_by` -- the operator the run is attributable to, and `source`
+  says where that value came from. Set `VIP_PERFORMED_BY` to name the person
+  accountable for the run (`source: "explicit"`); otherwise VIP reads the CI
+  system's own actor (`GITHUB_ACTOR`, `GITLAB_USER_LOGIN`, `BUILD_USER_ID`), then
+  falls back to the local login (`source: "login"`). `source` travels with the
+  value because a reader seeing a service-account name needs to know whether a
+  human typed it. FDA's Computer Software Assurance guidance asks the record of an
+  assurance activity to carry who performed the testing alongside the date, which
+  is why this exists; the rest of the block identifies a machine, not a person.
+
+The whole block is rendered into the HTML report and the PDF as well, so the
+archived artifact carries the attribution rather than only the machine-readable
+output.
 
 Be precise about what `python_version`, `platform`, and `execution.hostname`
 describe: they are properties of the machine that ran `vip verify` -- the VIP
diff --git a/website/src/pages/getting-started.astro b/website/src/pages/getting-started.astro
index 820b5cf9..1d3fafcd 100644
--- a/website/src/pages/getting-started.astro
+++ b/website/src/pages/getting-started.astro
@@ -212,6 +212,10 @@ vip verify --connect-url https://connect.example.com --no-interactive-authVIP_TEST_TOTP_SECRET
                   Base32 TOTP seed for the test service account. Optional; only used by --headless-auth when the IdP issues an MFA challenge.
                 
+                
+                  VIP_PERFORMED_BY
+                  Names the person the run is attributable to, recorded in the results file and shown in both report editions. Optional; VIP otherwise reads the CI system's actor, then the local login. Not a secret.
+                
               
             
           

From 4f9a479516f11536e91f5df2cedb43604ea9e07e Mon Sep 17 00:00:00 2001
From: Brian Deitte 
Date: Sat, 29 Aug 2026 20:51:26 -0500
Subject: [PATCH 095/106] fix(report): qualify every inherited performer
 identity with its source

A CI actor rendered bare read in the archived artifact exactly like an
explicitly named accountable operator, which is the distinction the source
field exists to make -- GITHUB_ACTOR on a scheduled run is whoever last
edited the workflow, and is frequently a service account. Only an explicit
VIP_PERFORMED_BY now renders unqualified. An unrecognized source, or a block
carrying an identity with no source at all, is labelled rather than promoted
to explicit by rendering it bare.

Also narrows two doc sentences that overclaimed: the report renders five
fields from the execution block, not the whole of it.
---
 docs/reporting.md                             | 11 ++++--
 .../VALIDATION-PACKAGE.md                     | 14 ++++---
 selftests/test_report_content.py              | 31 +++++++++++++--
 src/vip/report_content.py                     | 38 ++++++++++++++++---
 4 files changed, 77 insertions(+), 17 deletions(-)

diff --git a/docs/reporting.md b/docs/reporting.md
index e0ddac4a..3eb3f845 100644
--- a/docs/reporting.md
+++ b/docs/reporting.md
@@ -86,9 +86,14 @@ preset that turns on `json,junit,sarif` together with concise tracebacks.
   assurance activity to carry who performed the testing alongside the date, which
   is why this exists; the rest of the block identifies a machine, not a person.
 
-The whole block is rendered into the HTML report and the PDF as well, so the
-archived artifact carries the attribution rather than only the machine-readable
-output.
+Both report editions render the attribution too, so the archived artifact
+carries it rather than only the machine-readable output. They render five of
+these fields -- `performed_by` (qualified by its `source` unless that source is
+`explicit`), `hostname`, `git.commit` (flagged when `dirty`), `git.branch`, and
+`ci.run_url` or `ci.run_id`. `git.remote`, `ci.provider` and `ci.job` stay in
+`results.json` only: the remote and the job name add table width without adding
+much a reviewer can act on, and the provider is already evident from the run
+URL. Read `results.json` itself if you need the full block.
 
 Be precise about what `python_version`, `platform`, and `execution.hostname`
 describe: they are properties of the machine that ran `vip verify` -- the VIP
diff --git a/examples/21CFR_part11_validation/VALIDATION-PACKAGE.md b/examples/21CFR_part11_validation/VALIDATION-PACKAGE.md
index b96519c4..4818d85a 100644
--- a/examples/21CFR_part11_validation/VALIDATION-PACKAGE.md
+++ b/examples/21CFR_part11_validation/VALIDATION-PACKAGE.md
@@ -36,15 +36,19 @@ maintained by hand.
 
 Execution provenance. Each results file records who ran the tests, which host
 they ran on, which git commit and branch they came from, whether that tree was
-dirty, and which CI job produced them. The same block is rendered into the HTML
-report and the PDF, so the archived artifact carries it rather than only the
+dirty, and which CI job produced them. The HTML report and the PDF render the
+attribution as well, so the archived artifact carries it rather than only the
 machine-readable output. This is what makes a result attributable to a named
 operator and a pipeline execution rather than to an anonymous green tick.
 
 Set `VIP_PERFORMED_BY` to record the person accountable for the run. Without
-it, VIP falls back to the CI system's actor, then to the local login, and
-labels which one it used. `--vip-no-attribution` omits the whole block for
-anyone who does not want an operator identity written into an archived file.
+it, VIP falls back to the CI system's actor, then to the local login, and the
+report qualifies the name with where it came from -- `octocat (GitHub actor)`
+rather than a bare `octocat`. An explicitly named operator is the only one
+that renders unqualified, because it is the only one a human chose. A CI actor
+is frequently a service account, and on a scheduled run it is whoever last
+edited the workflow. `--vip-no-attribution` omits the whole block for anyone
+who does not want an operator identity written into an archived file.
 
 Tamper-evidence. A `results.json.sha256` sidecar detects corruption in transit,
 a truncated upload, or a file edited after the fact and not re-checksummed.
diff --git a/selftests/test_report_content.py b/selftests/test_report_content.py
index 6ca3d950..a75fd4d6 100644
--- a/selftests/test_report_content.py
+++ b/selftests/test_report_content.py
@@ -439,7 +439,7 @@ def _rows(execution):
 
     def test_every_execution_field_reaches_the_report(self):
         rows = self._rows(self.EXECUTION)
-        assert rows["Performed by"] == "octocat"
+        assert rows["Performed by"] == "octocat (GitHub actor)"
         assert rows["Run host"] == "runner-07"
         assert rows["Commit"] == "a1b2c3d4e5f6"
         assert rows["Branch"] == "main"
@@ -461,9 +461,32 @@ def test_a_dirty_tree_is_flagged_next_to_the_commit(self):
         execution = {**self.EXECUTION, "git": {**self.EXECUTION["git"], "dirty": True}}
         assert self._rows(execution)["Commit"] == "a1b2c3d4e5f6 (uncommitted changes present)"
 
-    def test_a_local_login_is_labelled_as_weaker_than_a_named_operator(self):
-        execution = {**self.EXECUTION, "performed_by": {"identity": "bd", "source": "login"}}
-        assert self._rows(execution)["Performed by"] == "bd (local login)"
+    @pytest.mark.parametrize(
+        ("source", "expected"),
+        [
+            ("login", "bd (local login)"),
+            ("github", "bd (GitHub actor)"),
+            ("gitlab", "bd (GitLab user)"),
+            ("jenkins", "bd (Jenkins build user)"),
+        ],
+    )
+    def test_every_inherited_identity_says_where_it_came_from(self, source, expected):
+        """A CI actor is often a service account. Unlabelled, it would read in
+        the archived artifact exactly like a named accountable operator."""
+        execution = {**self.EXECUTION, "performed_by": {"identity": "bd", "source": source}}
+        assert self._rows(execution)["Performed by"] == expected
+
+    def test_an_identity_with_no_source_is_never_rendered_bare(self):
+        """A malformed block must not read as an explicitly named operator,
+        nor render the literal string "None"."""
+        execution = {**self.EXECUTION, "performed_by": {"identity": "bd"}}
+        assert self._rows(execution)["Performed by"] == "bd (source not recorded)"
+
+    def test_an_unrecognized_source_is_still_labelled(self):
+        """A source this version does not know about is not an explicitly named
+        operator, and must not be promoted to one by rendering it bare."""
+        execution = {**self.EXECUTION, "performed_by": {"identity": "bd", "source": "buildkite"}}
+        assert self._rows(execution)["Performed by"] == "bd (buildkite)"
 
     def test_an_explicit_operator_is_not_labelled(self):
         performer = {"identity": "QA Lead", "source": "explicit"}
diff --git a/src/vip/report_content.py b/src/vip/report_content.py
index 79c934fb..1e1152d9 100644
--- a/src/vip/report_content.py
+++ b/src/vip/report_content.py
@@ -364,6 +364,38 @@ def skip_reason_parts(item: TestResult) -> tuple[str, str]:
 NOT_RECORDED = "not recorded"
 
 
+# How each `performed_by.source` reads in the report. `explicit` is the only
+# unlabelled state, because it is the only one where a human named the person
+# accountable for the run. Every other value was inherited from the
+# environment and must say so: `GITHUB_ACTOR` on a scheduled run is whoever
+# last touched the workflow, and a CI actor is often a service account, so an
+# unlabelled one would be indistinguishable from a named operator in the
+# archived artifact.
+PERFORMER_SOURCE_LABELS = {
+    "github": "GitHub actor",
+    "gitlab": "GitLab user",
+    "jenkins": "Jenkins build user",
+    "login": "local login",
+}
+
+
+def _performer_label(performer: dict) -> str | None:
+    """The operator identity as the report shows it, qualified by its source."""
+    identity = performer.get("identity")
+    source = performer.get("source")
+    if not identity or source == "explicit":
+        return identity
+    # An unrecognized source renders verbatim rather than bare: a value this
+    # version does not know about is still not an explicitly named operator,
+    # and rendering it unlabelled would promote it to one. Both backends
+    # escape the result, so an edited results.json cannot inject markup.
+    # A block carrying an identity but no source at all is malformed, and it
+    # gets the same treatment for the same reason -- never bare, and never the
+    # literal string "None".
+    label = PERFORMER_SOURCE_LABELS.get(source, source) if source else f"source {NOT_RECORDED}"
+    return f"{identity} ({label})"
+
+
 def _execution_rows(execution: dict | None) -> list[tuple[str, str | None]]:
     """Who ran this, on which host, from which commit, under which CI job.
 
@@ -391,11 +423,7 @@ def _execution_rows(execution: dict | None) -> list[tuple[str, str | None]]:
         # commit alone. That belongs next to the commit, not in a footnote.
         commit = f"{commit} (uncommitted changes present)"
 
-    identity = performer.get("identity")
-    if identity and performer.get("source") == "login":
-        # A local login is who was at the keyboard, which is weaker than a
-        # named operator or a CI actor. Say which one the reader is looking at.
-        identity = f"{identity} (local login)"
+    identity = _performer_label(performer)
 
     return [
         ("Performed by", identity),

From a6185282722a00f164ef108f7fb7aa292b86560a Mon Sep 17 00:00:00 2001
From: Brian Deitte 
Date: Sat, 29 Aug 2026 20:54:07 -0500
Subject: [PATCH 096/106] fix(traceability): reject unknown control keys and
 add an extra table

load_controls silently dropped any key it did not recognise, so a customer
who wrote phase = "OQ" got no error and no column -- and so did a typo like
referance, from a file whose whole job is to be the regulatory mapping of
record. Unknown keys are now an error naming the key and the recognised set.

Rejecting alone would only fix half of it: a regulated customer's control
list carries fields VIP cannot anticipate, such as an IQ/OQ/PQ phase, a SOP
reference or a control owner. [controls..extra] takes those and carries
them into the CSV as trailing columns and into the JSON as a per-control
object. Values must be strings for the same reason the built-in pass-through
fields must, and a key colliding with an existing column is rejected rather
than emitting a duplicate the spreadsheet resolves by taking the last one.

Extra columns append after CSV_COLUMNS rather than slotting in beside the
control metadata, so the fixed set stays an identical leading prefix whatever
a given customer's control list carries. Neither report edition renders them.
---
 docs/reporting.md                          | 18 +++++
 examples/21CFR_part11_validation/README.md | 19 ++++++
 selftests/test_traceability_controls.py    | 57 ++++++++++++++++
 selftests/test_traceability_render.py      | 47 +++++++++++++
 src/vip/traceability.py                    | 76 +++++++++++++++++++++-
 5 files changed, 215 insertions(+), 2 deletions(-)

diff --git a/docs/reporting.md b/docs/reporting.md
index 3eb3f845..563973dd 100644
--- a/docs/reporting.md
+++ b/docs/reporting.md
@@ -179,6 +179,24 @@ carries `reference`, `risk`, `responsibility`, and `notes` through to the output
 verbatim without interpreting them. The `[controls.]` key is the id a scenario
 references with `@control-` (with the `control-` tag prefix stripped).
 
+Those six keys plus `extra` are the whole recognised set, and any other key is an
+error. Rejecting rather than ignoring is what catches a misspelled `referance`
+before it vanishes from the matrix a reviewer reads. Carry your own fields in an
+`extra` table:
+
+``` toml
+[controls.audit-trail.extra]
+phase = "OQ"
+sop = "SOP-QA-014"
+```
+
+Its values must be strings, must not collide with an existing column name, and
+are carried through untouched: each becomes a trailing CSV column (appended after
+the fixed set, so `CSV_COLUMNS` stays an identical leading prefix across every
+customer's export) and an `extra` object per control in the JSON. Neither report
+edition renders them, because the table has no width for a variable number of
+columns.
+
 ### The three coverage outcomes
 
 A control's row in the matrix gets one of three `coverage` values:
diff --git a/examples/21CFR_part11_validation/README.md b/examples/21CFR_part11_validation/README.md
index 1a1974fc..467486fb 100644
--- a/examples/21CFR_part11_validation/README.md
+++ b/examples/21CFR_part11_validation/README.md
@@ -96,6 +96,25 @@ regulatory mapping uses. Only `description` is required, and every field must be
 a quoted string -- TOML would otherwise read `reference = 2024-01-01` as a date,
 which the JSON export cannot serialise.
 
+The recognised keys are `description`, `reference`, `risk`, `verification`,
+`responsibility` and `notes`. Anything else is an error rather than a silent
+drop, which is what catches `referance` before it disappears from the matrix a
+reviewer reads. Your own fields go in an `extra` table:
+
+```toml
+[controls.audit-trail-publish]
+description = "Deployment of content is recorded with actor and timestamp"
+reference = "21 CFR 11.10(e)"
+
+[controls.audit-trail-publish.extra]
+phase = "OQ"
+sop = "SOP-QA-014"
+```
+
+Those become trailing columns in the CSV export, and an `extra` object per
+control in the JSON. VIP does not interpret them and the report does not render
+them -- its table has no width for a variable number of columns.
+
 Control ids become pytest marker names, so use only letters, digits, `-`, `.`
 and `_`. Write `11-10-a`, not `11.10(a)`: a `:` or `(` truncates the marker name
 pytest registers and breaks collection under `--strict-markers`.
diff --git a/selftests/test_traceability_controls.py b/selftests/test_traceability_controls.py
index dc0e10a8..751c6163 100644
--- a/selftests/test_traceability_controls.py
+++ b/selftests/test_traceability_controls.py
@@ -134,3 +134,60 @@ def test_whitespace_only_description_is_an_error(tmp_path):
     p.write_text('[controls.x]\ndescription = "   "\n')
     with pytest.raises(ControlListError, match="empty"):
         load_controls(p)
+
+
+class TestUnknownKeysAndExtras:
+    """A control list is the regulatory mapping of record, so a key that goes
+    nowhere is worse than a rejected one. Before this, `phase = "OQ"` produced
+    no error and no column, and so did a typo like `referance`.
+    """
+
+    @staticmethod
+    def _write(tmp_path, body):
+        p = tmp_path / "controls.toml"
+        p.write_text(f'[controls.c1]\ndescription = "A control"\n{body}\n', encoding="utf-8")
+        return p
+
+    def test_an_unknown_key_is_rejected_and_the_message_points_at_extra(self, tmp_path):
+        path = self._write(tmp_path, 'phase = "OQ"')
+        with pytest.raises(ControlListError) as exc:
+            load_controls(path)
+        assert "unknown key phase" in str(exc.value)
+        assert "[controls.c1.extra]" in str(exc.value)
+
+    def test_a_misspelled_known_key_is_caught_rather_than_dropped(self, tmp_path):
+        """The failure this actually prevents: a reference that silently
+        vanishes from the matrix a reviewer reads."""
+        path = self._write(tmp_path, 'referance = "21 CFR 11.10(e)"')
+        with pytest.raises(ControlListError, match="unknown key referance"):
+            load_controls(path)
+
+    def test_several_unknown_keys_are_listed_together(self, tmp_path):
+        path = self._write(tmp_path, 'phase = "OQ"\nowner = "QA"')
+        with pytest.raises(ControlListError, match="unknown keys owner, phase"):
+            load_controls(path)
+
+    def test_an_extra_table_is_carried_through_verbatim(self, tmp_path):
+        path = self._write(tmp_path, '[controls.c1.extra]\nphase = "OQ"\nsop = "SOP-QA-014"')
+        assert load_controls(path)["c1"].extra == {"phase": "OQ", "sop": "SOP-QA-014"}
+
+    def test_a_control_with_no_extra_table_gets_an_empty_dict(self, tmp_path):
+        assert load_controls(self._write(tmp_path, "")).get("c1").extra == {}
+
+    def test_a_non_string_extra_value_is_rejected_like_the_built_ins(self, tmp_path):
+        """TOML reads a bare date as datetime.date, which the JSON encoder
+        refuses and the CSV writer silently stringifies."""
+        path = self._write(tmp_path, "[controls.c1.extra]\nqualified = 2024-01-01")
+        with pytest.raises(ControlListError, match="expected a string"):
+            load_controls(path)
+
+    def test_an_extra_key_colliding_with_a_column_is_rejected(self, tmp_path):
+        """Two `risk` columns in one CSV, and a spreadsheet takes the last."""
+        path = self._write(tmp_path, '[controls.c1.extra]\nrisk = "high"')
+        with pytest.raises(ControlListError, match="already a column"):
+            load_controls(path)
+
+    def test_an_extra_that_is_not_a_table_is_rejected(self, tmp_path):
+        path = self._write(tmp_path, 'extra = "OQ"')
+        with pytest.raises(ControlListError, match=r"\[controls.c1.extra\] must be a table"):
+            load_controls(path)
diff --git a/selftests/test_traceability_render.py b/selftests/test_traceability_render.py
index 0141776a..a54d57d4 100644
--- a/selftests/test_traceability_render.py
+++ b/selftests/test_traceability_render.py
@@ -233,3 +233,50 @@ def test_json_non_ascii_appears_literally():
     rendered = render_json(matrix)
     assert "Café français" in rendered
     assert "\\u" not in rendered
+
+
+class TestExtraColumns:
+    """[controls..extra] reaches both exports."""
+
+    @staticmethod
+    def _matrix(extras):
+        from vip.reporting import ReportData
+        from vip.traceability import ControlSpec, build_traceability_matrix
+
+        controls = {
+            cid: ControlSpec(control_id=cid, description=f"control {cid}", extra=extra)
+            for cid, extra in extras.items()
+        }
+        return build_traceability_matrix(ReportData(results=[]), controls)
+
+    def test_csv_appends_the_extra_columns_after_the_fixed_set(self):
+        csv_text = render_csv(self._matrix({"c1": {"phase": "OQ"}}))
+        header = csv_text.splitlines()[0].split(",")
+        assert header[: len(CSV_COLUMNS)] == CSV_COLUMNS
+        assert header[len(CSV_COLUMNS) :] == ["phase"]
+
+    def test_the_column_set_is_the_sorted_union_across_controls(self):
+        matrix = self._matrix({"c1": {"phase": "OQ"}, "c2": {"owner": "QA"}})
+        header = render_csv(matrix).splitlines()[0].split(",")
+        assert header[len(CSV_COLUMNS) :] == ["owner", "phase"]
+
+    def test_a_control_missing_an_extra_key_gets_an_empty_cell_not_a_ragged_row(self):
+        matrix = self._matrix({"c1": {"phase": "OQ"}, "c2": {"owner": "QA"}})
+        rows = list(csv.reader(io.StringIO(render_csv(matrix))))
+        assert all(len(r) == len(rows[0]) for r in rows)
+        by_id = {r[0]: dict(zip(rows[0], r)) for r in rows[1:]}
+        assert by_id["c1"]["owner"] == ""
+        assert by_id["c2"]["phase"] == ""
+
+    def test_an_extra_value_is_formula_neutralized_like_every_other_cell(self):
+        """A future column must inherit the CSV-injection protection."""
+        csv_text = render_csv(self._matrix({"c1": {"phase": "=SUM(1,2)"}}))
+        assert "'=SUM(1,2)" in csv_text
+
+    def test_json_nests_extra_under_each_control(self):
+        payload = json.loads(render_json(self._matrix({"c1": {"phase": "OQ"}})))
+        assert payload["controls"][0]["extra"] == {"phase": "OQ"}
+
+    def test_a_matrix_with_no_extras_keeps_the_exact_fixed_column_set(self):
+        header = render_csv(self._matrix({"c1": {}})).splitlines()[0].split(",")
+        assert header == CSV_COLUMNS
diff --git a/src/vip/traceability.py b/src/vip/traceability.py
index 1385b641..d1d1e68e 100644
--- a/src/vip/traceability.py
+++ b/src/vip/traceability.py
@@ -44,6 +44,55 @@ class ControlSpec:
     verification: str = "automated"
     responsibility: str | None = None
     notes: str | None = None
+    # Free-form pass-through columns from [controls..extra]. A regulated
+    # customer's control list is their own regulatory mapping, and the fields
+    # above cannot anticipate it -- IQ/OQ/PQ phase, a SOP reference, a control
+    # owner. These reach the CSV and JSON exports verbatim and are not
+    # rendered in the report, whose table has no width for a variable number
+    # of columns. Nothing here is interpreted.
+    extra: dict[str, str] = field(default_factory=dict)
+
+
+# Every key a [controls.] table may carry. Anything else is rejected
+# rather than ignored -- see the error raised in load_controls for why.
+_KNOWN_CONTROL_KEYS = frozenset(
+    {"description", "reference", "risk", "verification", "responsibility", "notes", "extra"}
+)
+
+
+def _load_extra(control_id: str, body: dict) -> dict[str, str]:
+    """Validate and return the [controls..extra] pass-through table.
+
+    Values must be strings for the same reason the built-in pass-through
+    fields must: TOML has native date and integer types, and a bare
+    ``2024-01-01`` becomes a ``datetime.date`` that the CSV writer silently
+    stringifies while the JSON encoder refuses outright.
+
+    A key that collides with a built-in column is rejected rather than
+    silently shadowed or duplicated: ``extra.risk`` would otherwise emit two
+    ``risk`` columns in the CSV, and a spreadsheet reader would take whichever
+    it saw last.
+    """
+    raw = body.get("extra")
+    if raw is None:
+        return {}
+    if not isinstance(raw, dict):
+        raise ControlListError(f"[controls.{control_id}.extra] must be a table")
+    extra: dict[str, str] = {}
+    for key, value in raw.items():
+        if key in CSV_COLUMNS:
+            raise ControlListError(
+                f"[controls.{control_id}.extra] has key {key!r}, which is already a "
+                "column in the exported matrix. Choose another name."
+            )
+        if not isinstance(value, str):
+            raise ControlListError(
+                f"[controls.{control_id}.extra] has {key}={value!r} "
+                f"({type(value).__name__}); expected a string. Quote it if it is a "
+                "date or a number."
+            )
+        extra[key] = value
+    return extra
 
 
 def load_controls(path: str | Path) -> dict[str, ControlSpec]:
@@ -103,7 +152,22 @@ def load_controls(path: str | Path) -> dict[str, ControlSpec]:
                     "date or a number."
                 )
             optional[field_name] = value
+        extra = _load_extra(control_id, body)
+        unknown = set(body) - _KNOWN_CONTROL_KEYS
+        if unknown:
+            # Silently dropping these was worse than either alternative: a
+            # customer who wrote `phase = "OQ"` got no error and no column, and
+            # a typo like `referance` vanished the same way -- from a file
+            # whose whole job is to be the regulatory mapping of record.
+            raise ControlListError(
+                f"[controls.{control_id}] has unknown "
+                f"{'keys' if len(unknown) > 1 else 'key'} {', '.join(sorted(unknown))}. "
+                f"Known keys are {', '.join(sorted(_KNOWN_CONTROL_KEYS))}. "
+                f"Put your own fields in [controls.{control_id}.extra]; they are "
+                "carried into the CSV and JSON exports untouched."
+            )
         controls[control_id] = ControlSpec(
+            extra=extra,
             control_id=control_id,
             description=description,
             verification=verification,
@@ -393,16 +457,23 @@ def render_csv(matrix: TraceabilityMatrix) -> str:
     columns empty, so a coverage gap is visible rather than absent.
     """
     buf = io.StringIO()
-    writer = csv.DictWriter(buf, fieldnames=CSV_COLUMNS, lineterminator="\n")
+    # Extra columns append after the fixed set rather than slotting in beside
+    # the control metadata they belong with: CSV_COLUMNS then stays an
+    # identical leading prefix across every customer's export, whatever their
+    # own control list carries. Union across all controls, sorted, so a
+    # control that omits a key gets an empty cell rather than a ragged row.
+    extra_columns = sorted({k for e in matrix.entries for k in e.control.extra})
+    writer = csv.DictWriter(buf, fieldnames=[*CSV_COLUMNS, *extra_columns], lineterminator="\n")
     writer.writeheader()
     prov = _provenance_columns(matrix)
+    blank_extra = dict.fromkeys(extra_columns, "")
 
     def _neutralize_row(row: dict) -> dict:
         """Apply formula neutralization to all string values in the row."""
         return {k: _neutralize_formula(v) if isinstance(v, str) else v for k, v in row.items()}
 
     for entry in matrix.entries:
-        base = _control_columns(entry)
+        base = {**_control_columns(entry), **blank_extra, **entry.control.extra}
         if not entry.matches:
             row = {
                 **base,
@@ -458,6 +529,7 @@ def render_json(matrix: TraceabilityMatrix) -> str:
         "controls": [
             {
                 **_control_columns(entry),
+                "extra": entry.control.extra,
                 "matches": [
                     {
                         "nodeid": m.nodeid,

From 6fbc02b94f432694da5c27c87fc29f85e9b99ef7 Mon Sep 17 00:00:00 2001
From: Brian Deitte 
Date: Sat, 29 Aug 2026 21:12:24 -0500
Subject: [PATCH 097/106] fix(cli): make --report '' actually disable the
 results file

run_verify forwarded --vip-report only when the value was non-empty, so
`vip verify --report ''` left the plugin on its own default and wrote
report/results.json -- the one thing the invocation asked it not to do. The
plugin side already honored an empty value; only the CLI dropped it.

Nothing else reads args.report, and argparse's default is a non-empty path,
so the empty string is the only invocation whose behavior changes.

Honoring it exposes a second edge, closed in the same commit. junit.xml and
results.sarif are written as siblings of results.json and built by reloading
it, so they cannot exist without it. While --report '' was ignored the two
flags could be combined and junit still appeared; now the combination would
run the whole product suite and produce nothing. The CLI refuses it before
starting the suite, naming whichever of --format or --ci asked for the
sibling formats.

docs/reporting.md documented the old behavior as a known limitation, along
with the pass-through workaround it needed. Both are gone.
---
 docs/reporting.md            | 19 +++++++------
 selftests/test_cli_verify.py | 54 ++++++++++++++++++++++++++++++++++++
 src/vip/cli.py               | 28 +++++++++++++++++--
 3 files changed, 89 insertions(+), 12 deletions(-)

diff --git a/docs/reporting.md b/docs/reporting.md
index 563973dd..16df5838 100644
--- a/docs/reporting.md
+++ b/docs/reporting.md
@@ -10,20 +10,21 @@ top of `results.json`.
 
 ## Machine-readable outputs
 
-Every `vip verify` run writes `report/results.json` by default; override the path
-with `--report`. Note that `--report ''` does NOT disable the report -- `vip verify`
-only forwards the option when it is non-empty, so an empty value falls back to the
-default path. To suppress the file entirely, pass the pytest-level option through:
-`vip verify --config vip.toml -- --vip-report=`. `--format` selects which additional
-formats are written alongside it:
+Every `vip verify` run writes `report/results.json` by default. Override the path
+with `--report`, or pass `--report ''` to write no results file at all. `--format`
+selects which additional formats are written alongside it:
 
 ```bash
 vip verify --config vip.toml --format json,junit,sarif
 ```
 
-`json` (`results.json`) is always written regardless of `--format`; `junit` and
-`sarif` are added as sibling files in the same directory when requested. `--ci` is a
-preset that turns on `json,junit,sarif` together with concise tracebacks.
+`json` (`results.json`) is always written unless `--report ''` turns it off. `junit`
+and `sarif` are added as sibling files in the same directory when requested, and
+they are built by reloading `results.json`, so they cannot outlive it: asking for
+either while disabling the results file is refused up front rather than after a
+full run that would produce nothing. `--ci` is a preset that turns on
+`json,junit,sarif` together with concise tracebacks, so it conflicts with
+`--report ''` for the same reason.
 
 ### `results.json` field inventory
 
diff --git a/selftests/test_cli_verify.py b/selftests/test_cli_verify.py
index 2c4ac42e..c3ec1045 100644
--- a/selftests/test_cli_verify.py
+++ b/selftests/test_cli_verify.py
@@ -1439,3 +1439,57 @@ def test_no_warning_when_flags_reach_the_generated_config(self, capsys):
         )
         err = capsys.readouterr().err
         assert "ignored when a config file is used" not in err
+
+
+class TestDisablingTheResultsFile:
+    """`--report ''` is the documented way to write no results file.
+
+    The CLI forwarded `--vip-report` only when the value was non-empty, so the
+    plugin fell back to its own default and wrote report/results.json anyway --
+    the one thing the invocation asked it not to do.
+    """
+
+    def test_an_empty_report_path_is_forwarded_to_the_plugin(self):
+        assert "--vip-report=" in _capture_cmd(_make_args(report=""))
+
+    def test_a_normal_report_path_is_unchanged(self):
+        cmd = _capture_cmd(_make_args(report="out/results.json"))
+        assert "--vip-report=out/results.json" in cmd
+
+    def test_the_default_path_is_still_forwarded(self):
+        assert "--vip-report=report/results.json" in _capture_cmd(_make_args())
+
+    @staticmethod
+    def _run_for_real_exit(args):
+        """Call run_verify without mocking sys.exit, so the refusal propagates.
+
+        The check runs before subprocess.run, so the suite never starts. The
+        shared _capture_call helper patches sys.exit into a no-op, which would
+        let execution fall through past the refusal.
+        """
+        with patch("vip.cli.subprocess.run", side_effect=AssertionError("suite was started")):
+            from vip.cli import run_verify
+
+            run_verify(args)
+
+    @pytest.mark.parametrize("fmt", ["junit", "sarif", "json,junit"])
+    def test_asking_for_a_sibling_format_while_disabling_the_source_is_refused(self, fmt):
+        """junit.xml and results.sarif are built by reloading results.json, so
+        the combination would run the whole suite and produce nothing."""
+        with pytest.raises(SystemExit) as exc:
+            self._run_for_real_exit(_make_args(report="", format=fmt))
+        assert exc.value.code == 2
+
+    def test_the_refusal_names_the_flag_the_formats_came_from(self, capsys):
+        with pytest.raises(SystemExit):
+            self._run_for_real_exit(_make_args(report="", ci=True))
+        assert "--ci" in capsys.readouterr().err
+
+    def test_the_refusal_happens_before_the_suite_runs(self):
+        """A message after a full product run would be worse than no message."""
+        with pytest.raises(SystemExit):
+            self._run_for_real_exit(_make_args(report="", format="junit"))
+
+    def test_disabling_the_report_with_json_alone_is_allowed(self):
+        """json *is* results.json, so there is no sibling left to strand."""
+        assert "--vip-report=" in _capture_cmd(_make_args(report="", format="json"))
diff --git a/src/vip/cli.py b/src/vip/cli.py
index f02fabf9..fecb11f4 100644
--- a/src/vip/cli.py
+++ b/src/vip/cli.py
@@ -535,8 +535,13 @@ def run_verify(args: argparse.Namespace) -> None:
 
     if config_path:
         cmd.append(f"--vip-config={config_path}")
-    if args.report:
-        cmd.append(f"--vip-report={args.report}")
+    # Forward even an empty value. `--vip-report=` is how the plugin is told to
+    # write no report at all, and skipping the flag instead left the plugin on
+    # its own default -- so `vip verify --report ''` wrote report/results.json,
+    # the one thing it was asked not to do. Nothing else reads args.report, and
+    # argparse's default is a non-empty path, so the empty string is the only
+    # invocation whose behavior changes.
+    cmd.append(f"--vip-report={args.report}")
 
     fmt = "json,junit,sarif" if getattr(args, "ci", False) else getattr(args, "format", "json")
     requested = [f.strip().lower() for f in fmt.split(",") if f.strip()]
@@ -548,6 +553,21 @@ def run_verify(args: argparse.Namespace) -> None:
             file=sys.stderr,
         )
         sys.exit(2)
+    # junit.xml and results.sarif are written as siblings of results.json and
+    # are built by reloading it, so they cannot exist without it. Before
+    # --report '' was honored the two flags could be combined and junit still
+    # appeared; now the combination would run the whole suite and produce
+    # nothing. Refuse it up front instead.
+    siblings = [f for f in requested if f != "json"]
+    if not args.report and siblings:
+        source = "--ci" if getattr(args, "ci", False) else "--format"
+        print(
+            f"Error: --report '' disables the results file, but {source} asks for "
+            f"{', '.join(siblings)}, which {'are' if len(siblings) > 1 else 'is'} "
+            "written from it. Drop one of the two.",
+            file=sys.stderr,
+        )
+        sys.exit(2)
     cmd.append(f"--vip-format={','.join(requested)}")
     if args.interactive_auth:
         cmd.append("--interactive-auth")
@@ -2096,7 +2116,9 @@ def main() -> None:
     verify_parser.add_argument(
         "--report",
         default="report/results.json",
-        help="Write JSON results to this path for Quarto report generation"
+        help="Write JSON results to this path for Quarto report generation."
+        " Pass an empty string to write no results file, which also rules out"
+        " the junit/sarif siblings built from it."
         " (default: report/results.json)",
     )
     verify_parser.add_argument(

From 1f41422562a90d7ad5ce4255ac371edb416c150c Mon Sep 17 00:00:00 2001
From: Brian Deitte 
Date: Sat, 29 Aug 2026 21:18:41 -0500
Subject: [PATCH 098/106] fix(traceability): neutralize customer-supplied csv
 header cells

Adding [controls..extra] made a CSV header cell user-controlled for the
first time -- every fieldname was previously a hardcoded CSV_COLUMNS entry, so
writer.writeheader() writing them raw was safe. TOML permits a quoted key, so
'"=HYPERLINK(...)" = "v"' is a legal control list whose column name reaches
the header unneutralized, past the protection every row value gets.

Closed at both layers. load_controls rejects a key starting with one of the
formula characters, which is the better error because a column named after a
formula is never a legitimate regulatory field, and rejecting keeps the CSV
header and the JSON key identical rather than making one sprout an apostrophe.
render_csv then writes the header through the same neutralization as the rows,
covering a ControlSpec built in code rather than loaded from TOML.

The prefix set is now named once and shared, so the two checks cannot drift.
TOML rejects a raw newline or tab inside a key on its own, so the tests reach
those two through TOML escape sequences rather than assuming a second gate
that is not the one under test.
---
 docs/reporting.md                          |  5 +++-
 examples/21CFR_part11_validation/README.md |  4 ++-
 selftests/test_traceability_controls.py    | 21 ++++++++++++++++
 selftests/test_traceability_render.py      | 12 +++++++++
 src/vip/traceability.py                    | 29 +++++++++++++++++++---
 5 files changed, 66 insertions(+), 5 deletions(-)

diff --git a/docs/reporting.md b/docs/reporting.md
index 16df5838..c7333c18 100644
--- a/docs/reporting.md
+++ b/docs/reporting.md
@@ -191,7 +191,10 @@ phase = "OQ"
 sop = "SOP-QA-014"
 ```
 
-Its values must be strings, must not collide with an existing column name, and
+Its keys must not start with a character a spreadsheet evaluates as a formula,
+since a key becomes a CSV header cell and TOML permits a quoted key like
+`"=HYPERLINK(...)"`. Its values must be strings, must not collide with an
+existing column name, and
 are carried through untouched: each becomes a trailing CSV column (appended after
 the fixed set, so `CSV_COLUMNS` stays an identical leading prefix across every
 customer's export) and an `extra` object per control in the JSON. Neither report
diff --git a/examples/21CFR_part11_validation/README.md b/examples/21CFR_part11_validation/README.md
index 467486fb..fd59dad3 100644
--- a/examples/21CFR_part11_validation/README.md
+++ b/examples/21CFR_part11_validation/README.md
@@ -113,7 +113,9 @@ sop = "SOP-QA-014"
 
 Those become trailing columns in the CSV export, and an `extra` object per
 control in the JSON. VIP does not interpret them and the report does not render
-them -- its table has no width for a variable number of columns.
+them -- its table has no width for a variable number of columns. A key may not
+start with `=`, `+`, `-` or `@`, because the key becomes a CSV header cell and
+a spreadsheet would evaluate it.
 
 Control ids become pytest marker names, so use only letters, digits, `-`, `.`
 and `_`. Write `11-10-a`, not `11.10(a)`: a `:` or `(` truncates the marker name
diff --git a/selftests/test_traceability_controls.py b/selftests/test_traceability_controls.py
index 751c6163..700987a8 100644
--- a/selftests/test_traceability_controls.py
+++ b/selftests/test_traceability_controls.py
@@ -1,3 +1,5 @@
+from pathlib import Path
+
 import pytest
 
 from vip.traceability import ControlListError, load_controls
@@ -187,6 +189,25 @@ def test_an_extra_key_colliding_with_a_column_is_rejected(self, tmp_path):
         with pytest.raises(ControlListError, match="already a column"):
             load_controls(path)
 
+    # As written in the TOML source. The control characters go in as escape
+    # sequences because TOML rejects a raw newline or tab inside a key, which
+    # is a second gate rather than the one under test here.
+    @pytest.mark.parametrize("prefix", ["=", "+", "-", "@", "\\t", "\\r", "\\n"])
+    def test_a_formula_leading_extra_key_is_rejected(self, tmp_path, prefix):
+        """TOML allows a quoted key, so the column *name* is attacker-reachable
+        and lands in the CSV header, which row-value neutralization misses."""
+        path = self._write(tmp_path, f'[controls.c1.extra]\n"{prefix}HYPERLINK(1)" = "v"')
+        with pytest.raises(ControlListError, match="reads as a formula"):
+            load_controls(path)
+
+    def test_a_key_merely_containing_a_formula_character_is_allowed(self):
+        """Only the leading character matters to a spreadsheet."""
+        import tempfile
+
+        with tempfile.TemporaryDirectory() as d:
+            path = self._write(Path(d), '[controls.c1.extra]\n"risk-tier" = "2"')
+            assert load_controls(path)["c1"].extra == {"risk-tier": "2"}
+
     def test_an_extra_that_is_not_a_table_is_rejected(self, tmp_path):
         path = self._write(tmp_path, 'extra = "OQ"')
         with pytest.raises(ControlListError, match=r"\[controls.c1.extra\] must be a table"):
diff --git a/selftests/test_traceability_render.py b/selftests/test_traceability_render.py
index a54d57d4..b2091b02 100644
--- a/selftests/test_traceability_render.py
+++ b/selftests/test_traceability_render.py
@@ -273,6 +273,18 @@ def test_an_extra_value_is_formula_neutralized_like_every_other_cell(self):
         csv_text = render_csv(self._matrix({"c1": {"phase": "=SUM(1,2)"}}))
         assert "'=SUM(1,2)" in csv_text
 
+    def test_a_formula_leading_header_cell_is_neutralized(self):
+        """load_controls rejects such a key, so this covers the other door: a
+        ControlSpec built in code rather than loaded from TOML."""
+        csv_text = render_csv(self._matrix({"c1": {"=HYPERLINK(1)": "v"}}))
+        header = next(csv.reader(io.StringIO(csv_text)))
+        assert header[-1] == "'=HYPERLINK(1)"
+
+    def test_ordinary_header_cells_are_untouched(self):
+        header = next(csv.reader(io.StringIO(render_csv(self._matrix({"c1": {"phase": "OQ"}})))))
+        assert header[: len(CSV_COLUMNS)] == CSV_COLUMNS
+        assert header[-1] == "phase"
+
     def test_json_nests_extra_under_each_control(self):
         payload = json.loads(render_json(self._matrix({"c1": {"phase": "OQ"}})))
         assert payload["controls"][0]["extra"] == {"phase": "OQ"}
diff --git a/src/vip/traceability.py b/src/vip/traceability.py
index d1d1e68e..ce4b4eb1 100644
--- a/src/vip/traceability.py
+++ b/src/vip/traceability.py
@@ -80,6 +80,18 @@ def _load_extra(control_id: str, body: dict) -> dict[str, str]:
         raise ControlListError(f"[controls.{control_id}.extra] must be a table")
     extra: dict[str, str] = {}
     for key, value in raw.items():
+        if key and key[0] in _FORMULA_PREFIXES:
+            # TOML allows a quoted key, so `"=HYPERLINK(...)" = "x"` is a legal
+            # control list. That key becomes a CSV *header* cell, which the
+            # formula neutralization below applies to as a backstop -- but a
+            # column named after a formula is never a legitimate regulatory
+            # field, and rejecting it here keeps the CSV header and the JSON
+            # key identical rather than making one of them sprout an
+            # apostrophe.
+            raise ControlListError(
+                f"[controls.{control_id}.extra] has key {key!r}, which starts with a "
+                "character a spreadsheet reads as a formula. Rename the field."
+            )
         if key in CSV_COLUMNS:
             raise ControlListError(
                 f"[controls.{control_id}.extra] has key {key!r}, which is already a "
@@ -406,6 +418,12 @@ def build_traceability_matrix(
 ]
 
 
+# Leading characters a spreadsheet evaluates rather than displays. Named once
+# because both the row values and the (now customer-supplied) header cells have
+# to be checked against the same set.
+_FORMULA_PREFIXES = ("=", "+", "-", "@", "\t", "\r", "\n")
+
+
 def _neutralize_formula(value: str) -> str:
     """Prefix leading formula characters with apostrophe to prevent Excel evaluation.
 
@@ -421,7 +439,7 @@ def _neutralize_formula(value: str) -> str:
     reach the sheet as an unescaped formula (OWASP's CSV-injection guidance
     treats these control characters the same as the leading =/+/-/@).
     """
-    if value and value[0] in ("=", "+", "-", "@", "\t", "\r", "\n"):
+    if value and value[0] in _FORMULA_PREFIXES:
         return "'" + value
     return value
 
@@ -463,8 +481,13 @@ def render_csv(matrix: TraceabilityMatrix) -> str:
     # own control list carries. Union across all controls, sorted, so a
     # control that omits a key gets an empty cell rather than a ragged row.
     extra_columns = sorted({k for e in matrix.entries for k in e.control.extra})
-    writer = csv.DictWriter(buf, fieldnames=[*CSV_COLUMNS, *extra_columns], lineterminator="\n")
-    writer.writeheader()
+    fieldnames = [*CSV_COLUMNS, *extra_columns]
+    writer = csv.DictWriter(buf, fieldnames=fieldnames, lineterminator="\n")
+    # Not writer.writeheader(): the extra columns are customer-supplied, so a
+    # header cell is user-controlled data now and needs the same neutralization
+    # every row value gets. load_controls rejects such a key outright, so this
+    # only fires for a ControlSpec built in code rather than loaded from TOML.
+    writer.writerow({name: _neutralize_formula(name) for name in fieldnames})
     prov = _provenance_columns(matrix)
     blank_extra = dict.fromkeys(extra_columns, "")
 

From 80ed8bb29df5b5933e697bd4dac15b0acaa9722c Mon Sep 17 00:00:00 2001
From: Brian Deitte 
Date: Sun, 30 Aug 2026 10:36:08 -0500
Subject: [PATCH 099/106] docs(readme): add a section on writing your own tests
 and the examples

The README never mentioned vip scaffold, the three example templates, or
vip trace, so the extensibility that makes VIP useful to a regulated customer
was reachable only from the website. The new section lists the templates,
shows the scaffold-then-verify loop, and gives the 21 CFR Part 11 example its
own subsection, since that is the one a customer arrives asking about.

It points at VALIDATION-PACKAGE.md and repeats the scope limit rather than
selling the matrix, because a reader who takes a green matrix into a
validation meeting believing it is an attestation is the failure this example
is written to prevent.

Also adds scaffold and trace to the CLI command table, which had both
missing.
---
 README.md | 46 ++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 46 insertions(+)

diff --git a/README.md b/README.md
index bd667b26..05b7d6c1 100644
--- a/README.md
+++ b/README.md
@@ -102,12 +102,58 @@ content from Connect.
 | `vip status` | Quick health check for each configured product |
 | `vip cleanup` | Delete VIP `_vip_test` content from Connect |
 | `vip report` | Render the HTML report from test results (requires [Quarto CLI](https://quarto.org/docs/download/)) |
+| `vip scaffold` | Generate a ready-to-run custom test extension from a template |
+| `vip trace` | Build a compliance traceability matrix from test results and a control list |
 | `vip auth` | Authentication tools (e.g. mint Connect API keys) |
 | `vip version` | Print the vip version and the minimum supported Posit Team version |
 | `vip --version` | Print the installed vip version |
 
 Run `vip --help` or `vip  --help` for full usage details.
 
+## Writing your own tests
+
+VIP is extensible without changing its source. You point `vip verify` at a
+directory of your own tests, and they run alongside the built-in suite and land
+in the same report. `vip scaffold` writes a working starting point:
+
+```bash
+vip scaffold --list
+vip scaffold --template minimal --output ./my-tests
+vip verify --config vip.toml --extensions ./my-tests
+```
+
+| Template | What it shows |
+|---|---|
+| `minimal` | A single HTTP health-check scenario. Start here. |
+| `cross-product` | R and Python runtime versions, and package installability across Connect and Workbench. The GxP starting point. |
+| `21cfr-part11-validation` | Regulatory control tagging plus a `controls.toml` for `vip trace`. |
+
+Every scaffolded directory runs as-is against your deployment, and each one
+carries an `AGENTS.md` describing the extension contract, so a coding agent
+picking up the directory knows the auto-skip rules and which fixtures and
+markers it may use.
+
+### The 21 CFR Part 11 example
+
+The compliance template is the one worth reading even if you never run it. Each
+scenario declares the control it evidences with an `@control-` tag,
+`controls.toml` holds your regulatory mapping, and `vip trace` joins the two
+into a traceability matrix as CSV or JSON:
+
+```bash
+vip trace --results report/results.json --controls ./my-tests/controls.toml
+```
+
+`vip report --controls` renders the same matrix into the HTML report and the
+archivable PDF, so the artifact you hand an auditor carries the join from a
+control to its evidence rather than only a list of passing tests.
+
+It ships with `VALIDATION-PACKAGE.md`, which is the honest version of what this
+is worth: which documents in a GxP validation package VIP produces, which you
+author, and which nothing can automate. Most of 21 CFR Part 11 cannot be
+evidenced by testing a platform, and a green matrix is not an attestation of
+compliance. Read that file before taking the example into a validation meeting.
+
 ## CI / pipeline integration
 
 VIP emits machine-readable output for security-ops and CI/CD pipelines.

From c870e8097eed48f9835d2d5d1c00b499ca17abb3 Mon Sep 17 00:00:00 2001
From: Brian Deitte 
Date: Mon, 31 Aug 2026 19:01:14 -0500
Subject: [PATCH 100/106] fix(ci): update stale zizmor ignore line and fix
 non-hermetic CLI test

website-preview.yml grew 9 lines earlier in the file, shifting the
checkout step zizmor.yml's artipacked ignore targets from line 100 to
109, so CI started flagging it as a new finding. Separately,
TestDisablingTheResultsFile ran run_verify with no --config, which only
passed because a gitignored local vip.toml happened to sit in the repo
root; in CI's clean checkout the config-not-found check fired before
the refusal being tested, changing the expected exit code.
---
 .github/zizmor.yml           |  2 +-
 selftests/test_cli_verify.py | 18 ++++++++++++------
 2 files changed, 13 insertions(+), 7 deletions(-)

diff --git a/.github/zizmor.yml b/.github/zizmor.yml
index 3bb4c38b..24774d99 100644
--- a/.github/zizmor.yml
+++ b/.github/zizmor.yml
@@ -18,4 +18,4 @@ rules:
       - preview.yml:38:9
       - preview.yml:58:9
       - website-preview.yml:36:9
-      - website-preview.yml:100:9
+      - website-preview.yml:109:9
diff --git a/selftests/test_cli_verify.py b/selftests/test_cli_verify.py
index 1d876d0c..90c3ecbc 100644
--- a/selftests/test_cli_verify.py
+++ b/selftests/test_cli_verify.py
@@ -1473,22 +1473,28 @@ def _run_for_real_exit(args):
             run_verify(args)
 
     @pytest.mark.parametrize("fmt", ["junit", "sarif", "json,junit"])
-    def test_asking_for_a_sibling_format_while_disabling_the_source_is_refused(self, fmt):
+    def test_asking_for_a_sibling_format_while_disabling_the_source_is_refused(self, tmp_path, fmt):
         """junit.xml and results.sarif are built by reloading results.json, so
         the combination would run the whole suite and produce nothing."""
+        cfg = tmp_path / "vip.toml"
+        cfg.write_text("[general]\n")
         with pytest.raises(SystemExit) as exc:
-            self._run_for_real_exit(_make_args(report="", format=fmt))
+            self._run_for_real_exit(_make_args(config=str(cfg), report="", format=fmt))
         assert exc.value.code == 2
 
-    def test_the_refusal_names_the_flag_the_formats_came_from(self, capsys):
+    def test_the_refusal_names_the_flag_the_formats_came_from(self, tmp_path, capsys):
+        cfg = tmp_path / "vip.toml"
+        cfg.write_text("[general]\n")
         with pytest.raises(SystemExit):
-            self._run_for_real_exit(_make_args(report="", ci=True))
+            self._run_for_real_exit(_make_args(config=str(cfg), report="", ci=True))
         assert "--ci" in capsys.readouterr().err
 
-    def test_the_refusal_happens_before_the_suite_runs(self):
+    def test_the_refusal_happens_before_the_suite_runs(self, tmp_path):
         """A message after a full product run would be worse than no message."""
+        cfg = tmp_path / "vip.toml"
+        cfg.write_text("[general]\n")
         with pytest.raises(SystemExit):
-            self._run_for_real_exit(_make_args(report="", format="junit"))
+            self._run_for_real_exit(_make_args(config=str(cfg), report="", format="junit"))
 
     def test_disabling_the_report_with_json_alone_is_allowed(self):
         """json *is* results.json, so there is no sibling left to strand."""

From 893022ac2bf07434f25a7bcce622fb79f977c591 Mon Sep 17 00:00:00 2001
From: Brian Deitte 
Date: Mon, 31 Aug 2026 19:04:04 -0500
Subject: [PATCH 101/106] refactor(reporting): address Copilot review findings
 on PR #627

traceability_summary_rows called control_rows(matrix) purely to count
coverage values, duplicating the scenario-list build both renderers
already do for the table itself; count straight from matrix.entries
instead. unauthenticated_status hard-coded a 30s timeout that could
diverge from the client's configured/scaled timeout, and assumed a
customer-supplied endpoint path was already slash-prefixed.
---
 src/vip/clients/base.py   |  7 +++++--
 src/vip/report_content.py | 14 ++++++++++----
 2 files changed, 15 insertions(+), 6 deletions(-)

diff --git a/src/vip/clients/base.py b/src/vip/clients/base.py
index 0dd0f151..4219a1e1 100644
--- a/src/vip/clients/base.py
+++ b/src/vip/clients/base.py
@@ -132,6 +132,7 @@ def __init__(
         # None sentinel: scale the 30-second default. Callers that supply an
         # explicit value opt out of scaling (their choice is honored as-is).
         effective_timeout = scaled(30.0) if timeout is None else timeout
+        self._timeout = effective_timeout
         self._client = httpx.Client(
             base_url=f"{self._base_url}{api_prefix}",
             headers=headers,
@@ -215,12 +216,14 @@ def unauthenticated_status(self, path: str) -> int:
         """
         from vip.proxy import proxy_for_url, verify_with_env_ca
 
-        url = f"{self.base_url.rstrip('/')}{path}"
+        if not path.startswith("/"):
+            path = f"/{path}"
+        url = f"{self.base_url}{path}"
         with httpx.Client(
             verify=verify_with_env_ca(self._verify),
             proxy=proxy_for_url(url, self._proxy_map),
             trust_env=False,
-            timeout=30.0,
+            timeout=self._timeout,
         ) as client:
             return client.get(url).status_code
 
diff --git a/src/vip/report_content.py b/src/vip/report_content.py
index 7b5ed514..407a156a 100644
--- a/src/vip/report_content.py
+++ b/src/vip/report_content.py
@@ -584,11 +584,17 @@ def control_rows(matrix) -> list[ControlRow]:  # noqa: ANN001 - TraceabilityMatr
 
 
 def traceability_summary_rows(matrix) -> list[tuple[str, str]]:  # noqa: ANN001
-    """Label/value counts for the section's summary table."""
-    rows = control_rows(matrix)
-    counts = Counter(r.coverage for r in rows)
+    """Label/value counts for the section's summary table.
+
+    Counts straight from ``matrix.entries`` rather than via :func:`control_rows`,
+    which also builds each control's full scenario list -- both backends already
+    call :func:`control_rows` once for the table itself, so building it again
+    here would be a second, unused pass over every control just to count them.
+    """
+    entries = list(matrix.entries)
+    counts = Counter(display_coverage(entry) for entry in entries)
     return [
-        ("Controls", str(len(rows))),
+        ("Controls", str(len(entries))),
         ("Covered, executed and passing", str(counts.get("covered", 0))),
         ("Covered, not executed", str(counts.get("covered_not_executed", 0))),
         ("Covered, failing", str(counts.get("covered_failed", 0))),

From 6b92efb2c6c6a23b5095794651f84446f26ffbc6 Mon Sep 17 00:00:00 2001
From: Brian Deitte 
Date: Mon, 31 Aug 2026 19:22:49 -0500
Subject: [PATCH 102/106] feat(website): drop the separate compliance report
 page

A standalone Compliance Report page confused users and understated
that the standard Example Report already serves most compliance
needs, since both are rendered by the same vip report with an
optional --controls attached. Removes the page, its nav link, and the
CI work that built it (a second pytest pass against the Part 11
example extension, a second Quarto render, and the
example-compliance-report artifact). Getting Started and the Example
Report page now point readers at examples/21CFR_part11_validation on
GitHub instead.
---
 .github/workflows/example-report.yml      |  72 ---------
 .github/workflows/website-preview.yml     |   9 --
 .github/workflows/website.yml             |   6 -
 .github/zizmor.yml                        |   2 +-
 AGENTS.md                                 |   4 +-
 website/src/components/Header.astro       |   1 -
 website/src/pages/compliance-report.astro | 172 ----------------------
 website/src/pages/getting-started.astro   |  16 +-
 website/src/pages/report.astro            |  12 +-
 9 files changed, 18 insertions(+), 276 deletions(-)
 delete mode 100644 website/src/pages/compliance-report.astro

diff --git a/.github/workflows/example-report.yml b/.github/workflows/example-report.yml
index 2e686d3d..5b0a9b27 100644
--- a/.github/workflows/example-report.yml
+++ b/.github/workflows/example-report.yml
@@ -394,77 +394,11 @@ jobs:
       - name: Render Quarto report
         run: cd report && uv run quarto render
 
-      # Both renders write report/_output and both runs write
-      # report/results.json, so the standard report has to move out of the way
-      # before the compliance pass starts. Everything below runs before the
-      # product containers stop, because the Part 11 scenarios need the same
-      # live Connect.
       - name: Stage the standard report
         run: |
           mkdir -p _site
           mv report/_output _site/example-report
 
-      # A Connect-only config on purpose. The Part 11 example also maps
-      # Workbench and Package Manager controls, but this job stands up Connect
-      # for the compliance pass, and reusing vip.toml would print the other two
-      # into the products table of a report in which no scenario touched them.
-      #
-      # The published matrix therefore shows the four Workbench and Package
-      # Manager controls as gaps. That is the intended demonstration: VIP
-      # deselects an unconfigured product's scenarios rather than skipping
-      # them, so they never reach results.json and their controls read as gaps
-      # rather than as covered. The example README explains the same thing in
-      # prose; here a reader sees it in the artifact.
-      - name: Configure VIP for the compliance example
-        run: |
-          cat > vip-compliance.toml << EOF
-          [general]
-          deployment_name = "CI Connect ${{ steps.version.outputs.resolved }} (21 CFR Part 11 example extension)"
-
-          [connect]
-          enabled = true
-          url = "${{ steps.connect.outputs.CONNECT_SERVER }}"
-          api_key = "${{ steps.connect.outputs.CONNECT_API_KEY }}"
-          version = "${{ steps.version.outputs.resolved }}"
-          EOF
-
-      # Same exit-code contract as the smoke run above: 0 and 1 mean pytest
-      # ran to completion, anything else means it did not. Exit code 5 (no
-      # tests collected) is the one that matters most here -- VIP deselects
-      # @connect scenarios when Connect is not configured, so a broken
-      # vip-compliance.toml would otherwise render an empty compliance report
-      # and publish it. VIP_ENABLE_EXPECTED_FAILURE_DEMO is deliberately unset:
-      # the by-design failing demo scenario reads as a broken deployment to the
-      # audience this report is for.
-      - name: Run the 21 CFR Part 11 example extension
-        run: |
-          set +e
-          uv run pytest examples/21CFR_part11_validation/ -v --vip-config=vip-compliance.toml
-          PYTEST_EXIT_CODE=$?
-          set -e
-          echo "pytest exited with code ${PYTEST_EXIT_CODE}"
-          if [ "${PYTEST_EXIT_CODE}" -eq 0 ] || [ "${PYTEST_EXIT_CODE}" -eq 1 ]; then
-            exit 0
-          fi
-          echo "::error::the Part 11 example extension exited with code ${PYTEST_EXIT_CODE} (collection, usage, or internal error rather than a test failure); see the log above for details."
-          exit 1
-
-      # VIP_CONTROLS must be absolute. Quarto renders with report/ as the
-      # working directory, so a relative path resolves against report/ and the
-      # control list is not found. Both documents now print a visible "could
-      # not render the traceability section" marker instead of dropping it
-      # silently -- but quarto render still exits 0 and this job still goes
-      # green either way, so the marker only shows up in the rendered
-      # artifact, not in this job's status. An absolute path is still what
-      # makes it work.
-      - name: Render the compliance report
-        env:
-          VIP_CONTROLS: ${{ github.workspace }}/examples/21CFR_part11_validation/controls.toml
-        run: cd report && uv run quarto render
-
-      - name: Stage the compliance report
-        run: mv report/_output _site/example-compliance-report
-
       # Stop Connect
       - name: Stop Connect
         if: always()
@@ -489,9 +423,3 @@ jobs:
         with:
           name: example-report
           path: _site/example-report/
-
-      - name: Upload compliance report artifact
-        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
-        with:
-          name: example-compliance-report
-          path: _site/example-compliance-report/
diff --git a/.github/workflows/website-preview.yml b/.github/workflows/website-preview.yml
index 460b5c84..b78ea649 100644
--- a/.github/workflows/website-preview.yml
+++ b/.github/workflows/website-preview.yml
@@ -70,12 +70,6 @@ jobs:
           name: example-report
           path: website/dist/example-report/
 
-      - name: Download compliance example report
-        uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8
-        with:
-          name: example-compliance-report
-          path: website/dist/example-compliance-report/
-
       - uses: rossjrw/pr-preview-action@ffa7509e91a3ec8dfc2e5536c4d5c1acdf7a6de9 # v1
         with:
           source-dir: website/dist/
@@ -87,7 +81,6 @@ jobs:
         env:
           WEBSITE_URL: https://posit-dev.github.io/vip/pr-preview-site/pr-${{ github.event.pull_request.number }}/
           REPORT_URL: https://posit-dev.github.io/vip/pr-preview/pr-${{ github.event.pull_request.number }}/
-          COMPLIANCE_URL: https://posit-dev.github.io/vip/pr-preview-site/pr-${{ github.event.pull_request.number }}/compliance-report/
           QR_PREFIX: https://qr.rossjrw.com/?color.dark=0d1117&url=
         with:
           header: preview-links
@@ -99,8 +92,6 @@ jobs:
             | Website preview QR code | Report preview QR code |
             | ${{ env.WEBSITE_URL }} | ${{ env.REPORT_URL }} |
 
-            Compliance traceability report: ${{ env.COMPLIANCE_URL }}
-
   cleanup:
     if: github.event.action == 'closed'
     name: Clean up website preview
diff --git a/.github/workflows/website.yml b/.github/workflows/website.yml
index f52158cc..6633a695 100644
--- a/.github/workflows/website.yml
+++ b/.github/workflows/website.yml
@@ -86,12 +86,6 @@ jobs:
           name: example-report
           path: website/dist/example-report/
 
-      - name: Download compliance example report
-        uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8
-        with:
-          name: example-compliance-report
-          path: website/dist/example-compliance-report/
-
       - uses: JamesIves/github-pages-deploy-action@fa24774553152dd7873cd16ebd8d959b010c5445 # v4.9.0
         with:
           branch: gh-pages
diff --git a/.github/zizmor.yml b/.github/zizmor.yml
index 24774d99..3bb4c38b 100644
--- a/.github/zizmor.yml
+++ b/.github/zizmor.yml
@@ -18,4 +18,4 @@ rules:
       - preview.yml:38:9
       - preview.yml:58:9
       - website-preview.yml:36:9
-      - website-preview.yml:109:9
+      - website-preview.yml:100:9
diff --git a/AGENTS.md b/AGENTS.md
index 8ce02a69..549cdbf8 100644
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -314,8 +314,8 @@ Every render also produces `_output/vip-report.pdf` from `report/vip-report.qmd`
 ## CI workflows
 
 -   **`ci.yml`** -- on every PR/push: ruff lint/format (pinned to 0.15.0), mypy type-check, zizmor actions-lint, a runtime dependency audit, and selftests (Ubuntu + macOS, Python 3.10 and 3.12). A `changes` path-filter gates the expensive jobs, while `Lint & Format`, `Selftests Status` and `CI Status` always run as required checks. Uses uv cache. `CI Status` is the scope-aware aggregator for the four path-gated jobs (`Type Check`, `Actions Lint (zizmor)`, `Dependency Audit`, `Lockfile Guard`): none of them can be a required check directly, because each is conditional on `changes` and a failed change-detection job would skip them all and report a green gate. A legitimately skipped job counts as passing; only failure or cancellation is fatal.
--   **`preview.yml`** -- runs selftests, renders Quarto report, publishes PR preview to gh-pages via `rossjrw/pr-preview-action@v1`. Uses uv and Quarto caches. It previews the standard report only: its job is checking `report/` template changes, and the compliance edition below adds no template surface.
--   **`example-report.yml`** -- builds *two* reports from one live deployment and uploads them as separate artifacts. The `example-report` pass runs the smoke subset and renders plainly. The `example-compliance-report` pass then runs `examples/21CFR_part11_validation/` against the same Connect and re-renders with `VIP_CONTROLS` set, so the traceability matrix appears in both the HTML and the PDF. Three constraints hold that apart: both passes write `report/results.json` and `report/_output`, so the first report is staged into `_site/` before the second starts; `VIP_CONTROLS` must be an absolute path, because Quarto's working directory is `report/` and both `.qmd` files swallow every exception, so a relative path drops the section silently and the job still goes green; and `VIP_ENABLE_EXPECTED_FAILURE_DEMO` stays unset on the compliance pass, because a by-design failing scenario reads as a broken deployment to a regulated reader. `website.yml` and `website-preview.yml` download both artifacts into `website/dist/`, where `report.astro` and `compliance-report.astro` embed them.
+-   **`preview.yml`** -- runs selftests, renders Quarto report, publishes PR preview to gh-pages via `rossjrw/pr-preview-action@v1`. Uses uv and Quarto caches. Its job is checking `report/` template changes.
+-   **`example-report.yml`** -- builds the example report from one live deployment and uploads it as an artifact. Runs the smoke subset against Connect, Workbench and Package Manager and renders it with `vip report`. `website.yml` and `website-preview.yml` download the artifact into `website/dist/`, where `report.astro` embeds it. A control-list traceability matrix (`vip report --controls`) is documented and demonstrated separately, in `examples/21CFR_part11_validation/`, rather than published on the website -- see that directory's `VALIDATION-PACKAGE.md`.
 -   **`pr-title.yml`** -- validates PR titles follow conventional commit format. Squash merges use the PR title as the commit message.
 -   **`release.yml`** -- cuts VIP's calver release train: `schedule` (Thursday evenings) plus `workflow_dispatch` for out-of-band releases. `scripts/next_version.py` computes the version (`YYYY.M.0` for the first release of a calendar month, `YYYY.M.PATCH` for later ones that month); a scheduled or blank-`version` dispatch run exits cleanly when there are no commits since the last tag, while a dispatch run with an explicit `version` skips that gate but must still be strictly greater than the last release. `cliff.toml` (git-cliff) generates the `CHANGELOG.md` entry for the tag before it exists, so it lands inside the release commit rather than needing a second commit. `just relock` keeps `uv.lock`'s own `posit-vip` entry in sync with the version just stamped -- see docs/development.md ("Versioning and the release cadence") for the full rule and issue #559 for why the relock step matters.
 -   **Smoke workflows** (`connect-smoke.yml`, `workbench-smoke.yml`, `packagemanager-smoke.yml`, `mock-idp-e2e.yml`) -- run the product suites against real containers. On PR/push each tests a single latest version (change-gated via a `changes` paths-filter job); on `schedule` (nightly, staggered hourly) and `workflow_dispatch` a `set-matrix` job fans each out across the product version support window (current + 2 back). Bump the pinned tags in each workflow's `set-matrix` step when a new product release ships. Each workflow's `*-status` aggregation job is scope-aware: it passes when the suite was legitimately out of scope (the PR's paths didn't match) but **fails** when the suite was in scope (`changes.relevant == 'true'`) yet did not succeed -- so a path-gated skip, an excluded actor, or a missing license secret can no longer report a green required check without the suite having run. The Connect, Workbench, and Package Manager `*-status` jobs are required merge checks.
diff --git a/website/src/components/Header.astro b/website/src/components/Header.astro
index 074a1670..d1e394e2 100644
--- a/website/src/components/Header.astro
+++ b/website/src/components/Header.astro
@@ -8,7 +8,6 @@ const links = [
   { href: `${base}tests/`, label: "Tests" },
   { href: `${base}feature-matrix/`, label: "Feature Matrix" },
   { href: `${base}report/`, label: "Example Report" },
-  { href: `${base}compliance-report/`, label: "Compliance Report" },
 ];
 
 function isActive(href: string): boolean {
diff --git a/website/src/pages/compliance-report.astro b/website/src/pages/compliance-report.astro
deleted file mode 100644
index ab80aebf..00000000
--- a/website/src/pages/compliance-report.astro
+++ /dev/null
@@ -1,172 +0,0 @@
----
-import Layout from "../layouts/Layout.astro";
-import Header from "../components/Header.astro";
-import Footer from "../components/Footer.astro";
-
-const base = import.meta.env.BASE_URL;
----
-
-
-  

-
-  

-    

-      
Compliance Traceability Report

-      

-        A second, narrower run against the same CI Connect deployment: only the
-        21 CFR Part 11 example extension that ships with VIP, joined against that
-        example’s controls.toml. The Compliance Traceability
-        section is the point — every control, the scenarios that verify it,
-        their outcome, and the timestamp each check ran.
-      

-      

-        The four Workbench and Package Manager controls report as gaps here, and
-        that is worth reading rather than skipping past. This run configures Connect
-        alone, VIP deselects an unconfigured product’s scenarios rather than
-        skipping them, so those scenarios never reach the results file and the
-        controls they carry read as gaps. The suite has the checks; this run did not
-        exercise them. Confirm which products a run covered before you conclude a
-        control has no automated evidence.
-      

-      

-        The standard example report is a broad smoke
-        run across Connect, Workbench and Package Manager with no control list
-        attached, so it has no such section and its summary counts many more
-        scenarios. The two differ in scope, not in machinery: both are rendered by
-        the same vip report, and any VIP run can carry a matrix by
-        attaching a control list.
-      

-      

-        Generate your own: run
-        vip scaffold --template 21cfr-part11-validation --output ./my-tests
-        to get a worked controls.toml and tagged scenarios, replace them
-        with your own regulatory mapping, then
-        vip verify --extensions ./my-tests followed by
-        vip report --controls ./my-tests/controls.toml. The same matrix
-        exports to CSV or JSON with vip trace.
-      

-      

-        Read the matrix carefully. This example uses a nine-control list, seven of
-        them automated across Connect, Workbench and Package Manager. Controls you
-        mark manual or procedural are reported as not
-        verifiable by automated test, which is deliberately distinct from a coverage
-        gap, and a control whose scenario was skipped is reported as covered but not
-        executed. A green matrix evidences the controls you chose to automate, on the
-        deployment you pointed VIP at, at the time it ran. It is not an attestation
-        of compliance.
-      

-      

-        
-          
-          Download PDF report
-        
-        
-          Open in new tab ↗
-        
-      

-    

-  

-
-  

-    
-  

-
-  

-
-
-
diff --git a/website/src/pages/getting-started.astro b/website/src/pages/getting-started.astro
index 1d3fafcd..baca518f 100644
--- a/website/src/pages/getting-started.astro
+++ b/website/src/pages/getting-started.astro
@@ -2,8 +2,6 @@
 import Layout from "../layouts/Layout.astro";
 import Header from "../components/Header.astro";
 import Footer from "../components/Footer.astro";
-
-const base = import.meta.env.BASE_URL;
 ---
 
 
@@ -543,13 +541,13 @@ vip trace --results report/results.json --controls ./my-tests/controls.toml
           

-            A rendered example is published on this site. The
-            compliance traceability report
-            is a real VIP run against a CI Connect deployment with the matrix in it,
-            readable in the browser and downloadable as a PDF.
-          

-          

-            A worked example ships with VIP. Generate it with
+            A worked example ships with VIP:
+            examples/21CFR_part11_validation
+            pairs a controls.toml with the tagged scenarios that satisfy
+            it. Generate your own copy with
             vip scaffold --template 21cfr-part11-validation --output ./my-tests,
             then read its README.md: it is a template to edit, and it is
             explicit about what a matrix like this can and cannot claim. Posit Team
diff --git a/website/src/pages/report.astro b/website/src/pages/report.astro
index 8d671df2..745866b7 100644
--- a/website/src/pages/report.astro
+++ b/website/src/pages/report.astro
@@ -22,10 +22,14 @@ const base = import.meta.env.BASE_URL;
         see every scenario that applies to your configuration.
       

       

-        For a regulated environment, the
-        compliance traceability report is a
-        second example: a narrower run joined against a control list, so every
-        control links to the scenarios that verify it.
+        This same report format serves regulated environments too: attach a control
+        list with vip report --controls and the report gains a
+        traceability matrix joining each control to the scenarios that verify it. See
+        the worked
+        21 CFR Part 11 example.
       

       

         Generate your own: run vip verify to save test results to

From 92330c42d4a8ed02e09aa137682124dd3341cdaa Mon Sep 17 00:00:00 2001
From: Brian Deitte 
Date: Mon, 31 Aug 2026 20:36:06 -0500
Subject: [PATCH 103/106] fix(traceability): count an unproven skip as a
 non-execution

An unproven skip records a check VIP was asked to run and could not, so
no assertion ran. NON_EXECUTING_STATUSES listed only skipped and
na_version, which made a control whose scenarios were all unproven read
as executed and failing, so the matrix reported it as covered-and-failed
rather than covered-not-executed.
---
 selftests/conftest.py                 |  9 ++++++--
 selftests/test_traceability_matrix.py | 30 +++++++++++++++++++++++++++
 src/vip/traceability.py               | 14 ++++++++++---
 3 files changed, 48 insertions(+), 5 deletions(-)

diff --git a/selftests/conftest.py b/selftests/conftest.py
index 1b453590..db4eeca3 100644
--- a/selftests/conftest.py
+++ b/selftests/conftest.py
@@ -194,11 +194,15 @@ def sample_results_json(tmp_path: Path) -> Path:
     return p
 
 
+_SKIP_STATUSES = frozenset({"skipped", "na_version", "unproven"})
+
+
 def matrix_from_statuses(statuses: dict[str, list[str]]):
     """Build a TraceabilityMatrix from {control_id: [scenario status, ...]}.
 
     One TestResult per status, tagged `control-`. A status of "na_version"
-    is written as a version-gated skip, which is how the plugin records it.
+    is written as a version-gated skip, and "unproven" as an attested one,
+    which is how the plugin records each.
     """
     from vip.reporting import ReportData, TestResult
     from vip.traceability import ControlSpec, build_traceability_matrix
@@ -209,8 +213,9 @@ def matrix_from_statuses(statuses: dict[str, list[str]]):
             results.append(
                 TestResult(
                     nodeid=f"test_{control_id}.py::test_{i}",
-                    outcome="skipped" if status == "na_version" else status,
+                    outcome="skipped" if status in _SKIP_STATUSES else status,
                     na_version=status == "na_version",
+                    unproven=status == "unproven",
                     markers=[f"control-{control_id}"],
                 )
             )
diff --git a/selftests/test_traceability_matrix.py b/selftests/test_traceability_matrix.py
index bfab5b70..8d53f56b 100644
--- a/selftests/test_traceability_matrix.py
+++ b/selftests/test_traceability_matrix.py
@@ -255,3 +255,33 @@ def test_an_all_skipped_control_is_not_failing(self):
     def test_a_version_gated_control_is_not_failing(self):
         matrix = matrix_from_statuses(statuses={"c1": ["na_version"]})
         assert matrix.entries[0].failing is False
+
+    def test_an_unproven_control_is_not_run_rather_than_failed(self):
+        """An unproven skip ran no assertion, so it is a non-execution.
+
+        Reporting it as failing would say the control was verified and came
+        back bad, when VIP never got to check it. The distinction is the whole
+        point of ``vip.attest.unproven``.
+        """
+        matrix = matrix_from_statuses(statuses={"c1": ["unproven"]})
+        entry = matrix.entries[0]
+        assert entry.coverage == "covered"
+        assert entry.executed is False
+        assert entry.failing is False
+        assert matrix.covered_without_execution == ["c1"]
+        assert matrix.covered_with_failure == []
+
+    def test_a_pass_beside_an_unproven_skip_reads_as_covered(self):
+        """The known cost of treating unproven as non-executing.
+
+        One scenario proved something and one could not be checked, and the
+        control's own summary buckets say only that it ran and passed. The
+        unproven scenario stays visible in the matrix's per-scenario status
+        column and still fails the run with exit code 6, so the signal is not
+        lost, only absent from the control-level rollup.
+        """
+        matrix = matrix_from_statuses(statuses={"c1": ["passed", "unproven"]})
+        entry = matrix.entries[0]
+        assert entry.executed is True
+        assert entry.failing is False
+        assert [m.status for m in entry.matches] == ["passed", "unproven"]
diff --git a/src/vip/traceability.py b/src/vip/traceability.py
index ce4b4eb1..0f5e1d02 100644
--- a/src/vip/traceability.py
+++ b/src/vip/traceability.py
@@ -203,8 +203,12 @@ class ControlMatch:
 # Statuses that mean the check did not run. "na_version" is a
 # version-gated non-execution, so it counts here too: treating it as
 # executed would let a control gated off on an older product read as
-# evidenced by a scenario that never ran a single assertion.
-NON_EXECUTING_STATUSES = frozenset({"skipped", "na_version"})
+# evidenced by a scenario that never ran a single assertion. "unproven" is
+# the same shape -- ``vip.attest.unproven`` skips a check VIP was asked to
+# run and could not, so no assertion ran there either. Its loudness lives in
+# the run's exit code 6 and in the per-scenario status column, not in
+# ``executed``, which answers only whether a result was produced.
+NON_EXECUTING_STATUSES = frozenset({"skipped", "na_version", "unproven"})
 
 
 @dataclass
@@ -243,7 +247,11 @@ def failing(self) -> bool:
         ``error`` is a reachable outcome alongside ``failed``, and an
         enumerated list would let an errored control read as evidenced.
         Non-executing statuses are excluded via ``NON_EXECUTING_STATUSES``, so
-        a skip alongside a pass never counts against a control.
+        a skip alongside a pass never counts against a control. That includes
+        an ``unproven`` skip, which is the contentious member: it is a check
+        VIP could not run, so it belongs with the non-executing statuses
+        rather than here, where it would report a control that never ran as
+        one that ran and failed.
         """
         return any(
             m.status not in NON_EXECUTING_STATUSES and m.status != "passed" for m in self.matches

From 0eef40a5552f2b34fcf96eca75e28a58bd768d11 Mon Sep 17 00:00:00 2001
From: Brian Deitte 
Date: Mon, 31 Aug 2026 21:55:19 -0500
Subject: [PATCH 104/106] feat(traceability): surface controls with a check VIP
 could not verify

An unproven skip is neither an execution nor a failure, so a control with
one passing scenario beside one unproven scenario read as fully evidenced
from the existing covered_not_executed and covered_failed facts alone.

ControlEntry.has_unproven and TraceabilityMatrix.covered_with_unproven are
the fourth fact. They overlap the other two rather than partitioning them:
an unproven-only control is both not executed and not verified, and each
statement is true. The display column shows the loudest of the three,
FAILED then UNPROVEN then NOT RUN, and both report editions reuse the
existing scenario-level unproven style, so no new color enters the palette.

Matrix schema bumped to 1.1; the summary key and the control-id list are
additive.
---
 docs/reporting.md                             | 51 +++++++++-----
 .../VALIDATION-PACKAGE.md                     | 10 ++-
 selftests/test_report_content.py              | 66 +++++++++++++++----
 selftests/test_trace_cli.py                   | 50 +++++++++++++-
 selftests/test_traceability_matrix.py         | 59 ++++++++++++++---
 selftests/test_traceability_render.py         | 12 +++-
 src/vip/cli.py                                | 17 ++++-
 src/vip/report_content.py                     | 46 ++++++++++---
 src/vip/traceability.py                       | 39 ++++++++++-
 9 files changed, 301 insertions(+), 49 deletions(-)

diff --git a/docs/reporting.md b/docs/reporting.md
index c7333c18..19fd73ac 100644
--- a/docs/reporting.md
+++ b/docs/reporting.md
@@ -223,18 +223,26 @@ A control's row in the matrix gets one of three `coverage` values:
 
 ### Coverage display states
 
-The rendered report displays coverage as `COVERED`, `FAILED`, `NOT RUN`, `GAP`, or
-`N/A (manual)`:
+The rendered report displays coverage as `COVERED`, `FAILED`, `UNPROVEN`,
+`NOT RUN`, `GAP`, or `N/A (manual)`:
 
 - `COVERED` -- at least one tagged scenario ran and passed.
 - `FAILED` -- a covered control with at least one tagged scenario that ran and did
   not pass.
+- `UNPROVEN` -- a covered control with at least one tagged scenario that VIP was
+  asked to run and could not (`vip.attest.unproven`).
 - `NOT RUN` -- at least one tagged scenario is present, but all of them skipped or
   were not executed.
 - `GAP` -- no tagged scenario is present and `verification = "automated"`.
 - `N/A (manual)` -- no tagged scenario is present but `verification` is `"manual"`
   or `"procedural"`.
 
+A control can satisfy more than one of these at once, and the display column
+shows only the loudest. The order is `FAILED`, then `UNPROVEN`, then `NOT RUN`:
+a control that ran and failed is the strongest claim against it, and an
+all-unproven control is also not executed, where `UNPROVEN` is the more
+specific of the two. Read the per-scenario `status` column for the rest.
+
 ### Covered is not the same as executed
 
 A scenario can run and skip itself -- the endpoint it probes is absent from this
@@ -247,18 +255,25 @@ this export can do, so it is reported three ways rather than left implicit:
 - `vip trace` warns on stderr, naming the affected control ids.
 - The JSON `summary` reports `covered_and_executed` (a tagged scenario ran,
   whether or not it passed), `covered_not_executed` (a tagged scenario is
-  present but none ran), and `covered_failed` (a tagged scenario ran and did
-  not pass). These are not a three-way partition: a failing control counts
-  toward both `covered_and_executed` and `covered_failed`, since it did run
-  and it did not pass. The rendered report's own summary table partitions
-  differently -- it splits the display value into three mutually exclusive
-  rows, `Covered, executed and passing`, `Covered, not executed` and
-  `Covered, failing`, so a failing control is counted once. Do not expect the
-  report table and the JSON `summary` to add up the same way.
-- The JSON carries a `covered_without_execution` list of control ids.
+  present but none ran), `covered_failed` (a tagged scenario ran and did not
+  pass), and `covered_unproven` (a tagged scenario was a check VIP could not
+  run). These are not a partition: a failing control counts toward both
+  `covered_and_executed` and `covered_failed`, since it did run and it did not
+  pass, and an all-unproven control counts toward both `covered_not_executed`
+  and `covered_unproven`. The rendered report's own summary table partitions
+  differently -- it splits the display value into mutually exclusive rows,
+  `Covered, executed and passing`, `Covered, not executed`, `Covered, failing`
+  and `Covered, not verified`, so each control is counted once. Do not expect
+  the report table and the JSON `summary` to add up the same way.
+- The JSON carries a `covered_without_execution` list of control ids and a
+  `covered_with_unproven` list.
 
 A version-gated scenario (`na_version`) counts as not executed too, for the same
-reason: it ran no assertions.
+reason: it ran no assertions. So does an `unproven` one, which is the reason
+`covered_unproven` exists alongside the other counts: an unproven skip is
+neither an execution nor a failure, so a control with one passing scenario and
+one unproven scenario is invisible to `covered_not_executed` and
+`covered_failed` both, and part of the control still went unchecked.
 
 An unconfigured product is a different case, and it fails the opposite way.
 Those scenarios are deselected rather than skipped -- excluded from the run
@@ -269,8 +284,11 @@ safer direction, but a reader who takes the gap at face value concludes the
 suite lacks a check it has. The `products` block records what was actually
 under test; read it alongside the gaps.
 
-Read `gaps: 0` together with `covered_not_executed`. Zero gaps and a non-zero
-`covered_not_executed` means the controls are mapped and the evidence is missing.
+Read `gaps: 0` together with `covered_not_executed` and `covered_unproven`.
+Zero gaps and a non-zero `covered_not_executed` means the controls are mapped
+and the evidence is missing. A non-zero `covered_unproven` means VIP was asked
+for evidence it could not produce, which is the difference between a check with
+nothing to test and a check that never got to run.
 
 ### Worked example
 
@@ -322,8 +340,9 @@ set by hand.
 
 The section repeats the same caveat the CSV and JSON exports carry, because the
 report is the artifact that gets archived and handed on: coverage records that a
-scenario is tagged, and a control shown as NOT RUN has a tagged scenario that ran
-and skipped itself. See `examples/21CFR_part11_validation/VALIDATION-PACKAGE.md` for how these
+scenario is tagged, a control shown as NOT RUN has a tagged scenario that ran
+and skipped itself, and a control shown as UNPROVEN has one VIP was asked to run
+and could not. See `examples/21CFR_part11_validation/VALIDATION-PACKAGE.md` for how these
 outputs map onto a GxP
 validation package, and which parts of one VIP cannot supply.
 
diff --git a/examples/21CFR_part11_validation/VALIDATION-PACKAGE.md b/examples/21CFR_part11_validation/VALIDATION-PACKAGE.md
index 4818d85a..18b60ad4 100644
--- a/examples/21CFR_part11_validation/VALIDATION-PACKAGE.md
+++ b/examples/21CFR_part11_validation/VALIDATION-PACKAGE.md
@@ -180,7 +180,7 @@ cannot quietly regenerate.
 A fully green matrix evidences the controls you chose to automate, on the
 deployment you pointed it at, at the time it ran. That is all it claims.
 
-Four specific ways it can mislead if read carelessly:
+Five specific ways it can mislead if read carelessly:
 
 Coverage is not execution. A control counts as covered when a scenario is
 tagged for it, whether or not that scenario reached an assertion. A scenario
@@ -190,6 +190,14 @@ excluded it. The report shows those controls as NOT RUN, `vip trace` warns on
 stderr, and the JSON summary counts them under `covered_not_executed`. Read
 `gaps: 0` together with that number.
 
+Execution is not verification either. A scenario can run, find that it cannot
+check what it was asked to check, and record that as unproven rather than as a
+pass. Such a control shows as UNPROVEN, warns on stderr, and is counted under
+`covered_unproven`. This is the one case the two numbers above miss entirely:
+a control with one passing scenario beside one unproven scenario is both
+executed and not failing, so it reads as fully evidenced from those two alone
+while part of the control went unchecked.
+
 A gap can be an artifact of the run rather than of your suite. Scenarios
 belonging to a product you did not configure are deselected, not skipped, so
 they never appear in the results file at all, and a control tagged only by them
diff --git a/selftests/test_report_content.py b/selftests/test_report_content.py
index a6a725fe..d89b0c6d 100644
--- a/selftests/test_report_content.py
+++ b/selftests/test_report_content.py
@@ -342,7 +342,7 @@ def test_no_scenario_title_returns_empty_list(self):
 
 class TestFailedControlDisplay:
     def test_a_failing_control_displays_as_covered_failed(self):
-        entry = SimpleNamespace(coverage="covered", executed=True, failing=True)
+        entry = SimpleNamespace(coverage="covered", executed=True, failing=True, has_unproven=False)
         assert report_content.display_coverage(entry) == "covered_failed"
 
     def test_a_failing_control_uses_the_failed_style(self):
@@ -353,15 +353,19 @@ def test_a_failing_control_is_labelled_failed(self):
         assert report_content.COVERAGE_LABELS["covered_failed"] == "FAILED"
 
     def test_a_passing_control_still_displays_as_covered(self):
-        entry = SimpleNamespace(coverage="covered", executed=True, failing=False)
+        entry = SimpleNamespace(
+            coverage="covered", executed=True, failing=False, has_unproven=False
+        )
         assert report_content.display_coverage(entry) == "covered"
 
     def test_an_all_skipped_control_still_displays_as_not_executed(self):
-        entry = SimpleNamespace(coverage="covered", executed=False, failing=False)
+        entry = SimpleNamespace(
+            coverage="covered", executed=False, failing=False, has_unproven=False
+        )
         assert report_content.display_coverage(entry) == "covered_not_executed"
 
     def test_a_gap_is_unaffected(self):
-        entry = SimpleNamespace(coverage="gap", executed=False, failing=False)
+        entry = SimpleNamespace(coverage="gap", executed=False, failing=False, has_unproven=False)
         assert report_content.display_coverage(entry) == "gap"
 
     def test_every_coverage_value_has_a_style_and_a_label(self):
@@ -386,20 +390,60 @@ def test_a_real_mixed_control_is_counted_in_the_summary(self):
         assert rows["Covered, failing"] == "1"
         assert rows["Covered, executed and passing"] == "1"
 
+    def test_a_pass_beside_an_unproven_skip_displays_as_unproven(self):
+        """Otherwise the badge reads COVERED and the unproven scenario is invisible."""
+        matrix = matrix_from_statuses({"c1": ["passed", "unproven"]})
+        assert report_content.display_coverage(matrix.entries[0]) == "covered_unproven"
+
+    def test_an_all_unproven_control_prefers_unproven_over_not_run(self):
+        """Both are true; UNPROVEN says which kind of non-execution it was."""
+        matrix = matrix_from_statuses({"c1": ["unproven"]})
+        entry = matrix.entries[0]
+        assert entry.executed is False
+        assert report_content.display_coverage(entry) == "covered_unproven"
+
+    def test_a_failure_outranks_an_unproven_skip(self):
+        """A control that ran and failed is the louder fact, so FAILED wins."""
+        matrix = matrix_from_statuses({"c1": ["failed", "unproven"]})
+        assert report_content.display_coverage(matrix.entries[0]) == "covered_failed"
+
+    def test_an_unproven_control_is_counted_in_the_summary(self):
+        matrix = matrix_from_statuses({"c1": ["passed", "unproven"], "c2": ["passed"]})
+        rows = dict(report_content.traceability_summary_rows(matrix))
+        assert rows["Covered, not verified"] == "1"
+        assert rows["Covered, executed and passing"] == "1"
+
+    def test_every_display_value_has_a_label_and_a_style(self):
+        """A display value with no entry in either dict renders as a blank badge."""
+        assert set(report_content.COVERAGE_LABELS) == set(report_content.COVERAGE_STYLE_KEY)
+        for key in report_content.COVERAGE_STYLE_KEY.values():
+            assert report_content.outcome_style(key).label != "?"
+
 
 class TestTraceabilityWarnings:
+    @staticmethod
+    def _matrix(unexecuted=(), failing=(), unproven=()):
+        return SimpleNamespace(
+            covered_without_execution=list(unexecuted),
+            covered_with_failure=list(failing),
+            covered_with_unproven=list(unproven),
+        )
+
     def test_a_failing_control_produces_a_warning(self):
-        matrix = SimpleNamespace(covered_without_execution=[], covered_with_failure=["c1"])
-        warnings_out = report_content.traceability_warnings(matrix)
+        warnings_out = report_content.traceability_warnings(self._matrix(failing=["c1"]))
         assert any("did not pass" in w and "c1" in w for w in warnings_out)
 
-    def test_both_conditions_produce_two_warnings(self):
-        matrix = SimpleNamespace(covered_without_execution=["c2"], covered_with_failure=["c1"])
-        assert len(report_content.traceability_warnings(matrix)) == 2
+    def test_an_unproven_control_produces_a_warning(self):
+        warnings_out = report_content.traceability_warnings(self._matrix(unproven=["c1"]))
+        assert any("could not verify" in w and "c1" in w for w in warnings_out)
+
+    def test_each_condition_gets_its_own_line(self):
+        """Three independent conditions, so three lines: a reader needs to know which."""
+        matrix = self._matrix(unexecuted=["c2"], failing=["c1"], unproven=["c3"])
+        assert len(report_content.traceability_warnings(matrix)) == 3
 
     def test_a_clean_matrix_produces_none(self):
-        matrix = SimpleNamespace(covered_without_execution=[], covered_with_failure=[])
-        assert report_content.traceability_warnings(matrix) == []
+        assert report_content.traceability_warnings(self._matrix()) == []
 
 
 class TestRenderFailureMessage:
diff --git a/selftests/test_trace_cli.py b/selftests/test_trace_cli.py
index 23be53dd..551a10a9 100644
--- a/selftests/test_trace_cli.py
+++ b/selftests/test_trace_cli.py
@@ -366,7 +366,55 @@ def test_closing_line_reports_the_failing_count(self, tmp_path, capsys):
             )
         )
         captured = capsys.readouterr()
-        assert f"Wrote {out} (2 controls, 1 gaps, 1 failing)" in captured.out
+        assert f"Wrote {out} (2 controls, 1 gaps, 1 failing, 0 not verified)" in captured.out
+
+
+class TestCoveredButUnproven:
+    """The third way a covered control is not evidence: VIP could not check it."""
+
+    def _run(self, tmp_path, statuses):
+        results = tmp_path / "results.json"
+        results.write_text(
+            json.dumps(
+                {
+                    "schema_version": "1.0",
+                    "results": [
+                        {
+                            "nodeid": f"t.py::test_{i}",
+                            "outcome": "skipped" if status == "unproven" else status,
+                            "unproven": status == "unproven",
+                            "markers": ["control-c1"],
+                        }
+                        for i, status in enumerate(statuses)
+                    ],
+                }
+            ),
+            encoding="utf-8",
+        )
+        controls = _write_controls(
+            tmp_path, '[controls.c1]\ndescription = "d"\nverification = "automated"\n'
+        )
+        return results, controls
+
+    def test_a_pass_beside_an_unproven_skip_warns(self, tmp_path):
+        """Neither of the other two warnings fires here, which is why this one exists."""
+        results, controls = self._run(tmp_path, ["passed", "unproven"])
+        r = _run_trace(results, controls)
+        assert r.returncode == 0
+        assert "could not verify" in r.stderr
+        assert "c1" in r.stderr
+        assert "did not pass" not in r.stderr
+        assert "no scenario that ran" not in r.stderr
+
+    def test_the_json_summary_counts_it(self, tmp_path):
+        results, controls = self._run(tmp_path, ["passed", "unproven"])
+        payload = json.loads(_run_trace(results, controls, "--format", "json").stdout)
+        assert payload["summary"]["covered_unproven"] == 1
+        assert payload["covered_with_unproven"] == ["c1"]
+
+    def test_a_clean_run_does_not_warn(self, tmp_path):
+        results, controls = self._run(tmp_path, ["passed"])
+        assert "could not verify" not in _run_trace(results, controls).stderr
 
 
 class TestOutputFormatResolution:
diff --git a/selftests/test_traceability_matrix.py b/selftests/test_traceability_matrix.py
index 8d53f56b..f8391d75 100644
--- a/selftests/test_traceability_matrix.py
+++ b/selftests/test_traceability_matrix.py
@@ -271,17 +271,60 @@ def test_an_unproven_control_is_not_run_rather_than_failed(self):
         assert matrix.covered_without_execution == ["c1"]
         assert matrix.covered_with_failure == []
 
-    def test_a_pass_beside_an_unproven_skip_reads_as_covered(self):
-        """The known cost of treating unproven as non-executing.
-
-        One scenario proved something and one could not be checked, and the
-        control's own summary buckets say only that it ran and passed. The
-        unproven scenario stays visible in the matrix's per-scenario status
-        column and still fails the run with exit code 6, so the signal is not
-        lost, only absent from the control-level rollup.
+    def test_a_pass_beside_an_unproven_skip_is_not_failing(self):
+        """Treating unproven as non-executing means it cannot mark a control failing.
+
+        ``covered_with_unproven`` is what surfaces this control instead; see
+        ``TestUnprovenControls``.
         """
         matrix = matrix_from_statuses(statuses={"c1": ["passed", "unproven"]})
         entry = matrix.entries[0]
         assert entry.executed is True
         assert entry.failing is False
         assert [m.status for m in entry.matches] == ["passed", "unproven"]
+
+
+class TestUnprovenControls:
+    """A control VIP was asked to check and could not is its own fact.
+
+    Coverage, execution and outcome already fail to describe it: an unproven
+    skip runs no assertion, so it is not executed and not failing, and those
+    two alone leave a control with one pass and one unproven scenario reading
+    as fully evidenced.
+    """
+
+    def test_an_unproven_scenario_marks_the_control(self):
+        matrix = matrix_from_statuses(statuses={"c1": ["unproven"]})
+        entry = matrix.entries[0]
+        assert entry.has_unproven is True
+        assert matrix.covered_with_unproven == ["c1"]
+
+    def test_a_pass_beside_an_unproven_skip_is_surfaced(self):
+        """The case the third bucket exists for: the other two say nothing."""
+        matrix = matrix_from_statuses(statuses={"c1": ["passed", "unproven"]})
+        assert matrix.covered_with_unproven == ["c1"]
+        assert matrix.covered_without_execution == []
+        assert matrix.covered_with_failure == []
+
+    def test_the_three_lists_overlap_rather_than_partition(self):
+        """Not disjoint buckets, and deliberately so.
+
+        The matrix keeps coverage, execution and outcome as separate facts and
+        flattens them only for display. An unproven-only control did not run
+        *and* could not be checked, so it belongs in both lists; forcing a
+        single bucket would drop one of the two true statements.
+        """
+        matrix = matrix_from_statuses(statuses={"c1": ["unproven"], "c2": ["failed", "unproven"]})
+        assert matrix.covered_without_execution == ["c1"]
+        assert matrix.covered_with_failure == ["c2"]
+        assert matrix.covered_with_unproven == ["c1", "c2"]
+
+    def test_a_plain_skip_is_not_unproven(self):
+        """`skipped` says there was nothing to check; `unproven` says VIP could not."""
+        matrix = matrix_from_statuses(statuses={"c1": ["skipped"], "c2": ["na_version"]})
+        assert matrix.covered_with_unproven == []
+
+    def test_a_passing_control_is_not_unproven(self):
+        matrix = matrix_from_statuses(statuses={"c1": ["passed"]})
+        assert matrix.entries[0].has_unproven is False
+        assert matrix.covered_with_unproven == []
diff --git a/selftests/test_traceability_render.py b/selftests/test_traceability_render.py
index b2091b02..2f3e4fd5 100644
--- a/selftests/test_traceability_render.py
+++ b/selftests/test_traceability_render.py
@@ -82,7 +82,7 @@ def test_csv_is_byte_identical_across_invocations():
 
 def test_json_carries_provenance_and_schema_version():
     payload = json.loads(render_json(_matrix()))
-    assert payload["schema_version"] == "1.0"
+    assert payload["schema_version"] == "1.1"
     assert payload["provenance"]["vip_version"] == "2026.8.3"
     assert payload["summary"]["gaps"] == 0
     assert payload["summary"]["covered"] == 1
@@ -96,6 +96,16 @@ def test_json_summary_counts_failing_controls():
     assert summary["covered_and_executed"] == 2
 
 
+def test_json_summary_counts_unproven_controls():
+    """Additive in 1.1: a control VIP could not check is neither a pass nor a failure."""
+    matrix = matrix_from_statuses(statuses={"c1": ["passed", "unproven"], "c2": ["passed"]})
+    payload = json.loads(render_json(matrix))
+    assert payload["summary"]["covered_unproven"] == 1
+    assert payload["summary"]["covered_failed"] == 0
+    assert payload["summary"]["covered_and_executed"] == 2
+    assert payload["covered_with_unproven"] == ["c1"]
+
+
 def test_json_is_byte_identical_across_invocations():
     assert render_json(_matrix()) == render_json(_matrix())
 
diff --git a/src/vip/cli.py b/src/vip/cli.py
index 767cec8d..f741dfd3 100644
--- a/src/vip/cli.py
+++ b/src/vip/cli.py
@@ -1875,6 +1875,20 @@ def run_trace(args: argparse.Namespace) -> None:
             file=sys.stderr,
         )
 
+    # The third condition, and the only one that catches a control whose
+    # scenarios ran and passed while part of the control went unchecked. The
+    # two warnings above stay silent on that case, because an unproven skip is
+    # neither an execution nor a failure.
+    unproven = matrix.covered_with_unproven
+    if unproven:
+        print(
+            f"Warning: {len(unproven)} covered control(s) had a scenario VIP "
+            f"could not verify: {', '.join(unproven)}. An unproven check was asked "
+            "for and could not be run, which is not the same as one that found "
+            "nothing to test.",
+            file=sys.stderr,
+        )
+
     if out is None:
         sys.stdout.write(rendered)
         return
@@ -1892,7 +1906,8 @@ def run_trace(args: argparse.Namespace) -> None:
         sys.exit(1)
     print(
         f"Wrote {out} ({len(matrix.entries)} controls, {matrix.gap_count} gaps, "
-        f"{len(matrix.covered_with_failure)} failing)"
+        f"{len(matrix.covered_with_failure)} failing, "
+        f"{len(matrix.covered_with_unproven)} not verified)"
     )
 
 
diff --git a/src/vip/report_content.py b/src/vip/report_content.py
index 407a156a..51203ce3 100644
--- a/src/vip/report_content.py
+++ b/src/vip/report_content.py
@@ -507,6 +507,10 @@ def summary_status(data: ReportData) -> str:
     "covered": "passed",
     "covered_not_executed": "na_version",
     "covered_failed": "failed",
+    # Reuses the scenario-level "unproven" style, so a control that could not
+    # be verified is painted the same amber-red as the scenario rows beneath
+    # it and no new color enters the palette.
+    "covered_unproven": "unproven",
     "gap": "failed",
     "not_automatable": "skipped",
 }
@@ -515,6 +519,7 @@ def summary_status(data: ReportData) -> str:
     "covered": "COVERED",
     "covered_not_executed": "NOT RUN",
     "covered_failed": "FAILED",
+    "covered_unproven": "UNPROVEN",
     "gap": "GAP",
     "not_automatable": "N/A (manual)",
 }
@@ -536,8 +541,8 @@ class ControlRow:
     rank or validate the value -- ``risk = "banana"`` renders as "banana".
     """
     coverage: str
-    """"covered" | "covered_not_executed" | "covered_failed" | "gap" |
-    "not_automatable".
+    """"covered" | "covered_not_executed" | "covered_failed" |
+    "covered_unproven" | "gap" | "not_automatable".
 
     Distinct from ``ControlEntry.coverage``, which has no
     ``covered_not_executed`` or ``covered_failed`` value: the matrix keeps
@@ -550,10 +555,22 @@ class ControlRow:
 
 
 def display_coverage(entry) -> str:  # noqa: ANN001 - vip.traceability.ControlEntry
-    """Flatten coverage, execution and outcome into the one value the report shows."""
-    if entry.coverage == "covered" and entry.failing:
+    """Flatten coverage, execution and outcome into the one value the report shows.
+
+    Ordered by how loudly each fact demotes the control, because the matrix
+    keeps them as overlapping facts and one column can show only one. A
+    control that ran and failed is the strongest claim against it, so FAILED
+    wins over an unproven scenario on the same control. UNPROVEN then outranks
+    NOT RUN, which is the vaguer of the two: an all-unproven control is both,
+    and "VIP could not check this" tells the reader more than "nothing ran".
+    """
+    if entry.coverage != "covered":
+        return entry.coverage
+    if entry.failing:
         return "covered_failed"
-    if entry.coverage == "covered" and not entry.executed:
+    if entry.has_unproven:
+        return "covered_unproven"
+    if not entry.executed:
         return "covered_not_executed"
     return entry.coverage
 
@@ -598,6 +615,7 @@ def traceability_summary_rows(matrix) -> list[tuple[str, str]]:  # noqa: ANN001
         ("Covered, executed and passing", str(counts.get("covered", 0))),
         ("Covered, not executed", str(counts.get("covered_not_executed", 0))),
         ("Covered, failing", str(counts.get("covered_failed", 0))),
+        ("Covered, not verified", str(counts.get("covered_unproven", 0))),
         ("Gaps", str(counts.get("gap", 0))),
         ("Not automatable", str(counts.get("not_automatable", 0))),
     ]
@@ -614,7 +632,9 @@ def traceability_summary_rows(matrix) -> list[tuple[str, str]]:  # noqa: ANN001
     "a GAP may instead belong to a product this run did not test, since those "
     "scenarios are excluded from the run and reach no result at all. "
     "A control shown as FAILED has a tagged scenario that ran and did not pass, "
-    "so the control is not evidenced by this run. This "
+    "so the control is not evidenced by this run. A control shown as UNPROVEN "
+    "has a tagged scenario VIP was asked to run and could not, which is not the "
+    "same as a check that found nothing to test. This "
     "section evidences the controls chosen for automation. It is not an "
     "attestation of regulatory compliance."
 )
@@ -628,9 +648,11 @@ def traceability_summary_rows(matrix) -> list[tuple[str, str]]:  # noqa: ANN001
 def traceability_warnings(matrix) -> list[str]:  # noqa: ANN001
     """Lines naming controls that look covered but are not evidence.
 
-    Two independent conditions, so two lines rather than one combined
-    sentence: a control can be counted as covered because nothing ran, or
-    because what ran did not pass, and a reader needs to know which.
+    Three independent conditions, so three lines rather than one combined
+    sentence: a control can be counted as covered because nothing ran, because
+    what ran did not pass, or because VIP was asked to check it and could not,
+    and a reader needs to know which. The conditions overlap, so one control
+    can appear on more than one line.
     """
     lines = []
     failing = matrix.covered_with_failure
@@ -645,4 +667,10 @@ def traceability_warnings(matrix) -> list[str]:  # noqa: ANN001
             f"{pluralize(len(unexecuted), 'control')} counted as covered but had no "
             f"scenario that ran: {', '.join(unexecuted)}."
         )
+    unproven = matrix.covered_with_unproven
+    if unproven:
+        lines.append(
+            f"{pluralize(len(unproven), 'control')} counted as covered but had a "
+            f"scenario VIP could not verify: {', '.join(unproven)}."
+        )
     return lines
diff --git a/src/vip/traceability.py b/src/vip/traceability.py
index 0f5e1d02..524d00ff 100644
--- a/src/vip/traceability.py
+++ b/src/vip/traceability.py
@@ -23,7 +23,7 @@
     import tomli as tomllib
 
 # Matrix output schema, versioned independently of results.json.
-MATRIX_SCHEMA_VERSION = "1.0"
+MATRIX_SCHEMA_VERSION = "1.1"
 
 VERIFICATION_VALUES = frozenset({"automated", "manual", "procedural"})
 
@@ -257,6 +257,22 @@ def failing(self) -> bool:
             m.status not in NON_EXECUTING_STATUSES and m.status != "passed" for m in self.matches
         )
 
+    @property
+    def has_unproven(self) -> bool:
+        """Whether any tagged scenario was a check VIP could not run.
+
+        The fourth fact, and the one the other three cannot state. An
+        ``unproven`` skip is non-executing, so ``executed`` and ``failing``
+        are both false for it, and a control with one passing scenario beside
+        one unproven scenario reads as fully evidenced from those two alone.
+
+        Narrower than ``not executed``: a plain skip says there was nothing to
+        check here, and ``na_version`` says a version gate excluded it. This
+        says VIP was asked to check the control and could not, which is the
+        distinction ``vip.attest.unproven`` exists to record.
+        """
+        return any(m.status == "unproven" for m in self.matches)
+
 
 @dataclass
 class TraceabilityMatrix:
@@ -299,6 +315,21 @@ def covered_with_failure(self) -> list[str]:
         """
         return [e.control.control_id for e in self.entries if e.coverage == "covered" and e.failing]
 
+    @property
+    def covered_with_unproven(self) -> list[str]:
+        """Control ids that are covered but have a scenario VIP could not run.
+
+        The third of these lists, and the only one that catches a control
+        whose scenarios ran, passed, and still left part of the control
+        unchecked. Deliberately not disjoint from the other two: an
+        unproven-only control is both not executed and not verifiable, and it
+        appears in both lists, because each statement is true and a reader
+        needs both.
+        """
+        return [
+            e.control.control_id for e in self.entries if e.coverage == "covered" and e.has_unproven
+        ]
+
 
 def _provenance(
     data: ReportData,
@@ -554,8 +585,14 @@ def render_json(matrix: TraceabilityMatrix) -> str:
             # Covered and executed, but not a success. The third way a green
             # matrix can mislead, after "nothing is tagged" and "nothing ran".
             "covered_failed": len(matrix.covered_with_failure),
+            # The fourth: a control carrying a check VIP was asked to run and
+            # could not. Overlaps the two counts above rather than partitioning
+            # them -- an unproven-only control is counted here and in
+            # covered_not_executed -- so these five do not sum to `total`.
+            "covered_unproven": len(matrix.covered_with_unproven),
         },
         "covered_without_execution": matrix.covered_without_execution,
+        "covered_with_unproven": matrix.covered_with_unproven,
         "unrecognized_tags": matrix.unrecognized_tags,
         "controls": [
             {

From 587bdca7958d6c7f179eb494e729261e28bdaf5e Mon Sep 17 00:00:00 2001
From: Brian Deitte 
Date: Mon, 31 Aug 2026 21:56:11 -0500
Subject: [PATCH 105/106] test(clients): cover slashless paths and the timeout
 in unauthenticated_status

Neither behavior had a test: every existing case passed a slash-prefixed
path and none inspected the timeout the ad-hoc client is built with. Both
reverted cleanly against the old suite, so a regression in the customer-
overridable Part 11 endpoints or a return to httpx's default timeout would
have gone unnoticed.
---
 selftests/test_connect_audit_client.py | 28 ++++++++++++++++++++++++++
 1 file changed, 28 insertions(+)

diff --git a/selftests/test_connect_audit_client.py b/selftests/test_connect_audit_client.py
index 79f5f2cf..5c3c8f35 100644
--- a/selftests/test_connect_audit_client.py
+++ b/selftests/test_connect_audit_client.py
@@ -113,6 +113,34 @@ def test_unauthenticated_status_sends_no_credentials(recording_client):
     assert "SECRET_KEY" not in repr(kwargs.get("headers", {}))
 
 
+def test_unauthenticated_status_accepts_a_path_without_a_leading_slash(recording_client):
+    """The two endpoints are customer-overridable in the Part 11 example's conftest.
+
+    Someone editing that override writes `__api__/v1/users` as readily as
+    `/__api__/v1/users`. Without normalization the two concatenate into
+    `https://connect.example.com__api__/v1/users`, which is a different host,
+    so the scenario fails on a deployment that is fine.
+    """
+    c = recording_client(api_key="k")
+    assert c.unauthenticated_status("__api__/v1/users") == 401
+    assert _RecordingClient.instances[0].requested == (
+        "https://connect.example.com/__api__/v1/users"
+    )
+
+
+def test_unauthenticated_status_uses_the_configured_timeout(recording_client):
+    """The ad-hoc client must not fall back to httpx's own default.
+
+    BaseClient scales its default timeout, and a caller can override it; a probe
+    that ignores both hangs for a different length of time than every other
+    request the same client makes.
+    """
+    c = recording_client(api_key="k", timeout=7.5)
+    c.unauthenticated_status("/__api__/v1/users")
+    assert _RecordingClient.instances[0].kwargs["timeout"] == 7.5
+    assert _RecordingClient.instances[0].kwargs["timeout"] == c._timeout
+
+
 def test_unauthenticated_status_pins_trust_env_and_keeps_env_ca(recording_client):
     """trust_env=False also disables SSL_CERT_FILE; verify_with_env_ca restores it.
 

From 24350a1abc9e961a47b5469e4ca3c15181e349e8 Mon Sep 17 00:00:00 2001
From: Brian Deitte 
Date: Fri, 4 Sep 2026 10:32:28 -0500
Subject: [PATCH 106/106] docs: tighten wording in the new traceability prose

Replace the metaphorical "carry" verb with the literal one (has,
includes, keeps, passes through, records) and split semicolon joins
into separate sentences across the docs and examples added for
control tagging and vip trace.
---
 AGENTS.md                                     |  4 +-
 README.md                                     |  6 +--
 docs/reporting.md                             | 54 +++++++++----------
 docs/test-architecture.md                     |  4 +-
 examples/21CFR_part11_validation/README.md    |  6 +--
 .../VALIDATION-PACKAGE.md                     | 10 ++--
 website/src/pages/getting-started.astro       |  8 +--
 7 files changed, 46 insertions(+), 46 deletions(-)

diff --git a/AGENTS.md b/AGENTS.md
index 549cdbf8..d71b6dde 100644
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -120,7 +120,7 @@ Key rules:
 -   Step function names should be descriptive. Use `target_fixture` to pass state between steps.
 -   Tests must be non-destructive. Tag created content with `_vip_test` and clean it up in a final `then` step.
 -   Use version gating for version-specific features: `@pytest.mark.min_version(product="connect", version="2024.09.0")`
--   Say what a skip *means*. `vip.attest.not_applicable(reason)` says there was nothing to check here (product not configured, tier lacks the feature) and keeps the run green. `vip.attest.unproven(reason)` says VIP was asked to check something and could not, which fails the run with exit code 6 unless `--allow-unproven` is passed. A bare `pytest.skip()` still behaves like `not_applicable`; prefer the explicit helper so the next reader does not have to infer which one you meant.
+-   Say what a skip *means*. `vip.attest.not_applicable(reason)` says there was nothing to check here (product not configured, tier lacks the feature) and keeps the run green. `vip.attest.unproven(reason)` says VIP was asked to check something and could not, which fails the run with exit code 6 unless `--allow-unproven` is passed. A bare `pytest.skip()` still behaves like `not_applicable`. Prefer the explicit helper so the next reader does not have to infer which one you meant.
 
 ## Four-layer test architecture
 
@@ -161,7 +161,7 @@ Key principles:
 | `src/vip/report_html.py` | HTML backend: renders `report_content` into the fragments `index.qmd`/`details.qmd` display |
 | `src/vip/report_typst.py` | Typst backend: renders the same content as Typst markup for `report/vip-report.qmd` → `_output/vip-report.pdf`; every dynamic value passes through `_lit` (Typst-injection escaping) |
 | `src/vip/attribution.py` | Collects the `execution` block written into `results.json` — hostname, git (commit/branch/dirty/remote, with any userinfo redacted from the remote URL), CI (provider/run_id/run_url/job for GitHub Actions, GitLab CI, Jenkins), and `performed_by` (`VIP_PERFORMED_BY` → CI actor → local login, each tagged with its `source`; FDA's Computer Software Assurance guidance asks the record to say who performed the testing). Every probe degrades to `None` rather than failing or warning; omitted entirely with `--vip-no-attribution`, which is also the opt-out for writing an operator identity into an archived artifact. `report_content.provenance_rows` renders the whole block into both report editions — before that it reached only `vip trace --format json` |
-| `src/vip/traceability.py` | Control list model (`ControlSpec`, loaded from `controls.toml`) and `build_traceability_matrix`, which joins controls against `@control-`-tagged scenarios in a loaded `results.json`; CSV (`render_csv`, apostrophe-neutralizes formula-leading cells) and JSON (`render_json`) renderers; `verify_results_checksum` (per-line sidecar parse, case-insensitive digest, matched by recorded filename) and the schema-version gate (`check_results_schema` hard-errors on an unknown major, unlike `reporting.load_results`, which only warns). `ControlEntry.executed` / `TraceabilityMatrix.covered_without_execution` separate "a scenario is tagged" from "a scenario ran" — a scenario that runs and skips itself (absent endpoint, no data, version gate) still carries its control tag, so coverage alone would report a green matrix for a run that verified nothing. `ControlEntry.failing` and `TraceabilityMatrix.covered_with_failure` are the third fact, separating "a scenario ran" from "a scenario passed", and any failing scenario demotes the control's badge rather than only an all-failed one. An *unconfigured* product is the opposite case and is often misdescribed: `plugin.pytest_collection_modifyitems` deselects those scenarios rather than skipping them, so they never reach `results.json` and a control tagged only by them reports as a `gap`, not as covered-not-executed. Powers `vip trace` |
+| `src/vip/traceability.py` | Control list model (`ControlSpec`, loaded from `controls.toml`) and `build_traceability_matrix`, which joins controls against `@control-`-tagged scenarios in a loaded `results.json`; CSV (`render_csv`, apostrophe-neutralizes formula-leading cells) and JSON (`render_json`) renderers; `verify_results_checksum` (per-line sidecar parse, case-insensitive digest, matched by recorded filename) and the schema-version gate (`check_results_schema` hard-errors on an unknown major, unlike `reporting.load_results`, which only warns). `ControlEntry.executed` / `TraceabilityMatrix.covered_without_execution` separate "a scenario is tagged" from "a scenario ran" — a scenario that runs and skips itself (absent endpoint, no data, version gate) still keeps its control tag, so coverage alone would report a green matrix for a run that verified nothing. `ControlEntry.failing` and `TraceabilityMatrix.covered_with_failure` are the third fact, separating "a scenario ran" from "a scenario passed", and any failing scenario demotes the control's badge rather than only an all-failed one. An *unconfigured* product is the opposite case and is often misdescribed: `plugin.pytest_collection_modifyitems` deselects those scenarios rather than skipping them, so they never reach `results.json` and a control tagged only by them reports as a `gap`, not as covered-not-executed. Powers `vip trace` |
 | `src/vip/clients/connect.py` | httpx client for Connect API |
 | `src/vip/clients/workbench.py` | httpx client for Workbench API; `quit_vip_sessions` warns loudly (not silently) when a VIP session persists after all retries. `session_owner` / `is_vip_session_for_owner` decide whether a VIP session belongs to the sweeping worker — see "Session ownership" below |
 | `src/vip/clients/packagemanager.py` | httpx client for Package Manager API |
diff --git a/README.md b/README.md
index ea37f999..63b5abb0 100644
--- a/README.md
+++ b/README.md
@@ -129,7 +129,7 @@ vip verify --config vip.toml --extensions ./my-tests
 | `21cfr-part11-validation` | Regulatory control tagging plus a `controls.toml` for `vip trace`. |
 
 Every scaffolded directory runs as-is against your deployment, and each one
-carries an `AGENTS.md` describing the extension contract, so a coding agent
+includes an `AGENTS.md` describing the extension contract, so a coding agent
 picking up the directory knows the auto-skip rules and which fixtures and
 markers it may use.
 
@@ -145,7 +145,7 @@ vip trace --results report/results.json --controls ./my-tests/controls.toml
 ```
 
 `vip report --controls` renders the same matrix into the HTML report and the
-archivable PDF, so the artifact you hand an auditor carries the join from a
+archivable PDF, so the artifact you hand an auditor includes the join from a
 control to its evidence rather than only a list of passing tests.
 
 It ships with `VALIDATION-PACKAGE.md`, which is the honest version of what this
@@ -171,7 +171,7 @@ vip verify --format json,junit,sarif
 Skip messages in the JUnit and SARIF output carry the actual skip reason
 instead of a generic "skipped" label. A check VIP was asked to run but could
 not -- a configured product whose authentication never completed, say -- is
-reported as **unproven** rather than as an ordinary skip: it carries an
+reported as **unproven** rather than as an ordinary skip: it has an
 `UNPROVEN:` prefix in JUnit, SARIF level `warning`, and exits **6** so a
 pipeline can tell "the deployment is broken" (exit 1) from "the deployment
 could not be checked" (exit 6). Pass `--allow-unproven` to exit 0 anyway. `results.json` also records provenance
diff --git a/docs/reporting.md b/docs/reporting.md
index 19fd73ac..f8d6203e 100644
--- a/docs/reporting.md
+++ b/docs/reporting.md
@@ -63,7 +63,7 @@ full run that would produce nothing. `--ci` is a preset that turns on
 ```
 
 - `schema_version` -- versioned independently of VIP itself. The minor number bumps
-  for an additive change (a new field); the major number bumps for a removal, a
+  for an additive change (a new field). The major number bumps for a removal, a
   rename, or a change in the meaning of an existing field. A file with no
   `schema_version` at all predates versioning and is treated as pre-1.0.
 - `started_at` / `finished_at` -- per-test UTC ISO 8601 timestamps, covering the call
@@ -79,16 +79,16 @@ full run that would produce nothing. `--ci` is a preset that turns on
   operator identity in an archived artifact.
 - `execution.performed_by` -- the operator the run is attributable to, and `source`
   says where that value came from. Set `VIP_PERFORMED_BY` to name the person
-  accountable for the run (`source: "explicit"`); otherwise VIP reads the CI
+  accountable for the run (`source: "explicit"`). Otherwise VIP reads the CI
   system's own actor (`GITHUB_ACTOR`, `GITLAB_USER_LOGIN`, `BUILD_USER_ID`), then
-  falls back to the local login (`source: "login"`). `source` travels with the
+  falls back to the local login (`source: "login"`). `source` stays attached to the
   value because a reader seeing a service-account name needs to know whether a
   human typed it. FDA's Computer Software Assurance guidance asks the record of an
-  assurance activity to carry who performed the testing alongside the date, which
-  is why this exists; the rest of the block identifies a machine, not a person.
+  assurance activity to state who performed the testing alongside the date, which
+  is why this exists. The rest of the block identifies a machine, not a person.
 
 Both report editions render the attribution too, so the archived artifact
-carries it rather than only the machine-readable output. They render five of
+includes it rather than only the machine-readable output. They render five of
 these fields -- `performed_by` (qualified by its `source` unless that source is
 `explicit`), `hostname`, `git.commit` (flagged when `dirty`), `git.branch`, and
 `ci.run_url` or `ci.run_id`. `git.remote`, `ci.provider` and `ci.job` stay in
@@ -101,12 +101,12 @@ describe: they are properties of the machine that ran `vip verify` -- the VIP
 runner -- not the Connect/Workbench/Package Manager deployment under test. The
 `products` table is what identifies the system under test (its name, URL, version,
 and whether it was configured for this run). Don't read the runner's platform string
-as evidence about the deployment; it isn't.
+as evidence about the deployment. It isn't.
 
 ### Schema compatibility policy
 
 An unknown schema minor is accepted (fields you don't recognize are additive and
-safe to ignore); an unknown schema major is refused. The two consumers of
+safe to ignore), but an unknown schema major is refused. The two consumers of
 `schema_version` apply this policy differently on purpose:
 
 - `vip.reporting.load_results` -- used by `vip report` and the Quarto notebooks
@@ -174,15 +174,15 @@ responsibility = "customer"
 notes = "Evidenced by training records in your QMS. No automated test can establish this."
 ```
 
-`description` is required; `verification` defaults to `"automated"` and must be one
+`description` is required. `verification` defaults to `"automated"` and must be one
 of `"automated"`, `"manual"`, or `"procedural"`. VIP is regulation-agnostic: it
-carries `reference`, `risk`, `responsibility`, and `notes` through to the output
+passes `reference`, `risk`, `responsibility`, and `notes` through to the output
 verbatim without interpreting them. The `[controls.]` key is the id a scenario
 references with `@control-` (with the `control-` tag prefix stripped).
 
 Those six keys plus `extra` are the whole recognised set, and any other key is an
 error. Rejecting rather than ignoring is what catches a misspelled `referance`
-before it vanishes from the matrix a reviewer reads. Carry your own fields in an
+before it vanishes from the matrix a reviewer reads. Put your own fields in an
 `extra` table:
 
 ``` toml
@@ -195,7 +195,7 @@ Its keys must not start with a character a spreadsheet evaluates as a formula,
 since a key becomes a CSV header cell and TOML permits a quoted key like
 `"=HYPERLINK(...)"`. Its values must be strings, must not collide with an
 existing column name, and
-are carried through untouched: each becomes a trailing CSV column (appended after
+are passed through untouched: each becomes a trailing CSV column (appended after
 the fixed set, so `CSV_COLUMNS` stays an identical leading prefix across every
 customer's export) and an `extra` object per control in the JSON. Neither report
 edition renders them, because the table has no width for a variable number of
@@ -210,9 +210,9 @@ A control's row in the matrix gets one of three `coverage` values:
   skipped scenario still counts as covered, and its `skipped` status is reported
   alongside. See "Covered is not the same as executed" below, which matters more
   than it sounds like it should.
-- `gap` -- no scenario carries the tag, and `verification = "automated"` (the
+- `gap` -- no scenario has the tag, and `verification = "automated"` (the
   default). This is the one that should worry you.
-- `not_automatable` -- no scenario carries the tag, but `verification` is
+- `not_automatable` -- no scenario has the tag, but `verification` is
   `"manual"` or `"procedural"`. This is *not* a gap. A control satisfied by a
   personnel training record, a physical procedure, or a signature-manifestation
   requirement that Posit Team's platform doesn't implement has no automated
@@ -239,15 +239,15 @@ The rendered report displays coverage as `COVERED`, `FAILED`, `UNPROVEN`,
 
 A control can satisfy more than one of these at once, and the display column
 shows only the loudest. The order is `FAILED`, then `UNPROVEN`, then `NOT RUN`:
-a control that ran and failed is the strongest claim against it, and an
-all-unproven control is also not executed, where `UNPROVEN` is the more
-specific of the two. Read the per-scenario `status` column for the rest.
+a control that ran and failed is the strongest claim against it. An
+all-unproven control is also not executed, but `UNPROVEN` is the more
+specific of the two, so it takes priority. Read the per-scenario `status` column for the rest.
 
 ### Covered is not the same as executed
 
 A scenario can run and skip itself -- the endpoint it probes is absent from this
 deployment, there is no data to inspect, a version gate excludes it -- and it
-still carries its control tag into the results file. A control tagged only by
+still keeps its control tag in the results file. A control tagged only by
 such scenarios is therefore `covered`, and a matrix can read `covered: 3,
 gaps: 0` while nothing was verified at all. That is the most misleading thing
 this export can do, so it is reported three ways rather than left implicit:
@@ -265,7 +265,7 @@ this export can do, so it is reported three ways rather than left implicit:
   `Covered, executed and passing`, `Covered, not executed`, `Covered, failing`
   and `Covered, not verified`, so each control is counted once. Do not expect
   the report table and the JSON `summary` to add up the same way.
-- The JSON carries a `covered_without_execution` list of control ids and a
+- The JSON has a `covered_without_execution` list of control ids and a
   `covered_with_unproven` list.
 
 A version-gated scenario (`na_version`) counts as not executed too, for the same
@@ -282,7 +282,7 @@ them has nothing to join against, so it reports as a `gap`. An underconfigured
 run therefore understates coverage rather than overstating it, which is the
 safer direction, but a reader who takes the gap at face value concludes the
 suite lacks a check it has. The `products` block records what was actually
-under test; read it alongside the gaps.
+under test. Read it alongside the gaps.
 
 Read `gaps: 0` together with `covered_not_executed` and `covered_unproven`.
 Zero gaps and a non-zero `covered_not_executed` means the controls are mapped
@@ -303,10 +303,10 @@ to write to a file instead. With `--output` and no `--format`, the format is tak
 from the file extension, so `--output matrix.json` writes JSON. An explicit
 `--format` always wins and warns when it disagrees with the extension.
 
-Both formats carry the results digest. CSV repeats `generated_at`, `vip_version`,
-`results_sha256` and `exit_status` on every row, which is enough to tie the
-archived spreadsheet back to the exact `results.json` it came from. The full
-provenance block -- the products and versions under test, the runner host, the CI
+Both formats include the results digest. CSV repeats `generated_at`, `vip_version`,
+`results_sha256` and `exit_status` on every row, which is enough to match the
+archived spreadsheet to the exact `results.json` it came from. The full
+provenance block -- the products and versions under test, the runner host, and the CI
 run -- is JSON only, because it does not flatten into columns. CSV is the more portable format for spreadsheet tools,
 but it alters what a non-Excel reader sees: any cell whose value begins with
 `= + - @` or a leading tab/carriage-return/newline is apostrophe-prefixed
@@ -316,13 +316,13 @@ fidelity to the underlying value matters more than spreadsheet safety.
 
 Control ids become pytest marker names, so they may use only letters, digits,
 `-`, `.` and `_`. A `:` or `(` in an id (`11.10(a)`, `iso:27001`) truncates the
-name pytest registers, which aborts collection under `--strict-markers`; VIP
+name pytest registers, which aborts collection under `--strict-markers`. VIP
 warns and skips registering such a tag. Write `11-10-a` instead.
 
 ### In the rendered report
 
 `vip report --controls PATH` adds a Compliance Traceability section to both the
-HTML report and the PDF, carrying the summary counts, the per-control coverage,
+HTML report and the PDF, with the summary counts, the per-control coverage,
 and the scenario and timestamp evidencing each one. Without `--controls` there is
 no section and nothing changes, which is the case for every run that has no
 control list.
@@ -338,7 +338,7 @@ compliance section nobody asked for, built from a stale list. Rendering the
 report documents directly with `quarto render` therefore needs `VIP_CONTROLS`
 set by hand.
 
-The section repeats the same caveat the CSV and JSON exports carry, because the
+The section repeats the same caveat the CSV and JSON exports state, because the
 report is the artifact that gets archived and handed on: coverage records that a
 scenario is tagged, a control shown as NOT RUN has a tagged scenario that ran
 and skipped itself, and a control shown as UNPROVEN has one VIP was asked to run
diff --git a/docs/test-architecture.md b/docs/test-architecture.md
index 5d75c4a4..752f9f61 100644
--- a/docs/test-architecture.md
+++ b/docs/test-architecture.md
@@ -125,7 +125,7 @@ registering the truncated name. Write `@control-11-10-a`.
 Control tags flow into `results.json` only, as entries in a test's `markers` list.
 They do not appear in the `junit.xml` or `results.sarif` outputs produced by
 `--format` — those formats predate control tagging and were not extended to
-carry it. If you need traceability evidence in CI artifacts beyond `results.json`,
+include it. If you need traceability evidence in CI artifacts beyond `results.json`,
 consume it via `vip trace` (see `docs/reporting.md`), not by expecting it in JUnit
 or SARIF.
 
@@ -197,7 +197,7 @@ Two failure modes to watch for, both of which have bitten this suite:
 
 **Skip and unproven are not the same statement.** A skip says "there was nothing here to check, and the run is still complete". Unproven says "this was supposed to be checked and was not, so treat the run as incomplete". Collapsing the two is how `vip verify` came to exit 0 against a configured product whose every test had silently fallen away (#596): to anyone reading the report, "we did not look" was indistinguishable from "we looked and it was fine".
 
-An unproven result carries through the whole pipeline -- its own `UNPROVEN` badge in the HTML report, an `UNPROVEN:`-prefixed message in JUnit, SARIF level `warning`, and exit code 6 from the run itself. `--allow-unproven` restores the old behaviour for pipelines that need it. Reach for `attest.unproven` whenever a *configured* capability goes unverified; reach for `attest.not_applicable` when skipping is the correct and final answer for this deployment. When in doubt, ask which one the person reading the report would want to be told.
+An unproven result shows up throughout the pipeline -- its own `UNPROVEN` badge in the HTML report, an `UNPROVEN:`-prefixed message in JUnit, SARIF level `warning`, and exit code 6 from the run itself. `--allow-unproven` restores the old behaviour for pipelines that need it. Reach for `attest.unproven` whenever a *configured* capability goes unverified. Reach for `attest.not_applicable` when skipping is the correct and final answer for this deployment. When in doubt, ask which one the person reading the report would want to be told.
 
 Skips carry the same burden of accuracy as failures. A skip reason states *why* there was nothing to verify, so it must be true: `test_repos.py` used to report "package not available — repo may not be synced yet" after probing only the first repo whose name matched, when a synced mirror sitting beside it served the package fine. Probe every candidate before concluding anything, and name all of them in the reason.
 
diff --git a/examples/21CFR_part11_validation/README.md b/examples/21CFR_part11_validation/README.md
index fd59dad3..ba0f2312 100644
--- a/examples/21CFR_part11_validation/README.md
+++ b/examples/21CFR_part11_validation/README.md
@@ -66,7 +66,7 @@ unauthenticated caller does not on its own evidence that access is limited to
 authorised individuals, because a deployment that refuses everybody passes that
 half too.
 
-Package Manager carries the reproducibility control because a dated snapshot
+Package Manager has the reproducibility control because a dated snapshot
 URL is what lets you rebuild the package set an analysis ran against. Set
 `validated_repo_name` and `validated_snapshot` in `conftest.py` to a repository
 and date your deployment actually covers. An absent snapshot returns 404, which
@@ -91,7 +91,7 @@ derived marker feeds the HTML report's per-feature grouping and the generated
 test catalog and feature matrix, while the product markers themselves
 separately drive auto-skip when a product is not configured.
 
-`controls.toml` names each control and carries whatever metadata your
+`controls.toml` names each control and records whatever metadata your
 regulatory mapping uses. Only `description` is required, and every field must be
 a quoted string -- TOML would otherwise read `reference = 2024-01-01` as a date,
 which the JSON export cannot serialise.
@@ -128,7 +128,7 @@ vip verify --config vip.toml --extensions ./21CFR_part11_validation
 vip trace --results report/results.json --controls ./21CFR_part11_validation/controls.toml
 ```
 
-Add `--format json` for a machine-readable matrix carrying the full provenance
+Add `--format json` for a machine-readable matrix that includes the full provenance
 block, or `--output matrix.csv` to write to a file.
 
 ## Extending it
diff --git a/examples/21CFR_part11_validation/VALIDATION-PACKAGE.md b/examples/21CFR_part11_validation/VALIDATION-PACKAGE.md
index 18b60ad4..75615811 100644
--- a/examples/21CFR_part11_validation/VALIDATION-PACKAGE.md
+++ b/examples/21CFR_part11_validation/VALIDATION-PACKAGE.md
@@ -37,7 +37,7 @@ maintained by hand.
 Execution provenance. Each results file records who ran the tests, which host
 they ran on, which git commit and branch they came from, whether that tree was
 dirty, and which CI job produced them. The HTML report and the PDF render the
-attribution as well, so the archived artifact carries it rather than only the
+attribution as well, so the archived artifact includes it rather than only the
 machine-readable output. This is what makes a result attributable to a named
 operator and a pipeline execution rather than to an anonymous green tick.
 
@@ -87,14 +87,14 @@ or in part:
 
 - the intended use of the feature or function, insofar as your control
   descriptions state it
-- the result of your risk-based analysis, carried through from `controls.toml`
+- the result of your risk-based analysis, read from `controls.toml`
   and rendered in the matrix
 - a description of the testing conducted, as the tagged scenario and its steps
 - issues found during testing, as failed and skipped scenarios
 - who performed the testing and the date it was performed
 
 Two items are yours. VIP writes no conclusion statement declaring
-acceptability, and it carries no resolution against the issues it records:
+acceptability, and it makes no determination on the issues it records:
 both are judgements, not test results. Neither is there any review and
 approval signature, which the guidance asks for "when appropriate" and which
 belongs in your quality system.
@@ -117,13 +117,13 @@ structure and responsibilities, the inventory of systems in scope, and the
 timelines and training that go with them.
 
 Risk assessments. Which controls matter, how badly, and why. `controls.toml`
-carries a `risk` field and VIP passes it through untouched, without
+has a `risk` field and VIP passes it through untouched, without
 interpreting it. Deciding the value is the assessment.
 
 The qualification protocols themselves. Installation, Operational and
 Performance Qualification documents are structured, pre-approved, signed
 records. VIP supplies evidence that can go inside an OQ. It does not write the
-protocol, define the acceptance criteria, or carry the approval signatures.
+protocol, define the acceptance criteria, or include the approval signatures.
 
 Standard Operating Procedures, training records, and the change control and
 deviation records that surround a validated system.
diff --git a/website/src/pages/getting-started.astro b/website/src/pages/getting-started.astro
index baca518f..5c666882 100644
--- a/website/src/pages/getting-started.astro
+++ b/website/src/pages/getting-started.astro
@@ -489,14 +489,14 @@ Feature: 21 CFR Part 11 flavoured controls
     Then each entry records an actor and a timestamp

Name the controls in a controls.toml of your own — VIP - stays regulation-agnostic and carries your reference, risk and + stays regulation-agnostic and passes your reference, risk and responsibility fields through untouched. Then join the two: 

vip verify --connect-url https://connect.example.com --extensions ./my-tests
 vip trace --results report/results.json --controls ./my-tests/controls.toml

The result is a matrix in CSV (for a spreadsheet or a downstream - qualification-protocol generator) or JSON, carrying each control, the + qualification-protocol generator) or JSON, listing each control, the scenarios that verify it, their outcome, and the timestamp each check ran. Every control gets one of three coverage values, and the third exists because collapsing it into the second would overstate your @@ -505,7 +505,7 @@ vip trace --results report/results.json --controls ./my-tests/controls.toml

  • covered — at least one scenario is tagged for it
  • - gap — no scenario in this run carries the tag, and + gap — no scenario in this run has the tag, and verification is automated (the default)
  • @@ -533,7 +533,7 @@ vip trace --results report/results.json --controls ./my-tests/controls.toml

    - Both formats carry a SHA-256 of the results file the matrix was derived + Both formats include a SHA-256 of the results file the matrix was derived from, so the archived artifact points back at its evidence. CSV repeats that digest with the generation time, VIP version and exit status on every row. The full provenance block, covering the products and