feat(traceability): add 21 CFR Part 11 control traceability matrix

.dockerignore

@@ -13,6 +13,7 @@ examples/*
 # selftests/test_dockerignore_force_includes.py fails if one is missing.
 !examples/cross_product_validation
 !examples/custom_tests
+!examples/21CFR_part11_validation
 !examples/_shared
 .gitignore
 .gitattributes

.github/workflows/example-report.yml

@@ -394,6 +394,11 @@ jobs:
       - name: Render Quarto report
         run: cd report && uv run quarto render
 
+      - name: Stage the standard report
+        run: |
+          mkdir -p _site
+          mv report/_output _site/example-report
+
       # Stop Connect
       - name: Stop Connect
         if: always()
@@ -417,4 +422,4 @@ jobs:
         uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
         with:
           name: example-report
-          path: report/_output/
+          path: _site/example-report/

.gitignore

@@ -1,7 +1,10 @@
 # VIP-specific
 vip.toml
 report/results.json
+report/results.json.sha256
 report/failures.json
+report/junit.xml
+report/results.sarif
 report/connect_system_checks.json
 .vip-auth-cache.json
 .vip-auth-cache.meta.json
@@ -253,3 +256,9 @@ presentations/qa-overview/index_files/
 # local dev license files
 /*.lic
 
+
+# Superpowers SDD scratch (subagent-driven-development workspace)
+.superpowers/
+
+# Staging directory for the two example reports built by example-report.yml
+_site/

AGENTS.md

@@ -120,7 +120,7 @@ Key rules:
 -   Step function names should be descriptive. Use `target_fixture` to pass state between steps.
 -   Tests must be non-destructive. Tag created content with `_vip_test` and clean it up in a final `then` step.
 -   Use version gating for version-specific features: `@pytest.mark.min_version(product="connect", version="2024.09.0")`
--   Say what a skip *means*. `vip.attest.not_applicable(reason)` says there was nothing to check here (product not configured, tier lacks the feature) and keeps the run green. `vip.attest.unproven(reason)` says VIP was asked to check something and could not, which fails the run with exit code 6 unless `--allow-unproven` is passed. A bare `pytest.skip()` still behaves like `not_applicable`; prefer the explicit helper so the next reader does not have to infer which one you meant.
+-   Say what a skip *means*. `vip.attest.not_applicable(reason)` says there was nothing to check here (product not configured, tier lacks the feature) and keeps the run green. `vip.attest.unproven(reason)` says VIP was asked to check something and could not, which fails the run with exit code 6 unless `--allow-unproven` is passed. A bare `pytest.skip()` still behaves like `not_applicable`. Prefer the explicit helper so the next reader does not have to infer which one you meant.
 
 ## Four-layer test architecture
 
@@ -146,7 +146,7 @@ Key principles:
 
 | File | Purpose |
 |------------------------------------|------------------------------------|
-| `src/vip/cli.py` | CLI entry point: version, verify (including `--basic` to skip `@slow`-tagged scenarios), cleanup (Connect content + orphaned Workbench sessions via `--workbench-url`), install, uninstall, auth, scaffold commands; `--version` flag |
+| `src/vip/cli.py` | CLI entry point: version, verify (including `--basic` to skip `@slow`-tagged scenarios), cleanup (Connect content + orphaned Workbench sessions via `--workbench-url`), install, uninstall, auth, scaffold, trace (join `results.json` against a `controls.toml` control list and emit a CSV/JSON traceability matrix) commands; `--version` flag |
 | `src/vip/config.py` | TOML config loader and dataclasses (includes `[proxy]` → `ProxyConfig`) |
 | `src/vip/proxy.py` | Single source of truth for outbound-proxy resolution. `ProxyConfig` + `build_proxy_map` (mirrors httpx's `get_environment_proxies`, incl. NO_PROXY formatting), `build_mounts` (per-scheme `HTTPTransport` mounts that keep `verify`), `proxy_for_url` (httpx-identical most-specific-pattern selection, used by non-httpx probes), `playwright_proxy` (renders a Playwright `launch(proxy=)` dict). Every HTTP egress path routes through this so VIP never diverges from httpx's own env-proxy behavior — see "Outbound proxy support" below |
 | `src/vip/auth.py` | Interactive and headless browser authentication for OIDC providers; `authenticated_page` opens a headless page from a cached auth session for `vip cleanup --workbench-url`; `auth_cache_path()` is the single source of truth for the `.vip-auth-cache.json` location (both `plugin.py` and `cli.py` must use it), and `_load_cached_auth` probes Workbench before trusting a cached session; `refresh_auth_cache_from_storage_state` writes a live context's cookies back over a cache whose session has been invalidated (atomic, 0600, existing caches only) |
@@ -156,10 +156,12 @@ Key principles:
 | `src/vip/fixtures.py` | VIP's core pytest fixtures and shared BDD "Given" steps (`vip_config`, `connect_client`, `browser_context_args`, etc.), registered by `vip.plugin.pytest_configure` rather than defined in a `conftest.py` — pytest scopes `conftest.py` fixtures by directory ancestry, which made them invisible to extension directories (issue #609) |
 | `src/vip/version.py` | `ProductVersion` parsing/comparison for `min_version` gating; `MINIMUM_SUPPORTED_POSIT_TEAM` support floor (powers `vip version`) |
 | `src/vip/workbench_ui.py` | Browser-driven Workbench session-cleanup sweep (`quit_vip_sessions_via_ui`), shared by the per-test cleanup fixture and `vip cleanup --workbench-url`; takes an `owner` so a per-test sweep only quits its own xdist worker's sessions |
-| `src/vip/reporting.py` | Report data model for Quarto templates |
+| `src/vip/reporting.py` | Report data model for Quarto templates; owns `RESULTS_SCHEMA_VERSION` and `load_results`, which warns (not raises) on an unknown schema major |
 | `src/vip/report_content.py` | Format-neutral report content shared by both rendering backends: titles, outcome/badge styling (colors drift-guarded against `styles.css` by `selftests/test_report_content.py`), grouping, skip wording, provenance rows |
 | `src/vip/report_html.py` | HTML backend: renders `report_content` into the fragments `index.qmd`/`details.qmd` display |
 | `src/vip/report_typst.py` | Typst backend: renders the same content as Typst markup for `report/vip-report.qmd` → `_output/vip-report.pdf`; every dynamic value passes through `_lit` (Typst-injection escaping) |
+| `src/vip/attribution.py` | Collects the `execution` block written into `results.json` — hostname, git (commit/branch/dirty/remote, with any userinfo redacted from the remote URL), CI (provider/run_id/run_url/job for GitHub Actions, GitLab CI, Jenkins), and `performed_by` (`VIP_PERFORMED_BY` → CI actor → local login, each tagged with its `source`; FDA's Computer Software Assurance guidance asks the record to say who performed the testing). Every probe degrades to `None` rather than failing or warning; omitted entirely with `--vip-no-attribution`, which is also the opt-out for writing an operator identity into an archived artifact. `report_content.provenance_rows` renders the whole block into both report editions — before that it reached only `vip trace --format json` |
+| `src/vip/traceability.py` | Control list model (`ControlSpec`, loaded from `controls.toml`) and `build_traceability_matrix`, which joins controls against `@control-<slug>`-tagged scenarios in a loaded `results.json`; CSV (`render_csv`, apostrophe-neutralizes formula-leading cells) and JSON (`render_json`) renderers; `verify_results_checksum` (per-line sidecar parse, case-insensitive digest, matched by recorded filename) and the schema-version gate (`check_results_schema` hard-errors on an unknown major, unlike `reporting.load_results`, which only warns). `ControlEntry.executed` / `TraceabilityMatrix.covered_without_execution` separate "a scenario is tagged" from "a scenario ran" — a scenario that runs and skips itself (absent endpoint, no data, version gate) still keeps its control tag, so coverage alone would report a green matrix for a run that verified nothing. `ControlEntry.failing` and `TraceabilityMatrix.covered_with_failure` are the third fact, separating "a scenario ran" from "a scenario passed", and any failing scenario demotes the control's badge rather than only an all-failed one. An *unconfigured* product is the opposite case and is often misdescribed: `plugin.pytest_collection_modifyitems` deselects those scenarios rather than skipping them, so they never reach `results.json` and a control tagged only by them reports as a `gap`, not as covered-not-executed. Powers `vip trace` |
 | `src/vip/clients/connect.py` | httpx client for Connect API |
 | `src/vip/clients/workbench.py` | httpx client for Workbench API; `quit_vip_sessions` warns loudly (not silently) when a VIP session persists after all retries. `session_owner` / `is_vip_session_for_owner` decide whether a VIP session belongs to the sweeping worker — see "Session ownership" below |
 | `src/vip/clients/packagemanager.py` | httpx client for Package Manager API |
@@ -170,6 +172,8 @@ Key principles:
 | `src/vip/install/plan.py` | Pure `build_install_plan` / `build_uninstall_plan` builders |
 | `src/vip/install/runner.py` | Plan executor: dry-run formatting + execute (system packages, Playwright, manifest writes) |
 | `src/vip_tests/conftest.py` | Directory-scoped warning filter (kept out of the global plugin deliberately) plus the three autouse Connect content-cleanup fixtures — see that file's docstring for why those stay directory-scoped instead of moving to `src/vip/fixtures.py` |
+| `examples/21CFR_part11_validation/` | Worked control-to-scenario mapping for `vip trace`, one feature file per product (`test_21CFR_part11_connect`, `_packagemanager`, `_workbench`) because the product tag is feature-level. The refusal assertion Connect and Workbench share sits in `part11_refusal.py`, not in a step file -- one pytest-bdd step module cannot import another. `conftest.py` holds the override points a customer edits: the two privileged endpoints, and the Package Manager repository and snapshot date the reproducibility scenario pins to |
+| `examples/21CFR_part11_validation/VALIDATION-PACKAGE.md` | What VIP supplies toward a GxP validation package, what the customer authors, and what nothing can automate. It lives beside the scaffold template rather than under `docs/` so that `vip scaffold --template 21cfr-part11-validation` copies it onto the customer's disk with the tests it describes. The reference for any regulated-customer conversation: it refuses the strong claims (tamper-evidence is not an immutable audit trail, a green matrix is not an attestation) and states the scenario-level evidence gap |
 | `report/index.qmd` | Quarto summary page |
 | `report/details.qmd` | Quarto detailed results page |
 | `report/vip-report.qmd` | Quarto/Typst PDF edition (summary + full listing in one archivable file) |
@@ -229,6 +233,7 @@ Clients live in `src/vip/clients/` and use plain httpx. Rules:
 -   Return dicts from JSON responses, not custom model objects.
 -   Add methods only when tests need them.
 -   All clients take a base URL and optional API key in their constructor.
+-   `BaseClient.unauthenticated_status(path)` is the shared credential-free probe behind every product's access-control scenario (Connect's user API, Workbench's session API). It lives on the base class rather than on one product client so a new access-control test never reaches for a raw `httpx.get` that would bypass the proxy and the CA overrides.
 -   `BaseClient` needs a custom `transport=` (for `retries` and transport-level `verify`), which makes httpx ignore env proxies. It therefore resolves the proxy itself via `vip.proxy` and passes per-scheme `mounts=`. Any new ad-hoc `httpx.get`/`httpx.Client` in the client layer must route through the same proxy — pass `proxy=proxy_for_url(url, self._proxy_map)` (see `fetch_content`), never rely on httpx's ambient env pickup, so an explicit `[proxy]` config applies uniformly. See "Outbound proxy support" below.
 
 ## Configuration
@@ -309,7 +314,8 @@ Every render also produces `_output/vip-report.pdf` from `report/vip-report.qmd`
 ## CI workflows
 
 -   **`ci.yml`** -- on every PR/push: ruff lint/format (pinned to 0.15.0), mypy type-check, zizmor actions-lint, a runtime dependency audit, and selftests (Ubuntu + macOS, Python 3.10 and 3.12). A `changes` path-filter gates the expensive jobs, while `Lint & Format`, `Selftests Status` and `CI Status` always run as required checks. Uses uv cache. `CI Status` is the scope-aware aggregator for the four path-gated jobs (`Type Check`, `Actions Lint (zizmor)`, `Dependency Audit`, `Lockfile Guard`): none of them can be a required check directly, because each is conditional on `changes` and a failed change-detection job would skip them all and report a green gate. A legitimately skipped job counts as passing; only failure or cancellation is fatal.
--   **`preview.yml`** -- runs selftests, renders Quarto report, publishes PR preview to gh-pages via `rossjrw/pr-preview-action@v1`. Uses uv and Quarto caches.
+-   **`preview.yml`** -- runs selftests, renders Quarto report, publishes PR preview to gh-pages via `rossjrw/pr-preview-action@v1`. Uses uv and Quarto caches. Its job is checking `report/` template changes.
+-   **`example-report.yml`** -- builds the example report from one live deployment and uploads it as an artifact. Runs the smoke subset against Connect, Workbench and Package Manager and renders it with `vip report`. `website.yml` and `website-preview.yml` download the artifact into `website/dist/`, where `report.astro` embeds it. A control-list traceability matrix (`vip report --controls`) is documented and demonstrated separately, in `examples/21CFR_part11_validation/`, rather than published on the website -- see that directory's `VALIDATION-PACKAGE.md`.
 -   **`pr-title.yml`** -- validates PR titles follow conventional commit format. Squash merges use the PR title as the commit message.
 -   **`release.yml`** -- cuts VIP's calver release train: `schedule` (Thursday evenings) plus `workflow_dispatch` for out-of-band releases. `scripts/next_version.py` computes the version (`YYYY.M.0` for the first release of a calendar month, `YYYY.M.PATCH` for later ones that month); a scheduled or blank-`version` dispatch run exits cleanly when there are no commits since the last tag, while a dispatch run with an explicit `version` skips that gate but must still be strictly greater than the last release. `cliff.toml` (git-cliff) generates the `CHANGELOG.md` entry for the tag before it exists, so it lands inside the release commit rather than needing a second commit. `just relock` keeps `uv.lock`'s own `posit-vip` entry in sync with the version just stamped -- see docs/development.md ("Versioning and the release cadence") for the full rule and issue #559 for why the relock step matters.
 -   **Smoke workflows** (`connect-smoke.yml`, `workbench-smoke.yml`, `packagemanager-smoke.yml`, `mock-idp-e2e.yml`) -- run the product suites against real containers. On PR/push each tests a single latest version (change-gated via a `changes` paths-filter job); on `schedule` (nightly, staggered hourly) and `workflow_dispatch` a `set-matrix` job fans each out across the product version support window (current + 2 back). Bump the pinned tags in each workflow's `set-matrix` step when a new product release ships. Each workflow's `*-status` aggregation job is scope-aware: it passes when the suite was legitimately out of scope (the PR's paths didn't match) but **fails** when the suite was in scope (`changes.relevant == 'true'`) yet did not succeed -- so a path-gated skip, an excluded actor, or a missing license secret can no longer report a green required check without the suite having run. The Connect, Workbench, and Package Manager `*-status` jobs are required merge checks.

README.md

@@ -102,12 +102,58 @@ content from Connect.
 | `vip status` | Quick health check for each configured product |
 | `vip cleanup` | Delete VIP `_vip_test` content from Connect |
 | `vip report` | Render the HTML report from test results (requires [Quarto CLI](https://quarto.org/docs/download/)) |
+| `vip scaffold` | Generate a ready-to-run custom test extension from a template |
+| `vip trace` | Build a compliance traceability matrix from test results and a control list |
 | `vip auth` | Authentication tools (e.g. mint Connect API keys) |
 | `vip version` | Print the vip version and the minimum supported Posit Team version |
 | `vip --version` | Print the installed vip version |
 
 Run `vip --help` or `vip <command> --help` for full usage details.
 
+## Writing your own tests
+
+VIP is extensible without changing its source. You point `vip verify` at a
+directory of your own tests, and they run alongside the built-in suite and land
+in the same report. `vip scaffold` writes a working starting point:
+
+```bash
+vip scaffold --list
+vip scaffold --template minimal --output ./my-tests
+vip verify --config vip.toml --extensions ./my-tests
+```
+
+| Template | What it shows |
+|---|---|
+| `minimal` | A single HTTP health-check scenario. Start here. |
+| `cross-product` | R and Python runtime versions, and package installability across Connect and Workbench. The GxP starting point. |
+| `21cfr-part11-validation` | Regulatory control tagging plus a `controls.toml` for `vip trace`. |
+
+Every scaffolded directory runs as-is against your deployment, and each one
+includes an `AGENTS.md` describing the extension contract, so a coding agent
+picking up the directory knows the auto-skip rules and which fixtures and
+markers it may use.
+
+### The 21 CFR Part 11 example
+
+The compliance template is the one worth reading even if you never run it. Each
+scenario declares the control it evidences with an `@control-<slug>` tag,
+`controls.toml` holds your regulatory mapping, and `vip trace` joins the two
+into a traceability matrix as CSV or JSON:
+
+```bash
+vip trace --results report/results.json --controls ./my-tests/controls.toml
+```
+
+`vip report --controls` renders the same matrix into the HTML report and the
+archivable PDF, so the artifact you hand an auditor includes the join from a
+control to its evidence rather than only a list of passing tests.
+
+It ships with `VALIDATION-PACKAGE.md`, which is the honest version of what this
+is worth: which documents in a GxP validation package VIP produces, which you
+author, and which nothing can automate. Most of 21 CFR Part 11 cannot be
+evidenced by testing a platform, and a green matrix is not an attestation of
+compliance. Read that file before taking the example into a validation meeting.
+
 ## CI / pipeline integration
 
 VIP emits machine-readable output for security-ops and CI/CD pipelines.
@@ -125,7 +171,7 @@ vip verify --format json,junit,sarif
 Skip messages in the JUnit and SARIF output carry the actual skip reason
 instead of a generic "skipped" label. A check VIP was asked to run but could
 not -- a configured product whose authentication never completed, say -- is
-reported as **unproven** rather than as an ordinary skip: it carries an
+reported as **unproven** rather than as an ordinary skip: it has an
 `UNPROVEN:` prefix in JUnit, SARIF level `warning`, and exits **6** so a
 pipeline can tell "the deployment is broken" (exit 1) from "the deployment
 could not be checked" (exit 6). Pass `--allow-unproven` to exit 0 anyway. `results.json` also records provenance