diff --git a/.dockerignore b/.dockerignore index 3315e5e5..5f479e2f 100644 --- a/.dockerignore +++ b/.dockerignore @@ -13,6 +13,7 @@ examples/* # selftests/test_dockerignore_force_includes.py fails if one is missing. !examples/cross_product_validation !examples/custom_tests +!examples/21CFR_part11_validation !examples/_shared .gitignore .gitattributes diff --git a/.github/workflows/example-report.yml b/.github/workflows/example-report.yml index 8ce2979a..5b0a9b27 100644 --- a/.github/workflows/example-report.yml +++ b/.github/workflows/example-report.yml @@ -394,6 +394,11 @@ jobs: - name: Render Quarto report run: cd report && uv run quarto render + - name: Stage the standard report + run: | + mkdir -p _site + mv report/_output _site/example-report + # Stop Connect - name: Stop Connect if: always() @@ -417,4 +422,4 @@ jobs: uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7 with: name: example-report - path: report/_output/ + path: _site/example-report/ diff --git a/.gitignore b/.gitignore index 0ccf3b4c..2b95d695 100644 --- a/.gitignore +++ b/.gitignore @@ -1,7 +1,10 @@ # VIP-specific vip.toml report/results.json +report/results.json.sha256 report/failures.json +report/junit.xml +report/results.sarif report/connect_system_checks.json .vip-auth-cache.json .vip-auth-cache.meta.json @@ -253,3 +256,9 @@ presentations/qa-overview/index_files/ # local dev license files /*.lic + +# Superpowers SDD scratch (subagent-driven-development workspace) +.superpowers/ + +# Staging directory for the two example reports built by example-report.yml +_site/ diff --git a/AGENTS.md b/AGENTS.md index d3f5eca9..d71b6dde 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -120,7 +120,7 @@ Key rules: - Step function names should be descriptive. Use `target_fixture` to pass state between steps. - Tests must be non-destructive. Tag created content with `_vip_test` and clean it up in a final `then` step. - Use version gating for version-specific features: `@pytest.mark.min_version(product="connect", version="2024.09.0")` -- Say what a skip *means*. `vip.attest.not_applicable(reason)` says there was nothing to check here (product not configured, tier lacks the feature) and keeps the run green. `vip.attest.unproven(reason)` says VIP was asked to check something and could not, which fails the run with exit code 6 unless `--allow-unproven` is passed. A bare `pytest.skip()` still behaves like `not_applicable`; prefer the explicit helper so the next reader does not have to infer which one you meant. +- Say what a skip *means*. `vip.attest.not_applicable(reason)` says there was nothing to check here (product not configured, tier lacks the feature) and keeps the run green. `vip.attest.unproven(reason)` says VIP was asked to check something and could not, which fails the run with exit code 6 unless `--allow-unproven` is passed. A bare `pytest.skip()` still behaves like `not_applicable`. Prefer the explicit helper so the next reader does not have to infer which one you meant. ## Four-layer test architecture @@ -146,7 +146,7 @@ Key principles: | File | Purpose | |------------------------------------|------------------------------------| -| `src/vip/cli.py` | CLI entry point: version, verify (including `--basic` to skip `@slow`-tagged scenarios), cleanup (Connect content + orphaned Workbench sessions via `--workbench-url`), install, uninstall, auth, scaffold commands; `--version` flag | +| `src/vip/cli.py` | CLI entry point: version, verify (including `--basic` to skip `@slow`-tagged scenarios), cleanup (Connect content + orphaned Workbench sessions via `--workbench-url`), install, uninstall, auth, scaffold, trace (join `results.json` against a `controls.toml` control list and emit a CSV/JSON traceability matrix) commands; `--version` flag | | `src/vip/config.py` | TOML config loader and dataclasses (includes `[proxy]` → `ProxyConfig`) | | `src/vip/proxy.py` | Single source of truth for outbound-proxy resolution. `ProxyConfig` + `build_proxy_map` (mirrors httpx's `get_environment_proxies`, incl. NO_PROXY formatting), `build_mounts` (per-scheme `HTTPTransport` mounts that keep `verify`), `proxy_for_url` (httpx-identical most-specific-pattern selection, used by non-httpx probes), `playwright_proxy` (renders a Playwright `launch(proxy=)` dict). Every HTTP egress path routes through this so VIP never diverges from httpx's own env-proxy behavior — see "Outbound proxy support" below | | `src/vip/auth.py` | Interactive and headless browser authentication for OIDC providers; `authenticated_page` opens a headless page from a cached auth session for `vip cleanup --workbench-url`; `auth_cache_path()` is the single source of truth for the `.vip-auth-cache.json` location (both `plugin.py` and `cli.py` must use it), and `_load_cached_auth` probes Workbench before trusting a cached session; `refresh_auth_cache_from_storage_state` writes a live context's cookies back over a cache whose session has been invalidated (atomic, 0600, existing caches only) | @@ -156,10 +156,12 @@ Key principles: | `src/vip/fixtures.py` | VIP's core pytest fixtures and shared BDD "Given" steps (`vip_config`, `connect_client`, `browser_context_args`, etc.), registered by `vip.plugin.pytest_configure` rather than defined in a `conftest.py` — pytest scopes `conftest.py` fixtures by directory ancestry, which made them invisible to extension directories (issue #609) | | `src/vip/version.py` | `ProductVersion` parsing/comparison for `min_version` gating; `MINIMUM_SUPPORTED_POSIT_TEAM` support floor (powers `vip version`) | | `src/vip/workbench_ui.py` | Browser-driven Workbench session-cleanup sweep (`quit_vip_sessions_via_ui`), shared by the per-test cleanup fixture and `vip cleanup --workbench-url`; takes an `owner` so a per-test sweep only quits its own xdist worker's sessions | -| `src/vip/reporting.py` | Report data model for Quarto templates | +| `src/vip/reporting.py` | Report data model for Quarto templates; owns `RESULTS_SCHEMA_VERSION` and `load_results`, which warns (not raises) on an unknown schema major | | `src/vip/report_content.py` | Format-neutral report content shared by both rendering backends: titles, outcome/badge styling (colors drift-guarded against `styles.css` by `selftests/test_report_content.py`), grouping, skip wording, provenance rows | | `src/vip/report_html.py` | HTML backend: renders `report_content` into the fragments `index.qmd`/`details.qmd` display | | `src/vip/report_typst.py` | Typst backend: renders the same content as Typst markup for `report/vip-report.qmd` → `_output/vip-report.pdf`; every dynamic value passes through `_lit` (Typst-injection escaping) | +| `src/vip/attribution.py` | Collects the `execution` block written into `results.json` — hostname, git (commit/branch/dirty/remote, with any userinfo redacted from the remote URL), CI (provider/run_id/run_url/job for GitHub Actions, GitLab CI, Jenkins), and `performed_by` (`VIP_PERFORMED_BY` → CI actor → local login, each tagged with its `source`; FDA's Computer Software Assurance guidance asks the record to say who performed the testing). Every probe degrades to `None` rather than failing or warning; omitted entirely with `--vip-no-attribution`, which is also the opt-out for writing an operator identity into an archived artifact. `report_content.provenance_rows` renders the whole block into both report editions — before that it reached only `vip trace --format json` | +| `src/vip/traceability.py` | Control list model (`ControlSpec`, loaded from `controls.toml`) and `build_traceability_matrix`, which joins controls against `@control-`-tagged scenarios in a loaded `results.json`; CSV (`render_csv`, apostrophe-neutralizes formula-leading cells) and JSON (`render_json`) renderers; `verify_results_checksum` (per-line sidecar parse, case-insensitive digest, matched by recorded filename) and the schema-version gate (`check_results_schema` hard-errors on an unknown major, unlike `reporting.load_results`, which only warns). `ControlEntry.executed` / `TraceabilityMatrix.covered_without_execution` separate "a scenario is tagged" from "a scenario ran" — a scenario that runs and skips itself (absent endpoint, no data, version gate) still keeps its control tag, so coverage alone would report a green matrix for a run that verified nothing. `ControlEntry.failing` and `TraceabilityMatrix.covered_with_failure` are the third fact, separating "a scenario ran" from "a scenario passed", and any failing scenario demotes the control's badge rather than only an all-failed one. An *unconfigured* product is the opposite case and is often misdescribed: `plugin.pytest_collection_modifyitems` deselects those scenarios rather than skipping them, so they never reach `results.json` and a control tagged only by them reports as a `gap`, not as covered-not-executed. Powers `vip trace` | | `src/vip/clients/connect.py` | httpx client for Connect API | | `src/vip/clients/workbench.py` | httpx client for Workbench API; `quit_vip_sessions` warns loudly (not silently) when a VIP session persists after all retries. `session_owner` / `is_vip_session_for_owner` decide whether a VIP session belongs to the sweeping worker — see "Session ownership" below | | `src/vip/clients/packagemanager.py` | httpx client for Package Manager API | @@ -170,6 +172,8 @@ Key principles: | `src/vip/install/plan.py` | Pure `build_install_plan` / `build_uninstall_plan` builders | | `src/vip/install/runner.py` | Plan executor: dry-run formatting + execute (system packages, Playwright, manifest writes) | | `src/vip_tests/conftest.py` | Directory-scoped warning filter (kept out of the global plugin deliberately) plus the three autouse Connect content-cleanup fixtures — see that file's docstring for why those stay directory-scoped instead of moving to `src/vip/fixtures.py` | +| `examples/21CFR_part11_validation/` | Worked control-to-scenario mapping for `vip trace`, one feature file per product (`test_21CFR_part11_connect`, `_packagemanager`, `_workbench`) because the product tag is feature-level. The refusal assertion Connect and Workbench share sits in `part11_refusal.py`, not in a step file -- one pytest-bdd step module cannot import another. `conftest.py` holds the override points a customer edits: the two privileged endpoints, and the Package Manager repository and snapshot date the reproducibility scenario pins to | +| `examples/21CFR_part11_validation/VALIDATION-PACKAGE.md` | What VIP supplies toward a GxP validation package, what the customer authors, and what nothing can automate. It lives beside the scaffold template rather than under `docs/` so that `vip scaffold --template 21cfr-part11-validation` copies it onto the customer's disk with the tests it describes. The reference for any regulated-customer conversation: it refuses the strong claims (tamper-evidence is not an immutable audit trail, a green matrix is not an attestation) and states the scenario-level evidence gap | | `report/index.qmd` | Quarto summary page | | `report/details.qmd` | Quarto detailed results page | | `report/vip-report.qmd` | Quarto/Typst PDF edition (summary + full listing in one archivable file) | @@ -229,6 +233,7 @@ Clients live in `src/vip/clients/` and use plain httpx. Rules: - Return dicts from JSON responses, not custom model objects. - Add methods only when tests need them. - All clients take a base URL and optional API key in their constructor. +- `BaseClient.unauthenticated_status(path)` is the shared credential-free probe behind every product's access-control scenario (Connect's user API, Workbench's session API). It lives on the base class rather than on one product client so a new access-control test never reaches for a raw `httpx.get` that would bypass the proxy and the CA overrides. - `BaseClient` needs a custom `transport=` (for `retries` and transport-level `verify`), which makes httpx ignore env proxies. It therefore resolves the proxy itself via `vip.proxy` and passes per-scheme `mounts=`. Any new ad-hoc `httpx.get`/`httpx.Client` in the client layer must route through the same proxy — pass `proxy=proxy_for_url(url, self._proxy_map)` (see `fetch_content`), never rely on httpx's ambient env pickup, so an explicit `[proxy]` config applies uniformly. See "Outbound proxy support" below. ## Configuration @@ -309,7 +314,8 @@ Every render also produces `_output/vip-report.pdf` from `report/vip-report.qmd` ## CI workflows - **`ci.yml`** -- on every PR/push: ruff lint/format (pinned to 0.15.0), mypy type-check, zizmor actions-lint, a runtime dependency audit, and selftests (Ubuntu + macOS, Python 3.10 and 3.12). A `changes` path-filter gates the expensive jobs, while `Lint & Format`, `Selftests Status` and `CI Status` always run as required checks. Uses uv cache. `CI Status` is the scope-aware aggregator for the four path-gated jobs (`Type Check`, `Actions Lint (zizmor)`, `Dependency Audit`, `Lockfile Guard`): none of them can be a required check directly, because each is conditional on `changes` and a failed change-detection job would skip them all and report a green gate. A legitimately skipped job counts as passing; only failure or cancellation is fatal. -- **`preview.yml`** -- runs selftests, renders Quarto report, publishes PR preview to gh-pages via `rossjrw/pr-preview-action@v1`. Uses uv and Quarto caches. +- **`preview.yml`** -- runs selftests, renders Quarto report, publishes PR preview to gh-pages via `rossjrw/pr-preview-action@v1`. Uses uv and Quarto caches. Its job is checking `report/` template changes. +- **`example-report.yml`** -- builds the example report from one live deployment and uploads it as an artifact. Runs the smoke subset against Connect, Workbench and Package Manager and renders it with `vip report`. `website.yml` and `website-preview.yml` download the artifact into `website/dist/`, where `report.astro` embeds it. A control-list traceability matrix (`vip report --controls`) is documented and demonstrated separately, in `examples/21CFR_part11_validation/`, rather than published on the website -- see that directory's `VALIDATION-PACKAGE.md`. - **`pr-title.yml`** -- validates PR titles follow conventional commit format. Squash merges use the PR title as the commit message. - **`release.yml`** -- cuts VIP's calver release train: `schedule` (Thursday evenings) plus `workflow_dispatch` for out-of-band releases. `scripts/next_version.py` computes the version (`YYYY.M.0` for the first release of a calendar month, `YYYY.M.PATCH` for later ones that month); a scheduled or blank-`version` dispatch run exits cleanly when there are no commits since the last tag, while a dispatch run with an explicit `version` skips that gate but must still be strictly greater than the last release. `cliff.toml` (git-cliff) generates the `CHANGELOG.md` entry for the tag before it exists, so it lands inside the release commit rather than needing a second commit. `just relock` keeps `uv.lock`'s own `posit-vip` entry in sync with the version just stamped -- see docs/development.md ("Versioning and the release cadence") for the full rule and issue #559 for why the relock step matters. - **Smoke workflows** (`connect-smoke.yml`, `workbench-smoke.yml`, `packagemanager-smoke.yml`, `mock-idp-e2e.yml`) -- run the product suites against real containers. On PR/push each tests a single latest version (change-gated via a `changes` paths-filter job); on `schedule` (nightly, staggered hourly) and `workflow_dispatch` a `set-matrix` job fans each out across the product version support window (current + 2 back). Bump the pinned tags in each workflow's `set-matrix` step when a new product release ships. Each workflow's `*-status` aggregation job is scope-aware: it passes when the suite was legitimately out of scope (the PR's paths didn't match) but **fails** when the suite was in scope (`changes.relevant == 'true'`) yet did not succeed -- so a path-gated skip, an excluded actor, or a missing license secret can no longer report a green required check without the suite having run. The Connect, Workbench, and Package Manager `*-status` jobs are required merge checks. diff --git a/README.md b/README.md index 282aad5c..63b5abb0 100644 --- a/README.md +++ b/README.md @@ -102,12 +102,58 @@ content from Connect. | `vip status` | Quick health check for each configured product | | `vip cleanup` | Delete VIP `_vip_test` content from Connect | | `vip report` | Render the HTML report from test results (requires [Quarto CLI](https://quarto.org/docs/download/)) | +| `vip scaffold` | Generate a ready-to-run custom test extension from a template | +| `vip trace` | Build a compliance traceability matrix from test results and a control list | | `vip auth` | Authentication tools (e.g. mint Connect API keys) | | `vip version` | Print the vip version and the minimum supported Posit Team version | | `vip --version` | Print the installed vip version | Run `vip --help` or `vip --help` for full usage details. +## Writing your own tests + +VIP is extensible without changing its source. You point `vip verify` at a +directory of your own tests, and they run alongside the built-in suite and land +in the same report. `vip scaffold` writes a working starting point: + +```bash +vip scaffold --list +vip scaffold --template minimal --output ./my-tests +vip verify --config vip.toml --extensions ./my-tests +``` + +| Template | What it shows | +|---|---| +| `minimal` | A single HTTP health-check scenario. Start here. | +| `cross-product` | R and Python runtime versions, and package installability across Connect and Workbench. The GxP starting point. | +| `21cfr-part11-validation` | Regulatory control tagging plus a `controls.toml` for `vip trace`. | + +Every scaffolded directory runs as-is against your deployment, and each one +includes an `AGENTS.md` describing the extension contract, so a coding agent +picking up the directory knows the auto-skip rules and which fixtures and +markers it may use. + +### The 21 CFR Part 11 example + +The compliance template is the one worth reading even if you never run it. Each +scenario declares the control it evidences with an `@control-` tag, +`controls.toml` holds your regulatory mapping, and `vip trace` joins the two +into a traceability matrix as CSV or JSON: + +```bash +vip trace --results report/results.json --controls ./my-tests/controls.toml +``` + +`vip report --controls` renders the same matrix into the HTML report and the +archivable PDF, so the artifact you hand an auditor includes the join from a +control to its evidence rather than only a list of passing tests. + +It ships with `VALIDATION-PACKAGE.md`, which is the honest version of what this +is worth: which documents in a GxP validation package VIP produces, which you +author, and which nothing can automate. Most of 21 CFR Part 11 cannot be +evidenced by testing a platform, and a green matrix is not an attestation of +compliance. Read that file before taking the example into a validation meeting. + ## CI / pipeline integration VIP emits machine-readable output for security-ops and CI/CD pipelines. @@ -125,7 +171,7 @@ vip verify --format json,junit,sarif Skip messages in the JUnit and SARIF output carry the actual skip reason instead of a generic "skipped" label. A check VIP was asked to run but could not -- a configured product whose authentication never completed, say -- is -reported as **unproven** rather than as an ordinary skip: it carries an +reported as **unproven** rather than as an ordinary skip: it has an `UNPROVEN:` prefix in JUnit, SARIF level `warning`, and exits **6** so a pipeline can tell "the deployment is broken" (exit 1) from "the deployment could not be checked" (exit 6). Pass `--allow-unproven` to exit 0 anyway. `results.json` also records provenance diff --git a/docs/reporting.md b/docs/reporting.md index b8c63313..f8d6203e 100644 --- a/docs/reporting.md +++ b/docs/reporting.md @@ -1,5 +1,352 @@ -# Quarto Report +# Reporting -This documentation has moved to the VIP website: +The interactive HTML report (built with Quarto) is documented on the VIP website: -**https://posit-dev.github.io/vip/report/** +https://posit-dev.github.io/vip/report/ + +This page covers the machine-readable outputs a `vip verify` run produces -- +`results.json`, `junit.xml`, `results.sarif` -- and the traceability export built on +top of `results.json`. + +## Machine-readable outputs + +Every `vip verify` run writes `report/results.json` by default. Override the path +with `--report`, or pass `--report ''` to write no results file at all. `--format` +selects which additional formats are written alongside it: + +```bash +vip verify --config vip.toml --format json,junit,sarif +``` + +`json` (`results.json`) is always written unless `--report ''` turns it off. `junit` +and `sarif` are added as sibling files in the same directory when requested, and +they are built by reloading `results.json`, so they cannot outlive it: asking for +either while disabling the results file is refused up front rather than after a +full run that would produce nothing. `--ci` is a preset that turns on +`json,junit,sarif` together with concise tracebacks, so it conflicts with +`--report ''` for the same reason. + +### `results.json` field inventory + +```json +{ + "schema_version": "1.0", + "generated_at": "2026-08-28T12:00:00+00:00", + "deployment_name": "example", + "exit_status": 0, + "vip_version": "2026.8.3", + "run_duration_seconds": 42.1, + "python_version": "3.12.4", + "platform": "macOS-14.5-arm64-...", + "basic_mode": false, + "products": { + "connect": {"enabled": true, "url": "...", "version": null, "configured": true}, + "workbench": {"enabled": true, "url": "", "version": null, "configured": false} + }, + "results": [ + { + "nodeid": "...", + "outcome": "passed", + "markers": ["connect", "control-audit-trail-publish"], + "scenario_title": "...", + "started_at": "2026-08-28T12:00:01+00:00", + "finished_at": "2026-08-28T12:00:02+00:00" + } + ], + "execution": { + "hostname": "ci-runner-01", + "git": { "commit": "...", "branch": "...", "dirty": false, "remote": "https://github.com/org/repo" }, + "ci": { "provider": "github", "run_id": "...", "run_url": "...", "job": "verify" }, + "performed_by": { "identity": "octocat", "source": "github" } + } +} +``` + +- `schema_version` -- versioned independently of VIP itself. The minor number bumps + for an additive change (a new field). The major number bumps for a removal, a + rename, or a change in the meaning of an existing field. A file with no + `schema_version` at all predates versioning and is treated as pre-1.0. +- `started_at` / `finished_at` -- per-test UTC ISO 8601 timestamps, covering the call + phase only (fixture setup is excluded, except that a setup-phase skip records setup + start). `None` for a `results.json` written before these fields existed. +- `execution` -- attribution for the run that produced this evidence: which host ran + it, which git commit/branch it ran from (dirty flag, remote with any credential + stripped out of the URL), which CI job (GitHub Actions, GitLab CI, or Jenkins) + ran it, if any, and who performed it. To omit this block entirely, pass the + pytest-level option after `--`: `vip verify --config vip.toml -- + --vip-no-attribution`. There is no `vip verify` flag of its own for this. Useful + if a deployment's policy is not to record hostnames, CI identifiers or an + operator identity in an archived artifact. +- `execution.performed_by` -- the operator the run is attributable to, and `source` + says where that value came from. Set `VIP_PERFORMED_BY` to name the person + accountable for the run (`source: "explicit"`). Otherwise VIP reads the CI + system's own actor (`GITHUB_ACTOR`, `GITLAB_USER_LOGIN`, `BUILD_USER_ID`), then + falls back to the local login (`source: "login"`). `source` stays attached to the + value because a reader seeing a service-account name needs to know whether a + human typed it. FDA's Computer Software Assurance guidance asks the record of an + assurance activity to state who performed the testing alongside the date, which + is why this exists. The rest of the block identifies a machine, not a person. + +Both report editions render the attribution too, so the archived artifact +includes it rather than only the machine-readable output. They render five of +these fields -- `performed_by` (qualified by its `source` unless that source is +`explicit`), `hostname`, `git.commit` (flagged when `dirty`), `git.branch`, and +`ci.run_url` or `ci.run_id`. `git.remote`, `ci.provider` and `ci.job` stay in +`results.json` only: the remote and the job name add table width without adding +much a reviewer can act on, and the provider is already evident from the run +URL. Read `results.json` itself if you need the full block. + +Be precise about what `python_version`, `platform`, and `execution.hostname` +describe: they are properties of the machine that ran `vip verify` -- the VIP +runner -- not the Connect/Workbench/Package Manager deployment under test. The +`products` table is what identifies the system under test (its name, URL, version, +and whether it was configured for this run). Don't read the runner's platform string +as evidence about the deployment. It isn't. + +### Schema compatibility policy + +An unknown schema minor is accepted (fields you don't recognize are additive and +safe to ignore), but an unknown schema major is refused. The two consumers of +`schema_version` apply this policy differently on purpose: + +- `vip.reporting.load_results` -- used by `vip report` and the Quarto notebooks + (`report/index.qmd`, `report/details.qmd`) -- only warns on an unknown major. It + runs inside a notebook cell, where raising would surface as an unreadable + traceback instead of a rendered report. +- `vip trace` hard-errors on an unknown major (see `check_results_schema` in + `src/vip/traceability.py`). A traceability matrix built against a `results.json` + whose shape it doesn't understand is not something you want silently degraded -- + it runs from a shell, where a clear error message is the right outcome. + +### The `.sha256` sidecar + +Every `results.json` write also produces a `results.json.sha256` sidecar next to it, +in the standard `shasum` format: + +``` + results.json +``` + +Verify it with: + +```bash +shasum -a 256 -c results.json.sha256 +``` + +The recorded filename is matched on its exact value first, then on its basename. +A sidecar generated from a directory above the results file records the path it +was given (` report/results.json`) and still verifies. A multi-file +sidecar that names the file exactly keeps the stricter exact match, so it cannot +be satisfied by a same-named file in another directory. + +This is tamper-evidence within a trusted pipeline, not tamper-proofing. Anyone who +can edit `results.json` can also regenerate the sidecar to match, so it does not +resist a motivated forger. What it does catch is the class of accidents that +actually happen to archived CI artifacts: corruption in transit, a truncated +upload, or someone hand-editing the file and forgetting to update the checksum +alongside it. Treat a checksum mismatch as "this file is not what the pipeline +produced," not as "this file has not been tampered with." + +## Traceability export + +`vip trace` joins `results.json` against a `controls.toml` control list and emits a +control-to-scenario traceability matrix, for suites that tag scenarios with +`@control-` Gherkin tags (see `docs/test-architecture.md` for the tagging +convention). + +### `controls.toml` format + +```toml +[controls.audit-trail-publish] +description = "Deployment of content is recorded with actor and timestamp" +reference = "21 CFR 11.10(e)" +risk = "high" +verification = "automated" +responsibility = "shared" +notes = "Retention duration of the audit log is a customer configuration decision." + +[controls.personnel-training] +description = "Personnel have the education, training and experience to perform their tasks" +reference = "21 CFR 11.10(i)" +risk = "medium" +verification = "procedural" +responsibility = "customer" +notes = "Evidenced by training records in your QMS. No automated test can establish this." +``` + +`description` is required. `verification` defaults to `"automated"` and must be one +of `"automated"`, `"manual"`, or `"procedural"`. VIP is regulation-agnostic: it +passes `reference`, `risk`, `responsibility`, and `notes` through to the output +verbatim without interpreting them. The `[controls.]` key is the id a scenario +references with `@control-` (with the `control-` tag prefix stripped). + +Those six keys plus `extra` are the whole recognised set, and any other key is an +error. Rejecting rather than ignoring is what catches a misspelled `referance` +before it vanishes from the matrix a reviewer reads. Put your own fields in an +`extra` table: + +``` toml +[controls.audit-trail.extra] +phase = "OQ" +sop = "SOP-QA-014" +``` + +Its keys must not start with a character a spreadsheet evaluates as a formula, +since a key becomes a CSV header cell and TOML permits a quoted key like +`"=HYPERLINK(...)"`. Its values must be strings, must not collide with an +existing column name, and +are passed through untouched: each becomes a trailing CSV column (appended after +the fixed set, so `CSV_COLUMNS` stays an identical leading prefix across every +customer's export) and an `extra` object per control in the JSON. Neither report +edition renders them, because the table has no width for a variable number of +columns. + +### The three coverage outcomes + +A control's row in the matrix gets one of three `coverage` values: + +- `covered` -- at least one scenario is tagged `@control-`, and its result is + attached. Coverage records that a scenario is tagged, not that it executed: a + skipped scenario still counts as covered, and its `skipped` status is reported + alongside. See "Covered is not the same as executed" below, which matters more + than it sounds like it should. +- `gap` -- no scenario has the tag, and `verification = "automated"` (the + default). This is the one that should worry you. +- `not_automatable` -- no scenario has the tag, but `verification` is + `"manual"` or `"procedural"`. This is *not* a gap. A control satisfied by a + personnel training record, a physical procedure, or a signature-manifestation + requirement that Posit Team's platform doesn't implement has no automated + scenario to point to, and reporting it as a gap would train reviewers to ignore + every real gap alongside it. Distinguishing the two is why `verification` exists + at all -- collapsing `not_automatable` into `gap` would make the matrix useless + for exactly the controls that need a human process instead of a test. + +### Coverage display states + +The rendered report displays coverage as `COVERED`, `FAILED`, `UNPROVEN`, +`NOT RUN`, `GAP`, or `N/A (manual)`: + +- `COVERED` -- at least one tagged scenario ran and passed. +- `FAILED` -- a covered control with at least one tagged scenario that ran and did + not pass. +- `UNPROVEN` -- a covered control with at least one tagged scenario that VIP was + asked to run and could not (`vip.attest.unproven`). +- `NOT RUN` -- at least one tagged scenario is present, but all of them skipped or + were not executed. +- `GAP` -- no tagged scenario is present and `verification = "automated"`. +- `N/A (manual)` -- no tagged scenario is present but `verification` is `"manual"` + or `"procedural"`. + +A control can satisfy more than one of these at once, and the display column +shows only the loudest. The order is `FAILED`, then `UNPROVEN`, then `NOT RUN`: +a control that ran and failed is the strongest claim against it. An +all-unproven control is also not executed, but `UNPROVEN` is the more +specific of the two, so it takes priority. Read the per-scenario `status` column for the rest. + +### Covered is not the same as executed + +A scenario can run and skip itself -- the endpoint it probes is absent from this +deployment, there is no data to inspect, a version gate excludes it -- and it +still keeps its control tag in the results file. A control tagged only by +such scenarios is therefore `covered`, and a matrix can read `covered: 3, +gaps: 0` while nothing was verified at all. That is the most misleading thing +this export can do, so it is reported three ways rather than left implicit: + +- `vip trace` warns on stderr, naming the affected control ids. +- The JSON `summary` reports `covered_and_executed` (a tagged scenario ran, + whether or not it passed), `covered_not_executed` (a tagged scenario is + present but none ran), `covered_failed` (a tagged scenario ran and did not + pass), and `covered_unproven` (a tagged scenario was a check VIP could not + run). These are not a partition: a failing control counts toward both + `covered_and_executed` and `covered_failed`, since it did run and it did not + pass, and an all-unproven control counts toward both `covered_not_executed` + and `covered_unproven`. The rendered report's own summary table partitions + differently -- it splits the display value into mutually exclusive rows, + `Covered, executed and passing`, `Covered, not executed`, `Covered, failing` + and `Covered, not verified`, so each control is counted once. Do not expect + the report table and the JSON `summary` to add up the same way. +- The JSON has a `covered_without_execution` list of control ids and a + `covered_with_unproven` list. + +A version-gated scenario (`na_version`) counts as not executed too, for the same +reason: it ran no assertions. So does an `unproven` one, which is the reason +`covered_unproven` exists alongside the other counts: an unproven skip is +neither an execution nor a failure, so a control with one passing scenario and +one unproven scenario is invisible to `covered_not_executed` and +`covered_failed` both, and part of the control still went unchecked. + +An unconfigured product is a different case, and it fails the opposite way. +Those scenarios are deselected rather than skipped -- excluded from the run +entirely, so they never reach `results.json` -- and a control tagged only by +them has nothing to join against, so it reports as a `gap`. An underconfigured +run therefore understates coverage rather than overstating it, which is the +safer direction, but a reader who takes the gap at face value concludes the +suite lacks a check it has. The `products` block records what was actually +under test. Read it alongside the gaps. + +Read `gaps: 0` together with `covered_not_executed` and `covered_unproven`. +Zero gaps and a non-zero `covered_not_executed` means the controls are mapped +and the evidence is missing. A non-zero `covered_unproven` means VIP was asked +for evidence it could not produce, which is the difference between a check with +nothing to test and a check that never got to run. + +### Worked example + +```bash +vip verify --config vip.toml --extensions ./examples/21CFR_part11_validation +vip trace --results report/results.json --controls ./examples/21CFR_part11_validation/controls.toml +``` + +`vip trace` defaults to CSV on stdout. Pass `--format json` for full fidelity output +(including nested match details), or `--output matrix.csv` / `--output matrix.json` +to write to a file instead. With `--output` and no `--format`, the format is taken +from the file extension, so `--output matrix.json` writes JSON. An explicit +`--format` always wins and warns when it disagrees with the extension. + +Both formats include the results digest. CSV repeats `generated_at`, `vip_version`, +`results_sha256` and `exit_status` on every row, which is enough to match the +archived spreadsheet to the exact `results.json` it came from. The full +provenance block -- the products and versions under test, the runner host, and the CI +run -- is JSON only, because it does not flatten into columns. CSV is the more portable format for spreadsheet tools, +but it alters what a non-Excel reader sees: any cell whose value begins with +`= + - @` or a leading tab/carriage-return/newline is apostrophe-prefixed +(`'=SUM(...)` instead of `=SUM(...)`) to stop it from being evaluated as a formula +when opened in Excel. JSON output is not altered this way -- use it when exact +fidelity to the underlying value matters more than spreadsheet safety. + +Control ids become pytest marker names, so they may use only letters, digits, +`-`, `.` and `_`. A `:` or `(` in an id (`11.10(a)`, `iso:27001`) truncates the +name pytest registers, which aborts collection under `--strict-markers`. VIP +warns and skips registering such a tag. Write `11-10-a` instead. + +### In the rendered report + +`vip report --controls PATH` adds a Compliance Traceability section to both the +HTML report and the PDF, with the summary counts, the per-control coverage, +and the scenario and timestamp evidencing each one. Without `--controls` there is +no section and nothing changes, which is the case for every run that has no +control list. + +```bash +vip report --results report/results.json --controls ./my-tests/controls.toml +``` + +The control list is scoped to that one render, passed to Quarto as `VIP_CONTROLS` +rather than copied into the report directory. That directory survives between +runs, so a copied file would make every later plain `vip report` sprout a +compliance section nobody asked for, built from a stale list. Rendering the +report documents directly with `quarto render` therefore needs `VIP_CONTROLS` +set by hand. + +The section repeats the same caveat the CSV and JSON exports state, because the +report is the artifact that gets archived and handed on: coverage records that a +scenario is tagged, a control shown as NOT RUN has a tagged scenario that ran +and skipped itself, and a control shown as UNPROVEN has one VIP was asked to run +and could not. See `examples/21CFR_part11_validation/VALIDATION-PACKAGE.md` for how these +outputs map onto a GxP +validation package, and which parts of one VIP cannot supply. + +`vip scaffold --template 21cfr-part11-validation --output DIR` generates a starting point +with a worked `controls.toml`, a tagged feature file, and the client methods +(`list_audit_logs`, `audit_log_allowed_methods`, `unauthenticated_status`) the +example scenarios use. diff --git a/docs/test-architecture.md b/docs/test-architecture.md index 3685f5a2..752f9f61 100644 --- a/docs/test-architecture.md +++ b/docs/test-architecture.md @@ -80,6 +80,55 @@ VIP supports running the same scenario through different channels using product Every feature file **must** have a product marker tag (`@connect`, `@workbench`, or `@package_manager`). The tag controls auto-skip: when a product is not configured, all scenarios with its tag are skipped automatically. Forgetting the tag breaks this mechanism and causes confusing failures. +### Compliance control tagging + +A scenario can declare which regulatory or compliance control it verifies with an +`@control-` Gherkin tag: + +```gherkin +@connect +Feature: 21 CFR Part 11 flavoured controls + + @control-audit-trail-publish + Scenario: Publishing content is recorded with an actor and a timestamp + Given Connect is accessible at the configured URL + When I list recent audit log entries + Then each entry records an actor and a timestamp +``` + +Control tags do not affect the derived marker. `src/vip/gherkin.py` derives a +feature's pytest marker from the first non-control tag it finds, skipping +`@control-` entirely, so `@control-audit-trail @connect` and +`@connect @control-audit-trail` both yield the marker `connect`. + +The product tag should still be the first tag that is not a control tag. Only control +tags are skipped, so any other tag ahead of the product tag becomes the marker instead +-- `@slow @connect` derives `slow`, not `connect`. The derived marker feeds the HTML +report's per-feature grouping and the generated test catalog and feature matrix, and +the product markers (`@connect`, `@workbench`, `@package_manager`) separately drive +auto-skip when a product is not configured. Getting this wrong mislabels the feature +in those outputs rather than raising an error, so it is worth a glance when adding a +tag. + +Control tags become registered pytest markers automatically. `vip.plugin` pre-scans +the feature files about to be collected and registers every `@control-` tag it +finds via `config.addinivalue_line("markers", ...)`, so a run under `--strict-markers` +(which regulated CI is likely to enable) does not fail on an unrecognized marker. + +Keep the slug to letters, digits, `-`, `.` and `_`. pytest derives a registered +marker's name by cutting at the first `:` or `(`, so `@control-11.10(a)` would +register as `control-11.10` while pytest-bdd applies the full tag, and collection +would then fail under `--strict-markers` against a marker list that looks like it +should have matched. VIP warns and skips registering such a tag rather than +registering the truncated name. Write `@control-11-10-a`. + +Control tags flow into `results.json` only, as entries in a test's `markers` list. +They do not appear in the `junit.xml` or `results.sarif` outputs produced by +`--format` — those formats predate control tagging and were not extended to +include it. If you need traceability evidence in CI artifacts beyond `results.json`, +consume it via `vip trace` (see `docs/reporting.md`), not by expecting it in JUnit +or SARIF. + ### Version gating Use the `min_version` marker for features that only exist in certain product versions: @@ -148,7 +197,7 @@ Two failure modes to watch for, both of which have bitten this suite: **Skip and unproven are not the same statement.** A skip says "there was nothing here to check, and the run is still complete". Unproven says "this was supposed to be checked and was not, so treat the run as incomplete". Collapsing the two is how `vip verify` came to exit 0 against a configured product whose every test had silently fallen away (#596): to anyone reading the report, "we did not look" was indistinguishable from "we looked and it was fine". -An unproven result carries through the whole pipeline -- its own `UNPROVEN` badge in the HTML report, an `UNPROVEN:`-prefixed message in JUnit, SARIF level `warning`, and exit code 6 from the run itself. `--allow-unproven` restores the old behaviour for pipelines that need it. Reach for `attest.unproven` whenever a *configured* capability goes unverified; reach for `attest.not_applicable` when skipping is the correct and final answer for this deployment. When in doubt, ask which one the person reading the report would want to be told. +An unproven result shows up throughout the pipeline -- its own `UNPROVEN` badge in the HTML report, an `UNPROVEN:`-prefixed message in JUnit, SARIF level `warning`, and exit code 6 from the run itself. `--allow-unproven` restores the old behaviour for pipelines that need it. Reach for `attest.unproven` whenever a *configured* capability goes unverified. Reach for `attest.not_applicable` when skipping is the correct and final answer for this deployment. When in doubt, ask which one the person reading the report would want to be told. Skips carry the same burden of accuracy as failures. A skip reason states *why* there was nothing to verify, so it must be true: `test_repos.py` used to report "package not available — repo may not be synced yet" after probing only the first repo whose name matched, when a synced mirror sitting beside it served the package fine. Probe every candidate before concluding anything, and name all of them in the reason. @@ -331,14 +380,16 @@ vip scaffold --template cross-product --output ./my-custom-tests ``` `--template` defaults to `cross-product` (the pre-existing behavior of `vip scaffold --output DIR` -is unchanged). Two canonical templates ship with VIP: +is unchanged). Three canonical templates ship with VIP: - `minimal` (`examples/custom_tests/`) — a single-scenario HTTP health check against your own configured product; the best starting point for a new extension - `cross-product` (`examples/cross_product_validation/`) — a full GxP validation example that verifies R/Python runtime versions and package installability across Connect and Workbench +- `21cfr-part11-validation` (`examples/21CFR_part11_validation/`) — compliance control tagging plus a + `controls.toml`, the worked starting point for a `vip trace` traceability matrix -Both follow the same four-layer architecture as the built-in suite. Every scaffolded directory +All three follow the same four-layer architecture as the built-in suite. Every scaffolded directory also gets an `AGENTS.md`, generated from a single shared source (`examples/_shared/AGENTS.md`), documenting the extension contract: the auto-skip rules, `min_version` gating, and an enumerated inventory of the public fixtures, registered markers, and client entry points an extension may diff --git a/examples/21CFR_part11_validation/README.md b/examples/21CFR_part11_validation/README.md new file mode 100644 index 00000000..ba0f2312 --- /dev/null +++ b/examples/21CFR_part11_validation/README.md @@ -0,0 +1,144 @@ +# 21 CFR Part 11 validation example + +A worked example of mapping regulatory controls to automated tests, and +exporting the result as a traceability matrix with `vip trace`. + +`VALIDATION-PACKAGE.md`, next to this file, is the wider context: which +documents in a GxP validation package VIP produces, which ones you author, and +which ones no test tool can produce. Read it before deciding what this example +is worth in your process. + +## What this is not + +This is a template, not a certified 21 CFR Part 11 test set. A fully green +matrix is evidence for the subset of controls you chose to automate. It is not +an attestation of 21 CFR Part 11 compliance, and nobody should present it as +one. + +Most of 21 CFR Part 11 cannot be evidenced by an automated test against Posit +Team. Roughly six clauses are genuinely testable against a deployment -- +11.10(a), 11.10(d), 11.10(e), 11.10(g), 11.30, and partly 11.10(b). Several +more are shared with your own procedures. The rest are either procedural +(11.10(i), 11.10(j)) or properties of the application you build on top of +Posit Team. + +In particular, Posit Team does not implement electronic signatures. Clause +11.50 (signature manifestations), 11.70 (signature/record linking) and all of +subpart C (11.100, 11.200, 11.300) are satisfied by your application and your +SOPs, not by Connect, Workbench or Package Manager. `controls.toml` includes +two such controls so you can see how a non-automatable control appears in the +matrix: as "not verifiable by automated test", which is deliberately distinct +from "coverage gap". + +A green matrix is also not evidence that the tests ran. A scenario that ran and +skipped itself -- the endpoint it probes is absent here, there was no data to +inspect, a version gate excluded it -- still counts as covering its control. +`vip trace` warns when that happens and the JSON summary reports +`covered_not_executed` separately, but the CSV `coverage` column alone will not +tell you. Read the `status` column with it. + +A gap can also mean the run rather than the suite. Scenarios belonging to a +product you have not configured are deselected rather than skipped, so they +never reach the results file, and a control tagged only by them reports as a +gap. This example spans all three products, so a partial run is the normal +case: point it at a deployment with Connect alone and the four Package Manager +and Workbench controls report as gaps, not as covered. That understates your +coverage rather than overstating it, but it is still a misreading: check which +products the run actually tested first. + +## What it covers + +Seven controls across the three products, chosen to show the range rather than +to be complete: + +| Product | Control | Clause | +|---|---|---| +| Connect | Publishing is recorded with an actor and a timestamp | 11.10(e) | +| Connect | A privileged action requires authorisation | 11.10(g) | +| Connect | The audit log offers no deletion method | 11.10(e) | +| Package Manager | A defined repository set is served | 11.10(a) | +| Package Manager | A past package set can still be retrieved | 11.10(a) | +| Workbench | An unauthenticated caller cannot reach the session API | 11.10(d) | +| Workbench | An authorised caller can reach the session API | 11.10(d) | + +The two Workbench scenarios are one control read from both sides. Refusing an +unauthenticated caller does not on its own evidence that access is limited to +authorised individuals, because a deployment that refuses everybody passes that +half too. + +Package Manager has the reproducibility control because a dated snapshot +URL is what lets you rebuild the package set an analysis ran against. Set +`validated_repo_name` and `validated_snapshot` in `conftest.py` to a repository +and date your deployment actually covers. An absent snapshot returns 404, which +the scenario reports as a skip, and a skipped scenario still counts as +coverage. + +## How it works + +A scenario declares the control it satisfies with a Gherkin tag: + +```gherkin +@control-audit-trail-publish +Scenario: Publishing content is recorded with an actor and a timestamp +``` + +A control tag can sit before or after the product tag without changing anything +-- VIP derives the feature's marker from the first non-control tag it finds, +skipping `@control-` entirely. But the product tag (`@connect`, +`@workbench`) should still be the first tag that is not a control tag, because +only control tags are skipped: `@slow @connect` derives the marker `slow`. The +derived marker feeds the HTML report's per-feature grouping and the generated +test catalog and feature matrix, while the product markers themselves +separately drive auto-skip when a product is not configured. + +`controls.toml` names each control and records whatever metadata your +regulatory mapping uses. Only `description` is required, and every field must be +a quoted string -- TOML would otherwise read `reference = 2024-01-01` as a date, +which the JSON export cannot serialise. + +The recognised keys are `description`, `reference`, `risk`, `verification`, +`responsibility` and `notes`. Anything else is an error rather than a silent +drop, which is what catches `referance` before it disappears from the matrix a +reviewer reads. Your own fields go in an `extra` table: + +```toml +[controls.audit-trail-publish] +description = "Deployment of content is recorded with actor and timestamp" +reference = "21 CFR 11.10(e)" + +[controls.audit-trail-publish.extra] +phase = "OQ" +sop = "SOP-QA-014" +``` + +Those become trailing columns in the CSV export, and an `extra` object per +control in the JSON. VIP does not interpret them and the report does not render +them -- its table has no width for a variable number of columns. A key may not +start with `=`, `+`, `-` or `@`, because the key becomes a CSV header cell and +a spreadsheet would evaluate it. + +Control ids become pytest marker names, so use only letters, digits, `-`, `.` +and `_`. Write `11-10-a`, not `11.10(a)`: a `:` or `(` truncates the marker name +pytest registers and breaks collection under `--strict-markers`. + +## Running it + +```bash +vip verify --config vip.toml --extensions ./21CFR_part11_validation +vip trace --results report/results.json --controls ./21CFR_part11_validation/controls.toml +``` + +Add `--format json` for a machine-readable matrix that includes the full provenance +block, or `--output matrix.csv` to write to a file. + +## Extending it + +Replace `controls.toml` with your own mapping and tag your own scenarios. One +feature file per product, because the product tag is feature-level. The +refusal assertion the Connect and Workbench scenarios share lives in +`part11_refusal.py` rather than in either step file: one pytest-bdd step module +cannot import another, since `@scenario` inspects the caller's frame at import +time. See +`security/test_auth_policy.py` in the VIP source for a fuller reference +implementation of access-control testing, and +`examples/cross_product_validation/` for the broader GxP starting point. diff --git a/examples/21CFR_part11_validation/VALIDATION-PACKAGE.md b/examples/21CFR_part11_validation/VALIDATION-PACKAGE.md new file mode 100644 index 00000000..75615811 --- /dev/null +++ b/examples/21CFR_part11_validation/VALIDATION-PACKAGE.md @@ -0,0 +1,225 @@ +# VIP and a GxP validation package + +A regulated customer asking "can VIP validate our Posit deployment" is usually +asking a narrower question than it sounds: which of the documents in their +validation package can this tool produce, and which must they still author. + +This page answers that plainly. It exists so nobody has to infer the boundary +from a feature list, and so a green report is never mistaken for something it +is not. + +## The short version + +VIP produces executed test evidence and the traceability that links it to your +controls. It does not produce the qualification protocols that evidence goes +into, and it cannot produce the documents that describe your organisation, +your risk posture, or your procedures. + +A validation package is mostly writing. VIP automates the part that is mostly +running. + +## What VIP supplies + +Executed test evidence, at scenario granularity. Every check records its +outcome, a start and finish timestamp, and the deployment it ran against. This +is the raw material an Operational Qualification is built from: evidence that a +specific function behaved as specified, on a specific system, at a specific +time. + +A requirement-to-evidence traceability matrix. You author `controls.toml` with +your own control list. Scenarios declare which control they verify with an +`@control-` Gherkin tag. `vip trace` joins the two and exports CSV or +JSON, and `vip report --controls` renders the same content into the HTML report +and the archivable PDF. The join is the deliverable: it is what lets a reviewer +go from a control to the check that evidences it without a spreadsheet +maintained by hand. + +Execution provenance. Each results file records who ran the tests, which host +they ran on, which git commit and branch they came from, whether that tree was +dirty, and which CI job produced them. The HTML report and the PDF render the +attribution as well, so the archived artifact includes it rather than only the +machine-readable output. This is what makes a result attributable to a named +operator and a pipeline execution rather than to an anonymous green tick. + +Set `VIP_PERFORMED_BY` to record the person accountable for the run. Without +it, VIP falls back to the CI system's actor, then to the local login, and the +report qualifies the name with where it came from -- `octocat (GitHub actor)` +rather than a bare `octocat`. An explicitly named operator is the only one +that renders unqualified, because it is the only one a human chose. A CI actor +is frequently a service account, and on a scheduled run it is whoever last +edited the workflow. `--vip-no-attribution` omits the whole block for anyone +who does not want an operator identity written into an archived file. + +Tamper-evidence. A `results.json.sha256` sidecar detects corruption in transit, +a truncated upload, or a file edited after the fact and not re-checksummed. + +Read that last one precisely. It is tamper-evidence within a trusted pipeline, +not tamper-proofing and not an immutable audit trail. Anyone who can edit the +results file can regenerate the sidecar to match. It catches accidents, which +is the failure that actually happens to archived CI artifacts. It does not +resist a motivated forger, and it must never be presented as though it does. + +## Where this sits in FDA's current thinking + +FDA finalised [Computer Software Assurance for Production and Quality +Management System +Software](https://www.fda.gov/regulatory-information/search-fda-guidance-documents/computer-software-assurance-production-and-quality-management-system-software) +in September 2025 and updated it in February 2026. Two passages in section +V.A.6, "Establishing the Appropriate Record", matter for anyone deciding what +VIP is worth. + +The first describes what VIP produces almost by name. Advances in digital +technology, the guidance says, "may allow for manufacturers to leverage +digital retention of results, automated traceability, automated testing, and +electronic capture of work performed as objective evidence, reducing the need +for manual or paper-based documentation." It then recommends, as the +least-burdensome approach, "incorporating the use of digital records, such as +system logs, audit trails, and other data generated and maintained by the +software, as opposed to paper documentation, screenshots, or duplicating +results already digitally retained by the software when establishing the +record associated with the assurance activities." A machine-generated results +file with per-scenario timestamps and execution provenance is the artifact +those sentences describe. Screenshots pasted into a Word protocol are what +they discourage. + +The second is the list of what the record should include. VIP covers, in whole +or in part: + +- the intended use of the feature or function, insofar as your control + descriptions state it +- the result of your risk-based analysis, read from `controls.toml` + and rendered in the matrix +- a description of the testing conducted, as the tagged scenario and its steps +- issues found during testing, as failed and skipped scenarios +- who performed the testing and the date it was performed + +Two items are yours. VIP writes no conclusion statement declaring +acceptability, and it makes no determination on the issues it records: +both are judgements, not test results. Neither is there any review and +approval signature, which the guidance asks for "when appropriate" and which +belongs in your quality system. + +None of this makes a VIP run a computer software assurance activity on its +own. The approach is risk-based, and the risk analysis that decides how much +assurance a function needs is yours. + +## What you author + +These are yours. No test tool produces them, because each one describes your +organisation rather than the software. + +A User Requirements Specification, stating what the system must do in your +process. VIP consumes the result of this work as a control list. It cannot +derive one. + +A Validation Master Plan, covering your validation policy, the organisational +structure and responsibilities, the inventory of systems in scope, and the +timelines and training that go with them. + +Risk assessments. Which controls matter, how badly, and why. `controls.toml` +has a `risk` field and VIP passes it through untouched, without +interpreting it. Deciding the value is the assessment. + +The qualification protocols themselves. Installation, Operational and +Performance Qualification documents are structured, pre-approved, signed +records. VIP supplies evidence that can go inside an OQ. It does not write the +protocol, define the acceptance criteria, or include the approval signatures. + +Standard Operating Procedures, training records, and the change control and +deviation records that surround a validated system. + +The validation summary report, and the Quality review and approval that closes +the exercise. + +## What nothing can automate + +Some controls have no automated test by their nature, and a matrix that +reported them as coverage gaps would train reviewers to ignore every real gap +sitting next to them. This is why `verification` exists in `controls.toml` and +why `not_automatable` is a distinct coverage value rather than folded into +`gap`. + +Procedural controls, such as whether personnel have the training and experience +to perform their tasks. The evidence is a training record in your quality +system. + +Controls satisfied by a physical or organisational process rather than by +software behaviour. + +Controls that are properties of the application you build on top of Posit Team, +not of the platform. Posit Team does not implement electronic signatures, so +21 CFR Part 11 clauses 11.50 and 11.70, and all of subpart C, are satisfied by +your application and your procedures. No amount of testing Connect, Workbench +or Package Manager will evidence them. + +## The gap worth knowing about + +VIP captures nothing below the scenario. A qualification protocol is written as +numbered steps, each with an expected result and an observed result, and a +reviewer follows it step by step. VIP records that a scenario passed, not that +step 3 of 7 produced the expected value. + +This is the largest remaining distance between VIP's output and a document +shaped like an executed protocol. If your process requires step-level evidence, +plan to record it another way for now. + +Two smaller absences worth stating: there is no cross-run deviation log, which +would need history VIP does not keep, and there is no cryptographic signing of +results. + +If you need results to resist a motivated forger rather than only to detect +corruption, sign them in your pipeline rather than waiting for VIP to grow its +own crypto. On GitHub Actions, `actions/attest-build-provenance` produces a +Sigstore-backed attestation over `results.json` that is recorded in a public +transparency log, and `gh attestation verify` checks it later. That gives you +what the sha256 sidecar deliberately does not claim: a signature the producer +cannot quietly regenerate. + +## Reading a green report + +A fully green matrix evidences the controls you chose to automate, on the +deployment you pointed it at, at the time it ran. That is all it claims. + +Five specific ways it can mislead if read carelessly: + +Coverage is not execution. A control counts as covered when a scenario is +tagged for it, whether or not that scenario reached an assertion. A scenario +that ran and skipped itself still covers its control: the endpoint it probes is +absent from this deployment, there was no data to inspect, or a version gate +excluded it. The report shows those controls as NOT RUN, `vip trace` warns on +stderr, and the JSON summary counts them under `covered_not_executed`. Read +`gaps: 0` together with that number. + +Execution is not verification either. A scenario can run, find that it cannot +check what it was asked to check, and record that as unproven rather than as a +pass. Such a control shows as UNPROVEN, warns on stderr, and is counted under +`covered_unproven`. This is the one case the two numbers above miss entirely: +a control with one passing scenario beside one unproven scenario is both +executed and not failing, so it reads as fully evidenced from those two alone +while part of the control went unchecked. + +A gap can be an artifact of the run rather than of your suite. Scenarios +belonging to a product you did not configure are deselected, not skipped, so +they never appear in the results file at all, and a control tagged only by them +reports as a gap. This errs toward understating your coverage, which is the +safer direction, but the reading is still wrong: the suite has the check, the +run did not exercise it. Check the products the report says were under test +before you conclude a control has no automated evidence. The worked example +next to this file maps controls across Connect, Workbench and Package Manager, +so a run covering one product shows gaps for the other two. + +Coverage is not completeness. The matrix reports on the controls in your +control list. A control you never wrote down cannot appear as a gap. + +A green report is not an attestation of compliance. It is one input to a +validation exercise that you own. + +## See also + +- `README.md`, next to this file, for the worked control list and tagged + scenarios, including the scope limits specific to 21 CFR Part 11 +- [the reporting guide](https://github.com/posit-dev/vip/blob/main/docs/reporting.md) + for the machine-readable outputs, the `vip trace` export, and the checksum + sidecar +- [the test architecture guide](https://github.com/posit-dev/vip/blob/main/docs/test-architecture.md) + for control tagging and the four-layer test architecture diff --git a/examples/21CFR_part11_validation/conftest.py b/examples/21CFR_part11_validation/conftest.py new file mode 100644 index 00000000..23f5135a --- /dev/null +++ b/examples/21CFR_part11_validation/conftest.py @@ -0,0 +1,48 @@ +"""Override points for the 21 CFR Part 11 example. + +Redefine these fixtures in your own conftest.py to point the scenarios at the +endpoints, repositories and snapshot dates your deployment exposes. +""" + +import pytest + + +@pytest.fixture +def connect_privileged_endpoint() -> str: + """An administrative Connect endpoint that must refuse an unauthenticated caller.""" + return "/__api__/v1/users" + + +@pytest.fixture +def workbench_privileged_endpoint() -> str: + """A Workbench endpoint that must refuse an unauthenticated caller. + + The session API, not ``/health-check`` -- the health endpoint answers + anonymously by design, so a refusal scenario against it would assert + nothing about access control. + + Override this if your deployment serves an SPA fallback here. Such a + deployment answers an anonymous GET with a 200 carrying a login shell, and + a status-only probe reads that as access granted and fails the scenario. + An endpoint that answers 401, 403 or a redirect gives the control real + evidence instead. + """ + return "/api/sessions" + + +@pytest.fixture +def validated_repo_name() -> str: + """The Package Manager repository the validated environment installs from.""" + return "cran" + + +@pytest.fixture +def validated_snapshot() -> str: + """A Package Manager snapshot the validated environment pins to. + + A ``YYYY-MM-DD`` date, or the id of a frozen repository URL. Set this to a + date your repository actually covers: a date before the repository existed + 404s, which the scenario reports as a skip rather than a failure, and a + skipped scenario still counts as covering its control in the matrix. + """ + return "2024-01-02" diff --git a/examples/21CFR_part11_validation/controls.toml b/examples/21CFR_part11_validation/controls.toml new file mode 100644 index 00000000..89cd7080 --- /dev/null +++ b/examples/21CFR_part11_validation/controls.toml @@ -0,0 +1,97 @@ +# Sample control list for `vip trace`. This is a TEMPLATE: replace these +# entries with your organisation's own regulatory mapping. VIP does not +# interpret `reference`, `risk` or `responsibility` -- they are carried +# through to the matrix verbatim. + +[controls.audit-trail-publish] +description = "Deployment of content is recorded with actor and timestamp" +reference = "21 CFR 11.10(e)" +risk = "high" +verification = "automated" +responsibility = "shared" +notes = "Retention duration of the audit log is a customer configuration decision." + +[controls.access-control-privileged-action] +description = "Only authorised individuals may perform privileged actions" +reference = "21 CFR 11.10(g)" +risk = "high" +verification = "automated" +responsibility = "shared" + +[controls.record-retention] +description = "Audit trail entries cannot be altered or deleted by ordinary users" +reference = "21 CFR 11.10(e)" +risk = "high" +verification = "automated" +responsibility = "shared" + +[controls.package-source-controlled] +description = "Analyses install packages from a defined, served repository set" +reference = "21 CFR 11.10(a)" +risk = "medium" +verification = "automated" +responsibility = "shared" +notes = """ +Evidences only that a repository set is served and named. Whether it holds the +right packages at the right versions belongs in your own qualification tests. +""" + +[controls.package-environment-reproducible] +description = "A past package set can be retrieved so an analysis can be rebuilt" +reference = "21 CFR 11.10(a)" +risk = "high" +verification = "automated" +responsibility = "shared" +notes = """ +Skips when the deployment has snapshots disabled or the pinned date predates +the repository, and a skip still counts as coverage. Set validated_snapshot to +a date your repository covers, then read the status column to confirm it ran. +""" + +[controls.access-control-session-api] +description = "An unauthenticated caller cannot reach the Workbench session API" +reference = "21 CFR 11.10(d)" +risk = "high" +verification = "automated" +responsibility = "shared" +notes = """ +Reads a status code, so two deployment shapes need the endpoint fixture +overridden. A session API that 404s to anonymous callers skips, because +refusal-by-hiding and a wrong path look identical from here. One that answers +a 200 login shell fails, because a status-only probe cannot tell that shell +from session data. +""" + +[controls.access-control-authorised-caller] +description = "An authorised caller can reach the Workbench session API" +reference = "21 CFR 11.10(d)" +risk = "medium" +verification = "automated" +responsibility = "shared" +notes = """ +The positive half of the control above: a deployment that refuses everyone +passes the negative half alone. Weaker than its counterpart, because it can +only pass or skip -- a run without --headless-auth or --interactive-auth +carries no credentials, which is indistinguishable from credentials being +rejected. Skips in that case rather than blaming the deployment. +""" + +[controls.personnel-training] +description = "Personnel have the education, training and experience to perform their tasks" +reference = "21 CFR 11.10(i)" +risk = "medium" +verification = "procedural" +responsibility = "customer" +notes = "Evidenced by training records in your QMS. No automated test can establish this." + +[controls.signature-manifestation] +description = "Signed records display the signer's printed name, date/time and meaning" +reference = "21 CFR 11.50" +risk = "high" +verification = "manual" +responsibility = "customer" +notes = """ +Posit Team does not implement electronic signatures. This control is satisfied +by the application you build on top of Posit Team, not by the platform, and is +listed here to show how a non-automatable control appears in the matrix. +""" diff --git a/examples/21CFR_part11_validation/part11_refusal.py b/examples/21CFR_part11_validation/part11_refusal.py new file mode 100644 index 00000000..24ed5798 --- /dev/null +++ b/examples/21CFR_part11_validation/part11_refusal.py @@ -0,0 +1,67 @@ +"""The refusal assertion shared by every access-control scenario here. + +Connect and Workbench both evidence 21 CFR 11.10(d) by asking a privileged +endpoint for a response without credentials, and both need to read the answer +the same way. That logic lives in a plain module rather than in either step +file, because importing one pytest-bdd step module from another raises +``IndexError`` -- ``@scenario`` inspects the caller's frame at import time. +A ``conftest.py`` would not work either: ``selftests`` exercises this function +inside a ``pytester`` subprocess that writes its own ``conftest.py``. +""" + +import pytest + + +def assert_refused(status: int) -> None: + """Assert the control that matters: unauthenticated access is not GRANTED. + + A bare ``in (401, 403)`` check fails a correctly-secured deployment fronted + by OIDC/SAML or a forward-auth gateway, which answers an unauthenticated + API call with a redirect (302/307) to a login page rather than a 401/403 -- + a deployment shape VIP explicitly supports. That redirect IS a refusal: + the request never reached the privileged endpoint unauthenticated. + + So the assertion is inverted: any 2xx is the one outcome that is actually + unsafe (credentials were not required), and that is what fails the + scenario. 401/403 and any 3xx are accepted as refusals. Every other status + is handled explicitly rather than falling through a bare comparison: a 5xx + means the deployment errored, which is not evidence the access control + works (or that it's broken) -- it is inconclusive, so the scenario fails + with a message that says so rather than passing silently. Anything else + unrecognized also fails explicitly, so a new status code shows up as a + named failure instead of a silent pass. + + A 404 is the one status that neither passes nor fails. Hiding a privileged + endpoint from anonymous callers is a real pattern, and Workbench's session + API 404s that way on deployments that serve an SPA fallback -- so failing + would paint a red row on a correctly secured deployment. But a 404 is also + what a mistyped endpoint fixture returns, and accepting it would pass a + scenario that probed nothing. Skipping refuses both readings: the matrix + shows the control as covered-not-executed, which is what this run actually + established. Point the fixture at an endpoint your deployment serves to + turn it into evidence. + + A 200 carrying an HTML login shell is the remaining blind spot. A + status-only probe reads it as access granted and fails, which overclaims -- + the shell is not session data. That failure stands rather than being + softened, because a wrong red on an unusual deployment shape is safer here + than a wrong green, and overriding the endpoint fixture resolves it. + """ + if 200 <= status < 300: + pytest.fail( + f"unauthenticated request was granted (status {status}); access control is not enforced" + ) + if status in (401, 403) or 300 <= status < 400: + return + if status == 404: + pytest.skip( + "the endpoint answered 404 to an unauthenticated caller; that may be " + "refusal-by-hiding or a path this deployment does not serve, and this " + "probe cannot tell them apart" + ) + if 500 <= status < 600: + pytest.fail( + f"deployment returned {status} for an unauthenticated request; a server " + "error is not evidence of a working access control" + ) + pytest.fail(f"unexpected status {status}; cannot confirm the request was refused") diff --git a/examples/21CFR_part11_validation/test_21CFR_part11_connect.feature b/examples/21CFR_part11_validation/test_21CFR_part11_connect.feature new file mode 100644 index 00000000..5f968014 --- /dev/null +++ b/examples/21CFR_part11_validation/test_21CFR_part11_connect.feature @@ -0,0 +1,23 @@ +@connect +Feature: 21 CFR Part 11 flavoured controls + As a validation lead in a regulated environment + I want automated evidence for the controls that can be automated + So that my traceability matrix is generated rather than hand-maintained + + @control-audit-trail-publish + Scenario: Publishing content is recorded with an actor and a timestamp + Given Connect is accessible at the configured URL + When I list recent audit log entries + Then each entry records an actor and a timestamp + + @control-access-control-privileged-action + Scenario: A privileged action requires authorisation + Given Connect is accessible at the configured URL + When I request a privileged administrative endpoint without credentials + Then the request is refused + + @control-record-retention + Scenario: The audit log does not offer a deletion method + Given Connect is accessible at the configured URL + When I ask which methods the audit log endpoint allows + Then deletion is not among them diff --git a/examples/21CFR_part11_validation/test_21CFR_part11_connect.py b/examples/21CFR_part11_validation/test_21CFR_part11_connect.py new file mode 100644 index 00000000..548f5556 --- /dev/null +++ b/examples/21CFR_part11_validation/test_21CFR_part11_connect.py @@ -0,0 +1,91 @@ +"""Step definitions for the 21 CFR Part 11 example's Connect scenarios. + +Every @scenario function carries a literal @pytest.mark.connect decorator: +feature-level Gherkin tags alone do not drive VIP's auto-skip in extension +directories. +""" + +import pytest +from part11_refusal import assert_refused +from pytest_bdd import given, scenario, then, when + + +@pytest.mark.connect +@scenario( + "test_21CFR_part11_connect.feature", + "Publishing content is recorded with an actor and a timestamp", +) +def test_audit_trail_publish(): + pass + + +@pytest.mark.connect +@scenario("test_21CFR_part11_connect.feature", "A privileged action requires authorisation") +def test_privileged_action_denied(): + pass + + +@pytest.mark.connect +@scenario("test_21CFR_part11_connect.feature", "The audit log does not offer a deletion method") +def test_audit_log_not_deletable(): + pass + + +@given("Connect is accessible at the configured URL") +def connect_accessible(connect_client): + if connect_client is None: + pytest.skip("Connect is not configured") + return connect_client + + +@when("I list recent audit log entries", target_fixture="audit_entries") +def list_audit_entries(connect_client): + entries = connect_client.list_audit_logs() + if entries is None: + pytest.skip("this deployment does not expose an audit log endpoint") + return entries + + +@then("each entry records an actor and a timestamp") +def entries_have_actor_and_timestamp(audit_entries): + if not audit_entries: + pytest.skip("no audit entries to inspect") + for entry in audit_entries: + assert entry.get("user_id") or entry.get("user_description"), ( + f"audit entry has no actor: {entry}" + ) + assert entry.get("time") or entry.get("timestamp"), f"audit entry has no timestamp: {entry}" + + +@when( + "I request a privileged administrative endpoint without credentials", + target_fixture="unauthenticated_status", +) +def request_privileged_endpoint(connect_client, connect_privileged_endpoint): + return connect_client.unauthenticated_status(connect_privileged_endpoint) + + +@then("the request is refused") +def request_refused(unauthenticated_status): + assert_refused(unauthenticated_status) + + +@when("I ask which methods the audit log endpoint allows", target_fixture="allowed_methods") +def audit_log_allowed_methods(connect_client): + """Read the advertised method set. Never issue a mutating request. + + This scenario must not DELETE a real audit record to prove records cannot + be deleted -- in a regulated deployment that record is the evidence, and + destroying it is the exact harm this control exists to prevent. + """ + methods = connect_client.audit_log_allowed_methods() + if methods is None: + pytest.skip("this deployment does not advertise allowed methods for the audit log") + return methods + + +@then("deletion is not among them") +def deletion_not_offered(allowed_methods): + assert "DELETE" not in allowed_methods, ( + f"audit log endpoint advertises DELETE; allowed methods: {sorted(allowed_methods)}" + ) diff --git a/examples/21CFR_part11_validation/test_21CFR_part11_packagemanager.feature b/examples/21CFR_part11_validation/test_21CFR_part11_packagemanager.feature new file mode 100644 index 00000000..be35b625 --- /dev/null +++ b/examples/21CFR_part11_validation/test_21CFR_part11_packagemanager.feature @@ -0,0 +1,17 @@ +@package_manager +Feature: 21 CFR Part 11 flavoured controls for Package Manager + As a validation lead in a regulated environment + I want evidence that analyses draw on a controlled, reconstructable package set + So that a record can be traced back to the software that produced it + + @control-package-source-controlled + Scenario: The deployment serves a defined set of repositories + Given Package Manager is accessible at the configured URL + When I list the configured repositories + Then at least one repository is served, and each one is named + + @control-package-environment-reproducible + Scenario: A past package set can still be retrieved + Given Package Manager is accessible at the configured URL + When I request the package index for the validated snapshot + Then the snapshot's index is served diff --git a/examples/21CFR_part11_validation/test_21CFR_part11_packagemanager.py b/examples/21CFR_part11_validation/test_21CFR_part11_packagemanager.py new file mode 100644 index 00000000..6c7a6522 --- /dev/null +++ b/examples/21CFR_part11_validation/test_21CFR_part11_packagemanager.py @@ -0,0 +1,96 @@ +"""Step definitions for the 21 CFR Part 11 example's Package Manager +scenarios. + +Every @scenario function carries a literal @pytest.mark.package_manager +decorator: feature-level Gherkin tags alone do not drive VIP's auto-skip in +extension directories. +""" + +import pytest +from pytest_bdd import given, scenario, then, when + + +@pytest.mark.package_manager +@scenario( + "test_21CFR_part11_packagemanager.feature", + "The deployment serves a defined set of repositories", +) +def test_package_source_controlled(): + pass + + +@pytest.mark.package_manager +@scenario( + "test_21CFR_part11_packagemanager.feature", + "A past package set can still be retrieved", +) +def test_package_environment_reproducible(): + pass + + +@given("Package Manager is accessible at the configured URL") +def pm_accessible(pm_client): + if pm_client is None: + pytest.skip("Package Manager is not configured") + return pm_client + + +@when("I list the configured repositories", target_fixture="repositories") +def list_repositories(pm_client): + return pm_client.list_repos() + + +@then("at least one repository is served, and each one is named") +def repositories_are_named(repositories): + """Evidence for 11.10(a): the package supply is defined rather than ad hoc. + + A deployment whose analyses install from the open internet cannot say what + software produced a record. One named repository is the weakest form of + that evidence, and it is deliberately all this scenario claims -- whether + the repository holds the right packages is a question for your own + qualification tests, not for a control-mapping example. + """ + assert repositories, "Package Manager serves no repositories" + for repo in repositories: + assert repo.get("name"), f"repository has no name: {repo}" + + +@when("I request the package index for the validated snapshot", target_fixture="snapshot_result") +def request_snapshot_index(pm_client, validated_repo_name, validated_snapshot): + """Read a dated index. Never mutate the repository. + + Package Manager's contribution to 11.10(a) is that a package set can be + addressed at a point in time, so an analysis run last year can be rebuilt + from the same inputs today. + """ + return pm_client.snapshot_index_reachable(validated_repo_name, validated_snapshot) + + +@then("the snapshot's index is served") +def snapshot_index_served(snapshot_result, validated_repo_name, validated_snapshot): + """Separate "this deployment cannot answer" from "this deployment answered wrong". + + A 404 means snapshots are switched off, or that date predates the + repository. A 401/403 means the repository is an authenticated one and this + run carried no token. Both are configuration facts about the deployment or + the run rather than failed controls, so the scenario skips. A 5xx means the + server broke while answering, which fails. Read either skip in the matrix as + covered-not-executed: it is not evidence the control holds. + """ + found, status = snapshot_result + if found: + return + if status == 404: + pytest.skip( + f"no snapshot {validated_snapshot} for repository {validated_repo_name}; " + "snapshots may be disabled, or the date may predate the repository" + ) + if status in (401, 403): + pytest.skip( + f"repository {validated_repo_name} requires a token this run did not carry; " + "set VIP_PACKAGE_MANAGER_TOKEN, or point validated_repo_name at an open repository" + ) + assert found, ( + f"snapshot {validated_snapshot} of repository {validated_repo_name} " + f"did not serve a package index (status {status})" + ) diff --git a/examples/21CFR_part11_validation/test_21CFR_part11_workbench.feature b/examples/21CFR_part11_validation/test_21CFR_part11_workbench.feature new file mode 100644 index 00000000..e36fd859 --- /dev/null +++ b/examples/21CFR_part11_validation/test_21CFR_part11_workbench.feature @@ -0,0 +1,17 @@ +@workbench +Feature: 21 CFR Part 11 flavoured controls for Workbench + As a validation lead in a regulated environment + I want evidence that interactive analysis is limited to authorised individuals + So that the work behind a record can be attributed to an account + + @control-access-control-session-api + Scenario: An unauthenticated caller cannot reach the session API + Given Workbench is accessible at the configured URL + When I request the Workbench session API without credentials + Then the request is refused + + @control-access-control-authorised-caller + Scenario: An authorised caller can reach the session API + Given Workbench is accessible at the configured URL + When I request the Workbench session API with the test credentials + Then a session listing is returned diff --git a/examples/21CFR_part11_validation/test_21CFR_part11_workbench.py b/examples/21CFR_part11_validation/test_21CFR_part11_workbench.py new file mode 100644 index 00000000..75f13c5e --- /dev/null +++ b/examples/21CFR_part11_validation/test_21CFR_part11_workbench.py @@ -0,0 +1,91 @@ +"""Step definitions for the 21 CFR Part 11 example's Workbench scenarios. + +The two scenarios are one control read from both sides. Refusing an +unauthenticated caller is not on its own evidence that access is limited to +authorised individuals -- a deployment that refuses everybody passes that half +too. Pairing it with a granted request is the standard positive-and-negative +form of an access-control test, and 11.10(d) asks for both halves. + +Every @scenario function carries a literal @pytest.mark.workbench decorator: +feature-level Gherkin tags alone do not drive VIP's auto-skip in extension +directories. +""" + +import pytest +from part11_refusal import assert_refused +from pytest_bdd import given, scenario, then, when + + +@pytest.mark.workbench +@scenario( + "test_21CFR_part11_workbench.feature", + "An unauthenticated caller cannot reach the session API", +) +def test_session_api_refuses_anonymous(): + pass + + +@pytest.mark.workbench +@scenario( + "test_21CFR_part11_workbench.feature", + "An authorised caller can reach the session API", +) +def test_session_api_serves_authorised_caller(): + pass + + +@given("Workbench is accessible at the configured URL") +def workbench_accessible(workbench_client): + if workbench_client is None: + pytest.skip("Workbench is not configured") + return workbench_client + + +@when( + "I request the Workbench session API without credentials", + target_fixture="unauthenticated_status", +) +def request_session_api_anonymously(workbench_client, workbench_privileged_endpoint): + return workbench_client.unauthenticated_status(workbench_privileged_endpoint) + + +@then("the request is refused") +def request_refused(unauthenticated_status): + assert_refused(unauthenticated_status) + + +@when( + "I request the Workbench session API with the test credentials", + target_fixture="session_api_usable", +) +def request_session_api_authenticated(workbench_client): + """Ask whether the session API answers this client with a usable listing. + + ``sessions_api_reachable`` requires a 200 whose body parses as a JSON + array, so a login redirect or an HTML SPA fallback counts as unusable + rather than as success. + """ + return workbench_client.sessions_api_reachable() + + +@then("a session listing is returned") +def session_listing_returned(session_api_usable): + """Pass or skip. This scenario cannot fail, and that is a real limitation. + + The client only holds a Workbench session under ``--headless-auth`` / + ``--interactive-auth`` or with an API key. An unusable answer therefore has + two causes that cannot be told apart from here: the run carried no + credentials at all (correct behaviour, nothing to assert), or it carried + credentials the deployment rejected (a genuine control failure). Blaming + the deployment for the first case would fail this scenario on every run + that skips authentication, so it skips instead. + + That makes this half weaker than its negative counterpart, which does fail + on a real defect. Read the skip in the matrix as covered-not-executed, and + run with authentication if you want the control exercised. + """ + if not session_api_usable: + pytest.skip( + "the session API did not return a usable listing; run with " + "--headless-auth or --interactive-auth to exercise this control" + ) diff --git a/pyproject.toml b/pyproject.toml index 1cba8d6b..e3a06f7f 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -157,6 +157,7 @@ packages = ["src/vip", "src/vip_tests"] [tool.hatch.build.targets.wheel.force-include] "examples/cross_product_validation" = "vip/_scaffold/cross_product_validation" "examples/custom_tests" = "vip/_scaffold/custom_tests" +"examples/21CFR_part11_validation" = "vip/_scaffold/21CFR_part11_validation" # AGENTS.md source shared by every scaffold template; copied into the output # dir by run_scaffold() after copytree(). Keep this path in sync with # _resolve_scaffold_source("_shared") in src/vip/cli.py. diff --git a/report/index.qmd b/report/index.qmd index 2c8fe20e..008b457c 100644 --- a/report/index.qmd +++ b/report/index.qmd @@ -58,3 +58,52 @@ if data.results: )) display(HTML(report_html.render_actionable_cards(data, hints))) ``` + +```{python} +#| echo: false +# Rendered only when VIP_CONTROLS names a control list. Scoped to one render +# via the environment rather than a file copied into this directory: the +# report directory survives between runs, so a copied controls.toml would make +# every later plain `vip report` sprout a compliance section nobody asked for, +# built from a stale list. +# +# Nothing here may raise. A failure in a notebook cell renders as a traceback +# instead of a report, so a missing or malformed control list, or a results +# checksum that fails verification, becomes a visible marker in the section +# instead -- the except branch below still shows the heading rather than +# dropping the section silently. +import os + +_controls = os.environ.get("VIP_CONTROLS") +if _controls: + try: + from vip.traceability import ( + build_traceability_matrix, + load_controls, + verify_results_checksum, + ) + + _results_path = Path("results.json") + # Verify rather than only digest. --controls makes this a compliance + # artifact, and the sidecar attestation in the provenance block is + # worthless if nothing checked it. verify_results_checksum raises, and + # the except below is now a visible marker rather than a dropped + # section, so raising here is safe. + _digest, _verified = verify_results_checksum(_results_path) + _matrix = build_traceability_matrix( + data, + load_controls(_controls), + results_sha256=_digest, + results_sha256_sidecar_verified=_verified or None, + ) + display(Markdown("## Compliance Traceability")) + display(HTML(report_html.render_traceability(_matrix))) + except Exception as exc: # noqa: BLE001 - a report must render regardless + display(Markdown("## Compliance Traceability")) + # HTML, not Markdown: IPython.display.Markdown passes raw HTML + # through in Quarto, and this message carries the VIP_CONTROLS path + # and control ids out of a customer-authored controls.toml. + # render_traceability_error escapes it the way every other value + # report_html emits is escaped. + display(HTML(report_html.render_traceability_error(exc))) +``` diff --git a/report/styles.css b/report/styles.css index f5f5abb7..f0081b1c 100644 --- a/report/styles.css +++ b/report/styles.css @@ -140,6 +140,21 @@ letter-spacing: 0.03em; } +/* Compliance traceability section (see report_html.render_traceability). Both + HTML and Typst editions render the caveat in gray italic and the warning in + red bold, so they must be styled to match. */ +.trace-caveat { + font-style: italic; + font-size: 0.875rem; + color: #6b7280; +} + +.trace-warning { + font-size: 0.875rem; + color: #dc2626; + font-weight: 700; +} + .vip-test-meta { font-size: 0.75rem; color: #9ca3af; diff --git a/report/vip-report.qmd b/report/vip-report.qmd index 196dacb7..1acb4e49 100644 --- a/report/vip-report.qmd +++ b/report/vip-report.qmd @@ -34,6 +34,7 @@ prints survives inside it. #| echo: false #| output: asis +import os from pathlib import Path from vip import report_typst @@ -43,7 +44,31 @@ data = load_results(Path("results.json")) _troubleshooting_path = troubleshooting_path() hints = load_troubleshooting(_troubleshooting_path) if _troubleshooting_path else {} +# See index.qmd for why the control list arrives by environment variable and +# why nothing in this cell may raise. +matrix = None +trace_error = None +_controls = os.environ.get("VIP_CONTROLS") +if _controls: + try: + from vip.traceability import ( + build_traceability_matrix, + load_controls, + verify_results_checksum, + ) + + _digest, _verified = verify_results_checksum(Path("results.json")) + matrix = build_traceability_matrix( + data, + load_controls(_controls), + results_sha256=_digest, + results_sha256_sidecar_verified=_verified or None, + ) + except Exception as exc: # noqa: BLE001 - a report must render regardless + matrix = None + trace_error = str(exc) + print("```{=typst}") -print(report_typst.render_document(data, hints)) +print(report_typst.render_document(data, hints, matrix, trace_error)) print("```") ```` diff --git a/selftests/conftest.py b/selftests/conftest.py index 4d068621..db4eeca3 100644 --- a/selftests/conftest.py +++ b/selftests/conftest.py @@ -192,3 +192,35 @@ def sample_results_json(tmp_path: Path) -> Path: p = tmp_path / "results.json" p.write_text(json.dumps(data)) return p + + +_SKIP_STATUSES = frozenset({"skipped", "na_version", "unproven"}) + + +def matrix_from_statuses(statuses: dict[str, list[str]]): + """Build a TraceabilityMatrix from {control_id: [scenario status, ...]}. + + One TestResult per status, tagged `control-`. A status of "na_version" + is written as a version-gated skip, and "unproven" as an attested one, + which is how the plugin records each. + """ + from vip.reporting import ReportData, TestResult + from vip.traceability import ControlSpec, build_traceability_matrix + + results = [] + for control_id, control_statuses in statuses.items(): + for i, status in enumerate(control_statuses): + results.append( + TestResult( + nodeid=f"test_{control_id}.py::test_{i}", + outcome="skipped" if status in _SKIP_STATUSES else status, + na_version=status == "na_version", + unproven=status == "unproven", + markers=[f"control-{control_id}"], + ) + ) + controls = { + cid: ControlSpec(control_id=cid, description=f"control {cid}", verification="automated") + for cid in statuses + } + return build_traceability_matrix(ReportData(results=results), controls) diff --git a/selftests/test_21CFR_part11_example.py b/selftests/test_21CFR_part11_example.py new file mode 100644 index 00000000..4a024219 --- /dev/null +++ b/selftests/test_21CFR_part11_example.py @@ -0,0 +1,202 @@ +import subprocess +import sys +from pathlib import Path + +import pytest +from _pytest.outcomes import Skipped + +# tomllib is stdlib only from 3.11; tomli backfills it on 3.10, which CI runs. +# Same guard as src/vip/traceability.py. +if sys.version_info >= (3, 11): + import tomllib +else: + import tomli as tomllib + +REPO = Path(__file__).resolve().parent.parent +EXAMPLE = REPO / "examples" / "21CFR_part11_validation" + + +PRODUCTS = ("connect", "packagemanager", "workbench") + + +def test_example_directory_exists(): + for product in PRODUCTS: + assert (EXAMPLE / f"test_21CFR_part11_{product}.feature").is_file() + assert (EXAMPLE / f"test_21CFR_part11_{product}.py").is_file() + assert (EXAMPLE / "part11_refusal.py").is_file() + assert (EXAMPLE / "controls.toml").is_file() + assert (EXAMPLE / "README.md").is_file() + + +def test_template_is_registered(): + from vip.cli import _SCAFFOLD_TEMPLATES + + assert "21cfr-part11-validation" in _SCAFFOLD_TEMPLATES + assert _SCAFFOLD_TEMPLATES["21cfr-part11-validation"][0] == "21CFR_part11_validation" + + +def test_template_is_bundled_into_the_wheel(): + """A template missing from force-include works in-repo and breaks when installed.""" + config = tomllib.loads((REPO / "pyproject.toml").read_text()) + includes = config["tool"]["hatch"]["build"]["targets"]["wheel"]["force-include"] + assert includes["examples/21CFR_part11_validation"] == "vip/_scaffold/21CFR_part11_validation" + + +def test_every_control_tag_is_defined_in_controls_toml(): + tags = { + tok.lstrip("@") + for path in EXAMPLE.glob("test_21CFR_part11_*.feature") + for line in path.read_text().splitlines() + for tok in line.split() + if tok.startswith("@control-") + } + assert tags, "no control tags found -- the glob stopped matching the feature files" + controls = tomllib.loads((EXAMPLE / "controls.toml").read_text())["controls"] + for tag in tags: + assert tag.removeprefix("control-") in controls, f"{tag} missing from controls.toml" + + +def test_controls_toml_shows_the_not_automatable_path(): + """The worked example must demonstrate more than the happy path.""" + controls = tomllib.loads((EXAMPLE / "controls.toml").read_text())["controls"] + verifications = {c.get("verification", "automated") for c in controls.values()} + responsibilities = {c.get("responsibility") for c in controls.values()} + assert verifications & {"manual", "procedural"} + assert "customer" in responsibilities + + +def test_readme_states_it_is_not_an_attestation(): + text = (EXAMPLE / "README.md").read_text().lower() + assert "not a certified" in text or "not an attestation" in text + assert "electronic signature" in text + + +def test_scenarios_carry_literal_product_markers(): + """Feature-level Gherkin tags alone do not drive auto-skip in extensions. + + Counts per file so a product whose step file forgot the decorator cannot + hide behind another product's total. + """ + for product, marker in ( + ("connect", "@pytest.mark.connect"), + ("packagemanager", "@pytest.mark.package_manager"), + ("workbench", "@pytest.mark.workbench"), + ): + path = EXAMPLE / f"test_21CFR_part11_{product}.py" + lines = [line.strip() for line in path.read_text().splitlines()] + # Decorator lines only -- a docstring naming the marker is not one. + markers = lines.count(marker) + scenarios = sum(1 for line in lines if line.startswith("@scenario(")) + assert markers == scenarios, ( + f"{path.name} has {scenarios} scenarios but {markers} {marker} decorators" + ) + + +def test_all_three_products_are_covered(): + """The example maps controls for the whole of Posit Team, not just Connect.""" + tags = { + line.split()[0] + for path in EXAMPLE.glob("test_21CFR_part11_*.feature") + for line in path.read_text().splitlines() + if line.startswith("@") + } + assert {"@connect", "@package_manager", "@workbench"} <= tags + + +def test_request_refused_step(pytester): + """Unit-test ``assert_refused`` directly, no live deployment. + + The logic lives in ``part11_refusal`` rather than a step file because + Connect and Workbench both use it, and one pytest-bdd step module cannot + import another (``@scenario`` inspects the caller's frame at import time). + + A correctly-secured deployment fronted by OIDC/SAML or a forward-auth + gateway answers an unauthenticated call with a redirect (302/307), not a + bare 401/403 -- that must still count as a refusal. Only a 2xx (access + actually granted) is a real control failure. A 5xx is a broken + deployment, not evidence either way, so it fails with its own distinct + message rather than being silently accepted as a refusal. A 404 is neither: + it is what both refusal-by-hiding and a mistyped endpoint fixture return, + so it skips rather than claiming evidence the run does not have. + """ + (pytester.path / "conftest.py").write_text( + f"import sys\nsys.path.insert(0, {str(EXAMPLE)!r})\n" + ) + pytester.makepyfile( + test_refused=""" +import pytest +from part11_refusal import assert_refused + + +@pytest.mark.parametrize("status", [401, 403, 302, 307]) +def test_accepts_refusal_statuses(status): + assert_refused(status) + + +def test_fails_when_access_is_granted(): + assert_refused(200) + + +def test_fails_on_server_error_rather_than_passing(): + assert_refused(500) + + +def test_skips_on_404_rather_than_passing_or_failing(): + assert_refused(404) +""" + ) + result = pytester.runpytest_subprocess("-p", "no:cacheprovider", "-rs") + result.assert_outcomes(passed=4, failed=2, skipped=1) + output = result.stdout.str() + assert "access control is not enforced" in output + assert "not evidence of a working access control" in output + + +def test_refusal_404_skip_explains_both_readings(monkeypatch): + """The 404 skip must say why it is not a pass, in-process. + + Asserted here rather than in the subprocess above because VIP's plugin owns + the terminal reporter, which drops skip reasons from the log even under + ``-rs``. Importing ``part11_refusal`` directly is safe -- it holds no + ``@scenario``, which is the whole reason the assertion lives in its own + module. + """ + monkeypatch.syspath_prepend(str(EXAMPLE)) + import part11_refusal + + with pytest.raises(Skipped) as excinfo: + part11_refusal.assert_refused(404) + message = str(excinfo.value) + assert "refusal-by-hiding" in message + assert "cannot tell them apart" in message + + +def test_example_collects(tmp_path): + """Collect with all three products "configured" so nothing is deselected. + + Without a config naming a product, the plugin's product-config gate + (``_should_deselect_for_product``) deselects that product's scenarios outright + -- "no test at all", not "collected but skipped" -- which pytest reports + as exit code 5. See ``test_workbench_ordering.py`` for the same pattern. + """ + config_path = tmp_path / "vip.toml" + config_path.write_text( + '[connect]\nurl = "https://connect.example.com"\n' + '[workbench]\nurl = "https://workbench.example.com"\n' + '[package_manager]\nurl = "https://pm.example.com"\n' + ) + proc = subprocess.run( + [ + sys.executable, + "-m", + "pytest", + str(EXAMPLE), + "--collect-only", + "-q", + f"--vip-config={config_path}", + ], + capture_output=True, + text=True, + cwd=REPO, + ) + assert proc.returncode == 0, proc.stdout + proc.stderr diff --git a/selftests/test_attribution.py b/selftests/test_attribution.py new file mode 100644 index 00000000..9c43a3ad --- /dev/null +++ b/selftests/test_attribution.py @@ -0,0 +1,251 @@ +import json +import subprocess +import sys +from pathlib import Path + +import pytest + +from vip.attribution import ( + _git, + _performed_by, + collect_execution_metadata, + redact_userinfo, +) + + +def _init_repo(path): + def git(*args): + subprocess.run(["git", *args], cwd=path, check=True, capture_output=True) + + git("init") + git("config", "user.email", "t@example.com") + git("config", "user.name", "Test") + git("config", "commit.gpgsign", "false") + (path / "f.txt").write_text("hello") + git("add", "f.txt") + git("commit", "-m", "initial") + + +def test_git_metadata_from_a_real_repo(tmp_path): + _init_repo(tmp_path) + meta = collect_execution_metadata(cwd=tmp_path, env={}) + assert meta["git"]["commit"] is not None + assert len(meta["git"]["commit"]) == 40 + assert meta["git"]["dirty"] is False + + +def test_dirty_worktree_is_reported(tmp_path): + _init_repo(tmp_path) + (tmp_path / "f.txt").write_text("changed") + assert collect_execution_metadata(cwd=tmp_path, env={})["git"]["dirty"] is True + + +def test_non_repo_yields_null_git(tmp_path): + assert collect_execution_metadata(cwd=tmp_path, env={})["git"] is None + + +def test_no_ci_env_yields_null_ci(tmp_path): + assert collect_execution_metadata(cwd=tmp_path, env={})["ci"] is None + + +def test_github_actions_run_url_is_composed(tmp_path): + env = { + "GITHUB_ACTIONS": "true", + "GITHUB_RUN_ID": "42", + "GITHUB_RUN_ATTEMPT": "1", + "GITHUB_SERVER_URL": "https://github.com", + "GITHUB_REPOSITORY": "posit-dev/vip", + "GITHUB_JOB": "connect-smoke", + } + ci = collect_execution_metadata(cwd=tmp_path, env=env)["ci"] + assert ci["provider"] == "github" + assert ci["run_url"] == "https://github.com/posit-dev/vip/actions/runs/42" + assert ci["job"] == "connect-smoke" + + +def test_gitlab_and_jenkins_are_recognized(tmp_path): + gl = collect_execution_metadata( + cwd=tmp_path, env={"GITLAB_CI": "true", "CI_JOB_URL": "https://gl/x/-/jobs/9"} + )["ci"] + assert gl["provider"] == "gitlab" + assert gl["run_url"] == "https://gl/x/-/jobs/9" + + jk = collect_execution_metadata( + cwd=tmp_path, env={"JENKINS_URL": "https://j/", "BUILD_URL": "https://j/job/x/7/"} + )["ci"] + assert jk["provider"] == "jenkins" + assert jk["run_url"] == "https://j/job/x/7/" + + +def test_env_sha_takes_precedence_over_subprocess(tmp_path): + _init_repo(tmp_path) + env = {"GITHUB_SHA": "a" * 40, "GITHUB_REF_NAME": "feature/x"} + meta = collect_execution_metadata(cwd=tmp_path, env=env) + assert meta["git"]["commit"] == "a" * 40 + assert meta["git"]["branch"] == "feature/x" + + +@pytest.mark.parametrize( + ("raw", "expected"), + [ + ("https://x-access-token:ghs_SECRET@github.com/o/r", "https://github.com/o/r"), + ("https://user:pw@example.com:8443/o/r.git", "https://example.com:8443/o/r.git"), + ("https://github.com/o/r", "https://github.com/o/r"), + ("git@github.com:o/r.git", "github.com:o/r.git"), + # urlsplit accepts this and only raises when .port is read. + ("https://example.com:bad/repo", None), + ("", None), + (None, None), + ], +) +def test_userinfo_is_redacted(raw, expected): + from vip.attribution import redact_userinfo + + assert redact_userinfo(raw) == expected + + +def test_hostname_is_recorded(tmp_path): + assert collect_execution_metadata(cwd=tmp_path, env={})["hostname"] + + +def test_no_attribution_flag_omits_the_block(pytester): + pytester.makepyfile(test_x="def test_ok(): assert True") + report = pytester.path / "results.json" + pytester.runpytest_subprocess( + "--vip-report", str(report), "--vip-no-attribution", "-p", "no:cacheprovider" + ) + assert json.loads(report.read_text())["execution"] is None + + +def test_credential_in_remote_never_reaches_the_file(pytester, monkeypatch): + """Regression guard: assert the token is absent from the whole file.""" + secret = "ghs_THISMUSTNOTAPPEAR" + subprocess.run(["git", "init"], cwd=pytester.path, check=True, capture_output=True) + subprocess.run( + ["git", "remote", "add", "origin", f"https://x-access-token:{secret}@github.com/o/r"], + cwd=pytester.path, + check=True, + capture_output=True, + ) + for key, value in ( + ("user.email", "t@example.com"), + ("user.name", "T"), + ("commit.gpgsign", "false"), + ): + subprocess.run( + ["git", "config", key, value], cwd=pytester.path, check=True, capture_output=True + ) + pytester.makepyfile(test_x="def test_ok(): assert True") + subprocess.run(["git", "add", "-A"], cwd=pytester.path, check=True, capture_output=True) + subprocess.run(["git", "commit", "-m", "c"], cwd=pytester.path, check=True, capture_output=True) + + report = pytester.path / "results.json" + pytester.runpytest_subprocess("--vip-report", str(report), "-p", "no:cacheprovider") + assert secret not in report.read_text() + + +class TestNeverFails: + """The module's contract: a probe failure must never break a run. + + These sit at the top of a chain that ends in results.json, junit.xml, + results.sarif and failures.json all going unwritten -- attribution is + collected while building the payload, above the try/except that writes it. + """ + + def test_deleted_working_directory_degrades_instead_of_raising(self, tmp_path, monkeypatch): + def boom(): + raise FileNotFoundError(2, "No such file or directory") + + monkeypatch.setattr(Path, "cwd", staticmethod(boom)) + meta = collect_execution_metadata(env={}) + assert meta["git"] is None + assert meta["hostname"] is not None + + def test_undecodable_git_output_does_not_raise(self, tmp_path, monkeypatch): + """text=True would decode with the locale codec and errors='strict'.""" + import subprocess + + real_run = subprocess.run + + def fake_run(args, **kwargs): + assert kwargs.get("encoding") == "utf-8" + assert kwargs.get("errors") == "replace" + return real_run([sys.executable, "-c", "print('ok')"], **kwargs) + + monkeypatch.setattr(subprocess, "run", fake_run) + assert _git(["rev-parse", "HEAD"], tmp_path) == "ok" + + +class TestRedactUserinfoEdgeCases: + def test_ipv6_host_keeps_its_brackets(self): + assert redact_userinfo("https://u:p@[2001:db8::1]:8443/o/r.git") == ( + "https://[2001:db8::1]:8443/o/r.git" + ) + + def test_hostless_remote_is_preserved(self): + """file:// has no hostname and no credential; dropping it deletes provenance.""" + assert redact_userinfo("file:///srv/git/repo.git") == "file:///srv/git/repo.git" + + @pytest.mark.parametrize( + "url", + [ + "/srv/git/repo@2.git", + # Relative local paths: an "@" alone is not userinfo. scp syntax is + # user@host:path, so without a colon in the host there is nothing + # to strip. + "repo@2.git", + "releases@2026/repo.git", + ], + ) + def test_path_containing_an_at_sign_is_not_truncated(self, url): + assert redact_userinfo(url) == url + + def test_scp_style_still_strips_the_user(self): + assert redact_userinfo("git@github.com:org/repo.git") == "github.com:org/repo.git" + + def test_credentialless_url_is_returned_unchanged(self): + assert redact_userinfo("https://github.com/org/repo.git") == ( + "https://github.com/org/repo.git" + ) + + +class TestPerformedBy: + """Who ran the verification. + + FDA's Computer Software Assurance guidance asks the record of an assurance + activity to carry who performed the testing alongside the date. Every other + field this module collects identifies a machine or a commit. + """ + + def test_explicit_override_wins_over_every_ci_actor(self): + env = {"VIP_PERFORMED_BY": "QA Lead", "GITHUB_ACTOR": "octocat"} + assert _performed_by(env) == {"identity": "QA Lead", "source": "explicit"} + + def test_whitespace_only_override_falls_through(self): + """An empty variable exported by a shell must not shadow the real actor.""" + env = {"VIP_PERFORMED_BY": " ", "GITHUB_ACTOR": "octocat"} + assert _performed_by(env) == {"identity": "octocat", "source": "github"} + + @pytest.mark.parametrize( + ("var", "source"), + [("GITHUB_ACTOR", "github"), ("GITLAB_USER_LOGIN", "gitlab"), ("BUILD_USER_ID", "jenkins")], + ) + def test_each_ci_actor_is_recognized_with_its_source(self, var, source): + assert _performed_by({var: "runner"}) == {"identity": "runner", "source": source} + + def test_local_login_is_the_last_resort_and_says_so(self, monkeypatch): + monkeypatch.setattr("vip.attribution.getpass.getuser", lambda: "bdeitte") + assert _performed_by({}) == {"identity": "bdeitte", "source": "login"} + + def test_an_unresolvable_login_degrades_to_none(self, monkeypatch): + """No passwd entry and no LOGNAME/USER set -- a bare container.""" + + def boom(): + raise KeyError("uid not found") + + monkeypatch.setattr("vip.attribution.getpass.getuser", boom) + assert _performed_by({}) is None + + def test_collect_execution_metadata_carries_the_performer(self, tmp_path): + meta = collect_execution_metadata(cwd=tmp_path, env={"VIP_PERFORMED_BY": "QA Lead"}) + assert meta["performed_by"] == {"identity": "QA Lead", "source": "explicit"} diff --git a/selftests/test_cli_report.py b/selftests/test_cli_report.py index 0305321b..c5dc2b7c 100644 --- a/selftests/test_cli_report.py +++ b/selftests/test_cli_report.py @@ -411,3 +411,304 @@ def test_report_subcommand_help(self): ) assert result.returncode == 0 assert "--results" in result.stdout + + +class TestReportControls: + """`vip report --controls` scopes the control list to one render. + + Copying controls.toml into the report directory was the alternative and is + wrong: that directory survives between runs, so one --controls invocation + would leave a file behind that every later plain `vip report` picks up, + growing a compliance section nobody asked for from a stale list. + """ + + @pytest.fixture + def cli(self): + from vip import cli + + return cli + + def _args(self, tmp_path, controls=None): + results = tmp_path / "results.json" + results.write_text('{"schema_version": "1.0", "results": []}', encoding="utf-8") + return argparse.Namespace(results=str(results), controls=controls, open=False, output=None) + + def test_malformed_control_list_fails_before_quarto_starts(self, cli, tmp_path, monkeypatch): + """A notebook cell can only degrade to a warning, so validate out here.""" + monkeypatch.chdir(tmp_path) + bad = tmp_path / "c.toml" + bad.write_text("[controls]\n", encoding="utf-8") + + called = [] + monkeypatch.setattr(cli, "_quarto_render", lambda *a, **k: called.append(a) or 0) + with pytest.raises(SystemExit) as exc: + cli.run_report(self._args(tmp_path, str(bad))) + assert exc.value.code == 1 + assert called == [] + + def test_missing_control_list_fails_before_quarto_starts(self, cli, tmp_path, monkeypatch): + monkeypatch.chdir(tmp_path) + called = [] + monkeypatch.setattr(cli, "_quarto_render", lambda *a, **k: called.append(a) or 0) + with pytest.raises(SystemExit): + cli.run_report(self._args(tmp_path, str(tmp_path / "absent.toml"))) + assert called == [] + + def test_controls_reach_both_renders_through_the_environment(self, cli, tmp_path, monkeypatch): + """The HTML pages and the PDF are separate quarto invocations.""" + monkeypatch.chdir(tmp_path) + controls = tmp_path / "c.toml" + controls.write_text('[controls.x]\ndescription = "d"\n', encoding="utf-8") + + envs = [] + + def fake_render(document, report_dir, env): + envs.append((document, env.get("VIP_CONTROLS"))) + out = report_dir / "_output" + out.mkdir(parents=True, exist_ok=True) + (out / "index.html").write_text("") + (out / "vip-report.pdf").write_bytes(b"%PDF-") + return 0 + + monkeypatch.setattr(cli, "_quarto_render", fake_render) + cli.run_report(self._args(tmp_path, str(controls))) + + rendered = dict(envs) + assert rendered["index.qmd"] == str(controls.resolve()) + assert rendered["vip-report.qmd"] == str(controls.resolve()) + + def _results(self, tmp_path, body): + results = tmp_path / "results.json" + results.write_text(body, encoding="utf-8") + return results + + def _fake_render(self, envs): + def render(document, report_dir, env): + envs.append((document, env.get("VIP_CONTROLS"))) + out = report_dir / "_output" + out.mkdir(parents=True, exist_ok=True) + (out / "index.html").write_text("") + (out / "vip-report.pdf").write_bytes(b"%PDF-") + return 0 + + return render + + _UNREADABLE_MARKERS = ( + '{"schema_version": "1.0", "results": ' + '[{"nodeid": "t", "outcome": "passed", "markers": null}]}' + ) + + def test_unreadable_markers_are_refused_on_a_compliance_render( + self, cli, tmp_path, monkeypatch + ): + """A row whose markers cannot be read would print a GAP that does not exist. + + `load_results` normalizes a malformed `markers` to an empty list so the + plain report still renders. Under --controls that turns a tagged + scenario into a coverage gap, so the matrix claims the suite is missing + a check it actually has. Refuse the file instead, the way `vip trace` + does. + """ + monkeypatch.chdir(tmp_path) + controls = tmp_path / "c.toml" + controls.write_text('[controls.x]\ndescription = "d"\n', encoding="utf-8") + results = self._results(tmp_path, self._UNREADABLE_MARKERS) + + called = [] + monkeypatch.setattr(cli, "_quarto_render", lambda *a, **k: called.append(a) or 0) + args = argparse.Namespace( + results=str(results), controls=str(controls), open=False, output=None + ) + with pytest.raises(SystemExit) as exc: + cli.run_report(args) + assert exc.value.code == 1 + assert called == [] + + def test_unknown_schema_major_is_refused_on_a_compliance_render( + self, cli, tmp_path, monkeypatch + ): + monkeypatch.chdir(tmp_path) + controls = tmp_path / "c.toml" + controls.write_text('[controls.x]\ndescription = "d"\n', encoding="utf-8") + results = self._results(tmp_path, '{"schema_version": "99.0", "results": []}') + + called = [] + monkeypatch.setattr(cli, "_quarto_render", lambda *a, **k: called.append(a) or 0) + args = argparse.Namespace( + results=str(results), controls=str(controls), open=False, output=None + ) + with pytest.raises(SystemExit) as exc: + cli.run_report(args) + assert exc.value.code == 1 + assert called == [] + + def test_the_same_file_still_renders_without_controls(self, cli, tmp_path, monkeypatch): + """The asymmetry is the design, not an oversight. + + Without a control list the report is a pass/fail document and must + render regardless -- the strictness above belongs to the compliance + artifact, not to every render. + """ + monkeypatch.chdir(tmp_path) + results = self._results(tmp_path, self._UNREADABLE_MARKERS) + + envs = [] + monkeypatch.setattr(cli, "_quarto_render", self._fake_render(envs)) + args = argparse.Namespace(results=str(results), controls=None, open=False, output=None) + cli.run_report(args) + + assert [document for document, _ in envs] == ["index.qmd", "details.qmd", "vip-report.qmd"] + + def test_without_controls_the_variable_is_absent_and_nothing_warns( + self, cli, tmp_path, monkeypatch, capsys + ): + """The overwhelmingly common path: no control list, no section, no noise.""" + monkeypatch.chdir(tmp_path) + envs = [] + + def fake_render(document, report_dir, env): + envs.append(env.get("VIP_CONTROLS")) + out = report_dir / "_output" + out.mkdir(parents=True, exist_ok=True) + (out / "index.html").write_text("") + (out / "vip-report.pdf").write_bytes(b"%PDF-") + return 0 + + monkeypatch.setattr(cli, "_quarto_render", fake_render) + cli.run_report(self._args(tmp_path, None)) + + assert envs == [None, None, None] + assert "controls" not in capsys.readouterr().out.lower() + + def test_report_with_controls_refuses_a_mismatched_sidecar( + self, cli, tmp_path, monkeypatch, capsys + ): + """--controls makes this a compliance artifact; it inherits trace's strictness. + + `vip trace` already refuses a results.json whose sidecar disagrees + (`verify_results_checksum`); a compliance render must refuse the same + way rather than silently rendering a matrix built from evidence that + does not match its own attestation. + """ + monkeypatch.chdir(tmp_path) + controls = tmp_path / "c.toml" + controls.write_text('[controls.x]\ndescription = "d"\n', encoding="utf-8") + results = self._results(tmp_path, '{"schema_version": "1.0", "results": []}') + results.with_name("results.json.sha256").write_text(f"{'a' * 64} results.json\n") + + called = [] + monkeypatch.setattr(cli, "_quarto_render", lambda *a, **k: called.append(a) or 0) + args = argparse.Namespace( + results=str(results), controls=str(controls), open=False, output=None + ) + with pytest.raises(SystemExit) as exc: + cli.run_report(args) + + assert exc.value.code == 1 + assert called == [] + assert "checksum mismatch" in capsys.readouterr().err + + def test_report_with_controls_refuses_an_empty_source_sidecar( + self, cli, tmp_path, monkeypatch, capsys + ): + """An invalid attestation must not launder itself into a benign absence. + + `--results /external/path/results.json` copies the file into the + report directory, and _rehome_sidecar correctly refuses to + manufacture a destination sidecar out of an empty source one. A + *missing* destination sidecar is legal and benign, so a gate that + only ever looked at the destination let the compliance render + proceed on input `vip trace` refuses as a truncated attestation. + The compliance path must never be more permissive than `vip trace`. + """ + monkeypatch.chdir(tmp_path) + external = tmp_path / "external" + external.mkdir() + results = external / "results.json" + results.write_text('{"schema_version": "1.0", "results": []}', encoding="utf-8") + results.with_name("results.json.sha256").write_text(" \n\n", encoding="utf-8") + + controls = tmp_path / "c.toml" + controls.write_text('[controls.x]\ndescription = "d"\n', encoding="utf-8") + + called = [] + monkeypatch.setattr(cli, "_quarto_render", lambda *a, **k: called.append(a) or 0) + args = argparse.Namespace( + results=str(results), controls=str(controls), open=False, output=None + ) + with pytest.raises(SystemExit) as exc: + cli.run_report(args) + + assert exc.value.code == 1 + assert called == [] + assert "is empty" in capsys.readouterr().err + + def test_the_same_empty_sidecar_still_renders_without_controls( + self, cli, tmp_path, monkeypatch + ): + """Plain `vip report` stays lenient; only the compliance path is strict.""" + monkeypatch.chdir(tmp_path) + external = tmp_path / "external" + external.mkdir() + results = external / "results.json" + results.write_text('{"schema_version": "1.0", "results": []}', encoding="utf-8") + results.with_name("results.json.sha256").write_text(" \n\n", encoding="utf-8") + + envs = [] + monkeypatch.setattr(cli, "_quarto_render", self._fake_render(envs)) + args = argparse.Namespace(results=str(results), controls=None, open=False, output=None) + cli.run_report(args) + + assert [document for document, _ in envs] == ["index.qmd", "details.qmd", "vip-report.qmd"] + assert not (tmp_path / "report" / "results.json.sha256").exists() + + def test_a_source_with_no_sidecar_at_all_is_still_benign_under_controls( + self, cli, tmp_path, monkeypatch + ): + """No sidecar is a documented benign state on both paths. + + Results files written before the sidecar existed have none, and + `vip trace` accepts them; the compliance render must match rather + than exceed that strictness. + """ + monkeypatch.chdir(tmp_path) + external = tmp_path / "external" + external.mkdir() + results = external / "results.json" + results.write_text('{"schema_version": "1.0", "results": []}', encoding="utf-8") + + controls = tmp_path / "c.toml" + controls.write_text('[controls.x]\ndescription = "d"\n', encoding="utf-8") + + envs = [] + monkeypatch.setattr(cli, "_quarto_render", self._fake_render(envs)) + args = argparse.Namespace( + results=str(results), controls=str(controls), open=False, output=None + ) + cli.run_report(args) + + assert [document for document, _ in envs] == ["index.qmd", "details.qmd", "vip-report.qmd"] + + def test_report_without_controls_ignores_a_mismatched_sidecar(self, cli, tmp_path, monkeypatch): + """Plain vip report stays lenient: a report must render regardless. + + This is the other half of the asymmetry: the checksum gate must live + inside the `if args.controls` block, not ahead of it, or a plain + render would refuse to produce anything from a perfectly good local + results.json just because its stale sidecar disagrees. + """ + monkeypatch.chdir(tmp_path) + results = self._results(tmp_path, '{"schema_version": "1.0", "results": []}') + results.with_name("results.json.sha256").write_text(f"{'a' * 64} results.json\n") + + envs = [] + monkeypatch.setattr(cli, "_quarto_render", self._fake_render(envs)) + args = argparse.Namespace(results=str(results), controls=None, open=False, output=None) + + cli.run_report(args) + + assert [document for document, _ in envs] == [ + "index.qmd", + "details.qmd", + "vip-report.qmd", + ] diff --git a/selftests/test_cli_scaffold.py b/selftests/test_cli_scaffold.py index cfaacb79..c30acbc5 100644 --- a/selftests/test_cli_scaffold.py +++ b/selftests/test_cli_scaffold.py @@ -268,3 +268,44 @@ def test_scaffold_subcommand_help(self): ) assert result.returncode == 0 assert "--output" in result.stdout + + +class TestScaffoldExcludesBuildArtifacts: + """A scaffolded directory must not carry the source checkout's detritus.""" + + def test_no_pycache_in_scaffolded_output(self, tmp_path): + """A checkout that has run the examples must not leak __pycache__. + + Running the bundled examples locally leaves __pycache__ beside them, and + copytree without an ignore= copied it straight into the customer's brand + new extension directory -- making the scaffold output depend on whether + the VIP checkout happened to have run its own tests. + """ + from vip.cli import run_scaffold + + dest = tmp_path / "scaffolded" + run_scaffold(_make_args(template="21cfr-part11-validation", output=str(dest))) + + leaked = [p for p in dest.rglob("*") if p.name == "__pycache__" or p.suffix == ".pyc"] + assert leaked == [], f"scaffold leaked build artifacts: {leaked}" + + def test_scaffold_still_copies_the_real_files(self, tmp_path): + """Guard the ignore pattern against over-matching.""" + from vip.cli import run_scaffold + + dest = tmp_path / "scaffolded" + run_scaffold(_make_args(template="21cfr-part11-validation", output=str(dest))) + + for name in ( + "README.md", + "conftest.py", + "controls.toml", + "part11_refusal.py", + "test_21CFR_part11_connect.feature", + "test_21CFR_part11_connect.py", + "test_21CFR_part11_packagemanager.feature", + "test_21CFR_part11_packagemanager.py", + "test_21CFR_part11_workbench.feature", + "test_21CFR_part11_workbench.py", + ): + assert (dest / name).is_file(), f"{name} missing from scaffold output" diff --git a/selftests/test_cli_verify.py b/selftests/test_cli_verify.py index 6096ac72..90c3ecbc 100644 --- a/selftests/test_cli_verify.py +++ b/selftests/test_cli_verify.py @@ -1441,6 +1441,66 @@ def test_no_warning_when_flags_reach_the_generated_config(self, capsys): assert "ignored when a config file is used" not in err +class TestDisablingTheResultsFile: + """`--report ''` is the documented way to write no results file. + + The CLI forwarded `--vip-report` only when the value was non-empty, so the + plugin fell back to its own default and wrote report/results.json anyway -- + the one thing the invocation asked it not to do. + """ + + def test_an_empty_report_path_is_forwarded_to_the_plugin(self): + assert "--vip-report=" in _capture_cmd(_make_args(report="")) + + def test_a_normal_report_path_is_unchanged(self): + cmd = _capture_cmd(_make_args(report="out/results.json")) + assert "--vip-report=out/results.json" in cmd + + def test_the_default_path_is_still_forwarded(self): + assert "--vip-report=report/results.json" in _capture_cmd(_make_args()) + + @staticmethod + def _run_for_real_exit(args): + """Call run_verify without mocking sys.exit, so the refusal propagates. + + The check runs before subprocess.run, so the suite never starts. The + shared _capture_call helper patches sys.exit into a no-op, which would + let execution fall through past the refusal. + """ + with patch("vip.cli.subprocess.run", side_effect=AssertionError("suite was started")): + from vip.cli import run_verify + + run_verify(args) + + @pytest.mark.parametrize("fmt", ["junit", "sarif", "json,junit"]) + def test_asking_for_a_sibling_format_while_disabling_the_source_is_refused(self, tmp_path, fmt): + """junit.xml and results.sarif are built by reloading results.json, so + the combination would run the whole suite and produce nothing.""" + cfg = tmp_path / "vip.toml" + cfg.write_text("[general]\n") + with pytest.raises(SystemExit) as exc: + self._run_for_real_exit(_make_args(config=str(cfg), report="", format=fmt)) + assert exc.value.code == 2 + + def test_the_refusal_names_the_flag_the_formats_came_from(self, tmp_path, capsys): + cfg = tmp_path / "vip.toml" + cfg.write_text("[general]\n") + with pytest.raises(SystemExit): + self._run_for_real_exit(_make_args(config=str(cfg), report="", ci=True)) + assert "--ci" in capsys.readouterr().err + + def test_the_refusal_happens_before_the_suite_runs(self, tmp_path): + """A message after a full product run would be worse than no message.""" + cfg = tmp_path / "vip.toml" + cfg.write_text("[general]\n") + with pytest.raises(SystemExit): + self._run_for_real_exit(_make_args(config=str(cfg), report="", format="junit")) + + def test_disabling_the_report_with_json_alone_is_allowed(self): + """json *is* results.json, so there is no sibling left to strand.""" + assert "--vip-report=" in _capture_cmd(_make_args(report="", format="json")) + + class TestAllowUnprovenFlag: """`vip verify --allow-unproven` opts out of the unproven exit code.""" diff --git a/selftests/test_connect_audit_client.py b/selftests/test_connect_audit_client.py new file mode 100644 index 00000000..5c3c8f35 --- /dev/null +++ b/selftests/test_connect_audit_client.py @@ -0,0 +1,159 @@ +import httpx +import pytest + +from vip.clients.connect import ConnectClient + + +def _client(handler): + c = ConnectClient(base_url="https://connect.example.com", api_key="k") + c._client = httpx.Client( + transport=httpx.MockTransport(handler), base_url="https://connect.example.com" + ) + return c + + +def test_list_audit_logs_returns_results(): + def handler(request): + assert request.url.path == "/v1/audit_logs" + return httpx.Response(200, json={"results": [{"user_id": 1, "time": "t"}]}) + + assert _client(handler).list_audit_logs() == [{"user_id": 1, "time": "t"}] + + +@pytest.mark.parametrize("status", [403, 404]) +def test_list_audit_logs_returns_none_when_unavailable(status): + assert _client(lambda r: httpx.Response(status)).list_audit_logs() is None + + +def test_allowed_methods_parses_the_allow_header(): + handler = lambda r: httpx.Response(200, headers={"Allow": "GET, HEAD, OPTIONS"}) # noqa: E731 + assert _client(handler).audit_log_allowed_methods() == {"GET", "HEAD", "OPTIONS"} + + +def test_allowed_methods_returns_none_without_an_allow_header(): + assert _client(lambda r: httpx.Response(200)).audit_log_allowed_methods() is None + + +def test_allowed_methods_never_issues_a_mutating_request(): + """Regression guard for the non-destructive contract.""" + seen = [] + + def handler(request): + seen.append(request.method) + return httpx.Response(200, headers={"Allow": "GET"}) + + _client(handler).audit_log_allowed_methods() + assert seen == ["OPTIONS"] + + +class _RecordingClient: + """Stands in for httpx.Client so we can inspect how it was constructed. + + unauthenticated_status builds its own client rather than using + self._client, so MockTransport on the pooled client cannot see it. + """ + + instances: list["_RecordingClient"] = [] + + def __init__(self, **kwargs): + self.kwargs = kwargs + self.requested = None + _RecordingClient.instances.append(self) + + def __enter__(self): + return self + + def __exit__(self, *exc): + return False + + def get(self, url): + self.requested = url + return httpx.Response(401, request=httpx.Request("GET", url)) + + +@pytest.fixture +def recording_client(monkeypatch): + """Build a ConnectClient, THEN patch httpx.Client. Order is load-bearing. + + BaseClient.__init__ builds its own pooled httpx.Client (`base.py:135`). + Patching before construction would make instances[0] the pooled client -- + which legitimately does carry credentials -- so every assertion below + would inspect the wrong object and the credential test would pass or fail + for entirely the wrong reason. + """ + + def _make(**kwargs): + client = ConnectClient(base_url="https://connect.example.com", **kwargs) + _RecordingClient.instances = [] + monkeypatch.setattr(httpx, "Client", _RecordingClient) + return client + + return _make + + +def test_unauthenticated_status_returns_the_status(recording_client): + c = recording_client(api_key="k") + assert c.unauthenticated_status("/__api__/v1/users") == 401 + # Exactly one client was built, and it is the ad-hoc one. + assert len(_RecordingClient.instances) == 1 + assert _RecordingClient.instances[0].requested == ( + "https://connect.example.com/__api__/v1/users" + ) + + +def test_unauthenticated_status_sends_no_credentials(recording_client): + """The whole point of the method: an authorised caller would get 200.""" + c = recording_client(api_key="SECRET_KEY") + c.unauthenticated_status("/__api__/v1/users") + + kwargs = _RecordingClient.instances[0].kwargs + # No auth, no cookies, and no headers carrying the key were configured. + assert "auth" not in kwargs or kwargs["auth"] is None + assert not kwargs.get("cookies") + assert "SECRET_KEY" not in repr(kwargs.get("headers", {})) + + +def test_unauthenticated_status_accepts_a_path_without_a_leading_slash(recording_client): + """The two endpoints are customer-overridable in the Part 11 example's conftest. + + Someone editing that override writes `__api__/v1/users` as readily as + `/__api__/v1/users`. Without normalization the two concatenate into + `https://connect.example.com__api__/v1/users`, which is a different host, + so the scenario fails on a deployment that is fine. + """ + c = recording_client(api_key="k") + assert c.unauthenticated_status("__api__/v1/users") == 401 + assert _RecordingClient.instances[0].requested == ( + "https://connect.example.com/__api__/v1/users" + ) + + +def test_unauthenticated_status_uses_the_configured_timeout(recording_client): + """The ad-hoc client must not fall back to httpx's own default. + + BaseClient scales its default timeout, and a caller can override it; a probe + that ignores both hangs for a different length of time than every other + request the same client makes. + """ + c = recording_client(api_key="k", timeout=7.5) + c.unauthenticated_status("/__api__/v1/users") + assert _RecordingClient.instances[0].kwargs["timeout"] == 7.5 + assert _RecordingClient.instances[0].kwargs["timeout"] == c._timeout + + +def test_unauthenticated_status_pins_trust_env_and_keeps_env_ca(recording_client): + """trust_env=False also disables SSL_CERT_FILE; verify_with_env_ca restores it. + + verify_with_env_ca(True) returns a fresh ssl.SSLContext per call (via + httpx.create_ssl_context), and SSLContext has no __eq__, so two contexts + built from identical inputs are never `==`. Compare type instead of value. + """ + from vip.proxy import verify_with_env_ca + + c = recording_client(api_key="k") + c.unauthenticated_status("/__api__/v1/users") + + kwargs = _RecordingClient.instances[0].kwargs + assert kwargs["trust_env"] is False + assert type(kwargs["verify"]) is type(verify_with_env_ca(c._verify)) + assert "proxy" in kwargs diff --git a/selftests/test_control_marker_registration.py b/selftests/test_control_marker_registration.py new file mode 100644 index 00000000..61cc3dad --- /dev/null +++ b/selftests/test_control_marker_registration.py @@ -0,0 +1,105 @@ +import json + +FEATURE = """@connect @control-cfr-11-10-e +Feature: Audit trail + Scenario: Publish is recorded + Given a thing +""" + +STEPS = """ +from pytest_bdd import given, scenario + + +@scenario("t.feature", "Publish is recorded") +def test_tagged(): + pass + + +@given("a thing") +def a_thing(): + pass +""" + + +def _write_suite(pytester): + (pytester.path / "t.feature").write_text(FEATURE) + pytester.makepyfile(test_t=STEPS) + (pytester.path / "vip.toml").write_text('[connect]\nurl = "https://c.example.com"\n') + + +def test_collects_under_strict_markers(pytester): + _write_suite(pytester) + result = pytester.runpytest_subprocess( + "--vip-config", "vip.toml", "--strict-markers", "-p", "no:cacheprovider" + ) + result.assert_outcomes(passed=1) + + +def test_collects_under_warnings_as_errors(pytester): + _write_suite(pytester) + result = pytester.runpytest_subprocess( + "--vip-config", + "vip.toml", + "-W", + "error::pytest.PytestUnknownMarkWarning", + "-p", + "no:cacheprovider", + ) + result.assert_outcomes(passed=1) + + +def test_collects_under_both_together(pytester): + _write_suite(pytester) + result = pytester.runpytest_subprocess( + "--vip-config", + "vip.toml", + "--strict-markers", + "-W", + "error::pytest.PytestUnknownMarkWarning", + "-p", + "no:cacheprovider", + ) + result.assert_outcomes(passed=1) + + +def test_control_tags_still_reach_results_json(pytester): + """Registration must not cost us the evidence it exists to preserve.""" + _write_suite(pytester) + report = pytester.path / "results.json" + pytester.runpytest_subprocess( + "--vip-config", "vip.toml", "--vip-report", str(report), "-p", "no:cacheprovider" + ) + markers = json.loads(report.read_text())["results"][0]["markers"] + assert "control-cfr-11-10-e" in markers + assert "connect" in markers + + +def test_extension_dir_from_vip_toml_registers_control_tags(pytester, tmp_path): + """An extension dir named only via [general] extension_dirs (not + --vip-extensions) must still get its control tags registered.""" + ext_dir = tmp_path / "ext" + ext_dir.mkdir() + (ext_dir / "t.feature").write_text(FEATURE) + (ext_dir / "test_t.py").write_text(STEPS) + (pytester.path / "vip.toml").write_text( + f'[connect]\nurl = "https://c.example.com"\n\n[general]\nextension_dirs = ["{ext_dir}"]\n' + ) + result = pytester.runpytest_subprocess( + "--vip-config", "vip.toml", "--strict-markers", "-p", "no:cacheprovider" + ) + result.assert_outcomes(passed=1) + + +def test_targeted_step_file_registers_control_tags(pytester): + """Naming the step (.py) file directly -- a normal targeted dev run -- + must still discover the control tags in the adjacent .feature file.""" + _write_suite(pytester) + result = pytester.runpytest_subprocess( + "--vip-config", + "vip.toml", + "--strict-markers", + "-p", + "no:cacheprovider", + "test_t.py", + ) + result.assert_outcomes(passed=1) diff --git a/selftests/test_feature_roots.py b/selftests/test_feature_roots.py new file mode 100644 index 00000000..8d5af674 --- /dev/null +++ b/selftests/test_feature_roots.py @@ -0,0 +1,164 @@ +"""_feature_roots resolves relative pytest args the way pytest itself does. + +These go through a stub Config rather than `pytester`, which always runs with +cwd == rootdir and therefore cannot exercise the case these tests exist for: +pytest invoked from a subdirectory, where `invocation_params.dir` and +`rootpath` differ. +""" + +from __future__ import annotations + +import types +from pathlib import Path + +import pytest + +from vip.plugin import _discover_control_tags, _feature_roots + +FEATURE = """@connect @control-audit-trail +Feature: F + Scenario: S + Given a thing +""" + + +def _config(args, rootpath, invocation_dir, ext_dirs=(), norecursedirs=None): + """A stub with just the attributes _feature_roots and _discover_control_tags read.""" + stash: dict = {} + if ext_dirs: + from vip.plugin import _ext_dirs_key + + stash[_ext_dirs_key] = list(ext_dirs) + return types.SimpleNamespace( + args=list(args), + rootpath=Path(rootpath), + invocation_params=types.SimpleNamespace(dir=Path(invocation_dir)), + stash=stash, + getini=lambda name: norecursedirs if norecursedirs is not None else [], + ) + + +def test_relative_arg_resolves_against_the_invocation_dir(tmp_path): + """The regression: resolving against rootpath yields a path that does not exist.""" + root = tmp_path / "repo" + target = root / "src" / "tests" + target.mkdir(parents=True) + subdir = root / "selftests" + subdir.mkdir() + + cfg = _config(args=["../src/tests"], rootpath=root, invocation_dir=subdir) + roots = _feature_roots(cfg) + + assert [r.resolve() for r in roots] == [target.resolve()] + assert roots[0].exists() + + +def test_control_tags_are_found_when_invoked_from_a_subdirectory(tmp_path): + root = tmp_path / "repo" + target = root / "src" / "tests" + target.mkdir(parents=True) + (target / "t.feature").write_text(FEATURE, encoding="utf-8") + subdir = root / "selftests" + subdir.mkdir() + + cfg = _config(args=["../src/tests"], rootpath=root, invocation_dir=subdir) + assert _discover_control_tags(cfg) == {"control-audit-trail"} + + +def test_absolute_args_are_unaffected(tmp_path): + target = tmp_path / "tests" + target.mkdir() + cfg = _config(args=[str(target)], rootpath=tmp_path, invocation_dir=tmp_path / "elsewhere") + assert [r.resolve() for r in _feature_roots(cfg)] == [target.resolve()] + + +def test_nodeid_args_keep_only_the_path(tmp_path): + target = tmp_path / "tests" + target.mkdir() + cfg = _config( + args=[f"{target}::TestClass::test_thing"], rootpath=tmp_path, invocation_dir=tmp_path + ) + assert [r.resolve() for r in _feature_roots(cfg)] == [target.resolve()] + + +def test_duplicate_roots_are_collapsed(tmp_path): + """A targeted run passing many sibling paths must not rescan the same tree.""" + target = tmp_path / "tests" + target.mkdir() + cfg = _config( + args=[str(target), str(target), f"{target}::x"], rootpath=tmp_path, invocation_dir=tmp_path + ) + assert len(_feature_roots(cfg)) == 1 + + +def test_no_args_falls_back_to_rootpath(tmp_path): + cfg = _config(args=[], rootpath=tmp_path, invocation_dir=tmp_path / "elsewhere") + assert [r.resolve() for r in _feature_roots(cfg)] == [tmp_path.resolve()] + + +def test_extension_dirs_are_scanned(tmp_path): + ext = tmp_path / "ext" + ext.mkdir() + (ext / "t.feature").write_text(FEATURE, encoding="utf-8") + cfg = _config(args=[], rootpath=tmp_path / "repo", invocation_dir=tmp_path, ext_dirs=[str(ext)]) + assert _discover_control_tags(cfg) == {"control-audit-trail"} + + +@pytest.mark.parametrize("ignored", [".venv", "node_modules"]) +def test_norecursedirs_are_pruned(tmp_path, ignored): + """A feature file inside an ignored directory must not register a marker.""" + buried = tmp_path / ignored / "pkg" + buried.mkdir(parents=True) + (buried / "t.feature").write_text(FEATURE, encoding="utf-8") + + cfg = _config( + args=[str(tmp_path)], + rootpath=tmp_path, + invocation_dir=tmp_path, + norecursedirs=[".*", "node_modules", "venv"], + ) + assert _discover_control_tags(cfg) == set() + + unpruned = _config(args=[str(tmp_path)], rootpath=tmp_path, invocation_dir=tmp_path) + assert _discover_control_tags(unpruned) == {"control-audit-trail"} + + +def test_norecursedirs_pattern_with_a_separator_matches_the_full_path(tmp_path): + """pytest matches a pattern containing a separator against the whole path. + + Matching the basename only would scan a directory pytest itself would never + collect, registering a control tag from a feature file that cannot run. + """ + buried = tmp_path / "generated" / "pkg" + buried.mkdir(parents=True) + (buried / "t.feature").write_text(FEATURE, encoding="utf-8") + + cfg = _config( + args=[str(tmp_path)], + rootpath=tmp_path, + invocation_dir=tmp_path, + norecursedirs=[f"{tmp_path}/generated"], + ) + assert _discover_control_tags(cfg) == set() + + +def test_a_bare_pattern_still_matches_by_basename(tmp_path): + buried = tmp_path / "build" / "pkg" + buried.mkdir(parents=True) + (buried / "t.feature").write_text(FEATURE, encoding="utf-8") + + cfg = _config( + args=[str(tmp_path)], rootpath=tmp_path, invocation_dir=tmp_path, norecursedirs=["build"] + ) + assert _discover_control_tags(cfg) == set() + + +def test_a_non_matching_pattern_does_not_prune(tmp_path): + kept = tmp_path / "generated" / "pkg" + kept.mkdir(parents=True) + (kept / "t.feature").write_text(FEATURE, encoding="utf-8") + + cfg = _config( + args=[str(tmp_path)], rootpath=tmp_path, invocation_dir=tmp_path, norecursedirs=["build"] + ) + assert _discover_control_tags(cfg) == {"control-audit-trail"} diff --git a/selftests/test_gherkin_control_tags.py b/selftests/test_gherkin_control_tags.py new file mode 100644 index 00000000..7ec4af61 --- /dev/null +++ b/selftests/test_gherkin_control_tags.py @@ -0,0 +1,44 @@ +from vip.gherkin import CONTROL_TAG_PREFIX, parse_feature_file + +FEATURE = """@control-cfr-11-10-e @connect +Feature: Audit trail + Scenario: Publish is recorded + Given Connect is reachable +""" + +PRODUCT_FIRST = """@connect @control-cfr-11-10-e +Feature: Audit trail + Scenario: Publish is recorded + Given Connect is reachable +""" + + +def test_control_tag_does_not_become_the_marker(tmp_path): + """Tag order inside a feature file must not change the derived marker.""" + f = tmp_path / "t.feature" + f.write_text(FEATURE) + assert parse_feature_file(f)["marker"] == "connect" + + +def test_product_first_ordering_is_unchanged(tmp_path): + f = tmp_path / "t.feature" + f.write_text(PRODUCT_FIRST) + assert parse_feature_file(f)["marker"] == "connect" + + +def test_all_tags_are_collected(tmp_path): + f = tmp_path / "t.feature" + f.write_text(FEATURE) + assert set(parse_feature_file(f)["tags"]) == {"control-cfr-11-10-e", "connect"} + + +def test_scenario_level_tags_are_collected(tmp_path): + f = tmp_path / "t.feature" + f.write_text("@connect\nFeature: F\n @control-access-control\n Scenario: S\n Given x\n") + parsed = parse_feature_file(f) + assert parsed["marker"] == "connect" + assert "control-access-control" in parsed["tags"] + + +def test_prefix_constant(): + assert CONTROL_TAG_PREFIX == "control-" diff --git a/selftests/test_report_content.py b/selftests/test_report_content.py index ea6cc522..d89b0c6d 100644 --- a/selftests/test_report_content.py +++ b/selftests/test_report_content.py @@ -14,9 +14,11 @@ import re from pathlib import Path +from types import SimpleNamespace import pytest +from conftest import matrix_from_statuses from vip import report_content from vip.reporting import ReportData, TestResult @@ -333,6 +335,221 @@ def test_no_scenario_title_returns_empty_list(self): assert index.steps_for(item) == [] +# --------------------------------------------------------------------------- +# Coverage display (failing controls) +# --------------------------------------------------------------------------- + + +class TestFailedControlDisplay: + def test_a_failing_control_displays_as_covered_failed(self): + entry = SimpleNamespace(coverage="covered", executed=True, failing=True, has_unproven=False) + assert report_content.display_coverage(entry) == "covered_failed" + + def test_a_failing_control_uses_the_failed_style(self): + """Reuses the outcome palette so the styles.css drift guard still holds.""" + assert report_content.COVERAGE_STYLE_KEY["covered_failed"] == "failed" + + def test_a_failing_control_is_labelled_failed(self): + assert report_content.COVERAGE_LABELS["covered_failed"] == "FAILED" + + def test_a_passing_control_still_displays_as_covered(self): + entry = SimpleNamespace( + coverage="covered", executed=True, failing=False, has_unproven=False + ) + assert report_content.display_coverage(entry) == "covered" + + def test_an_all_skipped_control_still_displays_as_not_executed(self): + entry = SimpleNamespace( + coverage="covered", executed=False, failing=False, has_unproven=False + ) + assert report_content.display_coverage(entry) == "covered_not_executed" + + def test_a_gap_is_unaffected(self): + entry = SimpleNamespace(coverage="gap", executed=False, failing=False, has_unproven=False) + assert report_content.display_coverage(entry) == "gap" + + def test_every_coverage_value_has_a_style_and_a_label(self): + assert set(report_content.COVERAGE_STYLE_KEY) == set(report_content.COVERAGE_LABELS) + + def test_a_real_mixed_pass_and_failure_control_displays_as_failed(self): + """End to end through a real matrix, not a stub. + + The stub tests above assert display_coverage's branching. This asserts + the decision the branching exists to implement: one failing scenario + demotes a control that also has a passing one. + """ + matrix = matrix_from_statuses({"c1": ["passed", "failed"]}) + entry = matrix.entries[0] + assert entry.coverage == "covered" + assert report_content.display_coverage(entry) == "covered_failed" + + def test_a_real_mixed_control_is_counted_in_the_summary(self): + """The summary row reads the display value, so the count must follow.""" + matrix = matrix_from_statuses({"c1": ["passed", "failed"], "c2": ["passed"]}) + rows = dict(report_content.traceability_summary_rows(matrix)) + assert rows["Covered, failing"] == "1" + assert rows["Covered, executed and passing"] == "1" + + def test_a_pass_beside_an_unproven_skip_displays_as_unproven(self): + """Otherwise the badge reads COVERED and the unproven scenario is invisible.""" + matrix = matrix_from_statuses({"c1": ["passed", "unproven"]}) + assert report_content.display_coverage(matrix.entries[0]) == "covered_unproven" + + def test_an_all_unproven_control_prefers_unproven_over_not_run(self): + """Both are true; UNPROVEN says which kind of non-execution it was.""" + matrix = matrix_from_statuses({"c1": ["unproven"]}) + entry = matrix.entries[0] + assert entry.executed is False + assert report_content.display_coverage(entry) == "covered_unproven" + + def test_a_failure_outranks_an_unproven_skip(self): + """A control that ran and failed is the louder fact, so FAILED wins.""" + matrix = matrix_from_statuses({"c1": ["failed", "unproven"]}) + assert report_content.display_coverage(matrix.entries[0]) == "covered_failed" + + def test_an_unproven_control_is_counted_in_the_summary(self): + matrix = matrix_from_statuses({"c1": ["passed", "unproven"], "c2": ["passed"]}) + rows = dict(report_content.traceability_summary_rows(matrix)) + assert rows["Covered, not verified"] == "1" + assert rows["Covered, executed and passing"] == "1" + + def test_every_display_value_has_a_label_and_a_style(self): + """A display value with no entry in either dict renders as a blank badge.""" + assert set(report_content.COVERAGE_LABELS) == set(report_content.COVERAGE_STYLE_KEY) + for key in report_content.COVERAGE_STYLE_KEY.values(): + assert report_content.outcome_style(key).label != "?" + + +class TestTraceabilityWarnings: + @staticmethod + def _matrix(unexecuted=(), failing=(), unproven=()): + return SimpleNamespace( + covered_without_execution=list(unexecuted), + covered_with_failure=list(failing), + covered_with_unproven=list(unproven), + ) + + def test_a_failing_control_produces_a_warning(self): + warnings_out = report_content.traceability_warnings(self._matrix(failing=["c1"])) + assert any("did not pass" in w and "c1" in w for w in warnings_out) + + def test_an_unproven_control_produces_a_warning(self): + warnings_out = report_content.traceability_warnings(self._matrix(unproven=["c1"])) + assert any("could not verify" in w and "c1" in w for w in warnings_out) + + def test_each_condition_gets_its_own_line(self): + """Three independent conditions, so three lines: a reader needs to know which.""" + matrix = self._matrix(unexecuted=["c2"], failing=["c1"], unproven=["c3"]) + assert len(report_content.traceability_warnings(matrix)) == 3 + + def test_a_clean_matrix_produces_none(self): + assert report_content.traceability_warnings(self._matrix()) == [] + + +class TestRenderFailureMessage: + def test_render_failure_message_names_the_error(self): + msg = report_content.TRACEABILITY_RENDER_FAILURE.format(error="boom") + assert "boom" in msg + assert "traceability" in msg.lower() + + +class TestExecutionProvenanceRows: + """Attribution reaches the artifact the customer archives. + + ``results.json`` has recorded the execution block since attribution + landed, but only ``vip trace --format json`` rendered it. A result that is + attributable in the machine-readable output and anonymous in the PDF is + attributable in the wrong place, because the PDF is what goes to an + auditor. + """ + + EXECUTION = { + "hostname": "runner-07", + "git": { + "commit": "a1b2c3d4e5f6", + "branch": "main", + "dirty": False, + "remote": "https://github.com/posit-dev/vip.git", + }, + "ci": {"provider": "github", "run_url": "https://github.com/o/r/actions/runs/5"}, + "performed_by": {"identity": "octocat", "source": "github"}, + } + + @staticmethod + def _rows(execution): + from vip.reporting import ReportData + + return dict(report_content.provenance_rows(ReportData(results=[], execution=execution))) + + def test_every_execution_field_reaches_the_report(self): + rows = self._rows(self.EXECUTION) + assert rows["Performed by"] == "octocat (GitHub actor)" + assert rows["Run host"] == "runner-07" + assert rows["Commit"] == "a1b2c3d4e5f6" + assert rows["Branch"] == "main" + assert rows["CI run"] == "https://github.com/o/r/actions/runs/5" + + def test_an_absent_execution_block_omits_the_rows_entirely(self): + """``--vip-no-attribution`` asked for this; five "not recorded" rows + would read as a broken run rather than a deliberate one.""" + rows = self._rows(None) + for label in ("Performed by", "Run host", "Commit", "Branch", "CI run"): + assert label not in rows + + def test_the_pre_attribution_rows_still_render_without_an_execution_block(self): + assert "Exit status" in self._rows(None) + + def test_a_dirty_tree_is_flagged_next_to_the_commit(self): + """Evidence from an uncommitted tree cannot be reproduced from the + commit alone, so the caveat belongs in the same cell.""" + execution = {**self.EXECUTION, "git": {**self.EXECUTION["git"], "dirty": True}} + assert self._rows(execution)["Commit"] == "a1b2c3d4e5f6 (uncommitted changes present)" + + @pytest.mark.parametrize( + ("source", "expected"), + [ + ("login", "bd (local login)"), + ("github", "bd (GitHub actor)"), + ("gitlab", "bd (GitLab user)"), + ("jenkins", "bd (Jenkins build user)"), + ], + ) + def test_every_inherited_identity_says_where_it_came_from(self, source, expected): + """A CI actor is often a service account. Unlabelled, it would read in + the archived artifact exactly like a named accountable operator.""" + execution = {**self.EXECUTION, "performed_by": {"identity": "bd", "source": source}} + assert self._rows(execution)["Performed by"] == expected + + def test_an_identity_with_no_source_is_never_rendered_bare(self): + """A malformed block must not read as an explicitly named operator, + nor render the literal string "None".""" + execution = {**self.EXECUTION, "performed_by": {"identity": "bd"}} + assert self._rows(execution)["Performed by"] == "bd (source not recorded)" + + def test_an_unrecognized_source_is_still_labelled(self): + """A source this version does not know about is not an explicitly named + operator, and must not be promoted to one by rendering it bare.""" + execution = {**self.EXECUTION, "performed_by": {"identity": "bd", "source": "buildkite"}} + assert self._rows(execution)["Performed by"] == "bd (buildkite)" + + def test_an_explicit_operator_is_not_labelled(self): + performer = {"identity": "QA Lead", "source": "explicit"} + execution = {**self.EXECUTION, "performed_by": performer} + assert self._rows(execution)["Performed by"] == "QA Lead" + + def test_a_missing_field_inside_a_present_block_follows_the_none_contract(self): + """Present-but-partial is different from absent: the row stays, and the + backend renders NOT_RECORDED rather than a fabricated value.""" + rows = self._rows({"hostname": "runner-07"}) + assert rows["Run host"] == "runner-07" + assert rows["Performed by"] is None + assert rows["Commit"] is None + + def test_a_ci_run_without_a_url_falls_back_to_the_run_id(self): + execution = {**self.EXECUTION, "ci": {"provider": "gitlab", "run_id": "4412"}} + assert self._rows(execution)["CI run"] == "4412" + + class TestUnprovenRendering: """The report is the artifact an auditor reads; unproven must be visible. diff --git a/selftests/test_report_traceability.py b/selftests/test_report_traceability.py new file mode 100644 index 00000000..998793ba --- /dev/null +++ b/selftests/test_report_traceability.py @@ -0,0 +1,248 @@ +"""The compliance traceability section, in both rendering backends. + +The section exists because the report is the artifact a customer actually +receives. Before it, every field this feature added -- the matrix, the control +tags, the per-check timestamps -- lived only in results.json and `vip trace` +output, so a reader of the PDF saw none of it. +""" + +from __future__ import annotations + +from pathlib import Path + +import pytest + +from conftest import matrix_from_statuses +from vip import report_html, report_typst +from vip.report_content import ( + COVERAGE_LABELS, + COVERAGE_STYLE_KEY, + control_rows, + display_coverage, + traceability_summary_rows, + traceability_warnings, +) +from vip.reporting import ReportData, TestResult +from vip.traceability import ControlSpec, build_traceability_matrix + + +def _result(nodeid, control, outcome="passed", title="S", **kw): + return TestResult( + nodeid=nodeid, + outcome=outcome, + markers=["connect", f"control-{control}"], + scenario_title=title, + started_at="2026-08-29T10:00:00+00:00", + **kw, + ) + + +def _matrix(): + """One control of every coverage state the section can show.""" + data = ReportData( + results=[ + _result("t.py::ok", "ok", title="Audit trail is written"), + _result("t.py::sk", "skipped-only", outcome="skipped", skip_reason="not configured"), + _result("t.py::bad", "failing", outcome="failed", title="Privileged action"), + ] + ) + controls = { + "ok": ControlSpec("ok", "Audit trail recorded", reference="21 CFR 11.10(e)"), + "skipped-only": ControlSpec("skipped-only", "Privileged action refused"), + "failing": ControlSpec("failing", "Records cannot be deleted"), + "missing": ControlSpec("missing", "Nothing tests this"), + "manual": ControlSpec("manual", "Training records", verification="procedural"), + } + return build_traceability_matrix(data, controls) + + +class TestCoverageDisplay: + def test_all_five_states_are_distinguishable(self): + by_id = {r.control_id: r.coverage for r in control_rows(_matrix())} + assert by_id["ok"] == "covered" + assert by_id["failing"] == "covered_failed" + assert by_id["skipped-only"] == "covered_not_executed" + assert by_id["missing"] == "gap" + assert by_id["manual"] == "not_automatable" + + def test_a_failed_scenario_displays_as_covered_failed(self): + """Coverage folds in outcome: a failing scenario is not evidence.""" + row = next(r for r in control_rows(_matrix()) if r.control_id == "failing") + assert row.coverage == "covered_failed" + assert row.scenarios[0][1] == "failed" + + def test_every_coverage_value_has_a_label_and_a_style(self): + for value in COVERAGE_LABELS: + assert value in COVERAGE_STYLE_KEY + + def test_display_coverage_leaves_a_gap_alone(self): + entry = next(e for e in _matrix().entries if e.control.control_id == "missing") + assert display_coverage(entry) == "gap" + + def test_summary_counts_split_executed_from_covered(self): + rows = dict(traceability_summary_rows(_matrix())) + assert rows["Controls"] == "5" + assert rows["Covered, executed and passing"] == "1" + assert rows["Covered, not executed"] == "1" + assert rows["Covered, failing"] == "1" + assert rows["Gaps"] == "1" + assert rows["Not automatable"] == "1" + + def test_warning_names_the_unexecuted_control(self): + assert any("skipped-only" in w for w in traceability_warnings(_matrix())) + + def test_no_warning_when_everything_ran(self): + data = ReportData(results=[_result("t.py::ok", "ok")]) + matrix = build_traceability_matrix(data, {"ok": ControlSpec("ok", "d")}) + assert traceability_warnings(matrix) == [] + + +class TestHtmlBackend: + def test_renders_every_state_and_the_caveat(self): + html = report_html.render_traceability(_matrix()) + for label in ("COVERED", "NOT RUN", "GAP", "N/A (manual)"): + assert label in html + assert "not an attestation" in html + assert "21 CFR 11.10(e)" in html + + def test_a_gap_says_so_rather_than_rendering_empty(self): + assert "no tagged scenario" in report_html.render_traceability(_matrix()) + + def test_customer_supplied_text_is_escaped(self): + data = ReportData(results=[]) + matrix = build_traceability_matrix( + data, {"x": ControlSpec("x", " & co")} + ) + html = report_html.render_traceability(matrix) + assert "" + + def test_hostile_error_text_is_escaped(self): + html = report_html.render_traceability_error(ValueError(self.HOSTILE)) + assert "" not in html + assert "<img src=x onerror=alert(1)>" in html + assert "<script>alert(2)</script>" in html + + def test_the_failure_is_still_visible(self): + html = report_html.render_traceability_error(ValueError("boom")) + assert "Could not render the traceability section" in html + assert "boom" in html + + def test_index_qmd_routes_the_failure_through_the_escaping_helper(self): + """Drift guard on the cell itself, where the vulnerability lived. + + The helper cannot escape a value the .qmd never hands it, so assert + the except branch calls it through HTML() and no longer formats the + message into Markdown. + """ + qmd = (Path(__file__).parent.parent / "report" / "index.qmd").read_text() + assert "display(HTML(report_html.render_traceability_error(exc)))" in qmd + assert "TRACEABILITY_RENDER_FAILURE" not in qmd, ( + "the .qmd must not format the failure message itself; " + "report_html.render_traceability_error owns the escaping" + ) + + +class TestTypstBackend: + def test_renders_every_state(self): + typ = report_typst.render_traceability(_matrix()) + for label in ("COVERED", "NOT RUN", "GAP", "N/A (manual)"): + assert label in typ + + @pytest.mark.parametrize("hostile", ["#heading[x]", "*bold*", "$x^2$", 'quote " and \\ slash']) + def test_customer_text_cannot_inject_typst(self, hostile): + """A control list is authored outside VIP, so its text is untrusted.""" + matrix = build_traceability_matrix(ReportData(results=[]), {"x": ControlSpec("x", hostile)}) + typ = report_typst.render_traceability(matrix) + escaped = hostile.replace("\\", "\\\\").replace('"', '\\"') + assert f'"{escaped}"' in typ + + def test_a_multi_line_cell_is_one_expression(self): + """A table cell must be a single expression; text(..)#block(..) is not.""" + typ = report_typst.render_traceability(_matrix()) + assert ")#block(" not in typ + + def test_document_without_a_matrix_omits_the_section(self): + """The default path must be byte-identical to before the section existed.""" + data = ReportData(results=[_result("t.py::ok", "ok")]) + assert "Compliance Traceability" not in report_typst.render_document(data, {}) + + def test_document_with_a_matrix_includes_the_section(self): + data = ReportData(results=[_result("t.py::ok", "ok")]) + doc = report_typst.render_document(data, {}, _matrix()) + assert "Compliance Traceability" in doc + assert "NOT RUN" in doc + + +class TestCoverageBadge: + def test_html_coverage_badge_uses_a_class_that_exists_in_styles_css(self): + css = (Path(__file__).parent.parent / "report" / "styles.css").read_text() + matrix = matrix_from_statuses(statuses={"c1": ["passed"]}) + html = report_html.render_traceability(matrix) + for cls in ("vip-badge", "trace-caveat", "trace-warning"): + assert f".{cls}" in css, ( + f"{cls} is referenced by the renderer but absent from styles.css" + ) + assert "class='badge'" not in html + + +class TestRiskIsVisibleInBothEditions: + """The customer's risk rating, rendered rather than left in the CSV export. + + `risk` reached `vip trace --format csv/json` from the day the control + loader carried it, but neither report edition showed it -- and the report + is what an auditor is handed. CSA is a risk-based framework, so a matrix + with no risk column reads as a flat checklist. + """ + + @staticmethod + def _matrix(): + data = ReportData(results=[_result("t.py::ok", "ok", title="Audit trail is written")]) + controls = { + "ok": ControlSpec( + "ok", "Audit trail recorded", reference="21 CFR 11.10(e)", risk="high" + ), + "unrated": ControlSpec("unrated", "No risk assigned"), + } + return build_traceability_matrix(data, controls) + + def test_control_row_carries_the_risk_verbatim(self): + by_id = {r.control_id: r.risk for r in control_rows(self._matrix())} + assert by_id["ok"] == "high" + assert by_id["unrated"] == "" + + def test_html_edition_shows_the_risk(self): + html = report_html.render_traceability(self._matrix()) + assert "risk: high" in html + + def test_typst_edition_shows_the_risk(self): + typst = report_typst.render_traceability(self._matrix()) + assert "risk: high" in typst + + @pytest.mark.parametrize("backend", [report_html, report_typst]) + def test_an_unrated_control_renders_no_empty_risk_line(self, backend): + """An absent rating must not become a dangling "risk: " subline.""" + rendered = backend.render_traceability(self._matrix()) + assert rendered.count("risk: ") == 1 + + def test_a_risk_value_vip_does_not_recognize_is_still_rendered(self): + """VIP carries the rating through uninterpreted; it does not rank it.""" + data = ReportData(results=[_result("t.py::ok", "ok")]) + controls = {"ok": ControlSpec("ok", "Some control", risk="Class II / banana")} + matrix = build_traceability_matrix(data, controls) + assert "Class II / banana" in report_html.render_traceability(matrix) + assert "Class II / banana" in report_typst.render_traceability(matrix) diff --git a/selftests/test_report_typst.py b/selftests/test_report_typst.py index 8bc6be16..1b7e3a01 100644 --- a/selftests/test_report_typst.py +++ b/selftests/test_report_typst.py @@ -13,8 +13,9 @@ from __future__ import annotations +from conftest import matrix_from_statuses from vip import report_typst -from vip.report_content import NA_VERSION_EXPLANATION +from vip.report_content import COVERAGE_STYLE_KEY, NA_VERSION_EXPLANATION, outcome_style from vip.reporting import ReportData, TestResult # Typst metacharacters that must never reach the document outside a string @@ -161,6 +162,28 @@ def test_empty_results_render_the_placeholder(self): assert "No results found" in markup assert markup.startswith(report_typst.PREAMBLE) + def test_empty_results_still_carry_the_matrix_when_one_was_asked_for(self): + """index.qmd renders the section at zero results, so the PDF must too. + + Returning early on an empty results file dropped the section from the + PDF alone, splitting the two editions on exactly the run a reader is + most likely to misread -- one where every automated control is a gap. + """ + from vip.traceability import ControlSpec, build_traceability_matrix + + matrix = build_traceability_matrix( + ReportData(), {"audit-trail": ControlSpec("audit-trail", "An audit trail exists")} + ) + markup = report_typst.render_document(ReportData(), {}, matrix) + assert "No results found" in markup + assert report_typst._lit("Compliance Traceability") in markup + assert report_typst._lit("audit-trail") in markup + + def test_empty_results_without_a_matrix_are_unchanged(self): + assert report_typst.render_document(ReportData(), {}, None) == ( + report_typst.render_document(ReportData(), {}) + ) + def test_document_carries_every_section(self): data = ReportData( deployment_name="Acme", @@ -190,3 +213,59 @@ def test_not_recorded_provenance_renders_placeholder(self): data = ReportData(results=[TestResult(nodeid="a.py::t", outcome="passed")]) markup = report_typst.render_document(data, {}) assert '"not recorded"' in markup + + def test_render_document_shows_a_trace_error_instead_of_dropping_the_section(self): + out = report_typst.render_document(ReportData(), {}, matrix=None, trace_error="boom") + assert "boom" in out + + def test_a_trace_error_is_escaped_for_typst(self): + """An exception message can carry #, * or $, which are live Typst markup. + + HOSTILE (module-level, shared with TestEscaping) carries a `#panic(...)` + call, math, emphasis, and an unescaped quote. The live-markup form must + never appear, and the text must land inside an escaped string literal -- + the same invariant TestEscaping checks for card content. + """ + out = report_typst.render_document(ReportData(), {}, matrix=None, trace_error=HOSTILE) + assert '#panic("owned")' not in out + assert '\\"owned\\"' in out + # Both branches emit the same heading, so a PDF reader sees the section + # start whether the matrix built or the render failed. + assert report_typst._lit("Compliance Traceability") in out + + +class TestCoverageBadge: + def test_coverage_badge_uses_the_same_chip_as_an_outcome(self): + """vip-pill is a saturated fill with white text; the HTML edition is a chip.""" + matrix = matrix_from_statuses(statuses={"c1": ["passed"]}) + out = report_typst.render_traceability(matrix) + assert "vip-chip" in out + assert "vip-pill" not in out + + # Verify colors are in the right order: fg (color) before bg (background). + style = outcome_style(COVERAGE_STYLE_KEY["covered"]) + fg_color = f'"{style.color}"' + bg_color = f'"{style.background}"' + assert fg_color in out + assert bg_color in out + # The fg must appear before bg in the vip-chip(...) call. + fg_pos = out.find(fg_color) + bg_pos = out.find(bg_color) + assert fg_pos < bg_pos, "fg color must appear before bg color in vip-chip call" + + def test_caveat_renders_in_gray_italic(self): + """The caveat matches HTML: gray #6b7280 italic.""" + matrix = matrix_from_statuses(statuses={"c1": ["passed"]}) + out = report_typst.render_traceability(matrix) + # Caveat should have gray fill and italic style. + assert 'rgb("#6b7280")' in out + assert 'style: "italic"' in out + + def test_warning_renders_in_red_bold(self): + """Each warning matches HTML: red #dc2626 bold.""" + matrix = matrix_from_statuses(statuses={"c1": ["failed"]}) + out = report_typst.render_traceability(matrix) + # A failing control produces a covered-but-not-passing warning, which + # must render in red and bold. + assert 'fill: rgb("#dc2626")' in out + assert 'weight: "bold"' in out diff --git a/selftests/test_reporting.py b/selftests/test_reporting.py index dc27118e..cfab9796 100644 --- a/selftests/test_reporting.py +++ b/selftests/test_reporting.py @@ -3,8 +3,11 @@ from __future__ import annotations import json +import warnings import xml.etree.ElementTree as ET +import pytest + from vip.reporting import ( ProductInfo, ReportData, @@ -832,6 +835,47 @@ def test_na_version_wording_is_fallback_without_skip_reason(self, tmp_path): assert result["message"]["text"] == "N/A for this product version" +class TestLoadResultsSchemaVersionWarning: + def test_unknown_major_schema_warns_but_still_returns_data(self, tmp_path): + # load_results only warns on an unknown schema major -- it's called + # from index.qmd, details.qmd and `vip report`, where raising would + # surface as an unreadable traceback inside a Quarto notebook cell. + # `vip trace` is the one place that hard-errors on this condition. + data = { + "schema_version": "2.0", + "generated_at": "2026-01-01T00:00:00+00:00", + "exit_status": 0, + "products": {}, + "results": [ + { + "nodeid": "tests/connect/test_a.py::test_x", + "outcome": "passed", + "markers": ["connect"], + }, + ], + } + p = tmp_path / "results.json" + p.write_text(json.dumps(data)) + with pytest.warns(UserWarning, match="2.0"): + rd = load_results(p) + assert rd.schema_version == "2.0" + assert len(rd.results) == 1 + + def test_known_schema_version_does_not_warn(self, tmp_path): + data = { + "schema_version": "1.0", + "generated_at": "2026-01-01T00:00:00+00:00", + "exit_status": 0, + "products": {}, + "results": [], + } + p = tmp_path / "results.json" + p.write_text(json.dumps(data)) + with warnings.catch_warnings(): + warnings.simplefilter("error") + load_results(p) + + class TestUnprovenStatus: """An `unproven` skip is one VIP was asked to run but could not. diff --git a/selftests/test_results_checksum.py b/selftests/test_results_checksum.py new file mode 100644 index 00000000..6c4fbbf0 --- /dev/null +++ b/selftests/test_results_checksum.py @@ -0,0 +1,414 @@ +import hashlib +import shutil +import subprocess +import sys + +import pytest + +from vip.cli import _rehome_sidecar +from vip.traceability import ResultsIntegrityError, verify_results_checksum + + +def test_sidecar_matches_the_bytes_on_disk(pytester): + pytester.makepyfile(test_x="def test_ok(): assert True") + report = pytester.path / "results.json" + pytester.runpytest_subprocess("--vip-report", str(report), "-p", "no:cacheprovider") + + sidecar = report.parent / "results.json.sha256" + assert sidecar.exists() + + expected = hashlib.sha256(report.read_bytes()).hexdigest() + line = sidecar.read_text().strip() + digest, name = line.split() + assert digest == expected + assert name == "results.json" + + +def test_sidecar_is_written_even_for_json_only_format(pytester): + """The checksum is a property of the file, not an output format.""" + pytester.makepyfile(test_x="def test_ok(): assert True") + report = pytester.path / "results.json" + pytester.runpytest_subprocess( + "--vip-report", str(report), "--vip-format", "json", "-p", "no:cacheprovider" + ) + assert (report.parent / "results.json.sha256").exists() + + +def test_sidecar_verifies_with_shasum(pytester): + if sys.platform.startswith("win"): + pytest.skip("shasum not available on Windows") + pytester.makepyfile(test_x="def test_ok(): assert True") + report = pytester.path / "results.json" + pytester.runpytest_subprocess("--vip-report", str(report), "-p", "no:cacheprovider") + try: + proc = subprocess.run( + ["shasum", "-a", "256", "-c", "results.json.sha256"], + cwd=report.parent, + capture_output=True, + text=True, + ) + except FileNotFoundError: + pytest.skip("shasum binary not found on PATH") + assert proc.returncode == 0, proc.stdout + proc.stderr + + +def test_sidecar_failure_does_not_suppress_requested_outputs(pytester): + """Verify that a sidecar write failure does not gate requested outputs like junit.xml.""" + pytester.makepyfile(test_x="def test_ok(): assert True") + report = pytester.path / "results.json" + report_dir = report.parent + + # Create a directory at the sidecar path to cause OSError on sidecar write. + sidecar_dir = report_dir / "results.json.sha256" + sidecar_dir.mkdir(parents=True) + + # Run with both json and junit formats: both should succeed despite sidecar failure. + pytester.runpytest_subprocess( + "--vip-report", + str(report), + "--vip-format", + "json,junit", + "-p", + "no:cacheprovider", + ) + + # Both requested outputs must exist; sidecar failure is not fatal. + assert report.exists(), "results.json should exist despite sidecar failure" + junit_xml = report_dir / "junit.xml" + assert junit_xml.exists(), "junit.xml should exist despite sidecar failure" + + +class TestSidecarParsing: + """Both failure directions: a false tamper alarm and a false attestation.""" + + def _results(self, tmp_path): + p = tmp_path / "results.json" + p.write_text('{"schema_version": "1.0", "results": []}', encoding="utf-8") + return p, hashlib.sha256(p.read_bytes()).hexdigest() + + def test_uppercase_digest_verifies(self, tmp_path): + """Get-FileHash and 7-Zip emit uppercase; hex case is not a mismatch.""" + p, digest = self._results(tmp_path) + p.with_name("results.json.sha256").write_text(f"{digest.upper()} results.json\n") + assert verify_results_checksum(p) == (digest, True) + + def test_binary_mode_marker_is_tolerated(self, tmp_path): + p, digest = self._results(tmp_path) + p.with_name("results.json.sha256").write_text(f"{digest} *results.json\n") + assert verify_results_checksum(p) == (digest, True) + + def test_utf8_bom_is_tolerated(self, tmp_path): + p, digest = self._results(tmp_path) + p.with_name("results.json.sha256").write_bytes( + b"\xef\xbb\xbf" + f"{digest} results.json\n".encode() + ) + assert verify_results_checksum(p) == (digest, True) + + def test_multi_file_sidecar_matches_the_right_line(self, tmp_path): + """A flat .split() would compare the first line's digest to this file.""" + p, digest = self._results(tmp_path) + p.with_name("results.json.sha256").write_text( + f"{'0' * 64} failures.json\n{digest} results.json\n" + ) + assert verify_results_checksum(p) == (digest, True) + + def test_sidecar_naming_a_different_file_is_refused(self, tmp_path): + """The false-attestation case: matching digest, wrong file.""" + p, digest = self._results(tmp_path) + p.with_name("results.json.sha256").write_text(f"{digest} totally_other.json\n") + with pytest.raises(ResultsIntegrityError, match="does not record an entry"): + verify_results_checksum(p) + + def test_bare_digest_without_a_filename_still_verifies(self, tmp_path): + p, digest = self._results(tmp_path) + p.with_name("results.json.sha256").write_text(f"{digest}\n") + assert verify_results_checksum(p) == (digest, True) + + def test_genuine_mismatch_is_still_refused(self, tmp_path): + p, _ = self._results(tmp_path) + p.with_name("results.json.sha256").write_text(f"{'a' * 64} results.json\n") + with pytest.raises(ResultsIntegrityError, match="checksum mismatch"): + verify_results_checksum(p) + + def test_path_qualified_sidecar_verifies(self, tmp_path): + """`shasum -a 256 report/results.json` from a parent directory.""" + p, digest = self._results(tmp_path) + p.with_name("results.json.sha256").write_text(f"{digest} report/results.json\n") + assert verify_results_checksum(p) == (digest, True) + + def test_windows_path_qualified_sidecar_verifies(self, tmp_path): + p, digest = self._results(tmp_path) + p.with_name("results.json.sha256").write_text(f"{digest} report\\results.json\n") + assert verify_results_checksum(p) == (digest, True) + + def test_exact_match_still_wins_over_a_basename_collision(self, tmp_path): + """Exact match is the primary key: a basename fallback must not widen it. + + The exact-name line carries a wrong digest and the path-qualified line + carries the right one. Correct behaviour stops at the exact match and + reports a mismatch. An implementation that ran the fallback + unconditionally would union both digests and wrongly verify. + """ + p, digest = self._results(tmp_path) + p.with_name("results.json.sha256").write_text( + f"{'0' * 64} results.json\n{digest} archive/results.json\n" + ) + with pytest.raises(ResultsIntegrityError, match="checksum mismatch"): + verify_results_checksum(p) + + +class TestSidecarAmbiguity: + """A sidecar must never say two different things about one file. + + Both selection paths -- the exact recorded name and the basename fallback + -- can collect several entries. Accepting the file because *any* of them + agrees means the attestation may describe a different artifact entirely, + which is the false attestation the recorded-name match exists to prevent. + """ + + def _results(self, tmp_path): + p = tmp_path / "results.json" + p.write_text('{"schema_version": "1.0", "results": []}', encoding="utf-8") + return p, hashlib.sha256(p.read_bytes()).hexdigest() + + def test_two_path_qualified_entries_disagreeing_are_refused(self, tmp_path): + """The basename fallback must not verify against whichever line agrees.""" + p, digest = self._results(tmp_path) + p.with_name("results.json.sha256").write_text( + f"{digest} archive/results.json\n{'0' * 64} nightly/results.json\n" + ) + with pytest.raises(ResultsIntegrityError, match="different digests"): + verify_results_checksum(p) + + def test_two_exact_entries_disagreeing_are_refused(self, tmp_path): + """A rehomed sidecar that grew a second same-named line is caught here.""" + p, digest = self._results(tmp_path) + p.with_name("results.json.sha256").write_text( + f"{'0' * 64} results.json\n{digest} results.json\n" + ) + with pytest.raises(ResultsIntegrityError, match="different digests"): + verify_results_checksum(p) + + def test_the_message_says_how_to_proceed(self, tmp_path): + p, digest = self._results(tmp_path) + p.with_name("results.json.sha256").write_text( + f"{digest} archive/results.json\n{'0' * 64} nightly/results.json\n" + ) + with pytest.raises(ResultsIntegrityError) as exc: + verify_results_checksum(p) + assert "Regenerate it" in str(exc.value) + assert "delete it" in str(exc.value) + + def test_entries_that_agree_are_not_ambiguous(self, tmp_path): + """Saying the same thing twice is not a disagreement. + + Case and path qualification both vary, because hex case is not a + mismatch and a path-qualified line describes the same file. + """ + p, digest = self._results(tmp_path) + p.with_name("results.json.sha256").write_text( + f"{digest} archive/results.json\n{digest.upper()} nightly/results.json\n" + ) + assert verify_results_checksum(p) == (digest, True) + + def test_agreeing_exact_entries_still_verify(self, tmp_path): + p, digest = self._results(tmp_path) + p.with_name("results.json.sha256").write_text( + f"{digest} results.json\n{digest.upper()} results.json\n" + ) + assert verify_results_checksum(p) == (digest, True) + + +class TestStaleSidecarInvalidation: + def test_writing_results_removes_a_stale_sidecar_first(self, tmp_path, pytester): + """A sidecar write that fails must not leave the previous run's digest.""" + from vip.plugin import pytest_sessionfinish # noqa: F401 (import guard only) + + results = tmp_path / "results.json" + results.write_text("old", encoding="utf-8") + sidecar = tmp_path / "results.json.sha256" + sidecar.write_text(f"{'0' * 64} results.json\n", encoding="utf-8") + + pytester.makepyfile(test_x="def test_x():\n assert True\n") + pytester.runpytest_subprocess( + f"--vip-report={results}", "-p", "no:cacheprovider", "--vip-no-attribution" + ) + + digest, present = verify_results_checksum(results) + assert present is True + assert digest == hashlib.sha256(results.read_bytes()).hexdigest() + + +class TestReportSidecarRehoming: + """`vip report --results` copies a results file; the sidecar must follow it.""" + + def _sidecar_for(self, path): + return path.with_name(f"{path.name}.sha256") + + def test_source_sidecar_is_rehomed_under_the_destination_name(self, tmp_path): + src = tmp_path / "run-42.json" + src.write_text('{"results": []}', encoding="utf-8") + digest = hashlib.sha256(src.read_bytes()).hexdigest() + self._sidecar_for(src).write_text(f"{digest} run-42.json\n", encoding="utf-8") + + dest = tmp_path / "out" / "results.json" + dest.parent.mkdir() + shutil.copy2(src, dest) + _rehome_sidecar(self._sidecar_for(src), self._sidecar_for(dest), src.name, dest.name) + + assert verify_results_checksum(dest) == (digest, True) + + def test_digest_is_carried_not_recomputed(self, tmp_path): + """Recomputing would launder a tampered file into a verified one.""" + src = tmp_path / "results.json" + src.write_text("tampered", encoding="utf-8") + self._sidecar_for(src).write_text(f"{'0' * 64} results.json\n", encoding="utf-8") + + dest = tmp_path / "out" / "results.json" + dest.parent.mkdir() + shutil.copy2(src, dest) + _rehome_sidecar(self._sidecar_for(src), self._sidecar_for(dest), src.name, dest.name) + + with pytest.raises(ResultsIntegrityError, match="checksum mismatch"): + verify_results_checksum(dest) + + def test_missing_source_sidecar_removes_the_stale_destination_one(self, tmp_path): + dest = tmp_path / "results.json" + dest.write_text('{"results": []}', encoding="utf-8") + self._sidecar_for(dest).write_text(f"{'0' * 64} results.json\n", encoding="utf-8") + + _rehome_sidecar(tmp_path / "absent.json.sha256", self._sidecar_for(dest), "a", dest.name) + + assert not self._sidecar_for(dest).exists() + _, present = verify_results_checksum(dest) + assert present is False + + def test_path_qualified_source_line_is_rehomed(self, tmp_path): + """A recorded path must be rewritten, not copied through verbatim.""" + src = tmp_path / "results.json" + src.write_text('{"results": []}', encoding="utf-8") + digest = hashlib.sha256(src.read_bytes()).hexdigest() + self._sidecar_for(src).write_text(f"{digest} report/results.json\n", encoding="utf-8") + + dest = tmp_path / "out" / "results.json" + dest.parent.mkdir() + shutil.copy2(src, dest) + _rehome_sidecar(self._sidecar_for(src), self._sidecar_for(dest), src.name, dest.name) + + assert self._sidecar_for(dest).read_text().split()[1] == "results.json" + assert verify_results_checksum(dest) == (digest, True) + + def test_whitespace_only_source_removes_the_destination(self, tmp_path): + """An empty sidecar is the truncated-upload case; do not manufacture one.""" + src = tmp_path / "results.json" + src.write_text('{"results": []}', encoding="utf-8") + self._sidecar_for(src).write_text(" \n\n", encoding="utf-8") + + dest = tmp_path / "out" / "results.json" + dest.parent.mkdir() + shutil.copy2(src, dest) + self._sidecar_for(dest).write_text(f"{'0' * 64} results.json\n", encoding="utf-8") + _rehome_sidecar(self._sidecar_for(src), self._sidecar_for(dest), src.name, dest.name) + + assert not self._sidecar_for(dest).exists() + _, present = verify_results_checksum(dest) + assert present is False + + def test_a_basename_collision_does_not_produce_two_destination_entries(self, tmp_path): + """Exact-name precedence has to survive the rehome, not just verification. + + Rewriting every basename match turned one wrong exact entry plus one + right path-qualified entry into two `results.json` lines, and + verification's exact-match branch then found both and accepted the + agreeable one -- defeating the precedence + test_exact_match_still_wins_over_a_basename_collision protects. + """ + src = tmp_path / "results.json" + src.write_text('{"results": []}', encoding="utf-8") + digest = hashlib.sha256(src.read_bytes()).hexdigest() + self._sidecar_for(src).write_text( + f"{'0' * 64} results.json\n{digest} archive/results.json\n", encoding="utf-8" + ) + + dest = tmp_path / "out" / "results.json" + dest.parent.mkdir() + shutil.copy2(src, dest) + _rehome_sidecar(self._sidecar_for(src), self._sidecar_for(dest), src.name, dest.name) + + rehomed = self._sidecar_for(dest).read_text().splitlines() + named = [line for line in rehomed if line.split(None, 1)[1].strip() == "results.json"] + assert len(named) == 1, f"the rehome invented a second results.json entry: {rehomed}" + with pytest.raises(ResultsIntegrityError, match="checksum mismatch"): + verify_results_checksum(dest) + + def test_several_basename_matches_with_no_exact_entry_are_left_alone(self, tmp_path): + """The source never said which one describes the destination.""" + src = tmp_path / "run-42.json" + src.write_text('{"results": []}', encoding="utf-8") + digest = hashlib.sha256(src.read_bytes()).hexdigest() + self._sidecar_for(src).write_text( + f"{digest} archive/run-42.json\n{'0' * 64} nightly/run-42.json\n", encoding="utf-8" + ) + + dest = tmp_path / "out" / "results.json" + dest.parent.mkdir() + shutil.copy2(src, dest) + _rehome_sidecar(self._sidecar_for(src), self._sidecar_for(dest), src.name, dest.name) + + assert "results.json" not in self._sidecar_for(dest).read_text() + with pytest.raises(ResultsIntegrityError, match="does not record an entry"): + verify_results_checksum(dest) + + def test_several_basename_matches_that_agree_are_rehomed(self, tmp_path): + """Several basename matches with the *same* digest are unambiguous. + + This mirrors verify_results_checksum's distinct-digest rule: several + entries agreeing on one digest is not ambiguous, even though the + basename fallback found more than one match. + """ + src = tmp_path / "run-42.json" + src.write_text('{"results": []}', encoding="utf-8") + digest = hashlib.sha256(src.read_bytes()).hexdigest() + self._sidecar_for(src).write_text( + f"{digest} archive/run-42.json\n{digest} nightly/run-42.json\n", encoding="utf-8" + ) + + dest = tmp_path / "out" / "results.json" + dest.parent.mkdir() + shutil.copy2(src, dest) + _rehome_sidecar(self._sidecar_for(src), self._sidecar_for(dest), src.name, dest.name) + + assert verify_results_checksum(dest) == (digest, True) + + def test_several_basename_matches_that_agree_case_insensitively_are_rehomed(self, tmp_path): + """Digest comparison is case-insensitive, matching verify_results_checksum. + + PowerShell's Get-FileHash and 7-Zip emit uppercase hex; a sidecar + mixing an uppercase and a lowercase rendering of the same digest must + not be treated as disagreement. + """ + src = tmp_path / "run-42.json" + src.write_text('{"results": []}', encoding="utf-8") + digest = hashlib.sha256(src.read_bytes()).hexdigest() + self._sidecar_for(src).write_text( + f"{digest.upper()} archive/run-42.json\n{digest} nightly/run-42.json\n", + encoding="utf-8", + ) + + dest = tmp_path / "out" / "results.json" + dest.parent.mkdir() + shutil.copy2(src, dest) + _rehome_sidecar(self._sidecar_for(src), self._sidecar_for(dest), src.name, dest.name) + + assert verify_results_checksum(dest) == (digest, True) + + def test_undecodable_source_raises_unicode_error_for_the_caller(self, tmp_path): + """The call site catches this; it must not escape as a bare traceback.""" + src = tmp_path / "results.json" + src.write_text('{"results": []}', encoding="utf-8") + self._sidecar_for(src).write_bytes(b"\xff\xfe\x00\x00 not utf-8") + + dest = tmp_path / "out" / "results.json" + dest.parent.mkdir() + with pytest.raises(UnicodeDecodeError): + _rehome_sidecar(self._sidecar_for(src), self._sidecar_for(dest), src.name, dest.name) diff --git a/selftests/test_results_schema.py b/selftests/test_results_schema.py new file mode 100644 index 00000000..4e3fdea8 --- /dev/null +++ b/selftests/test_results_schema.py @@ -0,0 +1,40 @@ +import json + +import pytest + +from vip.reporting import RESULTS_SCHEMA_VERSION, load_results + + +def test_schema_version_is_loaded(tmp_path): + p = tmp_path / "results.json" + p.write_text(json.dumps({"schema_version": "1.0", "results": []})) + assert load_results(p).schema_version == "1.0" + + +def test_pre_1_0_results_load_with_null_schema_version(tmp_path): + """An archived results.json written before versioning must still load.""" + p = tmp_path / "results.json" + p.write_text( + json.dumps( + { + "generated_at": "2026-01-01T00:00:00+00:00", + "results": [{"nodeid": "a.py::test_x", "outcome": "passed"}], + } + ) + ) + data = load_results(p) + assert data.schema_version is None + assert len(data.results) == 1 + + +def test_current_schema_version_constant(): + assert RESULTS_SCHEMA_VERSION == "1.0" + + +def test_direction_is_numeric_not_lexicographic(monkeypatch, tmp_path): + """ "9" > "10" as strings; a results file at major 9 is older, not newer.""" + monkeypatch.setattr("vip.reporting.RESULTS_SCHEMA_VERSION", "10.0") + p = tmp_path / "results.json" + p.write_text('{"schema_version": "9.0", "results": []}', encoding="utf-8") + with pytest.warns(UserWarning, match="older than"): + load_results(p) diff --git a/selftests/test_results_timestamps.py b/selftests/test_results_timestamps.py new file mode 100644 index 00000000..0a5fa085 --- /dev/null +++ b/selftests/test_results_timestamps.py @@ -0,0 +1,41 @@ +import json +from datetime import datetime + + +def test_results_carry_per_test_timestamps(pytester): + pytester.makepyfile( + test_stamp=""" + def test_passes(): + assert True + + import pytest + + @pytest.mark.skip(reason="deliberate") + def test_skipped(): + pass + """ + ) + report = pytester.path / "results.json" + pytester.runpytest_subprocess("--vip-report", str(report), "-p", "no:cacheprovider") + + data = json.loads(report.read_text()) + assert data["results"], "expected at least one result" + for entry in data["results"]: + started, finished = entry["started_at"], entry["finished_at"] + assert started is not None and finished is not None + # Parses as ISO 8601 and is timezone-aware UTC. + start_dt = datetime.fromisoformat(started) + finish_dt = datetime.fromisoformat(finished) + assert start_dt.tzinfo is not None + assert start_dt.utcoffset().total_seconds() == 0 + assert start_dt <= finish_dt + + +def test_timestamps_absent_in_old_file_load_as_none(tmp_path): + from vip.reporting import load_results + + p = tmp_path / "results.json" + p.write_text(json.dumps({"results": [{"nodeid": "a.py::t", "outcome": "passed"}]})) + result = load_results(p).results[0] + assert result.started_at is None + assert result.finished_at is None diff --git a/selftests/test_trace_cli.py b/selftests/test_trace_cli.py new file mode 100644 index 00000000..551a10a9 --- /dev/null +++ b/selftests/test_trace_cli.py @@ -0,0 +1,513 @@ +import argparse +import csv +import hashlib +import io +import json +import subprocess +import sys + +import pytest + +from vip.cli import run_trace + +CONTROLS = """ +[controls.x] +description = "Audit trail" +reference = "21 CFR 11.10(e)" + +[controls.y] +description = "Training records" +verification = "procedural" +""" + + +def _results(tmp_path, schema_version="1.0", write_sidecar=True): + payload = { + "schema_version": schema_version, + "generated_at": "2026-08-28T12:00:00+00:00", + "vip_version": "2026.8.3", + "results": [ + { + "nodeid": "t.py::a", + "outcome": "passed", + "markers": ["connect", "control-x"], + "scenario_title": "Scenario A", + "started_at": "2026-08-28T12:00:00+00:00", + "finished_at": "2026-08-28T12:00:01+00:00", + } + ], + } + if schema_version is None: + payload.pop("schema_version") + p = tmp_path / "results.json" + data = json.dumps(payload, indent=2).encode() + p.write_bytes(data) + if write_sidecar: + p.with_name("results.json.sha256").write_text( + f"{hashlib.sha256(data).hexdigest()} results.json\n" + ) + controls = tmp_path / "controls.toml" + controls.write_text(CONTROLS) + return p, controls + + +def _run(*args, cwd=None): + return subprocess.run( + [sys.executable, "-m", "vip.cli", "trace", *args], + capture_output=True, + text=True, + cwd=cwd, + ) + + +def test_csv_to_stdout(tmp_path): + results, controls = _results(tmp_path) + proc = _run("--results", str(results), "--controls", str(controls)) + assert proc.returncode == 0, proc.stderr + assert proc.stdout.startswith("control_id,description,") + assert "Scenario A" in proc.stdout + + +def test_json_format(tmp_path): + results, controls = _results(tmp_path) + proc = _run("--results", str(results), "--controls", str(controls), "--format", "json") + assert proc.returncode == 0, proc.stderr + payload = json.loads(proc.stdout) + assert payload["summary"]["covered"] == 1 + assert payload["summary"]["not_automatable"] == 1 + + +def test_output_to_file(tmp_path): + results, controls = _results(tmp_path) + out = tmp_path / "matrix.csv" + proc = _run("--results", str(results), "--controls", str(controls), "--output", str(out)) + assert proc.returncode == 0, proc.stderr + assert out.read_text().startswith("control_id,") + + +def test_tampered_results_file_is_rejected(tmp_path): + results, controls = _results(tmp_path) + results.write_text(results.read_text().replace("passed", "failed")) + proc = _run("--results", str(results), "--controls", str(controls)) + assert proc.returncode != 0 + assert "checksum" in proc.stderr.lower() + + +def test_missing_sidecar_is_allowed(tmp_path): + results, controls = _results(tmp_path, write_sidecar=False) + assert _run("--results", str(results), "--controls", str(controls)).returncode == 0 + + +def test_provenance_carries_results_sha256_with_sidecar(tmp_path): + results, controls = _results(tmp_path, write_sidecar=True) + proc = _run("--results", str(results), "--controls", str(controls), "--format", "json") + assert proc.returncode == 0, proc.stderr + payload = json.loads(proc.stdout) + expected_digest = hashlib.sha256(results.read_bytes()).hexdigest() + assert payload["provenance"]["results_sha256"] == expected_digest + assert payload["provenance"]["results_sha256_sidecar_verified"] is True + + +def test_provenance_carries_results_sha256_without_sidecar(tmp_path): + results, controls = _results(tmp_path, write_sidecar=False) + proc = _run("--results", str(results), "--controls", str(controls), "--format", "json") + assert proc.returncode == 0, proc.stderr + payload = json.loads(proc.stdout) + expected_digest = hashlib.sha256(results.read_bytes()).hexdigest() + assert payload["provenance"]["results_sha256"] == expected_digest + assert payload["provenance"]["results_sha256_sidecar_verified"] is None + + +def test_pre_1_0_results_are_accepted(tmp_path): + results, controls = _results(tmp_path, schema_version=None) + assert _run("--results", str(results), "--controls", str(controls)).returncode == 0 + + +def test_unknown_major_schema_is_rejected(tmp_path): + results, controls = _results(tmp_path, schema_version="2.0") + proc = _run("--results", str(results), "--controls", str(controls)) + assert proc.returncode != 0 + assert "2.0" in proc.stderr + + +def test_unknown_minor_schema_is_accepted(tmp_path): + results, controls = _results(tmp_path, schema_version="1.7") + assert _run("--results", str(results), "--controls", str(controls)).returncode == 0 + + +def test_unrecognized_tag_warns_without_failing(tmp_path): + results, controls = _results(tmp_path) + payload = json.loads(results.read_text()) + payload["results"][0]["markers"].append("control-typo") + data = json.dumps(payload, indent=2).encode() + results.write_bytes(data) + results.with_name("results.json.sha256").write_text( + f"{hashlib.sha256(data).hexdigest()} results.json\n" + ) + proc = _run("--results", str(results), "--controls", str(controls)) + assert proc.returncode == 0 + assert "control-typo" in proc.stderr + + +def test_missing_control_file_errors_clearly(tmp_path): + results, _ = _results(tmp_path) + proc = _run("--results", str(results), "--controls", str(tmp_path / "nope.toml")) + assert proc.returncode != 0 + assert "not found" in proc.stderr + + +def _raw_results(tmp_path, payload, write_sidecar=True): + p = tmp_path / "results.json" + data = json.dumps(payload, indent=2).encode() + p.write_bytes(data) + if write_sidecar: + p.with_name("results.json.sha256").write_text( + f"{hashlib.sha256(data).hexdigest()} results.json\n" + ) + controls = tmp_path / "controls.toml" + controls.write_text(CONTROLS) + return p, controls + + +def test_unknown_major_schema_with_malformed_results_errors_cleanly_before_load(tmp_path): + # The schema gate must run BEFORE load_results ever indexes into the + # results list, so a genuinely incompatible major (empty-dict results, + # not just a future version number over current-shape rows) is refused + # cleanly instead of crashing with a KeyError inside load_results. + results, controls = _raw_results( + tmp_path, {"schema_version": "2.0", "results": [{}]}, write_sidecar=False + ) + proc = _run("--results", str(results), "--controls", str(controls)) + assert proc.returncode != 0 + assert "Error:" in proc.stderr + assert "Traceback" not in proc.stderr + + +def test_current_major_with_structurally_malformed_results_errors_cleanly(tmp_path): + # No schema_version mismatch here -- this is a current-major file whose + # results are the wrong shape, which must not escape as a raw KeyError. + results, controls = _raw_results(tmp_path, {"results": [{}]}, write_sidecar=False) + proc = _run("--results", str(results), "--controls", str(controls)) + assert proc.returncode != 0 + assert "Error:" in proc.stderr + assert "Traceback" not in proc.stderr + + +def test_malformed_json_without_sidecar_errors_cleanly(tmp_path): + # No sidecar, so verify_results_checksum can't catch this as a checksum + # mismatch -- this is exactly the population "missing sidecar is fine" + # exists to serve (older results files), so it must not fall through to + # a raw traceback. + _, controls = _results(tmp_path, write_sidecar=False) + results = tmp_path / "results.json" + results.write_text("{not valid json") + proc = _run("--results", str(results), "--controls", str(controls)) + assert proc.returncode != 0 + assert "Error" in proc.stderr + assert "Traceback" not in proc.stderr + + +def test_valid_json_non_object_without_sidecar_errors_cleanly(tmp_path): + _, controls = _results(tmp_path, write_sidecar=False) + results = tmp_path / "results.json" + results.write_text("[]") + proc = _run("--results", str(results), "--controls", str(controls)) + assert proc.returncode != 0 + assert "Error" in proc.stderr + assert "Traceback" not in proc.stderr + + +def test_unknown_major_schema_does_not_print_raw_warning(tmp_path): + results, controls = _results(tmp_path, schema_version="2.0") + proc = _run("--results", str(results), "--controls", str(controls)) + assert proc.returncode != 0 + assert "Error:" in proc.stderr + assert "UserWarning" not in proc.stderr + + +def _write_controls(tmp_path, text): + c = tmp_path / "c.toml" + c.write_text(text, encoding="utf-8") + return c + + +def _run_trace(results, controls, *extra): + return _run("--results", str(results), "--controls", str(controls), *extra) + + +def _skipped_results(tmp_path, outcome="skipped"): + p = tmp_path / "results.json" + p.write_text( + json.dumps( + { + "schema_version": "1.0", + "results": [ + { + "nodeid": "t.py::a", + "outcome": outcome, + "markers": ["connect", "control-x"], + "skip_reason": "Connect is not configured", + "scenario_title": "S", + } + ], + } + ), + encoding="utf-8", + ) + return p + + +class TestCoveredButNotExecuted: + """A green matrix from a run that verified nothing must say so.""" + + def test_skipped_only_control_warns_on_stderr(self, tmp_path): + results = _skipped_results(tmp_path) + controls = _write_controls(tmp_path, '[controls.x]\ndescription = "d"\n') + r = _run_trace(results, controls) + assert r.returncode == 0 + assert "no scenario that ran" in r.stderr + assert "control-x" in r.stderr or "x" in r.stderr + + def test_json_summary_separates_executed_from_covered(self, tmp_path): + results = _skipped_results(tmp_path) + controls = _write_controls(tmp_path, '[controls.x]\ndescription = "d"\n') + payload = json.loads(_run_trace(results, controls, "--format", "json").stdout) + assert payload["summary"]["covered"] == 1 + assert payload["summary"]["gaps"] == 0 + assert payload["summary"]["covered_and_executed"] == 0 + assert payload["summary"]["covered_not_executed"] == 1 + assert payload["covered_without_execution"] == ["x"] + + def test_na_version_counts_as_not_executed(self, tmp_path): + results = _skipped_results(tmp_path, outcome="skipped") + raw = json.loads(results.read_text()) + raw["results"][0]["na_version"] = True + results.write_text(json.dumps(raw), encoding="utf-8") + controls = _write_controls(tmp_path, '[controls.x]\ndescription = "d"\n') + payload = json.loads(_run_trace(results, controls, "--format", "json").stdout) + assert payload["summary"]["covered_not_executed"] == 1 + + def test_an_executed_control_does_not_warn(self, tmp_path): + results = _skipped_results(tmp_path, outcome="passed") + controls = _write_controls(tmp_path, '[controls.x]\ndescription = "d"\n') + r = _run_trace(results, controls) + assert "no scenario that ran" not in r.stderr + + +class TestCoveredButFailing: + """The mirror of the covered-not-executed warning.""" + + def test_trace_warns_when_a_covered_control_failed(self, tmp_path, capsys): + results = tmp_path / "results.json" + results.write_text( + json.dumps( + { + "schema_version": "1.0", + "results": [ + { + "nodeid": "t.py::test_a", + "outcome": "failed", + "markers": ["control-c1"], + } + ], + } + ), + encoding="utf-8", + ) + controls = tmp_path / "controls.toml" + controls.write_text( + '[controls.c1]\ndescription = "a control"\nverification = "automated"\n', + encoding="utf-8", + ) + run_trace( + argparse.Namespace( + results=str(results), controls=str(controls), format="json", output=None + ) + ) + captured = capsys.readouterr() + assert "did not pass" in captured.err + assert "c1" in captured.err + + def test_closing_line_reports_the_failing_count(self, tmp_path, capsys): + """The closing "Wrote ..." line only prints when writing to a real file. + + One control (c1) is covered by a failed scenario; a second (c2) has no + tagged scenario at all, so it is a gap. matrix.entries therefore has 2 + controls, matrix.gap_count is 1 (c2), and matrix.covered_with_failure + is ["c1"] (length 1) -- so the closing line must read + "(2 controls, 1 gaps, 1 failing)". + """ + results = tmp_path / "results.json" + results.write_text( + json.dumps( + { + "schema_version": "1.0", + "results": [ + { + "nodeid": "t.py::test_a", + "outcome": "failed", + "markers": ["control-c1"], + } + ], + } + ), + encoding="utf-8", + ) + controls = tmp_path / "controls.toml" + controls.write_text( + '[controls.c1]\ndescription = "a control"\nverification = "automated"\n' + '[controls.c2]\ndescription = "an untested control"\nverification = "automated"\n', + encoding="utf-8", + ) + out = tmp_path / "matrix.json" + run_trace( + argparse.Namespace( + results=str(results), controls=str(controls), format="json", output=str(out) + ) + ) + captured = capsys.readouterr() + assert f"Wrote {out} (2 controls, 1 gaps, 1 failing, 0 not verified)" in captured.out + + +class TestCoveredButUnproven: + """The third way a covered control is not evidence: VIP could not check it.""" + + def _run(self, tmp_path, statuses): + results = tmp_path / "results.json" + results.write_text( + json.dumps( + { + "schema_version": "1.0", + "results": [ + { + "nodeid": f"t.py::test_{i}", + "outcome": "skipped" if status == "unproven" else status, + "unproven": status == "unproven", + "markers": ["control-c1"], + } + for i, status in enumerate(statuses) + ], + } + ), + encoding="utf-8", + ) + controls = _write_controls( + tmp_path, '[controls.c1]\ndescription = "d"\nverification = "automated"\n' + ) + return results, controls + + def test_a_pass_beside_an_unproven_skip_warns(self, tmp_path): + """Neither of the other two warnings fires here, which is why this one exists.""" + results, controls = self._run(tmp_path, ["passed", "unproven"]) + r = _run_trace(results, controls) + assert r.returncode == 0 + assert "could not verify" in r.stderr + assert "c1" in r.stderr + assert "did not pass" not in r.stderr + assert "no scenario that ran" not in r.stderr + + def test_the_json_summary_counts_it(self, tmp_path): + results, controls = self._run(tmp_path, ["passed", "unproven"]) + payload = json.loads(_run_trace(results, controls, "--format", "json").stdout) + assert payload["summary"]["covered_unproven"] == 1 + assert payload["covered_with_unproven"] == ["c1"] + + def test_a_clean_run_does_not_warn(self, tmp_path): + results, controls = self._run(tmp_path, ["passed"]) + assert "could not verify" not in _run_trace(results, controls).stderr + + +class TestOutputFormatResolution: + def test_json_extension_infers_json(self, tmp_path): + results = _skipped_results(tmp_path) + controls = _write_controls(tmp_path, '[controls.x]\ndescription = "d"\n') + out = tmp_path / "matrix.json" + assert _run_trace(results, controls, "--output", str(out)).returncode == 0 + json.loads(out.read_text()) # would raise if CSV had been written + + def test_explicit_format_wins_and_warns_on_a_mismatch(self, tmp_path): + results = _skipped_results(tmp_path) + controls = _write_controls(tmp_path, '[controls.x]\ndescription = "d"\n') + out = tmp_path / "matrix.json" + r = _run_trace(results, controls, "--output", str(out), "--format", "csv") + assert "does not match" in r.stderr + assert out.read_text().startswith("control_id,") + + def test_unknown_extension_falls_back_to_csv(self, tmp_path): + results = _skipped_results(tmp_path) + controls = _write_controls(tmp_path, '[controls.x]\ndescription = "d"\n') + out = tmp_path / "matrix.txt" + _run_trace(results, controls, "--output", str(out)) + assert out.read_text().startswith("control_id,") + + +class TestMalformedInputIsReportedNotRaised: + @pytest.mark.parametrize("markers", [None, "control-x", 7, {"a": 1}]) + def test_non_list_markers_is_refused(self, tmp_path, markers): + """Reading it as untagged would report a gap the suite does not have.""" + p = tmp_path / "results.json" + p.write_text( + json.dumps( + { + "schema_version": "1.0", + "results": [{"nodeid": "t.py::a", "outcome": "passed", "markers": markers}], + } + ), + encoding="utf-8", + ) + controls = _write_controls(tmp_path, '[controls.x]\ndescription = "d"\n') + r = _run_trace(p, controls) + assert r.returncode == 1 + assert "Traceback" not in r.stderr + assert "expected a list" in r.stderr + assert "t.py::a" in r.stderr + + def test_a_row_that_is_not_an_object_is_refused(self, tmp_path): + p = tmp_path / "results.json" + p.write_text(json.dumps({"schema_version": "1.0", "results": ["nope"]}), encoding="utf-8") + controls = _write_controls(tmp_path, '[controls.x]\ndescription = "d"\n') + r = _run_trace(p, controls) + assert r.returncode == 1 + assert "expected an object" in r.stderr + + def test_absent_markers_is_still_accepted(self, tmp_path): + """Omitting the key is legal; only a wrong type is refused.""" + p = tmp_path / "results.json" + p.write_text( + json.dumps( + {"schema_version": "1.0", "results": [{"nodeid": "a", "outcome": "passed"}]} + ), + encoding="utf-8", + ) + controls = _write_controls(tmp_path, '[controls.x]\ndescription = "d"\n') + assert _run_trace(p, controls).returncode == 0 + + def test_toml_native_date_is_refused_by_both_formats(self, tmp_path): + results = _skipped_results(tmp_path) + controls = _write_controls( + tmp_path, '[controls.x]\ndescription = "d"\nreference = 2024-01-01\n' + ) + for extra in ([], ["--format", "json"]): + r = _run_trace(results, controls, *extra) + assert r.returncode == 1 + assert "Traceback" not in r.stderr + assert "expected a string" in r.stderr + + def test_output_to_a_directory_errors_cleanly(self, tmp_path): + results = _skipped_results(tmp_path) + controls = _write_controls(tmp_path, '[controls.x]\ndescription = "d"\n') + target = tmp_path / "adir" + target.mkdir() + r = _run_trace(results, controls, "--output", str(target)) + assert r.returncode == 1 + assert "Traceback" not in r.stderr + assert "could not write" in r.stderr + + +class TestCsvProvenance: + def test_csv_rows_carry_the_results_digest(self, tmp_path): + results = _skipped_results(tmp_path) + controls = _write_controls(tmp_path, '[controls.x]\ndescription = "d"\n') + rows = list(csv.DictReader(io.StringIO(_run_trace(results, controls).stdout))) + expected = hashlib.sha256(results.read_bytes()).hexdigest() + assert rows[0]["results_sha256"] == expected diff --git a/selftests/test_traceability_controls.py b/selftests/test_traceability_controls.py new file mode 100644 index 00000000..700987a8 --- /dev/null +++ b/selftests/test_traceability_controls.py @@ -0,0 +1,214 @@ +from pathlib import Path + +import pytest + +from vip.traceability import ControlListError, load_controls + +FULL = """ +[controls.cfr-11-10-e] +description = "Secure, computer-generated, time-stamped audit trails" +reference = "21 CFR 11.10(e)" +risk = "high" +verification = "automated" +responsibility = "shared" +notes = "Retention duration is a customer configuration decision." + +[controls.training] +description = "Personnel training records" +verification = "procedural" +responsibility = "customer" +""" + + +def test_loads_all_fields(tmp_path): + p = tmp_path / "controls.toml" + p.write_text(FULL) + controls = load_controls(p) + + audit = controls["cfr-11-10-e"] + assert audit.description.startswith("Secure") + assert audit.reference == "21 CFR 11.10(e)" + assert audit.risk == "high" + assert audit.verification == "automated" + assert audit.responsibility == "shared" + assert audit.notes + + +def test_optional_fields_default(tmp_path): + p = tmp_path / "controls.toml" + p.write_text('[controls.x]\ndescription = "only required key"\n') + spec = load_controls(p)["x"] + assert spec.reference is None + assert spec.risk is None + assert spec.responsibility is None + assert spec.verification == "automated" + + +def test_missing_description_is_an_error(tmp_path): + p = tmp_path / "controls.toml" + p.write_text('[controls.x]\nrisk = "high"\n') + with pytest.raises(ControlListError, match="description"): + load_controls(p) + + +def test_unknown_verification_value_is_an_error(tmp_path): + p = tmp_path / "controls.toml" + p.write_text('[controls.x]\ndescription = "d"\nverification = "vibes"\n') + with pytest.raises(ControlListError, match="verification"): + load_controls(p) + + +def test_missing_file_is_an_error(tmp_path): + with pytest.raises(ControlListError, match="not found"): + load_controls(tmp_path / "nope.toml") + + +def test_missing_controls_table_is_an_error(tmp_path): + p = tmp_path / "controls.toml" + p.write_text('title = "wrong shape"\n') + with pytest.raises(ControlListError, match="controls"): + load_controls(p) + + +def test_numeric_description_is_an_error(tmp_path): + p = tmp_path / "controls.toml" + p.write_text("[controls.x]\ndescription = 5\n") + with pytest.raises(ControlListError, match="x.*expected a string"): + load_controls(p) + + +def test_list_description_is_an_error(tmp_path): + p = tmp_path / "controls.toml" + p.write_text('[controls.y]\ndescription = ["a", "b"]\n') + with pytest.raises(ControlListError, match="y.*expected a string"): + load_controls(p) + + +def test_empty_description_is_an_error(tmp_path): + p = tmp_path / "controls.toml" + p.write_text('[controls.z]\ndescription = ""\n') + with pytest.raises(ControlListError, match="empty"): + load_controls(p) + + +def test_directory_path_is_an_error(tmp_path): + with pytest.raises(ControlListError, match="not a file"): + load_controls(tmp_path) + + +def test_empty_controls_table_is_an_error(tmp_path): + p = tmp_path / "controls.toml" + p.write_text("[controls]\n") + with pytest.raises(ControlListError, match="empty"): + load_controls(p) + + +def test_list_verification_is_an_error(tmp_path): + p = tmp_path / "controls.toml" + p.write_text('[controls.x]\ndescription = "d"\nverification = ["automated"]\n') + with pytest.raises(ControlListError, match="x.*expected a string"): + load_controls(p) + + +def test_dict_verification_is_an_error(tmp_path): + p = tmp_path / "controls.toml" + p.write_text('[controls.x]\ndescription = "d"\n[controls.x.verification]\nauto = true\n') + with pytest.raises(ControlListError, match="x.*expected a string"): + load_controls(p) + + +def test_numeric_verification_is_an_error(tmp_path): + p = tmp_path / "controls.toml" + p.write_text('[controls.x]\ndescription = "d"\nverification = 5\n') + with pytest.raises(ControlListError, match="verification"): + load_controls(p) + + +def test_boolean_verification_is_an_error(tmp_path): + p = tmp_path / "controls.toml" + p.write_text('[controls.x]\ndescription = "d"\nverification = true\n') + with pytest.raises(ControlListError, match="verification"): + load_controls(p) + + +def test_whitespace_only_description_is_an_error(tmp_path): + p = tmp_path / "controls.toml" + p.write_text('[controls.x]\ndescription = " "\n') + with pytest.raises(ControlListError, match="empty"): + load_controls(p) + + +class TestUnknownKeysAndExtras: + """A control list is the regulatory mapping of record, so a key that goes + nowhere is worse than a rejected one. Before this, `phase = "OQ"` produced + no error and no column, and so did a typo like `referance`. + """ + + @staticmethod + def _write(tmp_path, body): + p = tmp_path / "controls.toml" + p.write_text(f'[controls.c1]\ndescription = "A control"\n{body}\n', encoding="utf-8") + return p + + def test_an_unknown_key_is_rejected_and_the_message_points_at_extra(self, tmp_path): + path = self._write(tmp_path, 'phase = "OQ"') + with pytest.raises(ControlListError) as exc: + load_controls(path) + assert "unknown key phase" in str(exc.value) + assert "[controls.c1.extra]" in str(exc.value) + + def test_a_misspelled_known_key_is_caught_rather_than_dropped(self, tmp_path): + """The failure this actually prevents: a reference that silently + vanishes from the matrix a reviewer reads.""" + path = self._write(tmp_path, 'referance = "21 CFR 11.10(e)"') + with pytest.raises(ControlListError, match="unknown key referance"): + load_controls(path) + + def test_several_unknown_keys_are_listed_together(self, tmp_path): + path = self._write(tmp_path, 'phase = "OQ"\nowner = "QA"') + with pytest.raises(ControlListError, match="unknown keys owner, phase"): + load_controls(path) + + def test_an_extra_table_is_carried_through_verbatim(self, tmp_path): + path = self._write(tmp_path, '[controls.c1.extra]\nphase = "OQ"\nsop = "SOP-QA-014"') + assert load_controls(path)["c1"].extra == {"phase": "OQ", "sop": "SOP-QA-014"} + + def test_a_control_with_no_extra_table_gets_an_empty_dict(self, tmp_path): + assert load_controls(self._write(tmp_path, "")).get("c1").extra == {} + + def test_a_non_string_extra_value_is_rejected_like_the_built_ins(self, tmp_path): + """TOML reads a bare date as datetime.date, which the JSON encoder + refuses and the CSV writer silently stringifies.""" + path = self._write(tmp_path, "[controls.c1.extra]\nqualified = 2024-01-01") + with pytest.raises(ControlListError, match="expected a string"): + load_controls(path) + + def test_an_extra_key_colliding_with_a_column_is_rejected(self, tmp_path): + """Two `risk` columns in one CSV, and a spreadsheet takes the last.""" + path = self._write(tmp_path, '[controls.c1.extra]\nrisk = "high"') + with pytest.raises(ControlListError, match="already a column"): + load_controls(path) + + # As written in the TOML source. The control characters go in as escape + # sequences because TOML rejects a raw newline or tab inside a key, which + # is a second gate rather than the one under test here. + @pytest.mark.parametrize("prefix", ["=", "+", "-", "@", "\\t", "\\r", "\\n"]) + def test_a_formula_leading_extra_key_is_rejected(self, tmp_path, prefix): + """TOML allows a quoted key, so the column *name* is attacker-reachable + and lands in the CSV header, which row-value neutralization misses.""" + path = self._write(tmp_path, f'[controls.c1.extra]\n"{prefix}HYPERLINK(1)" = "v"') + with pytest.raises(ControlListError, match="reads as a formula"): + load_controls(path) + + def test_a_key_merely_containing_a_formula_character_is_allowed(self): + """Only the leading character matters to a spreadsheet.""" + import tempfile + + with tempfile.TemporaryDirectory() as d: + path = self._write(Path(d), '[controls.c1.extra]\n"risk-tier" = "2"') + assert load_controls(path)["c1"].extra == {"risk-tier": "2"} + + def test_an_extra_that_is_not_a_table_is_rejected(self, tmp_path): + path = self._write(tmp_path, 'extra = "OQ"') + with pytest.raises(ControlListError, match=r"\[controls.c1.extra\] must be a table"): + load_controls(path) diff --git a/selftests/test_traceability_matrix.py b/selftests/test_traceability_matrix.py new file mode 100644 index 00000000..f8391d75 --- /dev/null +++ b/selftests/test_traceability_matrix.py @@ -0,0 +1,330 @@ +import hashlib + +import pytest + +from conftest import matrix_from_statuses +from vip.reporting import ReportData, TestResult +from vip.traceability import ( + ControlSpec, + ResultsIntegrityError, + build_traceability_matrix, + verify_results_checksum, +) + + +def _result(nodeid, markers, outcome="passed", **kw): + return TestResult( + nodeid=nodeid, + outcome=outcome, + markers=markers, + scenario_title=kw.pop("title", "A scenario"), + started_at="2026-08-28T12:00:00+00:00", + finished_at="2026-08-28T12:00:01+00:00", + **kw, + ) + + +def _controls(**kw): + return { + cid: ControlSpec(control_id=cid, description=f"desc {cid}", **opts) + for cid, opts in kw.items() + } + + +def test_covered_control_lists_its_scenarios(): + data = ReportData(results=[_result("t.py::a", ["connect", "control-x"], title="Scenario A")]) + matrix = build_traceability_matrix(data, _controls(x={})) + entry = matrix.entries[0] + assert entry.coverage == "covered" + assert entry.matches[0].scenario_title == "Scenario A" + assert entry.matches[0].status == "passed" + assert entry.matches[0].started_at == "2026-08-28T12:00:00+00:00" + + +def test_uncovered_automated_control_is_a_gap(): + matrix = build_traceability_matrix(ReportData(results=[]), _controls(x={})) + assert matrix.entries[0].coverage == "gap" + assert matrix.gap_count == 1 + + +def test_procedural_control_is_not_a_gap(): + controls = _controls(x={"verification": "procedural"}) + matrix = build_traceability_matrix(ReportData(results=[]), controls) + assert matrix.entries[0].coverage == "not_automatable" + assert matrix.gap_count == 0 + + +def test_one_control_satisfied_by_several_scenarios(): + data = ReportData( + results=[ + _result("t.py::a", ["control-x"], title="A"), + _result("t.py::b", ["control-x"], title="B"), + ] + ) + entry = build_traceability_matrix(data, _controls(x={})).entries[0] + assert [m.scenario_title for m in entry.matches] == ["A", "B"] + + +def test_one_scenario_satisfies_several_controls(): + data = ReportData(results=[_result("t.py::a", ["control-x", "control-y"])]) + matrix = build_traceability_matrix(data, _controls(x={}, y={})) + assert all(e.coverage == "covered" for e in matrix.entries) + + +def test_unrecognized_tag_is_reported(): + data = ReportData(results=[_result("t.py::a", ["control-typo"])]) + matrix = build_traceability_matrix(data, _controls(x={})) + assert matrix.unrecognized_tags == ["control-typo"] + + +def test_failure_detail_is_carried(): + data = ReportData( + results=[ + _result("t.py::a", ["control-x"], outcome="failed", concise_error="boom"), + ] + ) + match = build_traceability_matrix(data, _controls(x={})).entries[0].matches[0] + assert match.status == "failed" + assert match.detail == "boom" + + +def test_skip_reason_is_carried(): + data = ReportData( + results=[ + _result("t.py::a", ["control-x"], outcome="skipped", skip_reason="not configured"), + ] + ) + assert build_traceability_matrix(data, _controls(x={})).entries[0].matches[0].detail == ( + "not configured" + ) + + +def test_na_version_status_is_distinct(): + data = ReportData( + results=[_result("t.py::a", ["control-x"], outcome="skipped", na_version=True)] + ) + assert build_traceability_matrix(data, _controls(x={})).entries[0].matches[0].status == ( + "na_version" + ) + + +def test_entries_and_matches_are_sorted_deterministically(): + data = ReportData( + results=[ + _result("t.py::z", ["control-b"], title="Z"), + _result("t.py::a", ["control-b"], title="A"), + ] + ) + matrix = build_traceability_matrix(data, _controls(b={}, a={})) + assert [e.control.control_id for e in matrix.entries] == ["a", "b"] + b_entry = next(e for e in matrix.entries if e.control.control_id == "b") + assert [m.nodeid for m in b_entry.matches] == ["t.py::a", "t.py::z"] + + +def test_custom_tag_prefix(): + data = ReportData(results=[_result("t.py::a", ["req-x"])]) + matrix = build_traceability_matrix(data, _controls(x={}), tag_prefix="req-") + assert matrix.entries[0].coverage == "covered" + + +def test_provenance_is_carried_from_the_report(): + data = ReportData( + generated_at="2026-08-28T12:00:00+00:00", + vip_version="2026.8.3", + schema_version="1.0", + basic_mode=True, + execution={"hostname": "runner-1", "git": None, "ci": None}, + ) + prov = build_traceability_matrix(data, _controls(x={})).provenance + assert prov["vip_version"] == "2026.8.3" + assert prov["basic_mode"] is True + assert prov["execution"]["hostname"] == "runner-1" + assert prov["results_schema_version"] == "1.0" + + +def test_verify_results_checksum_reports_sidecar_presence(tmp_path): + results = tmp_path / "results.json" + data = b'{"results": []}' + results.write_bytes(data) + expected_digest = hashlib.sha256(data).hexdigest() + results.with_name("results.json.sha256").write_text(f"{expected_digest} results.json\n") + + digest, sidecar_present = verify_results_checksum(results) + assert digest == expected_digest + assert sidecar_present is True + + +def test_verify_results_checksum_reports_missing_sidecar(tmp_path): + results = tmp_path / "results.json" + data = b'{"results": []}' + results.write_bytes(data) + expected_digest = hashlib.sha256(data).hexdigest() + + digest, sidecar_present = verify_results_checksum(results) + assert digest == expected_digest + assert sidecar_present is False + + +@pytest.mark.parametrize("body", ["", " ", "\n", "\t\n "]) +def test_verify_results_checksum_rejects_an_empty_sidecar(tmp_path, body): + """An empty sidecar must not report as verified. + + It is the truncated-upload case this function advertises catching, and + returning True would write `results_sha256_sidecar_verified: true` into the + provenance block while nothing had been compared -- a false attestation in + the one field whose purpose is attesting the check happened. + """ + results = tmp_path / "results.json" + results.write_bytes(b'{"results": []}') + results.with_name("results.json.sha256").write_text(body) + + with pytest.raises(ResultsIntegrityError, match="empty"): + verify_results_checksum(results) + + +def test_provenance_defaults_checksum_fields_to_none(): + # A matrix built directly from a ReportData -- as every test above does, + # with no results file on disk -- must still work with no digest to carry. + prov = build_traceability_matrix(ReportData(results=[]), _controls(x={})).provenance + assert prov["results_sha256"] is None + assert prov["results_sha256_sidecar_verified"] is None + + +def test_provenance_carries_results_sha256_when_supplied(): + prov = build_traceability_matrix( + ReportData(results=[]), + _controls(x={}), + results_sha256="deadbeef", + results_sha256_sidecar_verified=True, + ).provenance + assert prov["results_sha256"] == "deadbeef" + assert prov["results_sha256_sidecar_verified"] is True + + +def test_provenance_distinguishes_sidecar_absent_from_verified(): + verified = build_traceability_matrix( + ReportData(results=[]), + _controls(x={}), + results_sha256="deadbeef", + results_sha256_sidecar_verified=True, + ).provenance + absent = build_traceability_matrix( + ReportData(results=[]), + _controls(x={}), + results_sha256="deadbeef", + results_sha256_sidecar_verified=None, + ).provenance + assert verified["results_sha256_sidecar_verified"] is True + assert absent["results_sha256_sidecar_verified"] is None + assert verified["results_sha256_sidecar_verified"] != absent["results_sha256_sidecar_verified"] + + +class TestFailingControls: + """A control whose scenarios ran and did not pass is not evidence.""" + + def test_a_failed_scenario_marks_the_control_failing(self): + matrix = matrix_from_statuses(statuses={"c1": ["failed"]}) + entry = matrix.entries[0] + assert entry.coverage == "covered" + assert entry.executed is True + assert entry.failing is True + assert matrix.covered_with_failure == ["c1"] + + def test_an_errored_scenario_marks_the_control_failing(self): + """`error` is a reachable outcome; enumerating only "failed" would miss it.""" + matrix = matrix_from_statuses(statuses={"c1": ["error"]}) + assert matrix.entries[0].failing is True + + def test_a_mixed_pass_and_failure_marks_the_control_failing(self): + matrix = matrix_from_statuses(statuses={"c1": ["passed", "failed"]}) + assert matrix.entries[0].failing is True + assert matrix.covered_with_failure == ["c1"] + + def test_a_pass_beside_a_skip_is_not_failing(self): + matrix = matrix_from_statuses(statuses={"c1": ["passed", "skipped"]}) + assert matrix.entries[0].failing is False + assert matrix.covered_with_failure == [] + + def test_an_all_skipped_control_is_not_failing(self): + """Not executed and not failing are different states.""" + matrix = matrix_from_statuses(statuses={"c1": ["skipped"]}) + entry = matrix.entries[0] + assert entry.executed is False + assert entry.failing is False + + def test_a_version_gated_control_is_not_failing(self): + matrix = matrix_from_statuses(statuses={"c1": ["na_version"]}) + assert matrix.entries[0].failing is False + + def test_an_unproven_control_is_not_run_rather_than_failed(self): + """An unproven skip ran no assertion, so it is a non-execution. + + Reporting it as failing would say the control was verified and came + back bad, when VIP never got to check it. The distinction is the whole + point of ``vip.attest.unproven``. + """ + matrix = matrix_from_statuses(statuses={"c1": ["unproven"]}) + entry = matrix.entries[0] + assert entry.coverage == "covered" + assert entry.executed is False + assert entry.failing is False + assert matrix.covered_without_execution == ["c1"] + assert matrix.covered_with_failure == [] + + def test_a_pass_beside_an_unproven_skip_is_not_failing(self): + """Treating unproven as non-executing means it cannot mark a control failing. + + ``covered_with_unproven`` is what surfaces this control instead; see + ``TestUnprovenControls``. + """ + matrix = matrix_from_statuses(statuses={"c1": ["passed", "unproven"]}) + entry = matrix.entries[0] + assert entry.executed is True + assert entry.failing is False + assert [m.status for m in entry.matches] == ["passed", "unproven"] + + +class TestUnprovenControls: + """A control VIP was asked to check and could not is its own fact. + + Coverage, execution and outcome already fail to describe it: an unproven + skip runs no assertion, so it is not executed and not failing, and those + two alone leave a control with one pass and one unproven scenario reading + as fully evidenced. + """ + + def test_an_unproven_scenario_marks_the_control(self): + matrix = matrix_from_statuses(statuses={"c1": ["unproven"]}) + entry = matrix.entries[0] + assert entry.has_unproven is True + assert matrix.covered_with_unproven == ["c1"] + + def test_a_pass_beside_an_unproven_skip_is_surfaced(self): + """The case the third bucket exists for: the other two say nothing.""" + matrix = matrix_from_statuses(statuses={"c1": ["passed", "unproven"]}) + assert matrix.covered_with_unproven == ["c1"] + assert matrix.covered_without_execution == [] + assert matrix.covered_with_failure == [] + + def test_the_three_lists_overlap_rather_than_partition(self): + """Not disjoint buckets, and deliberately so. + + The matrix keeps coverage, execution and outcome as separate facts and + flattens them only for display. An unproven-only control did not run + *and* could not be checked, so it belongs in both lists; forcing a + single bucket would drop one of the two true statements. + """ + matrix = matrix_from_statuses(statuses={"c1": ["unproven"], "c2": ["failed", "unproven"]}) + assert matrix.covered_without_execution == ["c1"] + assert matrix.covered_with_failure == ["c2"] + assert matrix.covered_with_unproven == ["c1", "c2"] + + def test_a_plain_skip_is_not_unproven(self): + """`skipped` says there was nothing to check; `unproven` says VIP could not.""" + matrix = matrix_from_statuses(statuses={"c1": ["skipped"], "c2": ["na_version"]}) + assert matrix.covered_with_unproven == [] + + def test_a_passing_control_is_not_unproven(self): + matrix = matrix_from_statuses(statuses={"c1": ["passed"]}) + assert matrix.entries[0].has_unproven is False + assert matrix.covered_with_unproven == [] diff --git a/selftests/test_traceability_render.py b/selftests/test_traceability_render.py new file mode 100644 index 00000000..2f3e4fd5 --- /dev/null +++ b/selftests/test_traceability_render.py @@ -0,0 +1,304 @@ +import csv +import io +import json + +from conftest import matrix_from_statuses +from vip.reporting import ReportData, TestResult +from vip.traceability import ( + ControlSpec, + build_traceability_matrix, + render_csv, + render_json, +) + +CSV_COLUMNS = [ + "control_id", + "description", + "reference", + "risk", + "verification", + "responsibility", + "coverage", + "scenario", + "nodeid", + "status", + "started_at", + "finished_at", + "detail", + "notes", + "generated_at", + "vip_version", + "results_sha256", + "exit_status", +] + + +def _matrix(): + data = ReportData( + generated_at="2026-08-28T12:00:00+00:00", + vip_version="2026.8.3", + results=[ + TestResult( + nodeid="t.py::a", + outcome="passed", + markers=["control-x"], + scenario_title="Scenario A", + started_at="2026-08-28T12:00:00+00:00", + finished_at="2026-08-28T12:00:01+00:00", + ) + ], + ) + controls = { + "x": ControlSpec("x", "Audit trail", reference="21 CFR 11.10(e)", risk="high"), + "y": ControlSpec("y", "Training records", verification="procedural"), + } + return build_traceability_matrix(data, controls) + + +def test_csv_has_the_expected_header(): + reader = csv.reader(io.StringIO(render_csv(_matrix()))) + assert next(reader) == CSV_COLUMNS + + +def test_csv_emits_one_row_per_match_and_one_for_a_gap(): + rows = list(csv.DictReader(io.StringIO(render_csv(_matrix())))) + assert len(rows) == 2 + covered = next(r for r in rows if r["control_id"] == "x") + assert covered["scenario"] == "Scenario A" + assert covered["status"] == "passed" + assert covered["coverage"] == "covered" + + procedural = next(r for r in rows if r["control_id"] == "y") + assert procedural["coverage"] == "not_automatable" + assert procedural["scenario"] == "" + assert procedural["nodeid"] == "" + + +def test_csv_is_byte_identical_across_invocations(): + m = _matrix() + assert render_csv(m) == render_csv(m) + assert render_csv(_matrix()) == render_csv(_matrix()) + + +def test_json_carries_provenance_and_schema_version(): + payload = json.loads(render_json(_matrix())) + assert payload["schema_version"] == "1.1" + assert payload["provenance"]["vip_version"] == "2026.8.3" + assert payload["summary"]["gaps"] == 0 + assert payload["summary"]["covered"] == 1 + assert payload["summary"]["not_automatable"] == 1 + + +def test_json_summary_counts_failing_controls(): + matrix = matrix_from_statuses(statuses={"c1": ["failed"], "c2": ["passed"]}) + summary = json.loads(render_json(matrix))["summary"] + assert summary["covered_failed"] == 1 + assert summary["covered_and_executed"] == 2 + + +def test_json_summary_counts_unproven_controls(): + """Additive in 1.1: a control VIP could not check is neither a pass nor a failure.""" + matrix = matrix_from_statuses(statuses={"c1": ["passed", "unproven"], "c2": ["passed"]}) + payload = json.loads(render_json(matrix)) + assert payload["summary"]["covered_unproven"] == 1 + assert payload["summary"]["covered_failed"] == 0 + assert payload["summary"]["covered_and_executed"] == 2 + assert payload["covered_with_unproven"] == ["c1"] + + +def test_json_is_byte_identical_across_invocations(): + assert render_json(_matrix()) == render_json(_matrix()) + + +def test_json_round_trips(): + payload = json.loads(render_json(_matrix())) + entry = next(e for e in payload["controls"] if e["control_id"] == "x") + assert entry["matches"][0]["nodeid"] == "t.py::a" + assert entry["reference"] == "21 CFR 11.10(e)" + + +def test_csv_formula_injection_equals_sign(): + data = ReportData( + generated_at="2026-08-28T12:00:00+00:00", + vip_version="2026.8.3", + results=[], + ) + controls = { + "a": ControlSpec("a", "=SUM(A1:A10)", verification="automated"), + } + matrix = build_traceability_matrix(data, controls) + rows = list(csv.DictReader(io.StringIO(render_csv(matrix)))) + assert rows[0]["description"] == "'=SUM(A1:A10)" + + +def test_csv_formula_injection_plus_sign(): + data = ReportData( + generated_at="2026-08-28T12:00:00+00:00", + vip_version="2026.8.3", + results=[], + ) + controls = { + "a": ControlSpec("a", "+1+1", verification="automated"), + } + matrix = build_traceability_matrix(data, controls) + rows = list(csv.DictReader(io.StringIO(render_csv(matrix)))) + assert rows[0]["description"] == "'+1+1" + + +def test_csv_formula_injection_minus_sign(): + data = ReportData( + generated_at="2026-08-28T12:00:00+00:00", + vip_version="2026.8.3", + results=[], + ) + controls = { + "a": ControlSpec("a", "-2+3", verification="automated"), + } + matrix = build_traceability_matrix(data, controls) + rows = list(csv.DictReader(io.StringIO(render_csv(matrix)))) + assert rows[0]["description"] == "'-2+3" + + +def test_csv_formula_injection_at_sign(): + data = ReportData( + generated_at="2026-08-28T12:00:00+00:00", + vip_version="2026.8.3", + results=[], + ) + controls = { + "a": ControlSpec("a", "@SUM(1,2)", verification="automated"), + } + matrix = build_traceability_matrix(data, controls) + rows = list(csv.DictReader(io.StringIO(render_csv(matrix)))) + assert rows[0]["description"] == "'@SUM(1,2)" + + +def test_csv_formula_injection_leading_tab(): + data = ReportData( + generated_at="2026-08-28T12:00:00+00:00", + vip_version="2026.8.3", + results=[], + ) + controls = { + "a": ControlSpec("a", "\t=SUM(1,2)", verification="automated"), + } + matrix = build_traceability_matrix(data, controls) + rows = list(csv.DictReader(io.StringIO(render_csv(matrix)))) + assert rows[0]["description"] == "'\t=SUM(1,2)" + + +def test_csv_formula_injection_leading_carriage_return(): + data = ReportData( + generated_at="2026-08-28T12:00:00+00:00", + vip_version="2026.8.3", + results=[], + ) + controls = { + "a": ControlSpec("a", "\r=SUM(1,2)", verification="automated"), + } + matrix = build_traceability_matrix(data, controls) + rows = list(csv.DictReader(io.StringIO(render_csv(matrix)))) + assert rows[0]["description"] == "'\r=SUM(1,2)" + + +def test_csv_formula_injection_leading_newline(): + data = ReportData( + generated_at="2026-08-28T12:00:00+00:00", + vip_version="2026.8.3", + results=[], + ) + controls = { + "a": ControlSpec("a", "\n=SUM(1,2)", verification="automated"), + } + matrix = build_traceability_matrix(data, controls) + rows = list(csv.DictReader(io.StringIO(render_csv(matrix)))) + assert rows[0]["description"] == "'\n=SUM(1,2)" + + +def test_csv_normal_description_not_escaped(): + data = ReportData( + generated_at="2026-08-28T12:00:00+00:00", + vip_version="2026.8.3", + results=[], + ) + controls = { + "a": ControlSpec("a", "Normal Description", verification="automated"), + } + matrix = build_traceability_matrix(data, controls) + rows = list(csv.DictReader(io.StringIO(render_csv(matrix)))) + assert rows[0]["description"] == "Normal Description" + assert not rows[0]["description"].startswith("'") + + +def test_json_non_ascii_appears_literally(): + data = ReportData( + generated_at="2026-08-28T12:00:00+00:00", + vip_version="2026.8.3", + results=[], + ) + controls = { + "a": ControlSpec("a", "Café français", verification="automated"), + } + matrix = build_traceability_matrix(data, controls) + rendered = render_json(matrix) + assert "Café français" in rendered + assert "\\u" not in rendered + + +class TestExtraColumns: + """[controls..extra] reaches both exports.""" + + @staticmethod + def _matrix(extras): + from vip.reporting import ReportData + from vip.traceability import ControlSpec, build_traceability_matrix + + controls = { + cid: ControlSpec(control_id=cid, description=f"control {cid}", extra=extra) + for cid, extra in extras.items() + } + return build_traceability_matrix(ReportData(results=[]), controls) + + def test_csv_appends_the_extra_columns_after_the_fixed_set(self): + csv_text = render_csv(self._matrix({"c1": {"phase": "OQ"}})) + header = csv_text.splitlines()[0].split(",") + assert header[: len(CSV_COLUMNS)] == CSV_COLUMNS + assert header[len(CSV_COLUMNS) :] == ["phase"] + + def test_the_column_set_is_the_sorted_union_across_controls(self): + matrix = self._matrix({"c1": {"phase": "OQ"}, "c2": {"owner": "QA"}}) + header = render_csv(matrix).splitlines()[0].split(",") + assert header[len(CSV_COLUMNS) :] == ["owner", "phase"] + + def test_a_control_missing_an_extra_key_gets_an_empty_cell_not_a_ragged_row(self): + matrix = self._matrix({"c1": {"phase": "OQ"}, "c2": {"owner": "QA"}}) + rows = list(csv.reader(io.StringIO(render_csv(matrix)))) + assert all(len(r) == len(rows[0]) for r in rows) + by_id = {r[0]: dict(zip(rows[0], r)) for r in rows[1:]} + assert by_id["c1"]["owner"] == "" + assert by_id["c2"]["phase"] == "" + + def test_an_extra_value_is_formula_neutralized_like_every_other_cell(self): + """A future column must inherit the CSV-injection protection.""" + csv_text = render_csv(self._matrix({"c1": {"phase": "=SUM(1,2)"}})) + assert "'=SUM(1,2)" in csv_text + + def test_a_formula_leading_header_cell_is_neutralized(self): + """load_controls rejects such a key, so this covers the other door: a + ControlSpec built in code rather than loaded from TOML.""" + csv_text = render_csv(self._matrix({"c1": {"=HYPERLINK(1)": "v"}})) + header = next(csv.reader(io.StringIO(csv_text))) + assert header[-1] == "'=HYPERLINK(1)" + + def test_ordinary_header_cells_are_untouched(self): + header = next(csv.reader(io.StringIO(render_csv(self._matrix({"c1": {"phase": "OQ"}}))))) + assert header[: len(CSV_COLUMNS)] == CSV_COLUMNS + assert header[-1] == "phase" + + def test_json_nests_extra_under_each_control(self): + payload = json.loads(render_json(self._matrix({"c1": {"phase": "OQ"}}))) + assert payload["controls"][0]["extra"] == {"phase": "OQ"} + + def test_a_matrix_with_no_extras_keeps_the_exact_fixed_column_set(self): + header = render_csv(self._matrix({"c1": {}})).splitlines()[0].split(",") + assert header == CSV_COLUMNS diff --git a/src/vip/attribution.py b/src/vip/attribution.py new file mode 100644 index 00000000..72ae8f29 --- /dev/null +++ b/src/vip/attribution.py @@ -0,0 +1,221 @@ +"""Execution attribution for the results.json evidence record. + +Answers "which pipeline execution, on which host, from which commit produced +this evidence" — the fields that make an automated test result attributable. + +Every probe here degrades to None. A missing git binary, a detached worktree, +a non-repo working directory or an unrecognized CI system must never fail or +warn a verification run; provenance is not worth breaking a run over. +""" + +from __future__ import annotations + +import getpass +import os +import platform +import subprocess +from collections.abc import Mapping +from pathlib import Path +from typing import Any +from urllib.parse import urlsplit, urlunsplit + +_GIT_TIMEOUT_SECONDS = 5 + + +def redact_userinfo(url: str | None) -> str | None: + """Strip any userinfo component from a remote URL. + + CI checkouts rewrite the origin to embed a credential + (``https://x-access-token:ghs_...@github.com/org/repo``). results.json is + an uploaded artifact, so the credential must never reach it. Userinfo is + never needed to identify a repository, so it is dropped unconditionally + rather than pattern-matched against known token shapes. + """ + if not url: + return None + if "://" not in url: + # scp-style (git@host:org/repo.git). No password component, but drop + # the user anyway so there is exactly one rule to reason about. + # + # Require the full scp shape rather than just an "@": a local path may + # legitimately contain one, and stripping on the "@" alone rewrites + # repo@2.git to 2.git and releases@2026/repo.git to 2026/repo.git, + # corrupting a provenance remote to remove a credential that was never + # there. scp syntax is user@host:path, so the host part must carry a + # colon before any slash. + user, sep, rest = url.partition("@") + if not sep or "/" in user: + return url + host = rest.split("/", 1)[0] + return rest if ":" in host else url + try: + parts = urlsplit(url) + hostname = parts.hostname + # urlsplit is lazy: it accepts "https://host:bad/x" and only raises + # when .port parses. Both accesses must sit inside the guard, or a + # malformed remote escapes the never-fail contract and takes down + # report writing at the end of an otherwise good run. + port = parts.port + has_userinfo = parts.username is not None or parts.password is not None + except ValueError: + return None + # Nothing to redact means nothing to rebuild. Returning the URL untouched + # is what keeps a host-less remote intact: file:///srv/git/repo.git has no + # hostname, and rebuilding it from the decomposed parts would drop it to + # None -- deleting provenance from an evidence record to strip a + # credential that was never there. + if not has_userinfo: + return url + if not hostname: + return None + # An IPv6 literal must keep its brackets or the port fuses into the + # address: [2001:db8::1]:8443 would otherwise rebuild as 2001:db8::1:8443, + # which no longer parses back to a host. + host = f"[{hostname}]" if ":" in hostname else hostname + netloc = f"{host}:{port}" if port else host + return urlunsplit((parts.scheme, netloc, parts.path, "", "")) + + +def _git(args: list[str], cwd: Path) -> str | None: + """Run a git command, returning stripped stdout, or None if it failed. + + An empty string is a valid successful result (``status --porcelain`` on a + clean tree), so success-with-no-output must stay distinguishable from + failure. Callers rely on that difference. + """ + try: + proc = subprocess.run( + ["git", *args], + cwd=cwd, + capture_output=True, + # Decode explicitly rather than via text=True, which uses the + # locale codec with errors="strict". Branch names and remote URLs + # come out of git config as raw bytes, so a non-ASCII one on a + # cp1252/cp932 host raises UnicodeDecodeError -- a ValueError, not + # caught below, which would escape this module's never-fail + # contract and take the whole report write down with it. + encoding="utf-8", + errors="replace", + timeout=_GIT_TIMEOUT_SECONDS, + check=False, + ) + except (OSError, subprocess.SubprocessError): + return None + if proc.returncode != 0: + return None + return proc.stdout.strip() + + +def _git_metadata(cwd: Path, env: Mapping[str, str]) -> dict[str, Any] | None: + commit = env.get("GITHUB_SHA") or _git(["rev-parse", "HEAD"], cwd) + if not commit: + return None + status = _git(["status", "--porcelain"], cwd) + return { + "commit": commit, + "branch": env.get("GITHUB_REF_NAME") or _git(["rev-parse", "--abbrev-ref", "HEAD"], cwd), + "dirty": bool(status) if status is not None else None, + "remote": redact_userinfo(_git(["remote", "get-url", "origin"], cwd)), + } + + +def _ci_metadata(env: Mapping[str, str]) -> dict[str, Any] | None: + if env.get("GITHUB_ACTIONS") == "true": + run_id = env.get("GITHUB_RUN_ID") + repo = env.get("GITHUB_REPOSITORY") + server = env.get("GITHUB_SERVER_URL") or "https://github.com" + run_url = f"{server}/{repo}/actions/runs/{run_id}" if run_id and repo else None + return { + "provider": "github", + "run_id": run_id, + "run_attempt": env.get("GITHUB_RUN_ATTEMPT"), + "run_url": run_url, + "job": env.get("GITHUB_JOB"), + } + if env.get("GITLAB_CI") == "true": + return { + "provider": "gitlab", + "run_id": env.get("CI_PIPELINE_ID"), + "run_attempt": None, + "run_url": env.get("CI_JOB_URL"), + "job": env.get("CI_JOB_NAME"), + } + if env.get("JENKINS_URL"): + return { + "provider": "jenkins", + "run_id": env.get("BUILD_NUMBER"), + "run_attempt": None, + "run_url": env.get("BUILD_URL"), + "job": env.get("JOB_NAME"), + } + return None + + +def _performed_by(env: Mapping[str, str]) -> dict[str, Any] | None: + """Who ran this verification. + + FDA's Computer Software Assurance guidance asks for a record of who + performed the testing alongside the date. Every other field in this module + identifies a *machine* or a *commit*, which answers "which execution" but + not "which person is accountable for it". + + Resolution order, most deliberate first: ``VIP_PERFORMED_BY`` names the + person on whose behalf the run happens, which is what a QA engineer + kicking off a scheduled pipeline needs to record; then the CI system's own + actor; then the local login. ``source`` travels with the value because an + auditor reading "svc-vip-runner" needs to know whether a human typed that + or a service account inherited it. + """ + explicit = (env.get("VIP_PERFORMED_BY") or "").strip() + if explicit: + return {"identity": explicit, "source": "explicit"} + for var, source in ( + ("GITHUB_ACTOR", "github"), + ("GITLAB_USER_LOGIN", "gitlab"), + ("BUILD_USER_ID", "jenkins"), + ): + value = (env.get(var) or "").strip() + if value: + return {"identity": value, "source": source} + try: + login = getpass.getuser().strip() + except Exception: + # getpass.getuser() raises on a container with no passwd entry and no + # LOGNAME/USER/LNAME/USERNAME set. Same never-fail contract as the rest + # of this module. + return None + return {"identity": login, "source": "login"} if login else None + + +def collect_execution_metadata( + *, cwd: Path | None = None, env: Mapping[str, str] | None = None +) -> dict[str, Any]: + """Collect host, git and CI attribution for the current run. + + ``hostname`` is the VIP runner's host, not the system under test. Anything + rendering it must label it that way. + + ``performed_by`` records an operator identity, so the whole block is what + ``--vip-no-attribution`` exists to omit for anyone who does not want that + written into an archived artifact. + """ + resolved_env = os.environ if env is None else env + if cwd is None: + # Path.cwd() raises FileNotFoundError when the working directory has + # been removed underneath the process -- a fixture that deletes its own + # cwd, or a CI step unmounting the workspace. Unguarded it escapes this + # module's never-fail contract. + try: + resolved_cwd: Path | None = Path.cwd() + except OSError: + resolved_cwd = None + else: + resolved_cwd = cwd + return { + "hostname": platform.node() or None, + # No cwd means no directory to resolve a repository against, so skip + # git rather than hand subprocess a None it would reject. + "git": _git_metadata(resolved_cwd, resolved_env) if resolved_cwd is not None else None, + "ci": _ci_metadata(resolved_env), + "performed_by": _performed_by(resolved_env), + } diff --git a/src/vip/cli.py b/src/vip/cli.py index 9ea2436a..f741dfd3 100644 --- a/src/vip/cli.py +++ b/src/vip/cli.py @@ -535,8 +535,13 @@ def run_verify(args: argparse.Namespace) -> None: if config_path: cmd.append(f"--vip-config={config_path}") - if args.report: - cmd.append(f"--vip-report={args.report}") + # Forward even an empty value. `--vip-report=` is how the plugin is told to + # write no report at all, and skipping the flag instead left the plugin on + # its own default -- so `vip verify --report ''` wrote report/results.json, + # the one thing it was asked not to do. Nothing else reads args.report, and + # argparse's default is a non-empty path, so the empty string is the only + # invocation whose behavior changes. + cmd.append(f"--vip-report={args.report}") fmt = "json,junit,sarif" if getattr(args, "ci", False) else getattr(args, "format", "json") requested = [f.strip().lower() for f in fmt.split(",") if f.strip()] @@ -548,6 +553,21 @@ def run_verify(args: argparse.Namespace) -> None: file=sys.stderr, ) sys.exit(2) + # junit.xml and results.sarif are written as siblings of results.json and + # are built by reloading it, so they cannot exist without it. Before + # --report '' was honored the two flags could be combined and junit still + # appeared; now the combination would run the whole suite and produce + # nothing. Refuse it up front instead. + siblings = [f for f in requested if f != "json"] + if not args.report and siblings: + source = "--ci" if getattr(args, "ci", False) else "--format" + print( + f"Error: --report '' disables the results file, but {source} asks for " + f"{', '.join(siblings)}, which {'are' if len(siblings) > 1 else 'is'} " + "written from it. Drop one of the two.", + file=sys.stderr, + ) + sys.exit(2) cmd.append(f"--vip-format={','.join(requested)}") if args.interactive_auth: cmd.append("--interactive-auth") @@ -767,7 +787,44 @@ def run_report(args: argparse.Namespace) -> None: if not results_src.exists(): print(f"Error: results file not found: {results_src}", file=sys.stderr) sys.exit(1) + if getattr(args, "controls", None): + # Verify the SOURCE before the copy, not only the destination + # after it. _rehome_sidecar is right to discard an empty or + # unreadable source sidecar rather than manufacture one at the + # destination -- but a missing destination sidecar is legal and + # benign, so the gate below would then wave through the very + # input `vip trace` refuses as a truncated attestation. The + # compliance render must never be more permissive than + # `vip trace` on identical bytes. A source with genuinely no + # sidecar stays benign here, exactly as it is for `vip trace`. + from vip.traceability import ResultsIntegrityError, verify_results_checksum + + try: + verify_results_checksum(results_src) + except ResultsIntegrityError as exc: + print(f"Error: {exc}", file=sys.stderr) + sys.exit(1) + except (OSError, UnicodeDecodeError) as exc: + print(f"Error: could not read results file {results_src}: {exc}", file=sys.stderr) + sys.exit(1) shutil.copy2(results_src, results_dest) + # Keep the checksum sidecar with the results it describes. Copying a + # results.json from a CI artifact over the local one leaves the + # previous run's sidecar in place, and the next `vip trace` then + # reports a checksum mismatch on a file nobody tampered with. Carry + # the source's sidecar across when it has one; otherwise remove the + # stale local one, because no sidecar is a documented benign state + # and a wrong one is a false tamper alarm. + src_sidecar = results_src.with_name(f"{results_src.name}.sha256") + dest_sidecar = results_dest.with_name(f"{results_dest.name}.sha256") + try: + _rehome_sidecar(src_sidecar, dest_sidecar, results_src.name, results_dest.name) + except (OSError, UnicodeDecodeError) as exc: + # UnicodeDecodeError as well as OSError: _rehome_sidecar reads with + # encoding="utf-8-sig" and a corrupt sidecar would otherwise reach the + # user as a traceback. verify_results_checksum already catches both on + # the identical read, and the two paths should agree. + print(f"Warning: could not update {dest_sidecar}: {exc}", file=sys.stderr) elif not results_dest.exists(): print( f"Error: no results found at {results_dest}. " @@ -792,6 +849,56 @@ def run_report(args: argparse.Namespace) -> None: # is the vip install itself, which always has both. See issue #554. env = {**os.environ, "QUARTO_PYTHON": sys.executable} + # Scope the control list to this render via the environment. Copying + # controls.toml into the report directory was the obvious alternative and + # is wrong: that directory survives between runs, so one + # `vip report --controls ...` would leave a file behind that every later + # plain `vip report` silently picks up, growing a compliance section + # nobody asked for out of a stale list. Validate it here so a malformed + # file fails before Quarto starts, rather than inside a notebook cell + # where the .qmd can only degrade to a warning. + if getattr(args, "controls", None): + from vip.traceability import ( + ControlListError, + ResultsIntegrityError, + check_results_rows, + check_results_schema, + load_controls, + read_results_schema_version, + verify_results_checksum, + ) + + controls_path = Path(args.controls).resolve() + try: + # --controls turns the report into a compliance artifact, so it + # inherits `vip trace`'s strictness about its evidence. Plain + # `vip report` stays lenient on purpose: `load_results` normalizes + # a malformed `markers` to an empty list and only warns on an + # unknown schema major, because a report must render regardless. + # That leniency is wrong here for one specific reason -- a row + # whose markers cannot be read looks untagged, so the control it + # was tagged for is printed as a GAP that does not exist, and the + # matrix claims the suite is missing a check it actually has. + # Refuse the file rather than render a compliance section that + # understates coverage. Same order as run_trace: the schema gate + # runs first, because the row check assumes current-shape rows. + check_results_schema(read_results_schema_version(results_dest)) + check_results_rows(results_dest) + # The sidecar too, not only the schema and the rows. A compliance + # render is an evidence artifact, so it inherits `vip trace`'s + # strictness in full rather than in part. + verify_results_checksum(results_dest) + load_controls(controls_path) + except (ResultsIntegrityError, ControlListError) as exc: + print(f"Error: {exc}", file=sys.stderr) + sys.exit(1) + except (json.JSONDecodeError, OSError, UnicodeDecodeError) as exc: + # read_results_schema_version raises these raw. A malformed + # results.json must not reach the user as a traceback. + print(f"Error: could not read results file {results_dest}: {exc}", file=sys.stderr) + sys.exit(1) + env["VIP_CONTROLS"] = str(controls_path) + # The HTML pages and the PDF render as separate quarto invocations on # purpose. One combined `quarto render` ties their fates together: on a # Quarto too old to know Typst (pre-1.4), the PDF document fails the @@ -1147,6 +1254,10 @@ def cleanup_callable(_url: str) -> None: # noqa: F811 "cross_product_validation", "R/Python runtime versions and package installability across Connect and Workbench", ), + "21cfr-part11-validation": ( + "21CFR_part11_validation", + "Compliance control tagging plus a controls.toml for `vip trace`", + ), } _DEFAULT_SCAFFOLD_TEMPLATE = "cross-product" @@ -1201,6 +1312,14 @@ def _scaffold_next_steps(template: str, dest: Path) -> str: f" vip verify --config vip.toml --extensions {dest}\n" f"\nSee {dest / 'README.md'} for full customization instructions." ) + if template == "21cfr-part11-validation": + return ( + f"\nNext steps:\n" + f" 1. Replace {dest / 'controls.toml'} with your own control list.\n" + f" 2. Tag your scenarios with @control- matching those ids.\n" + f" 3. Run: vip verify --extensions {dest}\n" + f" 4. Run: vip trace --controls {dest / 'controls.toml'}\n" + ) return ( f"\nNext steps:\n" f" 1. Edit {dest / 'test_custom_check.feature'} and" @@ -1258,7 +1377,16 @@ def run_scaffold(args: argparse.Namespace) -> None: else: dest.unlink() - shutil.copytree(src, dest) + # Skip build/test detritus. A source checkout that has run the example + # accumulates __pycache__ and .pytest_cache beside it, and without this + # they land in the customer's brand-new extension directory. Harmless + # but scruffy, and it makes the scaffold output differ depending on + # whether the VIP checkout happened to have run its own tests. + shutil.copytree( + src, + dest, + ignore=shutil.ignore_patterns("__pycache__", "*.pyc", "*.pyo", ".pytest_cache"), + ) # AGENTS.md is shared across every template (single source of truth), so # it's copied in separately rather than living inside each template dir. @@ -1545,6 +1673,244 @@ def run_version(args: argparse.Namespace) -> None: print(_format_version_details()) +def _resolve_trace_format(explicit: str | None, out: Path | None) -> str: + """Pick the matrix output format: explicit flag, then --output suffix, then csv. + + Inferring from the suffix is what stops `--output matrix.json` writing CSV + bytes into a .json file and reporting success -- the archived artifact then + fails to parse in whatever downstream consumer reads it, and unlike the + stdout case the caller never sees the bytes to notice. + """ + inferred = {".json": "json", ".csv": "csv"}.get(out.suffix.lower()) if out else None + if explicit is None: + return inferred or "csv" + if out is not None and inferred and inferred != explicit: + print( + f"Warning: --format {explicit} does not match the {out.suffix} extension of " + f"{out}; writing {explicit}.", + file=sys.stderr, + ) + return explicit + + +def _rehome_sidecar(src: Path, dest: Path, src_name: str, dest_name: str) -> None: + """Move a checksum sidecar alongside a copied results file. + + The digest is carried across unchanged -- recomputing it from the copy + would launder a tampered file into a verified one, which is the opposite + of what the sidecar is for. Only the recorded filename is rewritten, so a + source named run-42.json still verifies once copied to results.json. + + No source sidecar means the stale destination one is removed rather than + left behind: no sidecar is a documented benign state, a wrong one is a + false tamper alarm. + + Exact-name precedence is preserved across the rehome, because the + destination sidecar must never record two different digests under the + destination name. A source that names both ``results.json`` and + ``archive/results.json`` used to rewrite *both* lines, so the copy said + two things about one file and ``verify_results_checksum`` picked whichever + one agreed. Rewrite the exact matches when there are any; fall back to the + basename otherwise, using the same distinct-digest rule + ``verify_results_checksum`` applies: several basename matches that all + carry the same digest (compared case-insensitively) are unambiguous and + are rewritten together, same as a single match. Basename matches that + disagree on the digest are left alone -- the source never had the + authority to say which one describes the destination, so the copy does + not invent it, and verification reports that rather than guessing. + """ + from vip.traceability import sidecar_basename + + if not src.is_file(): + dest.unlink(missing_ok=True) + return + parsed: list[tuple[str, str, str | None]] = [] + for line in src.read_text(encoding="utf-8-sig").splitlines(): + parts = line.split(None, 1) + if not parts: + continue + recorded = parts[1].strip().lstrip("*") if len(parts) > 1 else None + parsed.append((line, parts[0], recorded)) + + # A bare digest counts as an exact entry: it names no other file, so it + # can only be describing the one being copied. + rewrite = {i for i, (_, _, r) in enumerate(parsed) if r is None or r == src_name} + if not rewrite: + # Compare basenames, not the raw recorded name. A sidecar generated + # from a parent directory records a path, and copying that line + # through verbatim produces a rehomed sidecar that then fails + # verification at the destination -- the false tamper alarm this + # function exists to prevent. + src_base = sidecar_basename(src_name) + matches = [i for i, (_, _, r) in enumerate(parsed) if r and sidecar_basename(r) == src_base] + distinct = {parsed[i][1].lower() for i in matches} + rewrite = set(matches) if len(distinct) == 1 else set() + lines = [ + f"{digest} {dest_name}" if i in rewrite else raw + for i, (raw, digest, _) in enumerate(parsed) + ] + if not lines: + # A source that parses to zero entries (whitespace-only, truncated) + # would otherwise produce an empty destination sidecar, which + # verify_results_checksum refuses as the truncated-upload case. No + # sidecar is the documented benign state, so produce that instead. + dest.unlink(missing_ok=True) + return + dest.write_text("\n".join(lines) + "\n", encoding="utf-8") + + +def run_trace(args: argparse.Namespace) -> None: + """Join a results.json against a control list and emit a traceability matrix.""" + import warnings + + from vip.reporting import load_results + from vip.traceability import ( + ControlListError, + ResultsIntegrityError, + build_traceability_matrix, + check_results_rows, + check_results_schema, + load_controls, + read_results_schema_version, + render_csv, + render_json, + verify_results_checksum, + ) + + results_path = Path(args.results) + if not results_path.is_file(): + print(f"Error: results file not found: {results_path}", file=sys.stderr) + sys.exit(1) + + try: + results_sha256, sidecar_present = verify_results_checksum(results_path) + # Read and validate the schema version BEFORE load_results ever + # indexes into the results list. load_results assumes current-shape + # rows (r["nodeid"], r["outcome"], ...) and raises KeyError on + # anything else, so the schema gate must run first or an + # incompatible/malformed file crashes before it can be refused + # cleanly. + check_results_schema(read_results_schema_version(results_path)) + # Structural validation before load_results normalizes the problem + # away. load_results turns a malformed `markers` into an empty list so + # the Quarto report still renders; for a matrix that silently converts + # a tagged scenario into a coverage gap. + check_results_rows(results_path) + # load_results only warns (not raises) on an unknown schema major -- + # it's also called from index.qmd/details.qmd/`vip report`, where that + # warning is the point. The check above already hard-errors on the + # same condition, so suppress the redundant warning here only. + with warnings.catch_warnings(): + warnings.simplefilter("ignore", UserWarning) + data = load_results(results_path) + controls = load_controls(args.controls) + except (ResultsIntegrityError, ControlListError) as exc: + print(f"Error: {exc}", file=sys.stderr) + sys.exit(1) + except ( + json.JSONDecodeError, + OSError, + UnicodeDecodeError, + AttributeError, + KeyError, + TypeError, + ) as exc: + # A malformed results.json must not surface as a traceback -- this + # catches structural failures (e.g. {"results": [{}]}) that pass JSON + # parsing and the schema gate but fail load_results' own field + # indexing. + print(f"Error: could not read results file {results_path}: {exc}", file=sys.stderr) + sys.exit(1) + + out = Path(args.output) if args.output else None + fmt = _resolve_trace_format(args.format, out) + + try: + matrix = build_traceability_matrix( + data, + controls, + results_sha256=results_sha256, + results_sha256_sidecar_verified=sidecar_present or None, + ) + rendered = render_json(matrix) if fmt == "json" else render_csv(matrix) + except (AttributeError, KeyError, TypeError, ValueError) as exc: + # Inside the guard, not outside it: a results.json can pass the + # checksum, the schema gate and load_results and still be structurally + # wrong in a way that only surfaces here -- an explicit `"markers": + # null`, say. A compliance tool reporting that as a raw traceback is + # the one presentation that tells an operator nothing. + print(f"Error: could not build the matrix from {results_path}: {exc}", file=sys.stderr) + sys.exit(1) + + if matrix.unrecognized_tags: + joined = ", ".join(matrix.unrecognized_tags) + print( + f"Warning: control tags present in results but absent from the control list: {joined}", + file=sys.stderr, + ) + + # A covered control whose scenarios ran and failed also counts toward + # "0 gaps". Coverage records that a scenario ran, not that it passed, and + # a compliance matrix that stays silent here is the more expensive of the + # two ways this tool can mislead. + failing = matrix.covered_with_failure + if failing: + print( + f"Warning: {len(failing)} covered control(s) had a scenario that did not " + f"pass: {', '.join(failing)}. Coverage records that a scenario ran, not " + "that it passed.", + file=sys.stderr, + ) + + # A covered control whose every scenario was skipped still counts toward + # "0 gaps". True, and on its own misleading: a scenario that runs and + # skips itself still counts as covering its control, so the greenest + # matrix this tool can print is one produced by verifying nothing. + unexecuted = matrix.covered_without_execution + if unexecuted: + print( + f"Warning: {len(unexecuted)} covered control(s) have no scenario that ran " + f"(all skipped): {', '.join(unexecuted)}. Coverage records that a scenario " + "is tagged, not that it was executed.", + file=sys.stderr, + ) + + # The third condition, and the only one that catches a control whose + # scenarios ran and passed while part of the control went unchecked. The + # two warnings above stay silent on that case, because an unproven skip is + # neither an execution nor a failure. + unproven = matrix.covered_with_unproven + if unproven: + print( + f"Warning: {len(unproven)} covered control(s) had a scenario VIP " + f"could not verify: {', '.join(unproven)}. An unproven check was asked " + "for and could not be run, which is not the same as one that found " + "nothing to test.", + file=sys.stderr, + ) + + if out is None: + sys.stdout.write(rendered) + return + + try: + out.parent.mkdir(parents=True, exist_ok=True) + # Write via a temp file in the destination directory, then replace. + # `write_text` truncates before it encodes, so a UnicodeEncodeError or + # a full disk would destroy a previously good matrix at this path. + tmp = out.with_name(f"{out.name}.tmp") + tmp.write_text(rendered, encoding="utf-8") + os.replace(tmp, out) + except (OSError, UnicodeError) as exc: + print(f"Error: could not write {out}: {exc}", file=sys.stderr) + sys.exit(1) + print( + f"Wrote {out} ({len(matrix.entries)} controls, {matrix.gap_count} gaps, " + f"{len(matrix.covered_with_failure)} failing, " + f"{len(matrix.covered_with_unproven)} not verified)" + ) + + def main() -> None: """Main entry point for the VIP CLI.""" from vip import __version__ @@ -1767,7 +2133,9 @@ def main() -> None: verify_parser.add_argument( "--report", default="report/results.json", - help="Write JSON results to this path for Quarto report generation" + help="Write JSON results to this path for Quarto report generation." + " Pass an empty string to write no results file, which also rules out" + " the junit/sarif siblings built from it." " (default: report/results.json)", ) verify_parser.add_argument( @@ -1928,6 +2296,14 @@ def main() -> None: default="report/results.json", help="Path to results.json (default: report/results.json)", ) + report_parser.add_argument( + "--controls", + default=None, + help=( + "Path to a controls.toml control list. Adds a compliance traceability " + "section to the HTML report and the PDF. Applies to this render only." + ), + ) report_parser.add_argument( "--open", action="store_true", @@ -1999,6 +2375,37 @@ def main() -> None: ) scaffold_parser.set_defaults(func=run_scaffold) + # vip trace + trace_parser = subparsers.add_parser( + "trace", + help="Generate a compliance traceability matrix from test results", + description=( + "Join a results.json against a control list (controls.toml) and emit a " + "control-to-scenario traceability matrix as CSV or JSON.\n\n" + "Scenarios declare the control they satisfy with an @control- " + "Gherkin tag. Controls with no matching scenario are reported as coverage " + 'gaps, except those marked verification = "manual" or "procedural", ' + "which are reported as not verifiable by automated test." + ), + formatter_class=argparse.RawDescriptionHelpFormatter, + ) + trace_parser.add_argument( + "--results", + default="report/results.json", + help="Path to results.json (default: report/results.json)", + ) + trace_parser.add_argument( + "--controls", required=True, help="Path to the controls.toml control list" + ) + trace_parser.add_argument( + "--format", + choices=("csv", "json"), + default=None, + help="Output format (default: inferred from --output's extension, else csv)", + ) + trace_parser.add_argument("--output", default=None, help="Write to this path instead of stdout") + trace_parser.set_defaults(func=run_trace) + # Map command names to their parsers for context-appropriate help subcommand_parsers = { "version": version_parser, @@ -2010,6 +2417,7 @@ def main() -> None: "report": report_parser, "status": status_parser, "scaffold": scaffold_parser, + "trace": trace_parser, } argv = _reorder_help_args(sys.argv[1:], set(subcommand_parsers)) diff --git a/src/vip/clients/base.py b/src/vip/clients/base.py index 1c0a5225..4219a1e1 100644 --- a/src/vip/clients/base.py +++ b/src/vip/clients/base.py @@ -132,6 +132,7 @@ def __init__( # None sentinel: scale the 30-second default. Callers that supply an # explicit value opt out of scaling (their choice is honored as-is). effective_timeout = scaled(30.0) if timeout is None else timeout + self._timeout = effective_timeout self._client = httpx.Client( base_url=f"{self._base_url}{api_prefix}", headers=headers, @@ -192,6 +193,40 @@ def cookies(self) -> httpx.Cookies | None: """ return self._cookies + def unauthenticated_status(self, path: str) -> int: + """Return the status code for `path` requested with no credentials. + + Uses a separate short-lived client so the configured API key and + cookies are not sent -- sending them would make the scenario assert + nothing at all, since an authorised caller is *supposed* to get 200. + + Lives on the base class because every product's access-control + scenario needs the same probe: the Part 11 example asserts it against + Connect's user API and Workbench's session API alike. + + Mirrors the ad-hoc-request contract that + :meth:`~vip.clients.connect.ConnectClient.fetch_content` follows: + route through the proxy the pooled client already resolved and pin + ``trust_env=False`` so that decision is authoritative, then fold the CA + env overrides back in by hand. ``trust_env=False`` disables httpx's own reading of + ``SSL_CERT_FILE``/``SSL_CERT_DIR`` along with the proxy vars, so + without ``verify_with_env_ca`` this probe would fail TLS against a + corporate CA that the pooled client verifies fine -- surfacing as a + Part 11 scenario erroring on a deployment that is actually healthy. + """ + from vip.proxy import proxy_for_url, verify_with_env_ca + + if not path.startswith("/"): + path = f"/{path}" + url = f"{self.base_url}{path}" + with httpx.Client( + verify=verify_with_env_ca(self._verify), + proxy=proxy_for_url(url, self._proxy_map), + trust_env=False, + timeout=self._timeout, + ) as client: + return client.get(url).status_code + def close(self) -> None: """Close the underlying httpx client.""" self._client.close() diff --git a/src/vip/clients/connect.py b/src/vip/clients/connect.py index d2f757ed..522509a1 100644 --- a/src/vip/clients/connect.py +++ b/src/vip/clients/connect.py @@ -497,3 +497,35 @@ def send_test_email(self, to: str) -> dict[str, Any]: resp = self._client.post("/v1/tasks/send-test-email", json={"to": to}) resp.raise_for_status() return resp.json() + + # -- Audit log ------------------------------------------------------------ + + def list_audit_logs(self, *, limit: int = 20) -> list[dict[str, Any]] | None: + """Return recent audit log entries, or None if unavailable. + + None (rather than an exception) for 404/403 so a caller can skip on a + deployment that does not expose the endpoint or a key that cannot read + it, without conflating that with an empty log. + """ + resp = self._client.get("/v1/audit_logs", params={"limit": limit}) + if resp.status_code in (403, 404): + return None + resp.raise_for_status() + payload = resp.json() + return payload.get("results", []) if isinstance(payload, dict) else payload + + def audit_log_allowed_methods(self) -> set[str] | None: + """Return the HTTP methods the audit log endpoint advertises, or None. + + Reads the Allow header from an OPTIONS request. Deliberately + read-only: proving the audit trail is immutable must never involve + deleting an audit record, which in a regulated deployment destroys the + very evidence the control protects. + """ + resp = self._client.request("OPTIONS", "/v1/audit_logs") + if resp.status_code in (401, 403, 404, 405, 501): + return None + allow = resp.headers.get("allow", "") + if not allow: + return None + return {m.strip().upper() for m in allow.split(",") if m.strip()} diff --git a/src/vip/clients/packagemanager.py b/src/vip/clients/packagemanager.py index 3cd4e0a7..6da1bbe5 100644 --- a/src/vip/clients/packagemanager.py +++ b/src/vip/clients/packagemanager.py @@ -81,6 +81,28 @@ def status(self) -> dict[str, Any]: resp.raise_for_status() return resp.json() + # -- Snapshots ---------------------------------------------------------- + + def snapshot_index_reachable(self, repo_name: str, snapshot: str) -> tuple[bool, int]: + """Check whether a point-in-time snapshot of a repo's CRAN index is served. + + *snapshot* is a Package Manager snapshot identifier -- a ``YYYY-MM-DD`` + date, or the id of a frozen repository URL. A dated URL is what lets a + regulated deployment reconstruct the exact package set an analysis ran + against, so this reads the snapshot's ``PACKAGES`` index rather than + just checking that the URL answers. + + Returns ``(found, status)`` on the same contract as + :meth:`cran_windows_binary_index_reachable`: *found* is True only for a + 200 carrying a real index body, so a caller can tell a broken server + (fail) from snapshots being switched off or that date predating the + repository (skip). + """ + resp = self._client.get(f"/{repo_name}/{snapshot}/src/contrib/PACKAGES") + if resp.status_code != 200: + return False, resp.status_code + return "Package:" in resp.text, resp.status_code + # -- CRAN --------------------------------------------------------------- def cran_package_available(self, repo_name: str, package: str) -> bool: diff --git a/src/vip/config.py b/src/vip/config.py index b152f6d4..710afb71 100644 --- a/src/vip/config.py +++ b/src/vip/config.py @@ -574,8 +574,9 @@ def load_config(path: str | Path | None = None) -> VIPConfig: path = Path(path) if not path.exists(): - # Return default config when no file is present - tests will be - # skipped for unconfigured products. + # Return default config when no file is present - every product is + # then unconfigured, so pytest_collection_modifyitems deselects its + # tests rather than skipping them (they never reach the report). warnings.warn(f"Config file not found: {path}", stacklevel=2) return VIPConfig() diff --git a/src/vip/gherkin.py b/src/vip/gherkin.py index 2523b973..4739f0a0 100644 --- a/src/vip/gherkin.py +++ b/src/vip/gherkin.py @@ -14,6 +14,30 @@ # Step keywords (prefix match after stripping whitespace). _STEP_PREFIXES = ("Given ", "When ", "Then ", "And ", "But ") +# Gherkin tags of the form @control- map a scenario to a compliance +# control. They are deliberately excluded from the derived feature marker: +# that value feeds the HTML report cards and the generated test catalog and +# feature matrix, so a control tag written before the product tag would +# otherwise silently mislabel the feature. +CONTROL_TAG_PREFIX = "control-" + + +def read_feature_tags(path: Path) -> list[str]: + """Every Gherkin tag in a feature file, feature-level and scenario-level. + + A tags-only reader, deliberately separate from :func:`parse_feature_file`: + the marker pre-scan in ``vip.plugin`` runs on every pytest invocation in + any environment where VIP is installed, and building the full scenario and + step model just to read the tag lines is most of that cost. + """ + tags: list[str] = [] + with path.open(encoding="utf-8") as fh: + for raw in fh: + line = raw.strip() + if line.startswith("@"): + tags.extend(tok.lstrip("@") for tok in line.split() if tok.startswith("@")) + return tags + def parse_feature_file(path: Path, *, relative_to: Path | None = None) -> dict: """Parse a ``.feature`` file and return a structured dict. @@ -28,8 +52,8 @@ def parse_feature_file(path: Path, *, relative_to: Path | None = None) -> dict: Returns ------- - dict with keys: ``title``, ``description``, ``marker``, ``file``, - ``scenarios`` (list of dicts with ``title`` and ``steps``). + dict with keys: ``title``, ``description``, ``marker``, ``tags``, + ``file``, ``scenarios`` (list of dicts with ``title`` and ``steps``). """ text = path.read_text(encoding="utf-8") lines = text.splitlines() @@ -38,6 +62,7 @@ def parse_feature_file(path: Path, *, relative_to: Path | None = None) -> dict: title = "" description_lines: list[str] = [] scenarios: list[dict] = [] + tags: list[str] = [] in_description = False current_scenario: dict | None = None @@ -49,12 +74,15 @@ def parse_feature_file(path: Path, *, relative_to: Path | None = None) -> dict: if not line or line.startswith("#"): continue - # Tag line — first tag becomes the marker. + # Tag line — first non-control tag becomes the marker. if line.startswith("@"): - # A tag after we've started collecting scenarios means a new - # tagged scenario, but we only care about the file-level marker. + line_tags = [tok.lstrip("@") for tok in line.split() if tok.startswith("@")] + tags.extend(line_tags) if not marker: - marker = line.lstrip("@").split()[0] if line.lstrip("@") else "" + for tag in line_tags: + if not tag.startswith(CONTROL_TAG_PREFIX): + marker = tag + break continue # Feature title. @@ -98,6 +126,7 @@ def parse_feature_file(path: Path, *, relative_to: Path | None = None) -> dict: "title": title, "description": "\n".join(description_lines).strip(), "marker": marker, + "tags": tags, "file": file_str, "scenarios": scenarios, } diff --git a/src/vip/plugin.py b/src/vip/plugin.py index 2b790d9a..c2d96903 100644 --- a/src/vip/plugin.py +++ b/src/vip/plugin.py @@ -17,7 +17,9 @@ from __future__ import annotations +import hashlib import json +import os import platform import re import sys @@ -30,9 +32,13 @@ from typing import Any import pytest +from _pytest.pathlib import fnmatch_ex from vip.attest import UNPROVEN_SENTINEL +from vip.attribution import collect_execution_metadata from vip.config import VIPConfig, load_config +from vip.gherkin import CONTROL_TAG_PREFIX, read_feature_tags +from vip.reporting import RESULTS_SCHEMA_VERSION from vip.version import ProductVersion # --------------------------------------------------------------------------- @@ -88,6 +94,125 @@ # --------------------------------------------------------------------------- +def _feature_roots(config: pytest.Config) -> list[Path]: + """Directories pytest is about to collect, plus any extension directories. + + Scanning these rather than walking rootpath keeps the pre-scan cheap in a + large monorepo and avoids registering controls from feature files that are + not part of this run. Extension directories are read from the merged + ``_ext_dirs_key`` stash (config file ``[general] extension_dirs`` plus + ``--vip-extensions``), not by re-reading the CLI option alone -- callers + must run this after ``config.stash[_ext_dirs_key]`` is populated in + ``pytest_configure``. + + Relative args resolve against ``config.invocation_params.dir`` -- the + directory pytest itself resolves them against -- NOT ``config.rootpath``. + The two differ whenever pytest is invoked from a subdirectory, and + resolving against rootpath there produces a path that does not exist. + ``rglob`` on a missing path yields nothing silently, so no control marker + gets registered and ``--strict-markers`` aborts collection: the exact + failure this pre-scan exists to prevent. + """ + invocation_dir = Path(config.invocation_params.dir) + roots: list[Path] = [] + for arg in config.args: + candidate = Path(str(arg).split("::")[0]) + roots.append(candidate if candidate.is_absolute() else invocation_dir / candidate) + roots.extend(Path(d) for d in config.stash.get(_ext_dirs_key, [])) + if not roots: + roots = [Path(config.rootpath)] + # Deduplicate on the resolved path: a targeted run can pass many step + # files from one directory (connect-smoke.yml passes 14 paths, 9 of them + # siblings), and without this each one re-reads the same feature files. + seen: dict[Path, Path] = {} + for root in roots: + try: + key = root.resolve() + except OSError: + key = root + seen.setdefault(key, root) + return list(seen.values()) + + +def _walk_features(root: Path, ignore: list[str]) -> list[Path]: + """Every ``.feature`` file under *root*, skipping ``norecursedirs`` matches. + + ``os.walk`` rather than ``rglob`` because only walk can prune a directory + before descending into it. Without pruning this descends into ``.venv`` + (which ``uv`` puts inside the project by default), ``.git`` and + ``.worktrees`` on every pytest run in any environment where VIP is + installed -- thousands of files walked to find feature files that could + never be collected. + """ + features: list[Path] = [] + for dirpath, dirnames, filenames in os.walk(root, onerror=lambda _: None): + dirnames[:] = [d for d in dirnames if not _is_ignored(Path(dirpath) / d, ignore)] + features.extend(Path(dirpath) / f for f in filenames if f.endswith(".feature")) + return sorted(features) + + +def _is_ignored(path: Path, patterns: list[str]) -> bool: + """Whether *path* matches a ``norecursedirs`` pattern, the way pytest matches it. + + Delegates to pytest's own ``fnmatch_ex`` so the two cannot drift: a pattern + containing a path separator matches against the whole path, while a bare + one matches the basename only. Matching the basename in both cases would + scan a directory pytest itself would never collect, which is how a control + tag from an ignored feature file ends up registered or warned about. + """ + for pattern in patterns: + try: + if fnmatch_ex(pattern, path): + return True + except (TypeError, ValueError): + continue + return False + + +def _discover_control_tags(config: pytest.Config) -> set[str]: + """Collect every @control-* tag from the feature files about to be collected.""" + try: + ignore = list(config.getini("norecursedirs") or []) + except (ValueError, KeyError): + ignore = [] + tags: set[str] = set() + seen_files: set[Path] = set() + for root in _feature_roots(config): + try: + if root.is_file(): + if root.suffix == ".feature": + features = [root] + else: + # pytest-bdd suites are usually collected by their step + # (.py) file, not the .feature file the tags live in -- + # e.g. a targeted `pytest test_x.py`. Scan only the + # containing directory; do not walk upward or widen. + features = sorted(root.parent.glob("*.feature")) + else: + features = _walk_features(root, ignore) + except OSError: + continue + for feature in features: + if feature in seen_files: + continue + seen_files.add(feature) + try: + found = read_feature_tags(feature) + except (OSError, UnicodeDecodeError): + continue + tags.update(t for t in found if t.startswith(CONTROL_TAG_PREFIX)) + return tags + + +# pytest derives a registered marker's name with +# ``line.split(":")[0].split("(")[0].strip()``, so either character truncates +# the name it registers under. Registering the truncated name is worse than +# not registering at all: pytest-bdd still applies the full tag, and +# --strict-markers then aborts collection against a marker list that looks +# like it should have matched. +_UNREGISTRABLE_TAG_CHARS = (":", "(") + + def pytest_addoption(parser: pytest.Parser) -> None: group = parser.getgroup("vip", "Verified Installation of Posit") group.addoption( @@ -153,6 +278,12 @@ def pytest_addoption(parser: pytest.Parser) -> None: default=False, help="Show full pytest tracebacks instead of concise error messages.", ) + group.addoption( + "--vip-no-attribution", + action="store_true", + default=False, + help="Omit host/git/CI attribution from results.json.", + ) def pytest_configure(config: pytest.Config) -> None: @@ -269,6 +400,28 @@ def pytest_configure(config: pytest.Config) -> None: ext_dirs.extend(config.getoption("--vip-extensions") or []) config.stash[_ext_dirs_key] = ext_dirs + # Compliance control tags (@control-) become pytest markers via + # pytest-bdd's default pytest_bdd_apply_tag hook. Their slugs are chosen by + # the customer, so they cannot be registered by name ahead of time -- but an + # unregistered mark warns by default and aborts collection outright under + # --strict-markers, which regulated CI is likely to enable. Registering the + # tags we are about to collect satisfies both paths at once. Run after the + # ext_dirs stash above so _feature_roots sees both extension sources + # (config-file [general] extension_dirs and --vip-extensions), and still + # well before collection starts. + for tag in sorted(_discover_control_tags(config)): + bad = [c for c in _UNREGISTRABLE_TAG_CHARS if c in tag] + if bad: + warnings.warn( + f"VIP: control tag @{tag} contains {' and '.join(repr(c) for c in bad)}, " + "which pytest cannot register as a marker name. Rename the control id " + "to use only letters, digits, '-', '.' and '_' (e.g. @control-11-10-a); " + "otherwise this scenario will fail collection under --strict-markers.", + stacklevel=1, + ) + continue + config.addinivalue_line("markers", f"{tag}: compliance control tag") + _any_product_configured = any( pc.is_configured for pc in (vip_cfg.connect, vip_cfg.workbench, vip_cfg.package_manager) ) @@ -1008,6 +1161,41 @@ def _extract_skip_reason(longrepr: object) -> str | None: return message.strip() or None +def _safe_execution_metadata(config: pytest.Config) -> dict[str, Any] | None: + """Attribution for results.json, or None if it was disabled or failed. + + ``collect_execution_metadata`` promises never to fail a run, and its own + probes are individually guarded. This is the belt-and-braces at the call + site: it is evaluated while building the results payload, which sits + ABOVE the try/except that writes the file, so anything escaping it takes + down results.json, the checksum sidecar, junit.xml, results.sarif and + failures.json together -- every artifact of an otherwise successful + verification run, lost for a provenance field. Provenance is never worth + that, so the catch here is deliberately broad. + """ + if config.getoption("--vip-no-attribution", default=False): + return None + try: + return collect_execution_metadata() + except Exception as exc: # noqa: BLE001 - see docstring + warnings.warn(f"VIP: could not collect execution attribution: {exc}", stacklevel=1) + return None + + +def _epoch_to_iso(value: float | None) -> str | None: + """Convert a pytest report epoch float to a UTC ISO 8601 string. + + Returns None rather than raising for a missing or unrepresentable value: + a provenance field is never worth failing a verification run over. + """ + if value is None: + return None + try: + return datetime.fromtimestamp(value, timezone.utc).isoformat() + except (OSError, OverflowError, ValueError): + return None + + def _format_concise_error( nodeid: str, exc_type: str, @@ -1285,6 +1473,8 @@ def pytest_runtest_logreport(report: pytest.TestReport) -> None: "scenario_title": getattr(report, "vip_scenario_title", None), "feature_description": getattr(report, "vip_feature_description", None), "na_version": getattr(report, "vip_na_version", False), + "started_at": _epoch_to_iso(getattr(report, "start", None)), + "finished_at": _epoch_to_iso(getattr(report, "stop", None)), "unproven": unproven, } ) @@ -1408,6 +1598,7 @@ def pytest_sessionfinish(session: pytest.Session, exitstatus: int) -> None: basic_mode = "not slow" in (session.config.getoption("markexpr", default="") or "") payload = { + "schema_version": RESULTS_SCHEMA_VERSION, "generated_at": datetime.now(timezone.utc).isoformat(), "deployment_name": cfg.deployment_name, "exit_status": exitstatus, @@ -1418,16 +1609,44 @@ def pytest_sessionfinish(session: pytest.Session, exitstatus: int) -> None: "basic_mode": basic_mode, "products": products, "results": results, + "execution": _safe_execution_metadata(session.config), } try: p = Path(report_path) p.parent.mkdir(parents=True, exist_ok=True) - p.write_text(json.dumps(payload, indent=2)) + # Invalidate any sidecar from a previous run BEFORE overwriting the + # results it describes. Otherwise a sidecar write that fails below + # leaves the old digest next to the new file, and the next `vip trace` + # reports a checksum mismatch -- a tamper alarm on a file the pipeline + # legitimately produced. No sidecar is a documented, benign state; a + # wrong one is not. + # + # Guarded separately from the results write: the sidecar path may be + # unremovable (a directory, a read-only mount), and results.json is + # what the user actually asked for. Letting an unlink failure abort + # the write would trade the whole evidence file for a checksum. + sidecar = p.with_name(f"{p.name}.sha256") + try: + sidecar.unlink(missing_ok=True) + except OSError: + pass + # Hash the exact bytes written, not a re-serialization: the sidecar is + # only useful if it verifies against the file actually on disk. + data = json.dumps(payload, indent=2).encode("utf-8") + p.write_bytes(data) except OSError as exc: warnings.warn(f"VIP: could not write report to {report_path}: {exc}", stacklevel=1) return + # A checksum is an optional artifact and must never suppress the outputs the + # user actually asked for (junit/sarif via --vip-format, and failures.json). + try: + digest = hashlib.sha256(data).hexdigest() + sidecar.write_text(f"{digest} {p.name}\n", encoding="utf-8") + except OSError as exc: + warnings.warn(f"VIP: could not write checksum sidecar for {p}: {exc}", stacklevel=1) + fmt = session.config.getoption("--vip-format", default="json") try: _emit_extra_formats(fmt, p) diff --git a/src/vip/report_content.py b/src/vip/report_content.py index 71da2ec6..51203ce3 100644 --- a/src/vip/report_content.py +++ b/src/vip/report_content.py @@ -380,8 +380,79 @@ def skip_reason_parts(item: TestResult) -> tuple[str, str]: NOT_RECORDED = "not recorded" +# How each `performed_by.source` reads in the report. `explicit` is the only +# unlabelled state, because it is the only one where a human named the person +# accountable for the run. Every other value was inherited from the +# environment and must say so: `GITHUB_ACTOR` on a scheduled run is whoever +# last touched the workflow, and a CI actor is often a service account, so an +# unlabelled one would be indistinguishable from a named operator in the +# archived artifact. +PERFORMER_SOURCE_LABELS = { + "github": "GitHub actor", + "gitlab": "GitLab user", + "jenkins": "Jenkins build user", + "login": "local login", +} + + +def _performer_label(performer: dict) -> str | None: + """The operator identity as the report shows it, qualified by its source.""" + identity = performer.get("identity") + source = performer.get("source") + if not identity or source == "explicit": + return identity + # An unrecognized source renders verbatim rather than bare: a value this + # version does not know about is still not an explicitly named operator, + # and rendering it unlabelled would promote it to one. Both backends + # escape the result, so an edited results.json cannot inject markup. + # A block carrying an identity but no source at all is malformed, and it + # gets the same treatment for the same reason -- never bare, and never the + # literal string "None". + label = PERFORMER_SOURCE_LABELS.get(source, source) if source else f"source {NOT_RECORDED}" + return f"{identity} ({label})" + + +def _execution_rows(execution: dict | None) -> list[tuple[str, str | None]]: + """Who ran this, on which host, from which commit, under which CI job. + + ``results.json`` has recorded this block since attribution landed, but + until now only ``vip trace --format json`` rendered it. The report is the + artifact a customer archives and hands to an auditor, so a result that is + attributable in the machine-readable output and anonymous in the PDF is + attributable in the wrong place. + + The whole block is omitted rather than shown as five ``NOT_RECORDED`` rows + when ``execution`` is absent: that is what ``--vip-no-attribution`` asked + for, and repeating "not recorded" five times reads as a broken run rather + than a deliberate one. Within a present block, an individual missing field + still follows the ``None`` contract above. + """ + if not execution: + return [] + git = execution.get("git") or {} + ci = execution.get("ci") or {} + performer = execution.get("performed_by") or {} + + commit = git.get("commit") + if commit and git.get("dirty"): + # An uncommitted tree means the evidence cannot be reproduced from the + # commit alone. That belongs next to the commit, not in a footnote. + commit = f"{commit} (uncommitted changes present)" + + identity = _performer_label(performer) + + return [ + ("Performed by", identity), + ("Run host", execution.get("hostname")), + ("Commit", commit), + ("Branch", git.get("branch")), + ("CI run", ci.get("run_url") or ci.get("run_id")), + ] + + def provenance_rows(data: ReportData) -> list[tuple[str, str | None]]: - """VIP version, run duration, interpreter/platform, mode, exit status (F9). + """VIP version, run duration, interpreter/platform, mode, exit status (F9), + then the execution attribution block when the run recorded one. A ``None`` value means the field is absent from this ``results.json`` and the backend must render ``NOT_RECORDED`` rather than a fabricated value. @@ -401,6 +472,7 @@ def provenance_rows(data: ReportData) -> list[tuple[str, str | None]]: ("Platform", data.platform), ("Mode", mode), ("Exit status", f"{data.exit_status} ({exit_label})"), + *_execution_rows(data.execution), ] @@ -417,3 +489,188 @@ def summary_status(data: ReportData) -> str: if data.unproven: return "UNPROVEN" return "PASS" + + +# --------------------------------------------------------------------------- +# Compliance traceability section +# --------------------------------------------------------------------------- + +# Coverage values reuse the outcome palette rather than introducing new colors, +# so `selftests/test_report_content.py`'s drift guard against styles.css keeps +# working unchanged. The mapping is the honest one: a gap reads like a failure, +# an executed covered control like a pass, a control with no automated test to +# point at like a skip, and a covered control whose scenarios never ran like +# na_version -- amber, because it is the state most likely to be misread as +# evidence. A covered control whose scenarios ran without passing uses the red +# of a gap, because both mean the control is not evidenced. +COVERAGE_STYLE_KEY = { + "covered": "passed", + "covered_not_executed": "na_version", + "covered_failed": "failed", + # Reuses the scenario-level "unproven" style, so a control that could not + # be verified is painted the same amber-red as the scenario rows beneath + # it and no new color enters the palette. + "covered_unproven": "unproven", + "gap": "failed", + "not_automatable": "skipped", +} + +COVERAGE_LABELS = { + "covered": "COVERED", + "covered_not_executed": "NOT RUN", + "covered_failed": "FAILED", + "covered_unproven": "UNPROVEN", + "gap": "GAP", + "not_automatable": "N/A (manual)", +} + + +@dataclass(frozen=True) +class ControlRow: + """One control's line in the rendered traceability section.""" + + control_id: str + description: str + reference: str + risk: str + """The customer's own risk rating, carried through uninterpreted. + + Rendered because FDA's Computer Software Assurance guidance asks the + record to carry the result of the risk-based analysis, and because a + reviewer triaging a matrix reads the high-risk gaps first. VIP does not + rank or validate the value -- ``risk = "banana"`` renders as "banana". + """ + coverage: str + """"covered" | "covered_not_executed" | "covered_failed" | + "covered_unproven" | "gap" | "not_automatable". + + Distinct from ``ControlEntry.coverage``, which has no + ``covered_not_executed`` or ``covered_failed`` value: the matrix keeps + coverage, execution and outcome as separate facts, and this flattens them + for display because a reader scanning one column must not read an + all-skipped or all-failing control as evidenced. + """ + scenarios: list[tuple[str, str, str]] + """``(scenario title, status, when it ran)`` for each matched scenario.""" + + +def display_coverage(entry) -> str: # noqa: ANN001 - vip.traceability.ControlEntry + """Flatten coverage, execution and outcome into the one value the report shows. + + Ordered by how loudly each fact demotes the control, because the matrix + keeps them as overlapping facts and one column can show only one. A + control that ran and failed is the strongest claim against it, so FAILED + wins over an unproven scenario on the same control. UNPROVEN then outranks + NOT RUN, which is the vaguer of the two: an all-unproven control is both, + and "VIP could not check this" tells the reader more than "nothing ran". + """ + if entry.coverage != "covered": + return entry.coverage + if entry.failing: + return "covered_failed" + if entry.has_unproven: + return "covered_unproven" + if not entry.executed: + return "covered_not_executed" + return entry.coverage + + +def control_rows(matrix) -> list[ControlRow]: # noqa: ANN001 - TraceabilityMatrix + """Every control in the matrix, ready for a backend to render as a table.""" + rows = [] + for entry in matrix.entries: + scenarios = [ + ( + m.scenario_title or m.nodeid, + m.status, + (m.started_at or "").replace("T", " ")[:19] or NOT_RECORDED, + ) + for m in entry.matches + ] + rows.append( + ControlRow( + control_id=entry.control.control_id, + description=entry.control.description, + reference=entry.control.reference or "", + risk=entry.control.risk or "", + coverage=display_coverage(entry), + scenarios=scenarios, + ) + ) + return rows + + +def traceability_summary_rows(matrix) -> list[tuple[str, str]]: # noqa: ANN001 + """Label/value counts for the section's summary table. + + Counts straight from ``matrix.entries`` rather than via :func:`control_rows`, + which also builds each control's full scenario list -- both backends already + call :func:`control_rows` once for the table itself, so building it again + here would be a second, unused pass over every control just to count them. + """ + entries = list(matrix.entries) + counts = Counter(display_coverage(entry) for entry in entries) + return [ + ("Controls", str(len(entries))), + ("Covered, executed and passing", str(counts.get("covered", 0))), + ("Covered, not executed", str(counts.get("covered_not_executed", 0))), + ("Covered, failing", str(counts.get("covered_failed", 0))), + ("Covered, not verified", str(counts.get("covered_unproven", 0))), + ("Gaps", str(counts.get("gap", 0))), + ("Not automatable", str(counts.get("not_automatable", 0))), + ] + + +# Shown under the section heading in both backends. The report is the artifact +# a customer archives, so the limits of the claim travel with it rather than +# living only in the docs they may never read. +TRACEABILITY_CAVEAT = ( + "Coverage records that a scenario is tagged for a control, not that the " + "scenario passed or even ran. A control shown as NOT RUN has a tagged " + "scenario that ran and skipped itself, because this deployment does not " + "expose what it probes or a version gate excluded it. A control shown as " + "a GAP may instead belong to a product this run did not test, since those " + "scenarios are excluded from the run and reach no result at all. " + "A control shown as FAILED has a tagged scenario that ran and did not pass, " + "so the control is not evidenced by this run. A control shown as UNPROVEN " + "has a tagged scenario VIP was asked to run and could not, which is not the " + "same as a check that found nothing to test. This " + "section evidences the controls chosen for automation. It is not an " + "attestation of regulatory compliance." +) + +# Both editions render this identically when the section cannot be built. A +# compliance report that drops the section without saying so is the one +# outcome a regulated reader cannot detect. +TRACEABILITY_RENDER_FAILURE = "Could not render the traceability section: {error}" + + +def traceability_warnings(matrix) -> list[str]: # noqa: ANN001 + """Lines naming controls that look covered but are not evidence. + + Three independent conditions, so three lines rather than one combined + sentence: a control can be counted as covered because nothing ran, because + what ran did not pass, or because VIP was asked to check it and could not, + and a reader needs to know which. The conditions overlap, so one control + can appear on more than one line. + """ + lines = [] + failing = matrix.covered_with_failure + if failing: + lines.append( + f"{pluralize(len(failing), 'control')} counted as covered but had a " + f"scenario that did not pass: {', '.join(failing)}." + ) + unexecuted = matrix.covered_without_execution + if unexecuted: + lines.append( + f"{pluralize(len(unexecuted), 'control')} counted as covered but had no " + f"scenario that ran: {', '.join(unexecuted)}." + ) + unproven = matrix.covered_with_unproven + if unproven: + lines.append( + f"{pluralize(len(unproven), 'control')} counted as covered but had a " + f"scenario VIP could not verify: {', '.join(unproven)}." + ) + return lines diff --git a/src/vip/report_html.py b/src/vip/report_html.py index ab91ec43..9ef240a7 100644 --- a/src/vip/report_html.py +++ b/src/vip/report_html.py @@ -24,12 +24,17 @@ from html import escape as _esc from vip.report_content import ( + COVERAGE_LABELS, + COVERAGE_STYLE_KEY, NOT_RECORDED, OUTCOME_LABELS, OUTCOME_ORDER, + TRACEABILITY_CAVEAT, + TRACEABILITY_RENDER_FAILURE, Badge, FeatureStepIndex, category_label, + control_rows, description_line, display_title, dominant_feature_description, @@ -43,6 +48,8 @@ secondary_badges_for, skip_reason_parts, summary_status, + traceability_summary_rows, + traceability_warnings, ) from vip.reporting import ReportData, TestResult @@ -411,3 +418,74 @@ def render_provenance_table(data: ReportData) -> str: for label, value in provenance_rows(data) ) return f"{body}
" + + +def render_traceability_error(error: object) -> str: + """The visible marker shown when the traceability section could not be built. + + ``error`` is the exception that stopped it -- a missing or malformed + control list, a checksum that failed verification. Its message carries + the ``VIP_CONTROLS`` path and control ids read straight out of a + customer-authored ``controls.toml``, which makes it the same fully + untrusted text ``render_traceability`` describes, and it goes through + ``_esc`` for the same reason. ``IPython.display.Markdown`` passes raw + HTML through in Quarto, so rendering this message as Markdown put + customer-controlled markup into a publicly published page; the Typst + edition already routes the identical value through ``_lit``. + + A dropped section is invisible to a regulated reader, so the failure is + a visible marker in both editions rather than a silent skip -- the + caller still emits the heading alongside this fragment. + """ + message = TRACEABILITY_RENDER_FAILURE.format(error=_esc(str(error))) + return f"

{message}

" + + +def render_traceability(matrix) -> str: # noqa: ANN001 - vip.traceability.TraceabilityMatrix + """The compliance traceability section: summary counts, then one row per control. + + Every customer-supplied value goes through ``_esc``. A control list is + authored outside VIP entirely, so its descriptions and references are the + first fully untrusted text this backend renders. + """ + summary = "".join( + f"{_esc(label)}{_esc(value)}" + for label, value in traceability_summary_rows(matrix) + ) + parts = [ + f"

{_esc(TRACEABILITY_CAVEAT)}

", + f"{summary}
", + ] + for warning in traceability_warnings(matrix): + parts.append(f"

{_esc(warning)}

") + + rows = [] + for row in control_rows(matrix): + style = outcome_style(COVERAGE_STYLE_KEY[row.coverage]) + badge = ( + f"{_esc(COVERAGE_LABELS[row.coverage])}" + ) + if row.scenarios: + evidence = "
".join( + f"{_esc(title)} — {_esc(status)} at {_esc(when)}" + for title, status, when in row.scenarios + ) + else: + evidence = "no tagged scenario" + # Reference and risk sit under the control id as sublines rather than + # as their own columns: the Typst edition's table is already at its + # width budget with four columns, and the two editions must match. + sublines = "".join( + f"
{_esc(text)}" + for text in (row.reference, f"risk: {row.risk}" if row.risk else "") + if text + ) + reference = sublines + rows.append( + f"{_esc(row.control_id)}{reference}" + f"{_esc(row.description)}{badge}{evidence}" + ) + header = "ControlDescriptionCoverageEvidence" + parts.append(f"{header}{''.join(rows)}
") + return "".join(parts) diff --git a/src/vip/report_typst.py b/src/vip/report_typst.py index fe048331..0741d5b4 100644 --- a/src/vip/report_typst.py +++ b/src/vip/report_typst.py @@ -38,14 +38,19 @@ import textwrap from vip.report_content import ( + COVERAGE_LABELS, + COVERAGE_STYLE_KEY, NOT_RECORDED, OUTCOME_LABELS, OUTCOME_ORDER, SECONDARY_BADGE_BACKGROUND, SECONDARY_BADGE_BORDER, + TRACEABILITY_CAVEAT, + TRACEABILITY_RENDER_FAILURE, Badge, FeatureStepIndex, category_label, + control_rows, description_line, display_title, dominant_feature_description, @@ -59,6 +64,8 @@ secondary_badges_for, skip_reason_parts, summary_status, + traceability_summary_rows, + traceability_warnings, ) from vip.reporting import ReportData, TestResult @@ -442,10 +449,20 @@ def render_provenance_table(data: ReportData) -> str: # --------------------------------------------------------------------------- -def _paragraph(value: str, *, italic: bool = False) -> str: +def _paragraph( + value: str, + *, + italic: bool = False, + fill: str | None = None, + weight: str | None = None, +) -> str: options = {"size": "10pt"} if italic: options["style"] = '"italic"' + if fill is not None: + options["fill"] = fill + if weight is not None: + options["weight"] = weight return _block(f"#{_text(value, **options)}", above="6pt", below="6pt") @@ -524,10 +541,116 @@ def render_details(data: ReportData, hints: dict[str, dict]) -> str: return "".join(parts) -def render_document(data: ReportData, hints: dict[str, dict]) -> str: - """The whole PDF body, preamble included, ready to emit as a ``{=typst}`` block.""" +def _stacked(parts: list[str]) -> str: + """Several Typst expressions as one table cell, separated by line breaks. + + A table cell must be a single expression, so multi-line content needs a + content block rather than concatenated ``#`` calls -- which is what + ``text(...)#block(...)`` produced, and Typst rejected. + """ + if len(parts) == 1: + return parts[0] + return "[" + "#linebreak()".join(f"#{part}" for part in parts) + "]" + + +def render_traceability(matrix) -> str: # noqa: ANN001 - TraceabilityMatrix + """The compliance traceability section as Typst markup. + + Every customer-supplied value passes through ``_lit`` (this module's + standing invariant). A control list is authored outside VIP, so a + description containing ``#``, ``*`` or ``$`` is live Typst markup + otherwise -- and these are the first fully customer-authored strings to + reach this backend. + """ + parts = [ + _paragraph(TRACEABILITY_CAVEAT, italic=True, fill='rgb("#6b7280")'), + _kv_table( + [ + (label, _text(value, size="9pt")) + for label, value in traceability_summary_rows(matrix) + ] + ), + ] + for warning in traceability_warnings(matrix): + parts.append(_paragraph(warning, fill='rgb("#dc2626")', weight='"bold"')) + + rows = [] + for row in control_rows(matrix): + style = outcome_style(COVERAGE_STYLE_KEY[row.coverage]) + control_parts = [_text(row.control_id, size="9pt")] + if row.reference: + control_parts.append(_text(row.reference, size="8pt", fill='rgb("#6b7280")')) + if row.risk: + control_parts.append(_text(f"risk: {row.risk}", size="8pt", fill='rgb("#6b7280")')) + if row.scenarios: + evidence = _stacked( + [_text(f"{t} - {s} at {w}", size="8.5pt") for t, s, w in row.scenarios] + ) + else: + evidence = _text("no tagged scenario", size="8.5pt", style='"italic"') + rows.append( + [ + _stacked(control_parts), + _text(row.description, size="9pt"), + # vip-chip, not vip-pill: the HTML edition renders dark text on + # a pale fill (outcome_badge_html), and vip-pill is a saturated + # fill with white text. The two editions must match. + _call( + "vip-chip", + _lit(COVERAGE_LABELS[row.coverage]), + _lit(style.color), + _lit(style.background), + ), + evidence, + ] + ) + parts.append( + _table( + "(auto, 1fr, auto, 1.2fr)", + ["Control", "Description", "Coverage", "Evidence"], + rows, + ) + ) + return "".join(parts) + + +def render_document(data: ReportData, hints: dict[str, dict], matrix=None, trace_error=None) -> str: # noqa: ANN001 + """The whole PDF body, preamble included, ready to emit as a ``{=typst}`` block. + + ``matrix`` is a ``vip.traceability.TraceabilityMatrix`` or ``None``. When + it is ``None`` -- every run without a control list, which is nearly all of + them -- the output is byte-identical to before the section existed. + + ``trace_error`` names why the section could not be built at all (a + missing/malformed control list, a results checksum mismatch). It renders + through ``_paragraph``, which routes the text through ``_text`` to + ``_lit``. An exception message is arbitrary text, and ``_lit`` escapes + the characters that could terminate the string literal early -- the + quote in particular, plus backslash -- so the message lands as inert + literal text inside the quotes rather than breaking out into live + markup. A dropped section is invisible to a regulated reader, so this + is a visible marker in both editions rather than a silent skip. + """ + # The error branch and the matrix branch emit the same heading, so a + # reader of the PDF sees the section start either way instead of the + # section silently disappearing. + if trace_error: + trace_section = _heading("Compliance Traceability", 2) + _paragraph( + TRACEABILITY_RENDER_FAILURE.format(error=trace_error) + ) + elif matrix is not None: + trace_section = _heading("Compliance Traceability", 2) + render_traceability(matrix) + else: + trace_section = "" + if data.total == 0: - return PREAMBLE + _paragraph("No results found. Run vip verify to generate results.") + empty = PREAMBLE + _paragraph("No results found. Run vip verify to generate results.") + # The HTML cell renders the section whenever a control list is set, + # including over an empty results file, where the matrix is all gaps + # and manual controls. Returning early here would drop it from the + # PDF alone and split the two editions on exactly the run a reader is + # most likely to misread. + return empty + trace_section parts = [ PREAMBLE, _heading("VIP Validation Report", 1), @@ -539,6 +662,7 @@ def render_document(data: ReportData, hints: dict[str, dict]) -> str: render_summary_table(data), _heading("Provenance", 2), render_provenance_table(data), + trace_section, _heading("Failures & Skips", 2), _paragraph( "Every check that did not pass, in full. Passing checks are counted above, " diff --git a/src/vip/reporting.py b/src/vip/reporting.py index 2f02ee3e..020d132b 100644 --- a/src/vip/reporting.py +++ b/src/vip/reporting.py @@ -5,6 +5,7 @@ import json import re import sys +import warnings import xml.etree.ElementTree as ET from dataclasses import dataclass, field from pathlib import Path @@ -17,6 +18,13 @@ VALID_FORMATS = frozenset({"json", "junit", "sarif"}) +# results.json schema version. Bump the minor for additive changes (a new +# field); bump the major for a removal, a rename, or a change in the meaning +# of an existing field. Consumers accept an unknown minor and refuse an +# unknown major. A file with no schema_version at all predates versioning +# and is treated as "pre-1.0". +RESULTS_SCHEMA_VERSION = "1.0" + @dataclass class TestResult: @@ -49,6 +57,12 @@ class TestResult: # that gets archived/published, and skip_reason already carries the part # a reader actually wants. skip_reason: str | None = None + # When this check began and ended, UTC ISO 8601, from pytest's report.start + # and report.stop. This is the call phase, so it excludes fixture setup + # (except for a setup-phase skip, where it is the setup start). None for a + # results.json written before these fields existed. + started_at: str | None = None + finished_at: str | None = None @property def category(self) -> str: @@ -134,11 +148,15 @@ class ReportData: # concrete-looking value (e.g. 0.0 or "unknown") so an older results.json # written before these fields existed loads as "not recorded" instead of # silently claiming a value that was never measured. + schema_version: str | None = None vip_version: str | None = None run_duration_seconds: float | None = None python_version: str | None = None platform: str | None = None basic_mode: bool | None = None + # Host / git / CI attribution; see vip.attribution. None when the run used + # --vip-no-attribution, or for a results.json predating the field. + execution: dict | None = None @property def total(self) -> int: @@ -199,7 +217,43 @@ def load_results(path: str | Path) -> ReportData: if not p.exists(): return ReportData() - raw = json.loads(p.read_text()) + raw = json.loads(p.read_text(encoding="utf-8")) + + # Guard the type before splitting. This loader's documented contract is to + # warn and carry on, never to raise: it runs inside the Quarto notebook + # cells (report/index.qmd, details.qmd, vip-report.qmd) where an exception + # renders as an unreadable traceback instead of a report. A hand-edited or + # third-party results.json carrying `"schema_version": 1.0` as a JSON + # number would otherwise raise AttributeError here. + schema_version = raw.get("schema_version") + if schema_version is not None and not isinstance(schema_version, str): + warnings.warn( + f"results.json schema_version is {schema_version!r}, not a string; " + "treating it as unversioned", + stacklevel=2, + ) + schema_version = None + if schema_version: + theirs = schema_version.split(".", 1)[0] + ours = RESULTS_SCHEMA_VERSION.split(".", 1)[0] + if theirs != ours: + # int, not string: "9" > "10" lexicographically, so a string + # compare misreports the direction once either major reaches two + # digits. A non-numeric major is possible in a hand-edited file, + # so fall back to the string compare rather than raising inside a + # loader whose contract is to warn and carry on. + try: + newer = int(theirs) > int(ours) + except ValueError: + newer = theirs > ours + direction = "newer than" if newer else "older than" + warnings.warn( + f"results.json schema version {schema_version} is {direction} this vip " + f"understands ({RESULTS_SCHEMA_VERSION}); some fields may be missing " + "or misinterpreted", + stacklevel=2, + ) + results = [ TestResult( nodeid=r["nodeid"], @@ -207,12 +261,22 @@ def load_results(path: str | Path) -> ReportData: duration=r.get("duration") or 0.0, longrepr=r.get("longrepr"), concise_error=r.get("concise_error"), - markers=r.get("markers", []), + # `or []` as well as the default: an explicit JSON null passes + # through .get() untouched and would reach every consumer as a + # None to iterate over. This loader is deliberately lenient -- + # it renders a report and must not raise inside a notebook cell. + # `vip trace` refuses the same input instead, via + # traceability.check_results_rows: silently reading a malformed + # row as untagged would drop its control tags and report a gap + # that does not exist. + markers=r.get("markers") or [], scenario_title=r.get("scenario_title"), feature_description=r.get("feature_description"), na_version=r.get("na_version", False), unproven=r.get("unproven", False), skip_reason=r.get("skip_reason"), + started_at=r.get("started_at"), + finished_at=r.get("finished_at"), ) for r in raw.get("results", []) ] @@ -235,11 +299,13 @@ def load_results(path: str | Path) -> ReportData: exit_status=raw.get("exit_status", 0), products=products, results=results, + schema_version=schema_version, vip_version=raw.get("vip_version"), run_duration_seconds=raw.get("run_duration_seconds"), python_version=raw.get("python_version"), platform=raw.get("platform"), basic_mode=raw.get("basic_mode"), + execution=raw.get("execution"), ) diff --git a/src/vip/traceability.py b/src/vip/traceability.py new file mode 100644 index 00000000..524d00ff --- /dev/null +++ b/src/vip/traceability.py @@ -0,0 +1,827 @@ +"""Traceability matrix: join compliance controls against tagged test results. + +VIP stays regulation-agnostic. The control list is supplied by whoever owns the +regulatory mapping; nothing here interprets ``reference``, ``risk`` or +``responsibility`` beyond carrying them through to the output. +""" + +from __future__ import annotations + +import csv +import hashlib +import io +import json +import sys +from dataclasses import dataclass, field +from pathlib import Path, PurePosixPath + +from vip.reporting import RESULTS_SCHEMA_VERSION, ReportData + +if sys.version_info >= (3, 11): + import tomllib +else: + import tomli as tomllib + +# Matrix output schema, versioned independently of results.json. +MATRIX_SCHEMA_VERSION = "1.1" + +VERIFICATION_VALUES = frozenset({"automated", "manual", "procedural"}) + + +class ControlListError(Exception): + """Raised when a controls.toml file is missing or malformed.""" + + +@dataclass +class ControlSpec: + control_id: str + description: str + reference: str | None = None + risk: str | None = None + # "automated" controls are expected to be covered by a tagged scenario. + # "manual" and "procedural" ones are reported as not verifiable by + # automated test rather than as coverage gaps. + verification: str = "automated" + responsibility: str | None = None + notes: str | None = None + # Free-form pass-through columns from [controls..extra]. A regulated + # customer's control list is their own regulatory mapping, and the fields + # above cannot anticipate it -- IQ/OQ/PQ phase, a SOP reference, a control + # owner. These reach the CSV and JSON exports verbatim and are not + # rendered in the report, whose table has no width for a variable number + # of columns. Nothing here is interpreted. + extra: dict[str, str] = field(default_factory=dict) + + +# Every key a [controls.] table may carry. Anything else is rejected +# rather than ignored -- see the error raised in load_controls for why. +_KNOWN_CONTROL_KEYS = frozenset( + {"description", "reference", "risk", "verification", "responsibility", "notes", "extra"} +) + + +def _load_extra(control_id: str, body: dict) -> dict[str, str]: + """Validate and return the [controls..extra] pass-through table. + + Values must be strings for the same reason the built-in pass-through + fields must: TOML has native date and integer types, and a bare + ``2024-01-01`` becomes a ``datetime.date`` that the CSV writer silently + stringifies while the JSON encoder refuses outright. + + A key that collides with a built-in column is rejected rather than + silently shadowed or duplicated: ``extra.risk`` would otherwise emit two + ``risk`` columns in the CSV, and a spreadsheet reader would take whichever + it saw last. + """ + raw = body.get("extra") + if raw is None: + return {} + if not isinstance(raw, dict): + raise ControlListError(f"[controls.{control_id}.extra] must be a table") + extra: dict[str, str] = {} + for key, value in raw.items(): + if key and key[0] in _FORMULA_PREFIXES: + # TOML allows a quoted key, so `"=HYPERLINK(...)" = "x"` is a legal + # control list. That key becomes a CSV *header* cell, which the + # formula neutralization below applies to as a backstop -- but a + # column named after a formula is never a legitimate regulatory + # field, and rejecting it here keeps the CSV header and the JSON + # key identical rather than making one of them sprout an + # apostrophe. + raise ControlListError( + f"[controls.{control_id}.extra] has key {key!r}, which starts with a " + "character a spreadsheet reads as a formula. Rename the field." + ) + if key in CSV_COLUMNS: + raise ControlListError( + f"[controls.{control_id}.extra] has key {key!r}, which is already a " + "column in the exported matrix. Choose another name." + ) + if not isinstance(value, str): + raise ControlListError( + f"[controls.{control_id}.extra] has {key}={value!r} " + f"({type(value).__name__}); expected a string. Quote it if it is a " + "date or a number." + ) + extra[key] = value + return extra + + +def load_controls(path: str | Path) -> dict[str, ControlSpec]: + """Load a controls.toml file into ControlSpec objects keyed by control id.""" + p = Path(path) + if not p.is_file(): + if p.exists(): + raise ControlListError(f"control list {p} is not a file") + raise ControlListError(f"control list not found: {p}") + try: + raw = tomllib.loads(p.read_text(encoding="utf-8")) + except (tomllib.TOMLDecodeError, OSError, UnicodeDecodeError) as exc: + raise ControlListError(f"could not read control list {p}: {exc}") from exc + + table = raw.get("controls") + if not isinstance(table, dict): + raise ControlListError(f"{p} has no [controls] table") + if not table: + raise ControlListError(f"{p} has an empty [controls] table") + + controls: dict[str, ControlSpec] = {} + for control_id, body in table.items(): + if not isinstance(body, dict): + raise ControlListError(f"[controls.{control_id}] must be a table") + description = body.get("description") + if not isinstance(description, str): + if description is None: + raise ControlListError(f"[controls.{control_id}] is missing a description") + raise ControlListError( + f"[controls.{control_id}] has description={description!r}; expected a string" + ) + if not description.strip(): + raise ControlListError(f"[controls.{control_id}] has an empty description") + verification = body.get("verification", "automated") + if not isinstance(verification, str): + raise ControlListError( + f"[controls.{control_id}] has verification={verification!r}; expected a string" + ) + if verification not in VERIFICATION_VALUES: + raise ControlListError( + f"[controls.{control_id}] has verification={verification!r};" + f" expected one of {sorted(VERIFICATION_VALUES)}" + ) + # The pass-through fields are carried verbatim into both renderers, so + # they must be validated here rather than at render time. TOML has + # native date and integer types: `reference = 2024-01-01` parses to a + # datetime.date, which the CSV writer silently stringifies while the + # JSON encoder refuses outright. Rejecting it once, here, is what keeps + # the two formats agreeing on what counts as a valid control list. + optional = {} + for field_name in ("reference", "risk", "responsibility", "notes"): + value = body.get(field_name) + if value is not None and not isinstance(value, str): + raise ControlListError( + f"[controls.{control_id}] has {field_name}={value!r} " + f"({type(value).__name__}); expected a string. Quote it if it is a " + "date or a number." + ) + optional[field_name] = value + extra = _load_extra(control_id, body) + unknown = set(body) - _KNOWN_CONTROL_KEYS + if unknown: + # Silently dropping these was worse than either alternative: a + # customer who wrote `phase = "OQ"` got no error and no column, and + # a typo like `referance` vanished the same way -- from a file + # whose whole job is to be the regulatory mapping of record. + raise ControlListError( + f"[controls.{control_id}] has unknown " + f"{'keys' if len(unknown) > 1 else 'key'} {', '.join(sorted(unknown))}. " + f"Known keys are {', '.join(sorted(_KNOWN_CONTROL_KEYS))}. " + f"Put your own fields in [controls.{control_id}.extra]; they are " + "carried into the CSV and JSON exports untouched." + ) + controls[control_id] = ControlSpec( + extra=extra, + control_id=control_id, + description=description, + verification=verification, + **optional, + ) + return controls + + +@dataclass +class ControlMatch: + """One scenario that carries a control's tag.""" + + nodeid: str + scenario_title: str | None + status: str + started_at: str | None + finished_at: str | None + detail: str | None + + +# Statuses that mean the check did not run. "na_version" is a +# version-gated non-execution, so it counts here too: treating it as +# executed would let a control gated off on an older product read as +# evidenced by a scenario that never ran a single assertion. "unproven" is +# the same shape -- ``vip.attest.unproven`` skips a check VIP was asked to +# run and could not, so no assertion ran there either. Its loudness lives in +# the run's exit code 6 and in the per-scenario status column, not in +# ``executed``, which answers only whether a result was produced. +NON_EXECUTING_STATUSES = frozenset({"skipped", "na_version", "unproven"}) + + +@dataclass +class ControlEntry: + control: ControlSpec + matches: list[ControlMatch] = field(default_factory=list) + # "covered" | "gap" | "not_automatable" + coverage: str = "gap" + + @property + def executed(self) -> bool: + """Whether any tagged scenario actually ran. + + Coverage says a scenario is tagged for this control. This says one of + them produced a result. The two come apart when a scenario runs and + skips itself (an absent endpoint, no data to inspect, a version gate), + because the result row it writes still lists its control tag, so a + matrix reads covered for a check that verified nothing. An + unconfigured product is the opposite case: + ``plugin.pytest_collection_modifyitems`` deselects those scenarios, so + they never reach ``results.json`` and a control tagged *only* by them + reports as a gap. + """ + return any(m.status not in NON_EXECUTING_STATUSES for m in self.matches) + + @property + def failing(self) -> bool: + """Whether any tagged scenario produced a result that was not a pass. + + The third fact about a control, after "a scenario is tagged" + (``coverage``) and "a scenario ran" (``executed``). Without it a + control whose only scenario failed reports as covered and executed, + which is true and reads as evidence. + + Defined by exclusion rather than by enumerating failure statuses: + ``error`` is a reachable outcome alongside ``failed``, and an + enumerated list would let an errored control read as evidenced. + Non-executing statuses are excluded via ``NON_EXECUTING_STATUSES``, so + a skip alongside a pass never counts against a control. That includes + an ``unproven`` skip, which is the contentious member: it is a check + VIP could not run, so it belongs with the non-executing statuses + rather than here, where it would report a control that never ran as + one that ran and failed. + """ + return any( + m.status not in NON_EXECUTING_STATUSES and m.status != "passed" for m in self.matches + ) + + @property + def has_unproven(self) -> bool: + """Whether any tagged scenario was a check VIP could not run. + + The fourth fact, and the one the other three cannot state. An + ``unproven`` skip is non-executing, so ``executed`` and ``failing`` + are both false for it, and a control with one passing scenario beside + one unproven scenario reads as fully evidenced from those two alone. + + Narrower than ``not executed``: a plain skip says there was nothing to + check here, and ``na_version`` says a version gate excluded it. This + says VIP was asked to check the control and could not, which is the + distinction ``vip.attest.unproven`` exists to record. + """ + return any(m.status == "unproven" for m in self.matches) + + +@dataclass +class TraceabilityMatrix: + entries: list[ControlEntry] = field(default_factory=list) + unrecognized_tags: list[str] = field(default_factory=list) + provenance: dict = field(default_factory=dict) + schema_version: str = MATRIX_SCHEMA_VERSION + + @property + def gap_count(self) -> int: + return sum(1 for e in self.entries if e.coverage == "gap") + + @property + def covered_count(self) -> int: + return sum(1 for e in self.entries if e.coverage == "covered") + + @property + def covered_without_execution(self) -> list[str]: + """Control ids that are covered but whose every scenario was skipped. + + The headline summary reports these as covered with zero gaps, which + is true and, on its own, badly misleading: nothing was verified. The + caller surfaces this so a reader cannot take a green matrix at face + value when every scenario behind it ran and skipped itself. + """ + return [ + e.control.control_id for e in self.entries if e.coverage == "covered" and not e.executed + ] + + @property + def covered_with_failure(self) -> list[str]: + """Control ids that are covered but whose scenarios did not all pass. + + The mirror of ``covered_without_execution``. That one catches a matrix + that is green because nothing ran; this one catches a matrix that is + green because a run that did happen was not a success. Any failing + scenario qualifies the control, not only an all-failed one: a green + badge above a visible failing scenario row is the misreading this + exists to prevent. + """ + return [e.control.control_id for e in self.entries if e.coverage == "covered" and e.failing] + + @property + def covered_with_unproven(self) -> list[str]: + """Control ids that are covered but have a scenario VIP could not run. + + The third of these lists, and the only one that catches a control + whose scenarios ran, passed, and still left part of the control + unchecked. Deliberately not disjoint from the other two: an + unproven-only control is both not executed and not verifiable, and it + appears in both lists, because each statement is true and a reader + needs both. + """ + return [ + e.control.control_id for e in self.entries if e.coverage == "covered" and e.has_unproven + ] + + +def _provenance( + data: ReportData, + results_sha256: str | None = None, + results_sha256_sidecar_verified: bool | None = None, +) -> dict: + products = { + p.name: {"url": p.url, "version": p.version, "configured": p.configured} + for p in data.products + } + return { + "generated_at": data.generated_at, + "deployment_name": data.deployment_name, + "vip_version": data.vip_version, + "results_schema_version": data.schema_version, + "exit_status": data.exit_status, + # basic_mode is surfaced deliberately: a matrix built from a + # `vip verify --basic` run omits every @slow scenario and would + # otherwise assert coverage that was never exercised. + "basic_mode": data.basic_mode, + "execution": data.execution, + # Describes the VIP runner, not the system under test. The products + # table below identifies the system under test. + "runner_python_version": data.python_version, + "runner_platform": data.platform, + "products": products, + # The digest `vip trace` computed over the results.json bytes it + # actually read -- the in-band link between this matrix and the exact + # evidence file it was derived from. None when the matrix was built + # without a file on disk (e.g. directly from a ReportData in tests). + "results_sha256": results_sha256, + # Tri-state, not a plain bool: True means a `.sha256` sidecar was + # present and matched (see verify_results_checksum); None means no + # sidecar existed to check, which is a legal, expected condition for + # results files written before the sidecar existed -- not a failure. + # False would mean the sidecar disagreed, but that raises + # ResultsIntegrityError before a matrix is ever built, so it can + # never actually appear here. + "results_sha256_sidecar_verified": results_sha256_sidecar_verified, + } + + +def build_traceability_matrix( + data: ReportData, + controls: dict[str, ControlSpec], + tag_prefix: str = "control-", + results_sha256: str | None = None, + results_sha256_sidecar_verified: bool | None = None, +) -> TraceabilityMatrix: + """Join control definitions against tagged test results. + + Sorted deterministically -- by control id, then by nodeid within a control + -- so the same results.json and control list always produce byte-identical + output for a downstream renderer to diff. + + ``results_sha256`` / ``results_sha256_sidecar_verified`` carry the + tamper-evidence digest ``vip trace`` already computed over the results + file (see ``verify_results_checksum``) into the matrix provenance. Both + default to ``None`` so a matrix built directly from a ``ReportData`` -- + with no results file on disk, as most tests do -- still works. + """ + by_tag: dict[str, list[ControlMatch]] = {} + seen_tags: set[str] = set() + for result in data.results: + for marker in result.markers: + if not marker.startswith(tag_prefix): + continue + seen_tags.add(marker) + by_tag.setdefault(marker, []).append( + ControlMatch( + nodeid=result.nodeid, + scenario_title=result.scenario_title, + status=result.status, + started_at=result.started_at, + finished_at=result.finished_at, + detail=result.concise_error or result.skip_reason, + ) + ) + + entries: list[ControlEntry] = [] + for control_id in sorted(controls): + control = controls[control_id] + matches = sorted(by_tag.get(f"{tag_prefix}{control_id}", []), key=lambda m: m.nodeid) + if matches: + coverage = "covered" + elif control.verification != "automated": + coverage = "not_automatable" + else: + coverage = "gap" + entries.append(ControlEntry(control=control, matches=matches, coverage=coverage)) + + known = {f"{tag_prefix}{cid}" for cid in controls} + return TraceabilityMatrix( + entries=entries, + unrecognized_tags=sorted(seen_tags - known), + provenance=_provenance(data, results_sha256, results_sha256_sidecar_verified), + ) + + +CSV_COLUMNS = [ + "control_id", + "description", + "reference", + "risk", + "verification", + "responsibility", + "coverage", + "scenario", + "nodeid", + "status", + "started_at", + "finished_at", + "detail", + "notes", + # Provenance, repeated on every row. CSV is the default format and the one + # that gets archived into a spreadsheet, so without these the artifact a + # reviewer actually holds has no link back to the results file it was + # derived from. results_sha256 is that link. The full block -- products + # under test, host, CI run -- is JSON only, because it does not flatten + # into columns. + "generated_at", + "vip_version", + "results_sha256", + "exit_status", +] + + +# Leading characters a spreadsheet evaluates rather than displays. Named once +# because both the row values and the (now customer-supplied) header cells have +# to be checked against the same set. +_FORMULA_PREFIXES = ("=", "+", "-", "@", "\t", "\r", "\n") + + +def _neutralize_formula(value: str) -> str: + """Prefix leading formula characters with apostrophe to prevent Excel evaluation. + + When a CSV is opened in Excel, a cell starting with =, +, -, or @ is evaluated + as a formula, which is a security and integrity risk for compliance artifacts. + The apostrophe forces literal interpretation. This alters the value as seen by + non-Excel CSV readers (they will see the leading apostrophe); JSON is the format + to use when exact fidelity matters. + + A leading \\t, \\r, or \\n is included in the dangerous prefix set too: a + spreadsheet importer commonly strips or normalizes leading control + characters before evaluating the cell, so "\\t=SUM(1,2)" would otherwise + reach the sheet as an unescaped formula (OWASP's CSV-injection guidance + treats these control characters the same as the leading =/+/-/@). + """ + if value and value[0] in _FORMULA_PREFIXES: + return "'" + value + return value + + +def _provenance_columns(matrix: TraceabilityMatrix) -> dict: + prov = matrix.provenance + return { + "generated_at": prov.get("generated_at") or "", + "vip_version": prov.get("vip_version") or "", + "results_sha256": prov.get("results_sha256") or "", + "exit_status": "" if prov.get("exit_status") is None else str(prov["exit_status"]), + } + + +def _control_columns(entry: ControlEntry) -> dict: + c = entry.control + return { + "control_id": c.control_id, + "description": c.description, + "reference": c.reference or "", + "risk": c.risk or "", + "verification": c.verification, + "responsibility": c.responsibility or "", + "coverage": entry.coverage, + "notes": c.notes or "", + } + + +def render_csv(matrix: TraceabilityMatrix) -> str: + """Render the matrix as CSV: one row per control/scenario pair. + + A control with no matching scenario still gets a row, with the scenario + columns empty, so a coverage gap is visible rather than absent. + """ + buf = io.StringIO() + # Extra columns append after the fixed set rather than slotting in beside + # the control metadata they belong with: CSV_COLUMNS then stays an + # identical leading prefix across every customer's export, whatever their + # own control list carries. Union across all controls, sorted, so a + # control that omits a key gets an empty cell rather than a ragged row. + extra_columns = sorted({k for e in matrix.entries for k in e.control.extra}) + fieldnames = [*CSV_COLUMNS, *extra_columns] + writer = csv.DictWriter(buf, fieldnames=fieldnames, lineterminator="\n") + # Not writer.writeheader(): the extra columns are customer-supplied, so a + # header cell is user-controlled data now and needs the same neutralization + # every row value gets. load_controls rejects such a key outright, so this + # only fires for a ControlSpec built in code rather than loaded from TOML. + writer.writerow({name: _neutralize_formula(name) for name in fieldnames}) + prov = _provenance_columns(matrix) + blank_extra = dict.fromkeys(extra_columns, "") + + def _neutralize_row(row: dict) -> dict: + """Apply formula neutralization to all string values in the row.""" + return {k: _neutralize_formula(v) if isinstance(v, str) else v for k, v in row.items()} + + for entry in matrix.entries: + base = {**_control_columns(entry), **blank_extra, **entry.control.extra} + if not entry.matches: + row = { + **base, + **prov, + "scenario": "", + "nodeid": "", + "status": "", + "started_at": "", + "finished_at": "", + "detail": "", + } + writer.writerow(_neutralize_row(row)) + continue + for match in entry.matches: + row = { + **base, + **prov, + "scenario": match.scenario_title or "", + "nodeid": match.nodeid, + "status": match.status, + "started_at": match.started_at or "", + "finished_at": match.finished_at or "", + "detail": match.detail or "", + } + writer.writerow(_neutralize_row(row)) + return buf.getvalue() + + +def render_json(matrix: TraceabilityMatrix) -> str: + """Render the matrix as JSON, carrying the full provenance block.""" + payload = { + "schema_version": matrix.schema_version, + "provenance": matrix.provenance, + "summary": { + "total": len(matrix.entries), + "covered": matrix.covered_count, + "gaps": matrix.gap_count, + "not_automatable": sum(1 for e in matrix.entries if e.coverage == "not_automatable"), + # Coverage counts controls that have a tagged scenario. This + # counts controls whose scenario actually ran. The two diverge + # when a tagged scenario runs and skips itself, which is exactly + # when a green matrix means least. + "covered_and_executed": sum( + 1 for e in matrix.entries if e.coverage == "covered" and e.executed + ), + "covered_not_executed": len(matrix.covered_without_execution), + # Covered and executed, but not a success. The third way a green + # matrix can mislead, after "nothing is tagged" and "nothing ran". + "covered_failed": len(matrix.covered_with_failure), + # The fourth: a control carrying a check VIP was asked to run and + # could not. Overlaps the two counts above rather than partitioning + # them -- an unproven-only control is counted here and in + # covered_not_executed -- so these five do not sum to `total`. + "covered_unproven": len(matrix.covered_with_unproven), + }, + "covered_without_execution": matrix.covered_without_execution, + "covered_with_unproven": matrix.covered_with_unproven, + "unrecognized_tags": matrix.unrecognized_tags, + "controls": [ + { + **_control_columns(entry), + "extra": entry.control.extra, + "matches": [ + { + "nodeid": m.nodeid, + "scenario": m.scenario_title, + "status": m.status, + "started_at": m.started_at, + "finished_at": m.finished_at, + "detail": m.detail, + } + for m in entry.matches + ], + } + for entry in matrix.entries + ], + } + return json.dumps(payload, indent=2, sort_keys=False, ensure_ascii=False) + "\n" + + +class ResultsIntegrityError(Exception): + """Raised when a results file fails checksum or schema validation.""" + + +def verify_results_checksum(path: str | Path) -> tuple[str, bool]: + """Verify a results file against its .sha256 sidecar. + + Returns ``(digest, sidecar_present)`` -- the sha256 of the file, and + whether a `.sha256` sidecar was found to check it against. Raises if a + sidecar exists and disagrees. A missing sidecar is not an error: results + files written before the sidecar existed have none, and callers use + ``sidecar_present`` to distinguish "verified" from "nothing to verify" + rather than treating both as the same success. + + Also raises when the entries selected for this file disagree with each + other: a sidecar that records two different digests under the same name + cannot attest to anything, so accepting the file because one of them + happens to match would be a false attestation. + + This is tamper-evidence within a trusted pipeline, not tamper-proofing -- + anyone who can edit the results file can regenerate the sidecar. It catches + corruption, truncated uploads and casual editing. + """ + p = Path(path) + digest = hashlib.sha256(p.read_bytes()).hexdigest() + sidecar = p.with_name(f"{p.name}.sha256") + if not sidecar.is_file(): + return digest, False + try: + text = sidecar.read_text(encoding="utf-8-sig") + except (OSError, UnicodeDecodeError) as exc: + raise ResultsIntegrityError(f"could not read checksum sidecar {sidecar}: {exc}") from exc + + entries = _parse_sidecar(text) + if not entries: + # An empty or whitespace-only sidecar is itself the truncated-upload + # case this function advertises catching. Returning True here would put + # `results_sha256_sidecar_verified: true` in the provenance block while + # nothing had actually been compared -- a false attestation in the one + # field whose entire purpose is attesting that the check happened. + raise ResultsIntegrityError( + f"checksum sidecar for {p} is empty; expected a sha256 digest. " + "Delete it to proceed without verification, or regenerate it." + ) + + # Match on the recorded filename rather than taking the first line. A + # sidecar may legitimately cover several files (`shasum -a 256 a b > s`), + # and without this the first line's digest is compared against a file it + # does not describe -- a mismatch on a good file, or, when the digests + # happen to agree, `results_sha256_sidecar_verified: true` attesting to a + # comparison against some other file entirely. + named = [d for d, name in entries if name == p.name] + if not named: + # Fall back to comparing basenames. `shasum -a 256 report/results.json` + # run from a directory above the file records the path it was given + # rather than the bare name, and refusing that sidecar reads to an + # operator as a tamper alarm on a file nobody touched. Exact match + # stays the primary key, so a multi-file sidecar that already names + # this file exactly never reaches here and keeps its strict behaviour. + named = [d for d, name in entries if name and sidecar_basename(name) == p.name] + if not named: + if len(entries) == 1 and entries[0][1] is None: + # A bare digest with no filename: nothing to disagree with. + named = [entries[0][0]] + else: + recorded_names = ", ".join(sorted({n or "" for _, n in entries})) + raise ResultsIntegrityError( + f"checksum sidecar {sidecar} does not record an entry for {p.name}; " + f"it names {recorded_names}. Regenerate it, or delete it to proceed " + "without verification." + ) + + # A sidecar must never say two different things about one file. Several + # selected entries carrying *different* digests means one of them is + # describing some other artifact, and accepting the file because *any* of + # them agrees turns the sidecar into an attestation about a file it does + # not describe -- the exact false attestation the recorded-name match + # above exists to prevent, reintroduced through the basename fallback (or + # through a rehomed sidecar that ended up with two same-named lines). + # Several entries agreeing on one digest is not ambiguous and still + # verifies: `shasum` run twice, or a rehomed line beside its original, + # says the same thing twice. + distinct = {d.lower() for d in named} + if len(distinct) > 1: + listed = ", ".join(sorted(distinct)) + raise ResultsIntegrityError( + f"checksum sidecar {sidecar} records {len(distinct)} different digests for " + f"{p.name} ({listed}); it cannot say which one describes this file. " + "Regenerate it, or delete it to proceed without verification." + ) + + # Case-insensitive: hex is hex. PowerShell's Get-FileHash and 7-Zip emit + # uppercase, and rejecting those as a mismatch reads to an operator as + # "this evidence file was tampered with" over nothing but letter case. + if not any(d.lower() == digest for d in named): + raise ResultsIntegrityError( + f"checksum mismatch for {p}: sidecar records {named[0]}, file hashes to {digest}" + ) + return digest, True + + +def sidecar_basename(name: str) -> str: + """The bare filename from a sidecar's recorded name. + + Backslashes are normalized first: shasum under Git Bash or MSYS can record + a Windows-style path, and PurePosixPath would treat the whole thing as one + filename. + """ + return PurePosixPath(name.replace("\\", "/")).name + + +def _parse_sidecar(text: str) -> list[tuple[str, str | None]]: + """Parse shasum-format lines into ``(digest, filename or None)`` pairs. + + One entry per line, not a flat ``.split()`` over the whole file: a + multi-file sidecar flattened that way puts the second file's digest where + a filename belongs and compares the wrong pair. + """ + entries: list[tuple[str, str | None]] = [] + for line in text.splitlines(): + parts = line.split(None, 1) + if not parts: + continue + digest = parts[0] + name = parts[1].strip() if len(parts) > 1 else None + # shasum marks binary-mode entries with a leading '*' on the filename. + if name: + name = name.lstrip("*") + entries.append((digest, name or None)) + return entries + + +def read_results_schema_version(path: str | Path) -> str | None: + """Read the top-level ``schema_version`` out of a results.json file. + + This is deliberately independent of ``load_results``: it must be callable + (and must raise cleanly) BEFORE ``load_results`` ever touches the file, so + an incompatible or structurally malformed results file is rejected by the + schema gate instead of crashing inside ``load_results``' own field + indexing (`r["nodeid"]`, `r["outcome"]`, ...). + """ + p = Path(path) + try: + raw = json.loads(p.read_text(encoding="utf-8")) + except (json.JSONDecodeError, OSError, UnicodeDecodeError) as exc: + raise ResultsIntegrityError(f"could not read results file {p}: {exc}") from exc + if not isinstance(raw, dict): + raise ResultsIntegrityError(f"could not read results file {p}: not a JSON object") + schema_version = raw.get("schema_version") + if schema_version is None: + return None + if not isinstance(schema_version, str): + raise ResultsIntegrityError( + f"could not read results file {p}: schema_version={schema_version!r} is not a string" + ) + return schema_version + + +def check_results_rows(path: str | Path) -> None: + """Refuse a results file whose rows are structurally wrong. + + ``reporting.load_results`` normalizes a malformed ``markers`` value to an + empty list, because it renders the HTML and PDF reports from inside Quarto + notebook cells where raising is an unreadable traceback. That leniency is + wrong for a traceability matrix: a row whose ``markers`` is a JSON null or + a string reads as carrying no control tags, so the control it was tagged + for is reported as a GAP that does not exist -- the matrix asserting the + suite is missing a check it actually has. Refuse the input instead. + """ + p = Path(path) + try: + raw = json.loads(p.read_text(encoding="utf-8")) + except (json.JSONDecodeError, OSError, UnicodeDecodeError) as exc: + raise ResultsIntegrityError(f"could not read results file {p}: {exc}") from exc + if not isinstance(raw, dict): + raise ResultsIntegrityError(f"could not read results file {p}: not a JSON object") + + results = raw.get("results", []) + if not isinstance(results, list): + raise ResultsIntegrityError(f"{p} has results={type(results).__name__}; expected a list") + for index, row in enumerate(results): + if not isinstance(row, dict): + raise ResultsIntegrityError( + f"{p} results[{index}] is a {type(row).__name__}; expected an object" + ) + markers = row.get("markers", []) + if not isinstance(markers, list): + nodeid = row.get("nodeid", f"results[{index}]") + raise ResultsIntegrityError( + f"{p}: {nodeid} has markers={markers!r} ({type(markers).__name__}); " + "expected a list. Control tags cannot be read from this file, so the " + "matrix would report gaps that may not exist." + ) + + +def check_results_schema(schema_version: str | None) -> None: + """Refuse an unknown major schema version; accept an unknown minor. + + A file with no schema_version predates versioning and is accepted. + """ + if not schema_version: + return + theirs = schema_version.split(".", 1)[0] + ours = RESULTS_SCHEMA_VERSION.split(".", 1)[0] + if theirs != ours: + raise ResultsIntegrityError( + f"results.json schema version {schema_version} is not supported by this " + f"vip (understands {RESULTS_SCHEMA_VERSION}); upgrade vip or regenerate the results" + ) diff --git a/website/src/pages/getting-started.astro b/website/src/pages/getting-started.astro index b61a6be9..5c666882 100644 --- a/website/src/pages/getting-started.astro +++ b/website/src/pages/getting-started.astro @@ -25,6 +25,7 @@ import Footer from "../components/Footer.astro";
  • Configuration file
  • Running behind a proxy
  • Generating reports
    • +
  • Compliance traceability
  • Extending VIP
  • Uninstalling
    • @@ -209,6 +210,10 @@ vip verify --connect-url https://connect.example.com --no-interactive-authVIP_TEST_TOTP_SECRET Base32 TOTP seed for the test service account. Optional; only used by --headless-auth when the IdP issues an MFA challenge. + + VIP_PERFORMED_BY + Names the person the run is attributable to, recorded in the results file and shown in both report editions. Optional; VIP otherwise reads the CI system's actor, then the local login. Not a secret. + @@ -466,6 +471,94 @@ quarto publish connect --server https://connect.example.com

    +
    +

    Compliance traceability

    +

    + In a regulated environment the question is rarely “did the tests + pass” but “which control does each test evidence, and when did + it run”. A scenario can declare the control it verifies with an + @control-<slug> Gherkin tag: +

    + 
    @connect
+Feature: 21 CFR Part 11 flavoured controls
+
+  @control-audit-trail-publish
+  Scenario: Publishing content is recorded with an actor and a timestamp
+    Given Connect is accessible at the configured URL
+    When I list recent audit log entries
+    Then each entry records an actor and a timestamp
    +

    + Name the controls in a controls.toml of your own — VIP + stays regulation-agnostic and passes your reference, risk and + responsibility fields through untouched. Then join the two: +

    + 
    vip verify --connect-url https://connect.example.com --extensions ./my-tests
+vip trace --results report/results.json --controls ./my-tests/controls.toml
    +

    + The result is a matrix in CSV (for a spreadsheet or a downstream + qualification-protocol generator) or JSON, listing each control, the + scenarios that verify it, their outcome, and the timestamp each check + ran. Every control gets one of three coverage values, and the third + exists because collapsing it into the second would overstate your + coverage: +

    +
      +
    • covered — at least one scenario is tagged for it
      • +
    • + gap — no scenario in this run has the tag, and + verification is automated (the default) +
      • +
    • + not_automatable — you marked it + manual or procedural, so no automated test is + expected. This is not a gap, and a matrix that showed it as one would + understate how complete your coverage is +
      • +
    +

    + Coverage is separate from outcome: a control is covered when a scenario + is tagged for it, and that scenario’s pass or fail is reported + alongside. A scenario can run and skip itself, when the deployment does + not expose what it probes or a version gate excludes it, and it covers + its control all the same. vip trace warns about those and + the JSON summary counts them separately, so read gaps: 0 + together with covered_not_executed. +

    +

    + Read a gap the same way. Scenarios belonging to a product you did not + configure are excluded from the run entirely rather than skipped, so + they reach no result and a control tagged only by them reports as a gap. + That understates your coverage rather than overstating it, but the + conclusion — that your suite has no check for the control — + is still wrong. Check which products the run tested first. +

    +

    + Both formats include a SHA-256 of the results file the matrix was derived + from, so the archived artifact points back at its evidence. CSV repeats + that digest with the generation time, VIP version and exit status on + every row. The full provenance block, covering the products and + versions under test plus the host and CI run, is in the JSON export, + which does not have to flatten into columns. +

    +

    + A worked example ships with VIP: + examples/21CFR_part11_validation + pairs a controls.toml with the tagged scenarios that satisfy + it. Generate your own copy with + vip scaffold --template 21cfr-part11-validation --output ./my-tests, + then read its README.md: it is a template to edit, and it is + explicit about what a matrix like this can and cannot claim. Posit Team + does not implement electronic signatures, so a green matrix evidences the + controls you chose to automate — it is not an attestation of + 21 CFR Part 11 compliance. The + reporting guide + covers the full field inventory. +

    +
    +

    Extending VIP

    @@ -475,7 +568,7 @@ quarto publish connect --server https://connect.example.com 

    vip scaffold --list
 vip scaffold --template minimal --output ./my-tests
    -

    Two templates ship with VIP:

    +

    Three templates ship with VIP:

    • minimal — a single-scenario health check against a @@ -486,6 +579,12 @@ vip scaffold --template minimal --output ./my-tests plus package installability across Connect and Workbench, the pattern regulated environments tend to need. This is the default.
      • +
    • + 21cfr-part11-validation — scenarios tagged with the + compliance controls they verify, plus a controls.toml to + feed the traceability matrix. + A starting point to edit, not a certified test set. +

    Every scaffolded directory also gets an AGENTS.md describing diff --git a/website/src/pages/report.astro b/website/src/pages/report.astro index c48c279a..745866b7 100644 --- a/website/src/pages/report.astro +++ b/website/src/pages/report.astro @@ -21,6 +21,16 @@ const base = import.meta.env.BASE_URL; a glance), not the full suite. Run vip verify against your own deployment to see every scenario that applies to your configuration.

    +

    + This same report format serves regulated environments too: attach a control + list with vip report --controls and the report gains a + traceability matrix joining each control to the scenarios that verify it. See + the worked + 21 CFR Part 11 example. +

    Generate your own: run vip verify to save test results to report/results.json, then vip report to render the report.