test(suites): port the conformance suites onto testkit and run conformance-integration on the runner

.agents/skills/watch-github-actions/SKILL.md

@@ -125,7 +125,7 @@ gh run list --json databaseId,status,headBranch,url --jq '.[] | {id: .databaseId
 
 ## View Job Logs
 
-For tmachine Kubernetes conformance failures, inspect the `tmachine-diagnostics-*` artifact. The K3s report includes Agent Sandbox controller state, leader leases, events, and current and previous controller logs. The conformance playbook checks reconciliation after guest boot with a temporary scheduling-gated Sandbox probe before running tests. Distinguish a failed controller probe from a failed conformance scenario; gateway connectivity and controller Deployment availability do not prove reconciliation works.
+For tmachine Kubernetes conformance failures, inspect the `e2e-results-conformance-sandbox-ubuntu-k3s-*` artifact. Its `diagnostics/` holds the K3s report with Agent Sandbox controller state, leader leases, events, and current and previous controller logs. The k3s installers check reconciliation after guest boot with a temporary scheduling-gated Sandbox probe, so a failed probe fails provisioning before any test runs. Distinguish a failed controller probe from a failed conformance scenario; gateway connectivity and controller Deployment availability do not prove reconciliation works.
 
 For the `e2e-suites` lane (`.github/workflows/e2e-suites.yml`), the last lines of the job log give the runner's result: PASS, FAIL after the end of the nextest log (which names the failed tests), or ERROR with the step that broke. Then inspect the `e2e-results-<family>-<package>-<environment>-<installer>` artifact, which holds `junit.xml`, `nextest.log` and, after a failure, the guest's `diagnostics/`.
 

.github/workflows/branch-e2e.yml

@@ -247,19 +247,11 @@ jobs:
       actions: read
       contents: read
       packages: read
-    uses: ./.github/workflows/integration-runner.yml
+    uses: ./.github/workflows/e2e-suites.yml
     with:
-      category: conformance
+      family: conformance
       source-sha: ${{ needs.prepare-integration.outputs.source_sha }}
       integration-inputs-artifact-id: ${{ needs.prepare-integration.outputs.integration_inputs_artifact_id }}
-      test-matrix: >-
-        [
-          {"environment":"ubuntu-docker-rootful","installer":"deb","testsuite":"conformance"},
-          {"environment":"ubuntu-k3s","installer":"k3s","testsuite":"conformance"},
-          {"environment":"ubuntu-k3s","installer":"k3s-ha","testsuite":"conformance"},
-          {"environment":"fedora-podman-rootful","installer":"rpm","testsuite":"conformance"},
-          {"environment":"fedora-podman-rootless","installer":"rpm","testsuite":"conformance"}
-        ]
 
   # Run feature-specific integration tests:
   feature-specific-integration:
@@ -303,9 +295,9 @@ jobs:
           {"environment":"fedora-podman-rootless","installer":"binaries","testsuite":"driver-podman"}
         ]
 
-  # Run the testkit suites in tests/suites/ on tmachine. The environments and
-  # suites are listed in e2e-suites.yml. Advisory: not part of
-  # core-e2e-result.needs.
+  # Run the testkit feature suites in tests/suites/ on tmachine. The
+  # environments and suites are listed in e2e-suites.yml; the conformance rows
+  # run in conformance-integration. Advisory: not part of core-e2e-result.needs.
   e2e-suites:
     needs: [pr_metadata, prepare-integration]
     if: needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_core_e2e == 'true'
@@ -315,6 +307,7 @@ jobs:
       packages: read
     uses: ./.github/workflows/e2e-suites.yml
     with:
+      family: features
       source-sha: ${{ needs.prepare-integration.outputs.source_sha }}
       integration-inputs-artifact-id: ${{ needs.prepare-integration.outputs.integration_inputs_artifact_id }}
 

.github/workflows/e2e-suites.yml

@@ -1,17 +1,24 @@
 # SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
 # SPDX-License-Identifier: Apache-2.0
 
-# Runs testkit suites in tmachine environments on CI. Each matrix row names a
-# tmachine environment and installer, the family (a workspace under
-# tests/suites/) and the package in it, and the runner label for the
-# environment's architecture. `openshell-test-runner` builds the tests, boots
-# the environment, runs them inside it and checks the results.
+# Runs testkit suites in tmachine environments on CI. Each row of `ROWS` below
+# names a tmachine environment and installer, the family (a workspace under
+# tests/suites/) and the package in it, the gateway name the installer
+# registers, and the runner label for the environment's architecture.
+# `openshell-test-runner` builds the tests, boots the environment, runs them
+# inside it and checks the results. The `family` input selects the rows of one
+# family; leave it empty to run every row.
 
 name: E2E Suites
 
 on:
   workflow_call:
     inputs:
+      family:
+        description: Run only the rows of this family (for example conformance); empty runs every row
+        required: false
+        type: string
+        default: ""
       source-sha:
         description: Source revision of the prepared candidate artifacts
         required: true
@@ -29,28 +36,52 @@ permissions:
 # github.workflow is the caller's name here, so scope the group to this
 # workflow to avoid colliding with (and cancelling) the caller's other jobs.
 concurrency:
-  group: ${{ github.workflow }}-e2e-suites-${{ github.ref }}
+  group: ${{ github.workflow }}-e2e-suites-${{ inputs.family }}-${{ github.ref }}
   cancel-in-progress: true
 
 jobs:
+  # The rows live here, in one place. Add an environment or a suite by adding a
+  # row; the runner needs no list of its own.
+  select:
+    name: Select E2E suites
+    runs-on: ubuntu-24.04
+    timeout-minutes: 5
+    outputs:
+      matrix: ${{ steps.rows.outputs.matrix }}
+    steps:
+      - name: Select the rows to run
+        id: rows
+        env:
+          FAMILY: ${{ inputs.family }}
+        run: |
+          set -euo pipefail
+          # Only amd64 for now: CI has no arm64 runner with KVM for tmachine,
+          # and the caller only builds amd64 inputs. The gateway is the name the
+          # installer registers with the CLI: `openshell` for the packages,
+          # `tmachine` for the k3s installers.
+          rows='[
+            {"environment":"ubuntu-docker-rootful","installer":"deb","family":"conformance","package":"sandbox","gateway":"openshell","runner":"ubuntu-24.04"},
+            {"environment":"fedora-podman-rootful","installer":"rpm","family":"conformance","package":"sandbox","gateway":"openshell","runner":"ubuntu-24.04"},
+            {"environment":"fedora-podman-rootless","installer":"rpm","family":"conformance","package":"sandbox","gateway":"openshell","runner":"ubuntu-24.04"},
+            {"environment":"ubuntu-k3s","installer":"k3s","family":"conformance","package":"sandbox","gateway":"tmachine","runner":"ubuntu-24.04"},
+            {"environment":"ubuntu-k3s","installer":"k3s-ha","family":"conformance","package":"sandbox","gateway":"tmachine","runner":"ubuntu-24.04"},
+            {"environment":"ubuntu-docker-rootful-keycloak","installer":"deb","family":"features","package":"keycloak","gateway":"openshell","runner":"ubuntu-24.04"}
+          ]'
+          matrix=$(jq -c --arg family "${FAMILY}" \
+            '{include: map(select($family == "" or .family == $family))}' <<< "$rows")
+          if [ "$(jq '.include | length' <<< "$matrix")" -eq 0 ]; then
+            echo "::error::No E2E suites match family '${FAMILY}'"
+            exit 1
+          fi
+          echo "$matrix"
+          echo "matrix=${matrix}" >> "$GITHUB_OUTPUT"
+
   suites:
     name: E2E suites (${{ matrix.family }}/${{ matrix.package }}, ${{ matrix.environment }}, ${{ matrix.installer }})
+    needs: select
     strategy:
       fail-fast: false
-      matrix:
-        include:
-          # amd64 only: CI has no arm64 runner with KVM for tmachine,
-          # and the caller only builds amd64 inputs.
-          - environment: ubuntu-docker-rootful
-            installer: deb
-            family: conformance
-            package: sandbox
-            runner: ubuntu-24.04
-          - environment: ubuntu-docker-rootful-keycloak
-            installer: deb
-            family: features
-            package: keycloak
-            runner: ubuntu-24.04
+      matrix: ${{ fromJSON(needs.select.outputs.matrix) }}
     runs-on: ${{ matrix.runner }}
     timeout-minutes: 90
     steps:
@@ -72,16 +103,20 @@ jobs:
 
       - uses: ./.github/actions/setup-nix
 
+      # integration-runner.yml skips the disk cache for ubuntu-k3s and doesn't say
+      # why, so the k3s environment skips it here too until that is understood.
       - name: Cache tmachine disks
+        if: matrix.environment != 'ubuntu-k3s'
         uses: actions/cache@caa296126883cff596d87d8935842f9db880ef25 # v5.1.0
         with:
           path: ~/.cache/tmachine
           # tmachine keys its disk layers on a hash of the Ansible tree.
           # actions/cache doesn't save after an exact key hit, so a fixed key
           # would rebuild the changed layers on every run. Key on the playbooks
           # with no restore-keys fallback: restoring an older entry would carry
-          # its stale disks into the new one, which then only grows.
-          key: tmachine-${{ runner.os }}-${{ runner.arch }}-${{ hashFiles('tests/tmachine/ansible/**', 'tests/tmachine/images.nix', 'scripts/keycloak-realm.json') }}
+          # its stale disks into the new one, which then only grows. The key
+          # includes the environment so each environment has its own entry.
+          key: tmachine-${{ matrix.environment }}-${{ runner.os }}-${{ runner.arch }}-${{ hashFiles('tests/tmachine/ansible/**', 'tests/tmachine/images.nix', 'scripts/keycloak-realm.json') }}
 
       - name: Download integration inputs
         uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
@@ -95,6 +130,7 @@ jobs:
           INSTALLER: ${{ matrix.installer }}
           FAMILY: ${{ matrix.family }}
           PACKAGE: ${{ matrix.package }}
+          GATEWAY: ${{ matrix.gateway }}
         run: |
           # Retry dependency preparation, then run the suites once. The runner
           # builds the suites for the guest, so it needs no prepared archives.
@@ -104,6 +140,7 @@ jobs:
             --installer "${INSTALLER}" \
             --family "${FAMILY}" \
             --package "${PACKAGE}" \
+            --gateway "${GATEWAY}" \
             --results artifacts/test-results
 
       # The results directory holds the JUnit report (junit.xml), the nextest
@@ -119,7 +156,7 @@ jobs:
 
   result:
     name: E2E suites result
-    needs: [suites]
+    needs: [select, suites]
     if: always()
     runs-on: ubuntu-24.04
     steps:

.github/workflows/integration-runner.yml

@@ -24,11 +24,8 @@ on:
         type: string
         default: >-
           [
-            {"environment":"ubuntu-docker-rootful","installer":"deb","testsuite":"conformance"},
-            {"environment":"ubuntu-k3s","installer":"k3s","testsuite":"conformance"},
-            {"environment":"ubuntu-k3s","installer":"k3s-ha","testsuite":"conformance"},
-            {"environment":"fedora-podman-rootful","installer":"rpm","testsuite":"conformance"},
-            {"environment":"fedora-podman-rootless","installer":"rpm","testsuite":"conformance"}
+            {"environment":"fedora-podman-rootful","installer":"rpm","testsuite":"provider-refresh"},
+            {"environment":"fedora-podman-rootless","installer":"rpm","testsuite":"provider-refresh"}
           ]
 
 permissions:

.github/workflows/integration-test.yml

@@ -20,18 +20,15 @@ on:
         description: Integration-test category, used for the job name and concurrency key
         required: true
         type: string
-        default: conformance
+        default: feature-specific
       test-matrix:
         description: JSON array of environment, installer, and testsuite combinations
         required: true
         type: string
         default: >-
           [
-            {"environment":"ubuntu-docker-rootful","installer":"deb","testsuite":"conformance"},
-            {"environment":"ubuntu-k3s","installer":"k3s","testsuite":"conformance"},
-            {"environment":"ubuntu-k3s","installer":"k3s-ha","testsuite":"conformance"},
-            {"environment":"fedora-podman-rootful","installer":"rpm","testsuite":"conformance"},
-            {"environment":"fedora-podman-rootless","installer":"rpm","testsuite":"conformance"}
+            {"environment":"fedora-podman-rootful","installer":"rpm","testsuite":"provider-refresh"},
+            {"environment":"fedora-podman-rootless","installer":"rpm","testsuite":"provider-refresh"}
           ]
 
 permissions:

.github/workflows/release-dev.yml

@@ -118,19 +118,11 @@ jobs:
       actions: read
       contents: read
       packages: read
-    uses: ./.github/workflows/integration-runner.yml
+    uses: ./.github/workflows/e2e-suites.yml
     with:
-      category: conformance
+      family: conformance
       source-sha: ${{ needs.prepare-integration.outputs.source_sha }}
       integration-inputs-artifact-id: ${{ needs.prepare-integration.outputs.integration_inputs_artifact_id }}
-      test-matrix: >-
-        [
-          {"environment":"ubuntu-docker-rootful","installer":"deb","testsuite":"conformance"},
-          {"environment":"ubuntu-k3s","installer":"k3s","testsuite":"conformance"},
-          {"environment":"ubuntu-k3s","installer":"k3s-ha","testsuite":"conformance"},
-          {"environment":"fedora-podman-rootful","installer":"rpm","testsuite":"conformance"},
-          {"environment":"fedora-podman-rootless","installer":"rpm","testsuite":"conformance"}
-        ]
 
   feature-specific-integration:
     needs: prepare-integration

.github/workflows/release-qualification.yml

@@ -71,19 +71,11 @@ jobs:
       actions: read
       contents: read
       packages: read
-    uses: ./.github/workflows/integration-runner.yml
+    uses: ./.github/workflows/e2e-suites.yml
     with:
-      category: conformance
+      family: conformance
       source-sha: ${{ needs.prepare-integration.outputs.source_sha }}
       integration-inputs-artifact-id: ${{ needs.prepare-integration.outputs.integration_inputs_artifact_id }}
-      test-matrix: >-
-        [
-          {"environment":"ubuntu-docker-rootful","installer":"deb","testsuite":"conformance"},
-          {"environment":"ubuntu-k3s","installer":"k3s","testsuite":"conformance"},
-          {"environment":"ubuntu-k3s","installer":"k3s-ha","testsuite":"conformance"},
-          {"environment":"fedora-podman-rootful","installer":"rpm","testsuite":"conformance"},
-          {"environment":"fedora-podman-rootless","installer":"rpm","testsuite":"conformance"}
-        ]
 
   feature-specific-integration:
     needs: prepare-integration

CI.md

@@ -174,7 +174,7 @@ push-triggered workflow jobs themselves.
 
 ### K3s conformance version baseline
 
-The tmachine `ubuntu-k3s` conformance lanes pin Agent Sandbox v0.5.0 as the
+The `ubuntu-k3s` conformance lanes pin Agent Sandbox v0.5.0 as the
 compatibility baseline for the v1beta1 Sandbox API. They do not track the local
 K3s development default, currently v1.0.3. OpenShell also supports v0.4.6 through
 its v1alpha1 fallback, so v0.5.0 is not the overall minimum supported version.
@@ -184,26 +184,26 @@ recreating the Service requires reprovisioning the fixture. The `k3s` lane runs
 one plaintext gateway. The `k3s-ha` lane runs the supported HA configuration:
 three gateway replicas sharing PostgreSQL, serving TLS, with the CLI connecting
 over mTLS. Before conformance, every installer waits up to five minutes for a
-connected gateway; interactive shells skip the wait. K3s failures upload
-diagnostics as `tmachine-diagnostics-*` artifacts.
+connected gateway; interactive shells skip the wait. Each lane uploads its
+JUnit report, nextest log and, after a failure, the guest's diagnostics as an
+`e2e-results-*` artifact.
 
-### Run only the policy advisor conformance tests
+### Run a subset of the conformance tests
 
-Manually dispatch `Integration Tests` on the candidate branch with an
-`artifact-run-id` from a build of the same commit. When a `Release` or `Pre-release` run was
-dispatched from a different commit, also set `source-sha` to the release tag's
-resolved commit so runtime images and test inputs match the candidate binaries.
-Otherwise, `source-sha` defaults to the artifact run's head SHA. Set `category`
-to `policy-advisor` and `test-matrix` to:
+The conformance lanes run the `sandbox` package of `tests/suites/conformance`
+through `.github/workflows/e2e-suites.yml`, which keeps the list of
+environments. To run one suite in one environment from a checkout, build the
+candidate inputs (see [TESTING.md](TESTING.md)) and narrow the tests with a
+nextest filterset:
 
-```json
-[{"environment":"ubuntu-docker-rootful","installer":"binaries","testsuite":"policy-advisor"}]
+```shell
+nix run .#openshell-test-runner -- \
+  --environment ubuntu-docker-rootful --installer deb \
+  --family conformance --package sandbox \
+  --filter 'binary_id(=sandbox::policy_advisor)'
 ```
 
-This runs the `mechanistic-proposal`, `new-hostname-proposal`, and
-`policy-local` conformance tests in the installed-artifact suite. The artifact run must contain the candidate CLI and
-gateway binaries and runtime images. This manual run does not replace the
-required PR E2E gate.
+A local run does not replace the required PR E2E gate.
 
 ## Informational security reports
 

TESTING.md

@@ -158,6 +158,25 @@ verifies this, and `uv run tasks/scripts/check_cargo_lockfile_alignment.py --fix
 re-locks a workspace from the root lockfile. Add new testkit workspaces to
 `ALIGNED_WORKSPACES` in that script.
 
+#### What a conformance test may assert
+
+Conformance tests run on every compute driver, so they check the public
+contract and nothing a driver is free to change.
+
+- Assert only what the public contract promises: the comments in `proto/` and
+  the published docs.
+- Assert on `phase` and `exit_code`. Don't assert on condition reasons or
+  messages.
+- To check that something did not happen, read the state that shows it. Don't
+  sleep and hope.
+- If a behavior isn't documented, open an issue to document it. Don't write a
+  test for it. A behavior that only some drivers have belongs in
+  `tests/suites/features` or `tests/suites/drivers`.
+- The only way to skip a test is `#[ignore = "<issue link>"]`, for a broken or
+  flaky test.
+
+Adapted from the handover notes in NVIDIA/OpenShell#4629 by Evan Lezar.
+
 ### Running suites in a tmachine environment (`tests/runner/`)
 
 `openshell-test-runner` runs a test family inside a tmachine VM, against the
@@ -348,9 +367,8 @@ Suites:
 - CLI conformance (`tests/suites/conformance`) - portable Cargo tests for
   lifecycle, mechanistic drafts, file transfer, and the sandbox-local API,
   including agent-authored permission requests. Driver E2E runs the complete
-  Cargo test package. The installed-artifact conformance suite runs the same
-  tests from a nextest archive and offers a focused `policy-advisor` testsuite
-  for manual integration runs.
+  Cargo test package. The installed-artifact conformance lanes run the `sandbox`
+  package through `openshell-test-runner` (see below).
 - Driver suites (`--features e2e-docker`, `e2e-podman`, `e2e-kubernetes`, or
   `e2e-vm`) - CLI conformance plus the common and driver-specific coverage for
   the selected deployment.
@@ -460,7 +478,9 @@ rebuilds only the RPMs. Run conformance against the installed RPMs:
 
 ```shell
 nix run .#build-artifacts
-nix run .#tmachine -- test fedora-podman-rootful rpm conformance
+nix run .#openshell-test-runner -- \
+  --environment fedora-podman-rootful --installer rpm \
+  --family conformance --package sandbox
 ```
 
 Run the portable subset in a disposable rootless Podman guest: