Skip to content

test: run a conformance smoke test on tmachine through a test runner #4934

Open

Description

@matthewgrossman

User Story

I'm moving OpenShell's end-to-end suites onto testkit (#3954). I need to run a testkit test against a real OpenShell install in a tmachine VM, locally and on CI, without writing an Ansible playbook per suite, so suites can move off the old tmachine testsuites one at a time.

Description

Add the smallest path that runs a testkit suite inside a tmachine environment:

  • a tmachine Session API that boots an environment with an installer, runs commands with timeouts, copies files in and out, collects diagnostics and always tears the VM down;
  • openshell-test-runner (tests/runner/), which builds a test family's packages for the guest, boots one named tmachine environment, waits for the gateway, runs the tests with nextest and checks the JUnit results;
  • a one-test sandbox smoke suite in tests/suites/conformance/sandbox;
  • an advisory CI lane (e2e-suites.yml) with an explicit matrix, starting with Ubuntu + rootful Docker + the Debian package.
nix run .#openshell-test-runner -- \
  --environment ubuntu-docker-rootful --installer deb \
  --family conformance --package sandbox

Context

Definition of Done

  • The runner passes the smoke suite in ubuntu-docker-rootful with the deb installer, locally (host architecture) and on CI (amd64).
  • A run fails if nextest fails or no test ran.
  • The CI lane uploads junit.xml, nextest.log and, after a failure, guest diagnostics, and is advisory.
  • TESTING.md documents how to run the runner.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions