Skip to content

test: the platform test covers what the CLI manages, not only runs - #82

Merged
achoimet merged 2 commits into
mainfrom
test/platform-coverage
Sep 30, 2026
Merged

achoimet merged 2 commits into
mainfrom
test/platform-coverage

Conversation

@achoimet

Copy link
Copy Markdown
Member

The platform test now goes through the acceptance criteria of card 14235 on dev, not only a pipeline's experiment runs. 81 checks in all, up from 15.

With the team token (CLI-E2E): run an experiment from a template with a placeholder, set and add run properties, list a run's artifacts, and a schedule's whole life (create, get, diff, update, apply, enable, disable, delete). Also services: apply, diff, the profile's template offered, link and unlink an experiment, risk, variables, delete. Plus target stats and the action list.

With the admin token (STEADYBIT_E2E_ADMIN_TOKEN): templates (apply, diff, placeholders), the run property's definition and association, service profiles, environments and their variables, and teams: apply, diff, environments, and members, which uses an unknown address and never a real person. Also access tokens (create, use, recreate, the old one stops working, delete), a hub (apply with sync, diff, resync, delete), webhook, preflight and preflight-action integrations (apply, diff, delete), and reads of the audit log, kill switch, reports and license. Without the secret these checks are skipped.

Safety on a shared platform:

  • Everything an admin creates is named cli-e2e-ci-*, and the cleanup deletes what matches by kind, including what a crashed run left.
  • Integrations are scoped to a team of the test's own, CLIX, which runs nothing, so no webhook is ever called.
  • The service and environment queries match no target.
  • The schedule starts in 2099 and is enabled only for a moment.
  • Nothing is imported from the hub.
  • The kill switch is only read, and no invitation is sent.
  • Tokens only go through the environment, never a command line, so a failing check, which shows its command, cannot show one. Tokens the test creates are masked in the job log and expire the next day at the latest.

A bug it found: execution property set with one --value sent a scalar, which the platform refuses for a list property ("must be a list of strings"). The CLI now reads the property's definition and sends a list for *_LIST types. When it cannot read the definition, it sends the value as given, as before. There is a changelog entry under v6.2.2.

Two things found, not changed here: right after apply, diff reports drift for an environment (the platform returns query beside predicate.query) and a service (a validation parameter the platform defaults). The test diffs what get wrote, which is what a repository keeps.

I ran it locally against dev with an admin token in place of both. All checks passed, and afterwards dev had no cli-e2e template, service, profile, environment, hub, token or integration left, and no CLIX team. This PR changes e2e/platform.sh, so it runs here with the real team token, which may show checks that need admin.

Templates, run properties, schedules, services and service profiles with the
team token; environments, teams, access tokens, hubs and integrations with an
admin token from STEADYBIT_E2E_ADMIN_TOKEN, skipped without it. What an admin
creates is named cli-e2e-ci-*, is scoped to a team of its own, targets nothing,
and is swept afterwards, also after a crashed run. The kill switch is only
read and no invitation is sent. Tokens only go through the environment, so a
failing check cannot show one, and new ones are masked in the job log.

It found that execution property set sent a single value as a scalar, which
the platform refuses for a list property; a list property now gets a list.
@achoimet
achoimet merged commit d6a1ff2 into main Sep 30, 2026
6 checks passed
@achoimet
achoimet deleted the test/platform-coverage branch September 30, 2026 07:38
@github-actions github-actions Bot locked and limited conversation to collaborators Sep 30, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant