test: the platform test covers what the CLI manages, not only runs - #82
Merged
Merged
Conversation
Templates, run properties, schedules, services and service profiles with the team token; environments, teams, access tokens, hubs and integrations with an admin token from STEADYBIT_E2E_ADMIN_TOKEN, skipped without it. What an admin creates is named cli-e2e-ci-*, is scoped to a team of its own, targets nothing, and is swept afterwards, also after a crashed run. The kill switch is only read and no invitation is sent. Tokens only go through the environment, so a failing check cannot show one, and new ones are masked in the job log. It found that execution property set sent a single value as a scalar, which the platform refuses for a list property; a list property now gets a list.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The platform test now goes through the acceptance criteria of card 14235 on dev, not only a pipeline's experiment runs. 81 checks in all, up from 15.
With the team token (CLI-E2E): run an experiment from a template with a placeholder, set and add run properties, list a run's artifacts, and a schedule's whole life (create, get, diff, update, apply, enable, disable, delete). Also services: apply, diff, the profile's template offered, link and unlink an experiment, risk, variables, delete. Plus target stats and the action list.
With the admin token (
STEADYBIT_E2E_ADMIN_TOKEN): templates (apply, diff, placeholders), the run property's definition and association, service profiles, environments and their variables, and teams: apply, diff, environments, and members, which uses an unknown address and never a real person. Also access tokens (create, use, recreate, the old one stops working, delete), a hub (apply with sync, diff, resync, delete), webhook, preflight and preflight-action integrations (apply, diff, delete), and reads of the audit log, kill switch, reports and license. Without the secret these checks are skipped.Safety on a shared platform:
cli-e2e-ci-*, and the cleanup deletes what matches by kind, including what a crashed run left.CLIX, which runs nothing, so no webhook is ever called.A bug it found:
execution property setwith one--valuesent a scalar, which the platform refuses for a list property ("must be a list of strings"). The CLI now reads the property's definition and sends a list for*_LISTtypes. When it cannot read the definition, it sends the value as given, as before. There is a changelog entry under v6.2.2.Two things found, not changed here: right after
apply,diffreports drift for an environment (the platform returnsquerybesidepredicate.query) and a service (a validation parameter the platform defaults). The test diffs whatgetwrote, which is what a repository keeps.I ran it locally against dev with an admin token in place of both. All checks passed, and afterwards dev had no
cli-e2etemplate, service, profile, environment, hub, token or integration left, and noCLIXteam. This PR changese2e/platform.sh, so it runs here with the real team token, which may show checks that need admin.