ci: use the CLI against the dev platform, weekly and after each release - #79
Merged
Merged
Conversation
Nothing checked that the released CLI, installed by the action, works against a real platform: the tests use a fake one, and the container tests run without one. e2e/platform.sh uses it the way a pipeline does, with the team token of the test team CLI-E2E: experiments applied from files (keys written back), drift found by diff, a parallel run with a JUnit report, SIGTERM canceling the run with exit 143, --no-wait failing on a run the platform refused, and an execution list gate. Only wait steps, and everything it creates is deleted afterwards, also what an earlier crashed run left. The Platform E2E workflow runs it weekly on Tuesday evenings with the latest release, after each stable release from ci.yml, by hand (optionally from source), and on pull requests that change it. A nightly job in ci.yml only reads the test team and its runs, to see early that the token and the API still work.
Other test suites run experiments on dev in the evening, so every run of the platform test allows running in parallel, now explicitly, except the one whose refusal its check is about. The weekly run moves to 17:00 UTC: 19:00 in Berlin in summer, 18:00 in winter, as cron has no time zone.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Nothing checked that the released CLI, installed by the Action, works against a real platform. The unit tests use a fake platform, the container tests run without one, and
verify-actiononly checks--version.e2e/platform.shuses the CLI the way a pipeline does, against dev, with the team token of the test team CLI-E2E (keyCLI, secretSTEADYBIT_E2E_TOKEN). It checks, in order:experiment apply -fcreates the experiments and writes their keys into the files, keeping comments.diffexits 0 right after apply, and 2 once a file changes;applythen updates it.run --parallel 2 --reportcompletes both runs, with a JUnit report holding both.--no-waitfails (exit 1) on a run the platform accepted and then cancelled because another was running.execution list --fail-on-matchgates on that cancelled run.Only wait steps. On exit, even after a failure, it deletes every experiment named
cli-e2e-ci-*in the team, including ones a crashed earlier run left behind. It first cancels anything still running.When it runs
ci.yml, afterverify-action, on the binary users now get.from-source).Setup done: the team
CLIhad no environment, so experiments couldn't be created. I addedGlobalwith my admin dev profile. The team can only usewaitandservice-validation, which suits a test team.Testing: run locally against dev with a build of
main: all 11 checks passed in 25 s, and the three experiments were deleted. With the old 6.0.0 binary the same run failed on--parallelandexecution list, so the test does catch a CLI without those features. This PR runs the test itself, with the real team token.