Repository navigation
CI: live sync test on throwaway instances, gating promotion to latest - #251
Conversation
|
First real run on GitHub, after the setup: run 41066708783 (attempt 2). Sign-in through OIDC, leasing, the test and the release all worked; both leases were deleted. The test fails for two CLI bugs, now ticketed:
This check stays red until those are fixed, which is the point of it. |
… (PROD-3016) Leases throwaway instances, syncs a source to a target with the CLI under test, runs verify, checks that a second sync would change nothing, and always deletes the instances. - ci/leases.mjs: dependency-free lease client (create and wait, release by label prefix, sweep of left-behind leases), with retries. - ci/permutation/run.mjs: the first slice of the test. Writes a results grid to the job summary and a JSON artifact. - sync-permutation.yml: quick tier on same-repo PRs, beta tier on v* tags (tests the published package), nightly tier with a sweep, and manual runs. Credentials come only from a GitHub Environment by OIDC, so fork PRs get none. Every identifier is an Environment secret, so nothing about the infrastructure appears in the repo or the public logs. - npm-publish.yml: promote to latest requires a passing tag run. - ci/README.md: when it runs, how to trigger it, what it checks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
9024f52 to
8e4605f
Compare
After the first sync, the test now runs rounds that alternate direction: odd rounds edit the source and sync forward, even rounds edit the target and reverse-sync. Every round also edits the receiving side (drift) and one item on both sides (a conflict). - ci/permutation/changes.mjs: change catalogue, discovered from whatever each instance holds. Content edit, add, add a linked chain, delete, publish/unpublish; list and component reordering; page edit, add, delete; component removal; model field add, label change and rename. - run.mjs: per-round sync, verify and no-op checks, plus conflict and drift outcomes (kept-and-reported passes, silently kept fails), and apply failures as findings. quick = fixed two-round set; beta and nightly = seeded random rounds, replayable with the printed seed. - Workflow: always installs dependencies (the change engine uses the Management SDK), takes a seed input, longer time budgets. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Handing this over to Kevin. It's already merged into
How it runs is in |
Blog Starter has no content that references itself or each other, so the live test could not catch a regression in the two-pass cycle push. Before the first sync, ci/permutation/fixtures.mjs adds a PermCycleNode model (a required single-item linked field with a companion field) and its list to the source: A <-> B and a self-reference, published. After round 0 the new `cycles` check confirms each copy on the target points at its partner's target ID, in both the link and the companion field. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Added a reference-cycle fixture (f266d0c) so the live test covers content that references itself and each other (PROD-3076). Blog Starter has no such content.
That check only passes with #253, which adds the CLI side (dependency-ordered linked content, PROD-3025; the same-run two-pass push for cycles, PROD-3076). #253 already contains this branch, so merge this first, then #253. Until #253 lands, the quick tier here also fails round 0's 🤖 Generated with Claude Code |
Adds a live, end-to-end test of
agility sync(PROD-3016, under PROD-3015). Each run leases throwaway instances, syncs a source to a target with the CLI under test, checks the result, and always deletes the instances.ci/README.mdexplains when it runs, how to trigger it, and what it checks.What it checks today (first slice):
syncexits 0 with no failed items;agility verifyfinds no differences;sync --preflightplans no changes.When it runs:
v*tagsmain, plus a sweep for left-behind instancesgh workflow runFork and Dependabot PRs skip it.
The gate:
npm-publish.yml's promote job now refuses to move a version tolatestunless that version's tag run passed.Cleanup, in three layers:
releasejob runs whatever the outcome and deletes every instance labelled with the run.Configuration: every identifier lives in the
cli-dev-liveGitHub Environment as a secret. Nothing is in the repo, and the public Actions logs mask it. Artifacts hold only the JSON results.Current result: the check fails on two CLI bugs this test found. PROD-3025: linked content isn't dependency-ordered within a sync, so items that reference other referenced items are skipped. PROD-3026: a page whose name already exists on the target isn't adopted. This check stays red until they're fixed, and merging this holds new versions off
latestuntil then.🤖 Generated with Claude Code