Skip to content

CI: live sync test on throwaway instances, gating promotion to latest - #251

Merged
5PK merged 3 commits into
mainfrom
ci/sync-permutation-pipeline
Nov 2, 2026
Merged

5PK merged 3 commits into
mainfrom
ci/sync-permutation-pipeline

Conversation

@joelvarty

@joelvarty joelvarty commented Oct 30, 2026 •

Copy link
Copy Markdown
Member

Adds a live, end-to-end test of agility sync (PROD-3016, under PROD-3015). Each run leases throwaway instances, syncs a source to a target with the CLI under test, checks the result, and always deletes the instances. ci/README.md explains when it runs, how to trigger it, and what it checks.

What it checks today (first slice):

  • sync exits 0 with no failed items;
  • agility verify finds no differences;
  • an immediate sync --preflight plans no changes.

When it runs:

Trigger Tier CLI under test
Pull requests from this repo's branches quick the PR's build
v* tags beta the published npm package
Nightly nightly main, plus a sweep for left-behind instances
By hand your choice via Run workflow or gh workflow run

Fork and Dependabot PRs skip it.

The gate: npm-publish.yml's promote job now refuses to move a version to latest unless that version's tag run passed.

Cleanup, in three layers:

  1. A release job runs whatever the outcome and deletes every instance labelled with the run.
  2. Short expiry times, so anything a dead runner leaves is deleted automatically.
  3. The nightly sweep.

Configuration: every identifier lives in the cli-dev-live GitHub Environment as a secret. Nothing is in the repo, and the public Actions logs mask it. Artifacts hold only the JSON results.

Current result: the check fails on two CLI bugs this test found. PROD-3025: linked content isn't dependency-ordered within a sync, so items that reference other referenced items are skipped. PROD-3026: a page whose name already exists on the target isn't adopted. This check stays red until they're fixed, and merging this holds new versions off latest until then.

🤖 Generated with Claude Code

@joelvarty

Copy link
Copy Markdown
Member Author

First real run on GitHub, after the setup: run 41066708783 (attempt 2). Sign-in through OIDC, leasing, the test and the release all worked; both leases were deleted.

The test fails for two CLI bugs, now ticketed:

  • PROD-3025: linked content isn't dependency-ordered inside the linked tier (get-content-item-types.ts puts every referenced item in one tier, pushed as one batch). So in FeaturedPost → Post → Category, the posts are remapped before their categories exist and get skipped. This causes 5 of the 6 failures, the verify differences, and most of the re-sync plan.
  • PROD-3026: a source page whose name already exists on the target, with no mapping, fails outright. Containers in the same situation get adopted with a mapping row.

This check stays red until those are fixed, which is the point of it.

… (PROD-3016)

Leases throwaway instances, syncs a source to a target with the CLI under
test, runs verify, checks that a second sync would change nothing, and
always deletes the instances.

- ci/leases.mjs: dependency-free lease client (create and wait, release
  by label prefix, sweep of left-behind leases), with retries.
- ci/permutation/run.mjs: the first slice of the test. Writes a results
  grid to the job summary and a JSON artifact.
- sync-permutation.yml: quick tier on same-repo PRs, beta tier on v* tags
  (tests the published package), nightly tier with a sweep, and manual
  runs. Credentials come only from a GitHub Environment by OIDC, so fork
  PRs get none. Every identifier is an Environment secret, so nothing
  about the infrastructure appears in the repo or the public logs.
- npm-publish.yml: promote to latest requires a passing tag run.
- ci/README.md: when it runs, how to trigger it, what it checks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@joelvarty
joelvarty force-pushed the ci/sync-permutation-pipeline branch from 9024f52 to 8e4605f Compare October 30, 2026 15:29
@joelvarty
joelvarty deployed to cli-dev-live October 30, 2026 15:29 — with GitHub Actions Active
@joelvarty joelvarty changed the title CI: live sync permutation test on dev, gating promotion to latest CI: live sync test on throwaway instances, gating promotion to latest Oct 30, 2026
@joelvarty
joelvarty deployed to cli-dev-live October 30, 2026 15:34 — with GitHub Actions Active
After the first sync, the test now runs rounds that alternate direction:
odd rounds edit the source and sync forward, even rounds edit the target
and reverse-sync. Every round also edits the receiving side (drift) and
one item on both sides (a conflict).

- ci/permutation/changes.mjs: change catalogue, discovered from whatever
  each instance holds. Content edit, add, add a linked chain, delete,
  publish/unpublish; list and component reordering; page edit, add,
  delete; component removal; model field add, label change and rename.
- run.mjs: per-round sync, verify and no-op checks, plus conflict and
  drift outcomes (kept-and-reported passes, silently kept fails), and
  apply failures as findings. quick = fixed two-round set; beta and
  nightly = seeded random rounds, replayable with the printed seed.
- Workflow: always installs dependencies (the change engine uses the
  Management SDK), takes a seed input, longer time budgets.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@joelvarty

Copy link
Copy Markdown
Member Author

Handing this over to Kevin. It's already merged into release/1.3.0-beta.0 (#254), where the first sync now passes. What still fails is verify and the no-op preflight reporting the conflicts and drift the test keeps on purpose. Next steps for the engine:

  1. Separate expected differences (kept conflicts, drift, known issue PROD-2775) from new ones, so a clean run passes.
  2. content.add-chain finds no linked item to start from on the blog template.
  3. Add change types for locales, moving items, delete-and-recreate under the same name, and damaged mapping files.

How it runs is in ci/README.md.

@joelvarty
joelvarty requested a review from 5PK November 2, 2026 15:21
Blog Starter has no content that references itself or each other, so the
live test could not catch a regression in the two-pass cycle push. Before
the first sync, ci/permutation/fixtures.mjs adds a PermCycleNode model
(a required single-item linked field with a companion field) and its list
to the source: A <-> B and a self-reference, published. After round 0 the
new `cycles` check confirms each copy on the target points at its
partner's target ID, in both the link and the companion field.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@5PK

5PK commented Nov 2, 2026

Copy link
Copy Markdown
Collaborator

Added a reference-cycle fixture (f266d0c) so the live test covers content that references itself and each other (PROD-3076). Blog Starter has no such content.

  • ci/permutation/fixtures.mjs adds a PermCycleNode model and list to the source before the first sync: A ↔ B and a self-reference, published. The link field is required and has a companion field.
  • After round 0, a new cycles check confirms each copy on the target points at its partner's target ID, in both the link and the companion field.

That check only passes with #253, which adds the CLI side (dependency-ordered linked content, PROD-3025; the same-run two-pass push for cycles, PROD-3076). #253 already contains this branch, so merge this first, then #253.

Until #253 lands, the quick tier here also fails round 0's cycles check, on top of the round 1–2 verify and no-op failures it already shows with main's CLI. A local run of this harness with #253's CLI passed all of round 0, including cycles. One of the round 1 failures looks like a timing gap in the conflict check. The pre-sync preflight reported 0 conflicts, but the sync itself then reported the conflict and kept the target's value. The check only credits the preflight's count, so it read this as "silently kept".

🤖 Generated with Claude Code

@5PK
5PK deployed to cli-dev-live November 2, 2026 18:33 — with GitHub Actions Active

@5PK 5PK left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving: the live sync test harness, plus the reference-cycle fixture. #253 follows with the CLI fixes the fixture needs.

@5PK
5PK merged commit 9ff223e into main Nov 2, 2026
4 checks passed
@5PK
5PK deployed to cli-dev-live November 2, 2026 18:39 — with GitHub Actions Active

This branch was successfully deployed

1 active deployment
cli-dev-live — f266d0c2 Deployed Nov 2, 2026 by 5PK via Release leases #13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants