Skip to content

PROD-3025, PROD-3076: push linked content in dependency order, two-pass for reference cycles - #253

Closed
5PK wants to merge 11 commits into
mainfrom
PROD-3025-linked-content-dependency-order
Closed

5PK wants to merge 11 commits into
mainfrom
PROD-3025-linked-content-dependency-order

Conversation

@5PK

@5PK 5PK commented Oct 30, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

getContentItemTypes sorted content into two tiers: normal and linked. Every item another item references went into one linked tier, pushed as one batch (batchSize 250) in reversed insertion order. In a chain like FeaturedPost → Post → Category, Post and Category are both linked and land in the same batch. The batch processor checks each item's references before the batch writes, so Post fails with Unresolved content reference(s) not present in the target mapping, and FeaturedPost fails because Post never got created. Category is created moments later in the same run. A re-sync "fixes" it by accident because Category exists by then, which is why it can look intermittent.

The sync permutation test (PROD-3016) reproduces this every run: Blog Starter into an empty instance gives 3 posts and 2 featured posts failed, verify differences, and a re-sync that still plans 6 creates.

Fix

  • get-content-item-types.ts: also returns linkedContentLevels, the linked items split into dependency levels, leaves first. A level's items reference only lower levels.
    • It builds a graph from the existing collectReferencedContentIDs (whole-list and single-item references, including companion fields), keeping only edges between linked items and dropping self-references.
    • Tarjan's algorithm collapses cycles. Each item sits one level above its deepest dependency, so this works for any depth, uneven depths and diamonds. It is iterative, so a long chain can't overflow the stack.
    • linkedContentItems is unchanged.
  • content-pusher.ts:
    • The linked tier is still filtered once. It is then grouped by level, and each level goes through its own processBatches call (Linked Content (level n/N)), so every mapping a level needs was written by an earlier level.
    • .reverse() is removed.
    • A level the filter emptied is skipped, so there is no empty processBatches call any more.
    • The duplicated result bookkeeping for the two tiers is now one recordBatchResult helper.
    • Preflight is unchanged.

Cycles: handled by the two-pass push below (PROD-3076).

Cost: each level is one more batch round trip, so deep chains take longer than a single batch. Depth comes from the content model, not the amount of content.

Also: empty page titles (found by the live test once linked content synced)

processPage sent page.title || page.menuText || page.name, so an untitled page reached the target titled with its name. A dynamic page's titles come from its titleFormula, so it is often untitled in the source. Blog Starter's posts-dynamic page is one, and verify reported posts-dynamic / title on every sync. Pulled pages come through the Content Sync SDK (the Fetch API), which returns an empty title as null, so "untitled" arrives as null, not "".

The Management API's PageItem.title is optional, the batch insert copies it without validating it, and the server stores a null title as "". There was never a reason to fill one in. The title is now sent as-is, with null as "".

Verified locally: a sync of a leased blog copy into a leased blank instance, then verify passes (0 failures) and sync --preflight plans nothing.

Tests: process-page.test.ts covers a null title on a dynamic page (fails without the fix), an empty title, a normal title, and no fallback to menu text or name.

Also: content in a reference cycle (PROD-3076)

Items that reference each other by content ID (A -> B -> A), or that reference themselves, were never created. Each failed the unresolved-reference check because the other had no target ID yet, and every later sync failed the same way. Dependency levels can't help, since a cycle has no leaf. Models already solve this with stub-then-fields (model-pusher.ts), and content now does the equivalent in the same run:

  • getContentItemTypes also returns linkedCycleReferences: for each linked item in a cycle, its by-ID references into that cycle, including a self-reference. Whole-list references resolve by container name, never block, and aren't included.
  • Pass 1: for each level, a cycle member whose partners have no mapping yet goes out with those references blanked (stripContentReferences: contentid set to -1, the ID removed from sortids and from companion fields).
  • Pass 2: right after, the same members go out in full (Linked Content …: completing reference cycles). They go through a batch processor with the new trustSameRunMappings option, so the update targets the item pass 1 created. That item isn't in the pulled target yet, and without the option it would look deleted and be created again.
  • Each item is counted and published once, from pass 2. A failed pass 2 is reported, and ContentItemMapper.markSourceChanged makes the next sync push the item again. A cycle whose members are already mapped (a re-sync) goes out once, unchanged.
  • The server accepts a required linked-content field saved blank (tested on dev), so pass 1 works for required fields.

Verified on dev with a leased blog copy plus a CycleNode model (A <-> B, C -> C, a required single-item linked field with a companion field). A first sync into an empty instance creates all 30 items with 0 failures, the references point at the right target IDs, verify passes, and a re-sync plans nothing. The live test now seeds the same kind of cycle into its source (see below).

Tests: strip-content-references.test.ts (new), cycle detection in get-content-item-types.test.ts, trustSameRunMappings in content-batch-processor.test.ts, and the two passes, counts, publishing, failed pass 2, failed stub and re-sync in content-pusher-linked-levels.test.ts.

Tests

  • get-content-item-types.test.ts:
    • a three-deep chain
    • a 50-deep chain (one level per item, no stack issues)
    • uneven depths
    • a diamond
    • whole-list references
    • a cycle above a leaf
    • a self-reference
    • levels covering exactly the linked items
  • New content-pusher-linked-levels.test.ts:
    • one processBatches call per level, in order, then the normal tier
    • totals and failure registration across levels
    • a level emptied by the filter
    • preflight
    • splitIntoLevels
  • content-pusher-validation.test.ts: no longer expects a processBatches call for an emptied linked tier.
  • Full suite: 136 suites, 2,594 tests pass; npm run build passes.

Live verification

The live sync test (#251, now on main) runs on this PR. It includes a reference-cycle fixture: A ↔ B and a self-reference through a required linked field with a companion field. After the first sync, its cycles check confirms each copy points at its partner's target ID. That check, together with round 0's sync, verify and no-op checks, is what this PR fixes. The later change rounds (both-direction edits, drift, conflicts) have failures of their own that happen with main's CLI too (see #251).

🤖 Generated with Claude Code

joelvarty and others added 3 commits October 30, 2026 11:29
… (PROD-3016)

Leases throwaway instances, syncs a source to a target with the CLI under
test, runs verify, checks that a second sync would change nothing, and
always deletes the instances.

- ci/leases.mjs: dependency-free lease client (create and wait, release
  by label prefix, sweep of left-behind leases), with retries.
- ci/permutation/run.mjs: the first slice of the test. Writes a results
  grid to the job summary and a JSON artifact.
- sync-permutation.yml: quick tier on same-repo PRs, beta tier on v* tags
  (tests the published package), nightly tier with a sweep, and manual
  runs. Credentials come only from a GitHub Environment by OIDC, so fork
  PRs get none. Every identifier is an Environment secret, so nothing
  about the infrastructure appears in the repo or the public logs.
- npm-publish.yml: promote to latest requires a passing tag run.
- ci/README.md: when it runs, how to trigger it, what it checks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
getContentItemTypes put every referenced item into one linked tier, pushed
as a single batch in reversed insertion order. In a chain A -> B -> C, B and
C landed in the same batch, and B's references were checked before C had a
target ID, so B (and A above it) failed with "Unresolved content
reference(s) not present in the target mapping".

getContentItemTypes now also returns linkedContentLevels: the linked items
split into dependency levels (Tarjan SCCs over the reference graph, so any
depth works and cycles collapse into one level). pushContent sends each
level through its own processBatches call, so every mapping a level needs
was written by an earlier one. The per-tier result bookkeeping moves into
one helper. Cycles still fail as before; they need a two-pass push.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…to PROD-3025-linked-content-dependency-order
After the first sync, the test now runs rounds that alternate direction:
odd rounds edit the source and sync forward, even rounds edit the target
and reverse-sync. Every round also edits the receiving side (drift) and
one item on both sides (a conflict).

- ci/permutation/changes.mjs: change catalogue, discovered from whatever
  each instance holds. Content edit, add, add a linked chain, delete,
  publish/unpublish; list and component reordering; page edit, add,
  delete; component removal; model field add, label change and rename.
- run.mjs: per-round sync, verify and no-op checks, plus conflict and
  drift outcomes (kept-and-reported passes, silently kept fails), and
  apply failures as findings. quick = fixed two-round set; beta and
  nightly = seeded random rounds, replayable with the printed seed.
- Workflow: always installs dependencies (the change engine uses the
  Management SDK), takes a seed input, longer time budgets.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@5PK
5PK deployed to cli-dev-live November 2, 2026 15:47 — with GitHub Actions Active
@5PK
5PK deployed to cli-dev-live November 2, 2026 15:51 — with GitHub Actions Active
@5PK
5PK deployed to cli-dev-live November 2, 2026 16:23 — with GitHub Actions Active
@5PK
5PK deployed to cli-dev-live November 2, 2026 16:27 — with GitHub Actions Active
processPage sent `page.title || page.menuText || page.name`, so a page
saved with an empty title reached the target titled with its name. A
dynamic page's titles come from its titleFormula, so it is often untitled
in the source; verify then reported a title mismatch on every sync.

Send the source title as-is when it is a string, including "", and fall
back only when the title is missing (folders come back with title null).
The batch insert copies Title without validating it, and folders ignore
it, so an empty title saves fine.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@5PK
5PK deployed to cli-dev-live November 2, 2026 16:34 — with GitHub Actions Active
@5PK
5PK deployed to cli-dev-live November 2, 2026 16:39 — with GitHub Actions Active
The previous commit kept "" but still fell back for a null title, and the
pulled page has title null: the Content Sync SDK reads the Fetch API,
which returns an empty title as null. So the dynamic page still reached
the target titled with its name.

The Management API's PageItem.title is optional and the server stores a
null title as "", so there was never a reason to fill one in. Send the
source title as-is, with null as "". Verified with a local sync of a
leased blog copy into a leased blank instance: verify passes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@5PK
5PK deployed to cli-dev-live November 2, 2026 16:47 — with GitHub Actions Active
@5PK
5PK deployed to cli-dev-live November 2, 2026 16:48 — with GitHub Actions Active
@5PK
5PK deployed to cli-dev-live November 2, 2026 16:51 — with GitHub Actions Active
Items that reference each other by content ID (A -> B -> A), or that
reference themselves, were never created: each failed the unresolved-
reference guard because the other had no target ID yet, and every later
sync failed the same way. Dependency levels (PROD-3025) can't help, since
a cycle has no leaf.

getContentItemTypes now reports, per linked item in a cycle, its by-ID
references into that cycle (whole-list references resolve by container
name and never block). For each level, the pusher sends those items with
the references to not-yet-mapped members blanked (contentid -1, the ID
removed from sortids and companion fields), then sends them again in full
once every member has a target ID. The second pass uses a new batch
option, trustSameRunMappings, so it updates the target item pass 1 made
instead of treating it as deleted (it isn't in the pulled target yet).

Each item is counted and published once, from pass 2. If pass 2 fails,
the item is reported and its mapping marked changed so the next sync
pushes it again. Cycles whose members are already mapped (a re-sync) go
out once, unchanged. The server accepts a required linked field saved
blank, so pass 1 works for required fields too.

Verified on dev with a leased blog copy plus a CycleNode model (A <-> B,
C -> C, required single linked field with a companion): first sync into
an empty instance creates all 30 items with 0 failures, the references
point at the right target IDs, verify passes, and a re-sync plans nothing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@5PK
5PK deployed to cli-dev-live November 2, 2026 18:07 — with GitHub Actions Active
@5PK 5PK changed the title PROD-3025: push linked content in dependency levels, leaves first PROD-3025, PROD-3076: push linked content in dependency order, two-pass for reference cycles Nov 2, 2026
@5PK
5PK deployed to cli-dev-live November 2, 2026 18:08 — with GitHub Actions Active
@5PK
5PK deployed to cli-dev-live November 2, 2026 18:11 — with GitHub Actions Active
5PK and others added 3 commits November 2, 2026 13:17
…to PROD-3025-linked-content-dependency-order
Blog Starter has no content that references itself or each other, so the
live test could not catch a regression in the two-pass cycle push. Before
the first sync, ci/permutation/fixtures.mjs adds a PermCycleNode model
(a required single-item linked field with a companion field) and its list
to the source: A <-> B and a self-reference, published. After round 0 the
new `cycles` check confirms each copy on the target points at its
partner's target ID, in both the link and the companion field.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@5PK
5PK deployed to cli-dev-live November 2, 2026 18:33 — with GitHub Actions Active
5PK added a commit that referenced this pull request Nov 2, 2026
…ge title fix

Conflict in process-page.ts: #255's PROD-3043 kept the source title only
for dynamic pages; #253 keeps it for every page (null as ""), since the
Management API's title is optional. Took #253's, and updated PROD-3043's
static-page test to match: an untitled static page stays untitled.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@5PK
5PK deployed to cli-dev-live November 2, 2026 18:41 — with GitHub Actions Active
…t sync

The first sync pulls the source through the Fetch API's preview sync
feed, which lists newly published items a minute or two after the
Management API save. In CI the fixture's items were not in the feed yet,
so round 0 synced 27 items instead of 30 and the cycles check failed on
items that were never pulled. After seeding, read the same feed (with the
source's preview key) until it lists every fixture item, up to 10 minutes,
and report how long it took.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@5PK
5PK deployed to cli-dev-live November 2, 2026 18:44 — with GitHub Actions Active
@5PK
5PK deployed to cli-dev-live November 2, 2026 18:50 — with GitHub Actions Active
@5PK

5PK commented Nov 2, 2026

Copy link
Copy Markdown
Collaborator Author

Closing: everything here is in #254 (release/1.3.0-beta.0), which takes these changes to main for the 1.3.0 beta. This branch has no commits that #254 doesn't. The description and review threads here still apply.

🤖 Generated with Claude Code

@5PK 5PK closed this Nov 2, 2026

This branch was successfully deployed

1 active deployment
cli-dev-live — df3988fb Deployed Nov 2, 2026 by 5PK via Release leases #16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants