PROD-3078: keep target-only edits to templates, URL redirections and list order

README.md

@@ -118,7 +118,7 @@ agility sync [options]
 
 | Option          | Type    | Default      | Description                                                                                                                                                                                                                                           |
 | --------------- | ------- | ------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| `--overwrite`   | boolean | `false`      | Conflict-scoped override. By default, when a target item has its own changes that conflict with the source (both sides changed since the last sync), the CLI skips it as a conflict to avoid data loss. An item changed only on the target, with the source unchanged, is kept with a warning instead (listed under "Target changes kept" and as `targetChangeKept` warnings in `--jsonSummary`); it isn't a conflict, and a later source change to it is. This applies to content, pages, models, containers, assets and galleries. With `--overwrite`, both kinds are overwritten with the source version. Non-conflicting updates are applied either way. Also covers a mapped target item that no longer exists (deleted or unpublished on the target): by default that is reported as a conflict and skipped, with `--overwrite` the item is recreated and the mapping repointed. |
+| `--overwrite`   | boolean | `false`      | Conflict-scoped override. By default, when a target item has its own changes that conflict with the source (both sides changed since the last sync), the CLI skips it as a conflict to avoid data loss. An item changed only on the target, with the source unchanged, is kept with a warning instead (listed under "Target changes kept" and as `targetChangeKept` warnings in `--jsonSummary`); it isn't a conflict, and a later source change to it is. This applies to content, pages, models, containers, assets, galleries, page templates, URL redirections and list order (page templates and URL redirections have no version, so the CLI stores a hash of each side in the mapping to tell which one changed; a mapping from an older CLI has none, so the first sync after upgrading lets the source win and records them). A page template kept on the target holds back the pages that use it only when it has no section for one of their zones. With `--overwrite`, both kinds are overwritten with the source version. Non-conflicting updates are applied either way. Also covers a mapped target item that no longer exists (deleted or unpublished on the target): by default that is reported as a conflict and skipped, with `--overwrite` the item is recreated and the mapping repointed. |
 | `--syncDeletes` | boolean | `false`      | After syncing, delete from the target the entities that were deleted in the source since the last sync (URL redirections, pages, content, containers, templates, models, assets, galleries — dependents first). Only entities the CLI previously synced (that have a mapping) are considered; nothing created directly on the target is touched. A target entity edited since the last sync is reported as a conflict and kept unless `--overwrite` is also set. An entity that something kept on the target still uses (a child page, a page or item that links to the content, a container's items, ...) is kept and reported. Content and page deletes are confirmed against the source first, so an item that was only unpublished is never deleted. Cannot be combined with `--pages`, `--containers`, `--models` or `--models-with-deps`; use `--elements` to limit the types. Works with `--preflight`. Not read from `.env`. See [Syncing deletes](#syncing-deletes). |
 | `--allowMassDeletes` | boolean | `false` | With `--syncDeletes`: allow a run to delete more than half (and more than 10) of the mapped entities of one type. Without it such a run deletes nothing and fails, because a stale or partial local cache looks the same as a mass delete. |
 | `--autoPublish` | string  | _(disabled)_ | Automatically publish synced items that were published in the source instance. Values: `content`, `pages`, `both`. If flag is provided without a value, defaults to `both`. Items that are only in staging (not published) in the source are skipped. |
@@ -194,6 +194,7 @@ A sync puts each synced content list on the target in the source's item order. I
 - The new order applies to the target's staging and live list at once. No item gets a new version, so nothing needs publishing for the order to show.
 - Items that exist only on the target keep their relative order after the synced ones.
 - A scoped run (`--pages`, `--containers`, `--models`) only orders the lists its content is in, but sends every synced item of those lists, so their order stays whole.
+- Each run records, per list, the source's order and the order it left on the target. A list reordered on the target since then, while the source's order didn't change, keeps the target's order with a warning ("Target changes kept"); reordered on both sides, it's a conflict. `--overwrite` sets the source's order either way. Adding or deleting an item isn't a reorder, and neither is a list that still shows the order the last sync replaced (the Fetch API shows an order change a few minutes later; the list is skipped until it does). The first sync after upgrading has no record, so it sets the source's order, as before, and records it.
 - `--preflight` lists the lists it would reorder under a `List order` phase, and `--jsonSummary` reports them as the `listOrder` phase.
 
 ### Auto-Publish

ci/README.md

@@ -37,17 +37,23 @@ directions:
 
 - **Odd rounds** edit A and `sync` A → B.
 - **Even rounds** edit B and `reverse-sync` B → A.
-- **Every round** also edits the other instance (drift) and one item on both sides (a conflict).
+- **Every round** also edits the other instance (drift: a content item, a list's order, a page
+  template's section name and a URL redirection's destination) and one item on both sides (a
+  conflict). Before syncing, the run waits for the Fetch API to show each edit, a list's new order
+  included (it shows up later than a new version). The redirection is one the run adds to A before the first sync; the Management SDK can
+  only delete redirections, so the run saves it through the Management API's batch endpoint and
+  waits for the Fetch API to list each edit, as the CLI does.
 
 The changes, in `ci/permutation/changes.mjs`, are discovered from whatever each instance holds:
 
 | Area       | Changes                                                                                            |
 | ---------- | -------------------------------------------------------------------------------------------------- |
 | Content    | edit, add, add a linked item together with an item that links to it, delete, publish and unpublish |
-| Ordering   | swap two items in a list; reverse the components in a page zone                                    |
+| Ordering   | move a list's last item to the top (as drag and drop does, no new versions); reverse the components in a page zone |
 | Pages      | edit title and menu text, add a page, delete it later                                              |
 | Components | remove one from a zone                                                                             |
 | Models     | add a field, change a label, rename a field                                                        |
+| Templates  | rename a section's display name                                                                    |
 
 The `quick` tier runs a fixed set covering every area in two rounds. `beta` (4 rounds) and `nightly`
 (6 rounds) pick changes at random from a seed printed in the summary. Replay one with
@@ -61,6 +67,11 @@ every sync it checks:
   conflict;
 - **drift:** an edit made only on the receiving side is overwritten, or kept with a "target changed"
   warning (not a conflict), never silently kept;
+- **drift:listOrder, drift:template, drift:urlRedirection:** a list reorder, a template edit or a
+  redirection edit made only on the receiving
+  side is kept with a "target changed" warning, or reported as a conflict when the round's own changes
+  touched the same list or template. These have no version, so the sync tells which side changed
+  from what it recorded at the last sync (PROD-3078); overwriting them without `--overwrite` fails;
 - **resolve:** when the sync kept such edits, the same sync with `--overwrite` settles them, as an
   editor would, before the checks below;
 - **verify:** `agility verify` finds no differences. A verify that finds some is tried once more a

ci/permutation/changes.mjs

@@ -206,21 +206,39 @@ export const CHANGES = {
     return { area: "content", change: "delete", detail: `#${id}` };
   },
 
+  // A reorder as the Manager's drag and drop makes it: the list item-order endpoint renumbers the
+  // list in place, with no new versions. The last item moves to the top. (Saving items with
+  // swapped itemOrder values doesn't reliably reorder: the server can leave the order as it was.)
   async "content.reorder"(ctx) {
-    const l = ctx.r.pick(await listsWithItems(ctx.c, ctx.guid, ctx.locale, 2));
+    // `ctx.avoidLists`: lists the round already reordered on the other side. The same move on both sides
+    // leaves them in the same order again, which is no change to report.
+    const avoid = new Set((ctx.avoidLists ?? []).map((x) => x.toLowerCase()));
+    const candidates = (await listsWithItems(ctx.c, ctx.guid, ctx.locale, 2)).filter(
+      (x) => x.items.length <= 30 && !avoid.has(x.list.referenceName.toLowerCase())
+    );
+    const l = ctx.r.pick(candidates);
     if (!l) return null;
-    const [a, b] = [l.items[0], l.items[l.items.length - 1]];
-    const fa = await ctx.c.contentMethods.getContentItem(a.contentID, ctx.guid, ctx.locale);
-    const fb = await ctx.c.contentMethods.getContentItem(b.contentID, ctx.guid, ctx.locale);
-    const oa = fa.properties.itemOrder;
-    fa.properties.itemOrder = fb.properties.itemOrder;
-    fb.properties.itemOrder = oa;
-    await save(ctx.c, ctx.guid, ctx.locale, fa);
-    await save(ctx.c, ctx.guid, ctx.locale, fb);
+    const readOrder = async () => {
+      const full = await Promise.all(
+        l.items.map((x) => ctx.c.contentMethods.getContentItem(x.contentID, ctx.guid, ctx.locale))
+      );
+      return full
+        .map((x) => [x.contentID, x.properties?.itemOrder ?? Number.MAX_SAFE_INTEGER])
+        .sort((a, b) => a[1] - b[1] || a[0] - b[0]);
+    };
+    const before = (await readOrder()).map(([id]) => id);
+    const wanted = [before[before.length - 1], ...before.slice(0, -1)];
+    await ctx.c.contentMethods.setContentListItemOrder(l.list.referenceName, ctx.guid, ctx.locale, wanted, true);
+    const after = await readOrder();
+    if (after.map(([id]) => id).join() !== wanted.join()) {
+      throw new Error(`${l.list.referenceName} is in order ${after.map(([id]) => id).join(", ")}, not ${wanted.join(", ")}`);
+    }
     return {
       area: "ordering",
-      change: "list-swap",
-      detail: `${l.list.referenceName} #${a.contentID} ↔ #${b.contentID}`,
+      change: "list-reorder",
+      detail: `${l.list.referenceName} #${wanted[0]} moved to the top`,
+      entity: l.list.referenceName,
+      orders: { guid: ctx.guid, items: after },
     };
   },
 
@@ -234,6 +252,25 @@ export const CHANGES = {
     return { area: "content", change: published ? "unpublish" : "publish", detail: `#${it.contentID}` };
   },
 
+  // A section's display name, not its reference name: a page's zones stay where they are.
+  async "template.section-rename"(ctx) {
+    const all = await ctx.c.pageMethods.getPageTemplates(ctx.guid, ctx.locale, true);
+    const t = ctx.r.pick(all.filter((x) => x.pageTemplateID > 0));
+    if (!t) return null;
+    const full = await ctx.c.pageMethods.getPageTemplate(ctx.guid, ctx.locale, t.pageTemplateID);
+    const s = full?.contentSectionDefinitions?.[0];
+    if (!s) return null;
+    const base = String(s.pageItemTemplateName ?? s.pageItemTemplateReferenceName).replace(/ \(r\d+[st]\)$/, "");
+    s.pageItemTemplateName = `${base} (${tag(ctx)})`;
+    await ctx.c.pageMethods.savePageTemplate(ctx.guid, ctx.locale, full);
+    return {
+      area: "templates",
+      change: "section-rename",
+      detail: `template ${full.pageTemplateName} section ${s.pageItemTemplateReferenceName}`,
+      entity: full.pageTemplateName,
+    };
+  },
+
   async "page.edit"(ctx) {
     const p = ctx.r.pick(await pages(ctx.c, ctx.guid, ctx.locale));
     if (!p) return null;
@@ -377,6 +414,14 @@ export async function drift(ctx, avoid = new Set()) {
   return CHANGES["content.edit"]({ ...ctx, avoid });
 }
 
+// Drift on the types with no version to compare (PROD-3078): the sync tells which side changed from
+// what it recorded at the last sync. `kind` is the --jsonSummary warning type of a kept change, and
+// `phase` the preflight phase a conflict is reported under.
+export const STRUCTURAL_DRIFTS = [
+  { kind: "listOrder", phase: "List order", change: "content.reorder" },
+  { kind: "template", phase: "Templates", change: "template.section-rename" },
+];
+
 // The same item edited on both sides with different values: a conflict. `avoidDstId` is the item
 // the round's drift edited, so the two checks grade different items.
 export async function conflict(srcCtx, dstCtx, mappingLookup, avoidDstId = null) {

ci/permutation/fixtures.mjs

@@ -222,6 +222,40 @@ export async function waitForFetchVersions(token, guid, locale, wanted, { timeou
   throw new Error(`Fetch API still behind after ${timeoutMs / 1000}s on items ${behind.slice(0, 10).join(", ")}`);
 }
 
+// An order change reaches the Fetch API's sync feed later than the item's new version. Wait until
+// the feed shows each given item at the given itemOrder. Returns the seconds waited; throws on timeout.
+export async function waitForFetchItemOrder(token, guid, locale, items, { timeoutMs = 10 * 60 * 1000 } = {}) {
+  if (!items.length) return 0;
+  const { mgmt, api } = hosts(guid);
+  const keyRes = await fetch(`${mgmt}/oauth/GetPreviewKey?guid=${guid}`, {
+    headers: { Authorization: `Bearer ${token}`, "Cache-Control": "no-cache" },
+  });
+  if (!keyRes.ok) throw new Error(`GetPreviewKey: HTTP ${keyRes.status}`);
+  const apiKey = (await keyRes.text()).replace(/^"|"$/g, "");
+
+  const started = Date.now();
+  let behind = [];
+  while (Date.now() - started < timeoutMs) {
+    // The feed is a log: the last entry for an item is its current state
+    const order = new Map();
+    let syncToken = 0;
+    for (let page = 0; page < 100; page++) {
+      const res = await fetch(`${api}/${guid}/preview/${locale}/sync/items?syncToken=${syncToken}&pageSize=500`, {
+        headers: { APIKey: apiKey, "Cache-Control": "no-cache" },
+      });
+      if (!res.ok) throw new Error(`Fetch API sync/items: HTTP ${res.status}`);
+      const body = await res.json();
+      for (const item of body.items ?? []) order.set(item.contentID, item.properties?.itemOrder);
+      if (!body.items?.length || !body.syncToken || body.syncToken === syncToken) break;
+      syncToken = body.syncToken;
+    }
+    behind = items.filter(([id, o]) => order.get(id) !== o).map(([id]) => id);
+    if (behind.length === 0) return Math.round((Date.now() - started) / 1000);
+    await new Promise((r) => setTimeout(r, 15_000));
+  }
+  throw new Error(`Fetch API still shows the old order after ${timeoutMs / 1000}s for items ${behind.join(", ")}`);
+}
+
 // After a sync: each target copy's Next (and NextID) must hold the TARGET ID of its partner.
 export async function checkReferenceCycles(c, guid, locale, fixture, toTarget) {
   const problems = [];
@@ -240,3 +274,89 @@ export async function checkReferenceCycles(c, guid, locale, fixture, toTarget) {
   }
   return problems;
 }
+
+// ── URL redirections (PROD-3078) ──────────────────────────────────────────────────────────────
+// The Management SDK can only delete redirections, so these call the endpoints the CLI uses: the
+// Management API's batch save to write, and the Fetch API's redirection list (what a pull reads).
+
+async function previewKey(token, guid) {
+  const { mgmt } = hosts(guid);
+  const res = await fetch(`${mgmt}/oauth/GetPreviewKey?guid=${guid}`, {
+    headers: { Authorization: `Bearer ${token}`, "Cache-Control": "no-cache" },
+  });
+  if (!res.ok) throw new Error(`GetPreviewKey: HTTP ${res.status}`);
+  return (await res.text()).replace(/^"|"$/g, "");
+}
+
+const sameOrigin = (a, b) => String(a ?? "").trim().toLowerCase() === String(b ?? "").trim().toLowerCase();
+
+// The instance's redirections as the Fetch API lists them.
+export async function listRedirections(token, guid) {
+  const { api } = hosts(guid);
+  const res = await fetch(`${api}/${guid}/preview/urlredirection`, {
+    headers: { APIKey: await previewKey(token, guid), "Cache-Control": "no-cache" },
+  });
+  if (!res.ok) throw new Error(`Fetch API urlredirection: HTTP ${res.status}`);
+  const body = await res.json();
+  return body.urlRedirections ?? body.items ?? [];
+}
+
+// Create (urlRedirectionID 0) or replace one redirection; returns its ID. A refusal is thrown.
+export async function saveRedirection(token, guid, r) {
+  const { mgmt } = hosts(guid);
+  const res = await fetch(`${mgmt}/api/v1/instance/${guid}/url-redirections`, {
+    method: "POST",
+    headers: { Authorization: `Bearer ${token}`, "Content-Type": "application/json", "Cache-Control": "no-cache" },
+    body: JSON.stringify([
+      {
+        urlRedirectionID: r.urlRedirectionID ?? 0,
+        originUrl: r.originUrl,
+        destinationUrl: r.destinationUrl,
+        httpCode: r.statusCode ?? 301,
+        destinationLocale: r.destinationLocale ?? null,
+        originLocales: r.originLocales ?? null,
+        content: r.content ?? null,
+      },
+    ]),
+  });
+  if (!res.ok) throw new Error(`url-redirections save: HTTP ${res.status} ${(await res.text()).slice(0, 200)}`);
+  const out = await res.json();
+  const done = out.created?.[0] ?? out.updated?.[0];
+  if (!done) throw new Error(`url-redirections save skipped it: ${out.skipped?.[0]?.reason ?? JSON.stringify(out)}`);
+  return done.urlRedirectionID ?? r.urlRedirectionID;
+}
+
+// Wait until the Fetch API lists the redirection with this origin in the wanted state.
+export async function waitForRedirection(token, guid, originUrl, test, { timeoutMs = 10 * 60 * 1000 } = {}) {
+  const started = Date.now();
+  while (Date.now() - started < timeoutMs) {
+    const found = (await listRedirections(token, guid)).find((x) => sameOrigin(x.originUrl, originUrl));
+    if (found && test(found)) return Math.round((Date.now() - started) / 1000);
+    await new Promise((r) => setTimeout(r, 10_000));
+  }
+  throw new Error(`redirection ${originUrl} not in the wanted state in the Fetch API after ${timeoutMs / 1000}s`);
+}
+
+// One redirection on the source before the first sync, for later rounds to edit.
+export async function seedRedirection(token, guid, seed) {
+  const originUrl = `~/perm-redirect-${seed}`;
+  const destinationUrl = "~/";
+  await saveRedirection(token, guid, { originUrl, destinationUrl });
+  const waited = await waitForRedirection(token, guid, originUrl, (x) => x.destinationUrl === destinationUrl);
+  return { originUrl, waited };
+}
+
+// Point the redirection somewhere new on one instance, as an editor would; waits for the Fetch API.
+export async function editRedirection(token, guid, originUrl, tag) {
+  const current = (await listRedirections(token, guid)).find((x) => sameOrigin(x.originUrl, originUrl));
+  if (!current) return null;
+  const destinationUrl = `~/perm-${tag}`;
+  await saveRedirection(token, guid, { ...current, urlRedirectionID: current.id, destinationUrl });
+  const waited = await waitForRedirection(token, guid, originUrl, (x) => x.destinationUrl === destinationUrl);
+  return {
+    area: "redirections",
+    change: "edit",
+    detail: `${current.originUrl} → ${destinationUrl} (in the Fetch API after ${waited}s)`,
+    entity: current.originUrl,
+  };
+}