From 0ea82031ca62f617dbf777dd4466b8876bf4d777 Mon Sep 17 00:00:00 2001 From: Myasnikov Daniil Date: Tue, 13 Oct 2026 17:27:38 +0500 Subject: [PATCH] docs(platform-migrations): propose a slug-keyed platform migration engine Platform migrations are identified by a dense integer that also serves as cluster state, so concurrent migration PRs conflict on the next number, a scalar stamp cannot describe a branched history, and the pending set is computed away from the migrations themselves. Replace the integer with a slug as identity and an applied-set ledger: one m. key per migration in the cozy-system/cozystack-version ConfigMap, carrying state, time, script sha256, batch, image and attempts. What runs is the set difference between what the chart ships and what the ledger records, so a skipped release, a failed run or a deleted script leaves a record instead of a silent gap. Execution order is derived at build time from the release tag trees into a generated, untracked order.d/. Migrations are pre-apply (the blocking Helm hook) or background (serial Jobs run by the operator with capped, backed-off retries). The proposal covers revocation, a retention floor that refuses an upgrade from below it, fresh-install seeding, conversion of the existing integer set through a legacy map, a migration contract with a linter and a generator, the test layers, and the phases of the rollout. Assisted-By: LLM Signed-off-by: Myasnikov Daniil --- .../platform-migrations/README.md | 556 ++++++++++++++++++ 1 file changed, 556 insertions(+) create mode 100644 design-proposals/platform-migrations/README.md diff --git a/design-proposals/platform-migrations/README.md b/design-proposals/platform-migrations/README.md new file mode 100644 index 00000000..7359b379 --- /dev/null +++ b/design-proposals/platform-migrations/README.md @@ -0,0 +1,556 @@ +# Platform migration engine + +- **Title:** `Platform migration engine: slug-keyed migrations, an applied-set ledger, and an authoring contract` +- **Author(s):** `@myasnikovdaniil` +- **Date:** `2026-08-12` +- **Status:** Draft + +## Overview + +Platform migrations are identified by a dense global integer that also serves as cluster state. Every new migration must claim the next number, which makes concurrent migration PRs conflict by construction; the single scalar recording progress cannot describe a branched history, which makes migrations unbackportable; and the pending set is computed from a number maintained in a different file than the migrations themselves, which makes silent skips possible. + +This proposal replaces the integer with a slug as identity and an applied-set ledger, splits execution into a blocking tier and a background tier, and puts a machine-checkable contract around migration authoring. Slot contention disappears, backports become safe by construction, and the "migration silently did not run" failure stops being expressible. + +## How it works + +A migration is a shell script under `pre-apply/` or `background/` whose filename is a slug and nothing else: `redis-failover-group-label`. That slug is its identity for the life of every cluster — it is the key a cluster records when the migration runs, it is the same on every branch, and it never changes. Two people adding migrations at the same time pick two different slugs and touch nothing in common, so there is no number to claim and nothing to renumber on a rebase. + +At build time CI derives one file per release into `order.d/`, listing the slugs that release is the first to ship, by reading the tags' own trees rather than any committed state. Those files only ever accumulate, so reading them in order gives the execution order, and each release's list begins with the previous release's list unchanged. That is what makes clusters agree on the blocking tier along one release line: a cluster upgrading 1.7.0 → 1.7.2 in one hop walks the same list in the same order as one that went through 1.7.1, and background migrations are first attempted in that list order among themselves (§3 says what the two tiers do and do not share, and where two lines differ). A migration written long ago but merged late sits at the end, where the release that shipped it put it, rather than in the middle where its authoring date would have put it. + +On the cluster, ConfigMap `cozy-system/cozystack-version` gains one key per migration that has a recorded outcome, and no key is ever removed. The runner reads the list once, skips anything already recorded, revoked or retired, and runs the rest — no sorting, no dependency resolution and no version arithmetic, because the build settled all three. A blocking migration runs in the existing `pre-upgrade` hook Job before the platform chart applies; a non-blocking one is run afterwards by cozystack-operator, one Job each. A fresh install records every slug it ships as skipped rather than running it, which is what a new cluster does today. + +## Scope and related proposals + +- [`kubernetes-nodes-split`](../kubernetes-nodes-split/) — its Phase 2 migration is used here as a worked example of the blocking tier. No ordering dependency between the two. +- Phase 4 below sketches two mechanisms for reducing the *rate* at which migrations are needed (admission-time defaulting, API-layer conversion). Each is large enough to deserve its own proposal; they are included here only to show where the migration count goes over time. +- Carrying migrations out of the monorepo alongside their package would need a scoping mechanism this proposal deliberately does not offer; §9 records why package scoping was withdrawn. Nothing here depends on decomposition work. + +## Context + +Migrations live in `packages/core/platform/images/migrations/` and are delivered as a Helm `pre-upgrade,pre-install` hook Job from `packages/core/platform/templates/migration-hook.yaml`, running with `cluster-admin`. All migration scripts are baked into a single image, pinned by digest in `packages/core/platform/values.yaml`. + +`run-migrations.sh` walks `seq $CURRENT_VERSION $((TARGET_VERSION - 1))`. `CURRENT_VERSION` comes from a scalar in ConfigMap `cozy-system/cozystack-version`; `TARGET_VERSION` is hand-maintained in `values.yaml`. The chart's `templates/cozystack-version.yaml` creates that ConfigMap on first install, and the hook only renders when `currentVersion < targetVersion`, so a cluster with nothing pending never starts a pod. + +At the time of writing there are 59 migrations on disk and `targetVersion` is 60. That count is a snapshot and this document does not keep it current — it is quoted because the scale is what the argument rests on, and every count below reads the same way. Statements that define the design never carry it: the set the conversion in §10 acts on is the integer set as of the Phase 1 freeze, whatever its size is then. + +### The problem + +One integer carries both a migration's identity and the cluster's progress through the sequence. Three consequences follow. + +**Slot contention.** Every new migration must claim the next integer, so every pair of concurrent migration PRs conflicts. [#3406](https://github.com/cozystack/cozystack/pull/3406), [#3379](https://github.com/cozystack/cozystack/pull/3379) and [#3315](https://github.com/cozystack/cozystack/pull/3315) all added `migrations/54` when this document was written. All three have since merged, as `54`, `55` and `56`, two of them renumbering on the way in, and `targetVersion` went to 57 with them. Renumbering is not a rename — it also means re-checking `targetVersion` and cross-branch alignment. The contention resolved itself the expensive way, in three PRs, while a document about it sat in review. + +**Backports are unsafe.** A scalar high-water mark can only express "everything below N is done", which holds only if the sequence is byte-identical on every branch. Branching voids that invariant. [#3534](https://github.com/cozystack/cozystack/issues/3534) is the realised failure: slot 45 holds a different migration on `release-1.5` than on `release-1.6`, so a cluster taking the normal patch-then-minor path is stamped past `release-1.6`'s Kubeadm keep-policy pin and never runs it. Nothing reports this class of divergence; it was found by hand while auditing backports. + +**The work list lives in a different file than the migrations.** Nothing derives the pending set from the migrations on disk. `hack/check-migrations-target.sh` exists solely to catch the resulting off-by-one — a lint compensating for a design flaw. + +A fourth cost is unbilled. Migration `43` matched the owning Helm release against the literal string `seaweedfs-system`, so it skipped every SeaweedFS instance not named `seaweedfs` — and `SeaweedFS` is a user-creatable kind, so an instance `foo` is owned by release `foo-system`. Those tenants never received `helm.sh/resource-policy: keep`, and the post-split `foo-system` upgrade pruned their `Cluster`. CNPG takes the PVC with the Cluster, so the filer metadata and every object in that tenant's S3 became unreachable. The migration was then edited in place, which fixed clusters that had not yet run it and did nothing for those that had — so migration `53` had to be written to repair them. + +## Goals + +- Two concurrent migration PRs never contend for the same identifier. +- A migration backported to a maintenance branch cannot cause a forward-upgrade skip. +- The pending set is derived from the migrations on disk, not from a separately maintained number. +- A migration that must run before the chart applies keeps that guarantee; one that does not, stops blocking upgrades. +- Authoring a migration follows a contract CI can check, rather than convention rediscovered per author. +- Obsolete migrations can be deleted rather than shipped forever. + +### Non-goals + +- Rewriting the existing integer migrations' logic. They are already applied on essentially every cluster; their content is frozen. They are renamed, but only mechanically, in Phase 5. +- Moving migrations out of shell. The operations are `kubectl` calls, and shell keeps them reviewable and testable against fixtures. +- Replacing the `cozystack-version` ConfigMap with a new API kind. It is extended in place. + +## Design + +### 1. Identity and layout + +Identity is the **slug** and nothing else: `redis-failover-group-label`. No date, no sequence number, no hash in the filename. The slug is the ledger key and is the one thing about a migration that may never change. + +That is forced rather than chosen. §3 shows that execution order cannot be derived from anything an author knows while writing the migration, so an ordinal in the name is a claim the name cannot keep, and maintaining it costs a rename on every stale rebase. Removing it also removes the only reason two concurrent PRs would ever have to look at each other. + +Grammar is `^[a-z0-9]+(-[a-z0-9]+)*$`, at most 47 characters, so that the re-run Job `migration-rerun-` of §7 fits the 63-character limit on a Job name, unique across **both** tiers — the ledger key has no tier dimension, and a migration may be moved between tiers while its PR is open. Deriving identity from a path instead would mint a fresh identity on every move. + +Tier is the **directory**, so nothing about tiering is parsed at render time; the converted integer set of §10 is the one exception and is `pre-apply` by definition. + +The integer set does not survive alongside the new one. It is carried untouched only while the conversion is staged, and §10 renames all of it, after which there is one identity scheme, one runner path and nothing a contributor has to know about integers. Leaving them in place permanently is explicitly rejected: it would mean a dual-mode runner forever, `targetVersion` forever, and every new contributor learning a deprecated scheme next to the live one in order to add a migration. + +During the transition: + +``` +packages/core/platform/images/migrations/migrations/ + 1 .. N # the integer set as of the Phase 1 freeze — then frozen, never extended, converted by §10 + lib/ + revoked # slugs that must not run (§8) + order.d/ # release-assigned execution order (§3); generated at build, not in git + 00001-v1.6.0 + 00002-v1.7.0-rc.1 + pre-apply/ # blocking, runs in the pre-upgrade hook Job + redis-failover-group-label + background/ # non-blocking, run by cozystack-operator + clickhouse-keeper-pvc-labels +``` + +After the conversion, which is the shape to design against: + +``` +packages/core/platform/images/migrations/migrations/ + lib/ + revoked + order.d/ # generated at build, not in git (§3) + 00000-legacy # the converted integer set, from legacy-map + 00001-v1.6.0 + … + mariadb-operator-secrets # was `1`, renamed in place beside lib/ (§10) + … # one file per converted integer + pre-apply/ + redis-failover-group-label + background/ + clickhouse-keeper-pvc-labels +``` + +### 2. Tiers + +The two tiers already exist informally, written into the migration headers. Migrations `48`, `49` and `51` each carry the same sentence: *"Best-effort by design: the worst case if a relabel is skipped is the pre-existing bug this PR set out to fix, which is strictly no worse than the status quo."* That is the background criterion, reasoned about case by case in prose. This proposal makes it declarable and executable. + +**`pre-apply` is the default**, and the test is not "is this a backfill" — it is *can something be named that breaks if this runs late?* Chart render guards, operator label selectors and Helm prunes all count: + +| migration | what depends on it having already run | +|---|---| +| `56` ([#3315](https://github.com/cozystack/cozystack/pull/3315)) worker pools | creates the child `kubernetes-nodes--` HelmReleases before the kubernetes chart stops rendering worker objects. Skip it and Helm prunes the tenant's running worker VMs. | +| `55` ([#3379](https://github.com/cozystack/cozystack/pull/3379)) FluxCD addon removal | suspends and drops the finalizer on tenant Flux HelmReleases so the delete does not uninstall Flux *inside* the tenant cluster. | +| `54` ([#3406](https://github.com/cozystack/cozystack/pull/3406)) RedisFailover | stamps the operator-group label and pins the engine image before redis-operator v3.3.5 rolls. An unlabelled CR never reaches the new operator's informer at all — no event, no error. | +| `46` k8s version bump | patches `kuberneteses.apps.cozystack.io` `spec.version` v1.30 → v1.31 **ahead of the chart upgrade**. The chart carries an explicit `fail()` guard on v1.30 in `templates/_versions.tpl`, so a tenant still on v1.30 when the platform upgrade lands gets a failed HelmRelease and the Talos worker rollover never starts. | +| `53` seaweedfs | hands `Cluster/seaweedfs-db` to the `-db` release before `-system` re-renders without it. CNPG takes the PVC with the Cluster. | +| `52` linstor-scheduler | deletes the old admission Deployment (immutable `spec.selector`) so the chart's new one can be created. | + +`46` is the instructive case: a plain data patch on a CR that looks exactly like a backfill, but a chart render guard depends on it. Anything whose failure mode is "a HelmRelease goes red" is `pre-apply`, however small the edit. + +`background` is for work whose only cost, if it lands late, is that a pre-existing bug persists a while longer: + +| migration | what it does, and what late costs | +|---|---| +| `51` | backfills `apps.cozystack.io/application.name` on VM/VL storage PVCs so the post-delete cleanup hook can select them. Late → the PVC leak it fixes persists. | +| `48` | same shape for pre-existing ClickHouse keeper PVCs ([#3057](https://github.com/cozystack/cozystack/issues/3057)). Late → the leak persists. | +| `49` | backfills `tenant.cozystack.io/` labels on tenant namespaces. The chart fix already makes every namespace self-compute its chain on next reconcile; the migration only *accelerates* healing for suspended or failed releases. | +| `44` | removes a leftover `flux-tenants` Deployment, and only once the shard has already drained. Explicitly defensive — flux-shard-operator retires it at runtime regardless. | + +Late includes running after a later release's `pre-apply` migrations, because the two tiers are not ordered against each other (§3). A background migration therefore has to stay correct on a cluster that has already taken them, and while one of them is running, because a background Job of one release can still be running or backing off when the next release's hook starts; that is the second half of the test for the tier, next to the question that decides `pre-apply`. + +### 3. Ordering + +Order is assigned when a release is cut, because that is the only moment the necessary fact exists. + +**Why nothing else can carry it.** Call the release that first shipped a migration its **shipping release**. Take two migrations of the same tier with different shipping releases, and a cluster that upgrades through the earlier one and later through the later one. On the first hop the earlier migration is pending and the later one is not in that image at all, so this cluster runs them in shipping-release order — it had no other option available to it. Every other cluster has to match, or they disagree. That is the entire requirement, and it can be stated in one line: + +> Within a tier, the execution order must never contradict shipping-release order. + +Nothing an author knows while writing a migration tells them its shipping release, because it depends on which release happens to be cut next and on whether the PR is still in review by then. A date, a name, a checksum of the contents, a random identifier, a declared dependency — none of these has any relationship to shipping-release order, so each of them agrees with it only by accident. The failure is not hypothetical: + +> This has already happened in tree. [#3315](https://github.com/cozystack/cozystack/pull/3315) was opened on 2026-07-15, before [#3379](https://github.com/cozystack/cozystack/pull/3379) and [#3406](https://github.com/cozystack/cozystack/pull/3406), and merged on 2026-08-26, after both. It shipped as migration `56` while those two took `54` and `55`. Under any authoring-time key — a date, a name, a hash — the relative order of `56` against the other two depends on which cluster is asked, and no dependency was declared because the author of #3315 never knew the other two would land first. + +**Is a total order needed at all?** Not proven, and this section should say so rather than let the machinery imply it. No migration in tree states an order requirement against another migration: the order-sensitive cases in §2 are ordered against the *chart* — `56` must create the child HelmReleases before the kubernetes chart stops rendering worker objects, `46` must patch `spec.version` before the chart's `fail()` guard sees v1.30 — and not against a sibling migration. What is demonstrably broken today is the other failure, a late-merged migration treated as already applied ([#3534](https://github.com/cozystack/cozystack/issues/3534)), and the slug-keyed ledger of §4 fixes that on its own with no ordering at all. + +So the batch mechanism is insurance, not a proven requirement, and it is kept on the strength of what it costs rather than what it has already prevented: an author picks no number, touches no file, waits on no slot and adds no line to any PR, because §3 is derived from tags the release already produces. The day a pair does need a declared order, the alternative to having this is finding out on a cluster. If that trade stops looking worth it, the thing to delete is this section, and §4 keeps working without it. + +**A dependency graph does not rescue this.** The idea is the natural one: declare edges between migrations, topologically sort the whole set once, and have every cluster run its pending migrations in that single order. It looks sound, because given one fixed list any two clusters that both run `X` and `Y` take their relative order from the same place. + +There is no one fixed list. Each release's image holds a different set of migrations, so each release produces a different sorted list. And sorting by declared dependencies does not produce one answer, it produces *an* answer: for a pair with no edge between them the dependency rule has no opinion about which comes first, and yet a list must put one of them somewhere. Some other rule has to decide, and in practice that rule is alphabetical order of the file names. Since almost no pair of migrations has a declared edge, that fallback rule is what actually produces the order. Worked through: + +- 1.7's image holds two `pre-apply` migrations, `etcd-crds-precreate` and `vm-disk-bus-default`, with no dependency declared between them. The fallback rule applies, so the list is alphabetical — `etcd-crds-precreate`, `vm-disk-bus-default` — and a cluster reaching 1.7 runs both. +- 1.8 adds a third `pre-apply` one, `vm-cloudinit-drive`, again with no edge. 1.8's list is now `etcd-crds-precreate`, `vm-cloudinit-drive`, `vm-disk-bus-default`: the new migration sorted into the **middle**. +- The cluster already at 1.7 has `vm-disk-bus-default` recorded, so at 1.8 it runs `vm-cloudinit-drive` *after* it. A cluster arriving at 1.8 from 1.6 runs the list as written and takes `vm-cloudinit-drive` *before* it. They disagree — and the graph played no part, because there were no edges to play one. + +So the requirement lands on the fallback rule, not on the dependencies: **adding a migration must only ever extend the order at its end, and never insert into its middle.** Alphabetical order of names does not do that, since a new name can land anywhere in the alphabet. Ordering by date does not either — that is the July PR inserting ahead of August. A batch number does, because a newly added migration always receives the newest batch, which is by definition at the end. + +**The manifest.** Order lives in `order.d/`, one file per release, and it is **generated rather than committed**: a migration's batch is the earliest tag whose tree already contains its slug. No commit carries it, and no author writes a line of it. What ships in the image and in the chart looks like this: + +``` +order.d/00002-v1.7.0-rc.1 +``` + +``` +# batch 00002, first shipped in v1.7.0-rc.1, generated by hack/generate-migration-order.sh +clickhouse-keeper-pvc-labels +vm-pool-adopt +etcd-crds-precreate +``` + +Bare slugs, one per line. Tier is not repeated — it is the directory the script lives in, and duplicating it here would let the two drift. + +**Within a batch, nothing needs assembling.** Two migrations first shipped by the same release are seen by every cluster in one pass off one image, so the agreement requirement says nothing about their relative order and any deterministic rule satisfies it. That collapses the build step to *reading which tag introduced each slug*: no reconstruction of merge order, no rank counter, no git archaeology. The rule is alphabetical order of the slugs, then reordered where a `requires` in the same batch demands it. + +**The prefix invariant, which is the whole guarantee in one check.** Because batches only ever append, each release's manifest extends the previous one's: every numbered file it shipped is in the next release unchanged, except that a line may gain `retired` (§11) and that `00000-legacy` appears once, when Phase 5 lands (§10). Deriving it from tags makes that structural rather than asserted: a batch is fixed by the first tag whose tree holds the slug, and that tag does not change afterwards, so there is no file for a later step to edit and no lint asserting that the first N lines did not move. One check is still worth running, because it catches what a generator regression or a moved tag that stays among this build's ancestors would break: fetch the manifest the previous release of this line published, the newest earlier tag in the walk of step 1, which its packages artifact carries (step 6 below), and compare it file by file with this release's, ignoring the `retired` flag and `00000-legacy`. It is the published copy and not a regeneration, because a regeneration reads the same tags this build does, and a moved tag changes both sides alike. The check has a limit: a tag moved off this commit's ancestry drops out of the walk of step 1 and the baseline falls back to an older release, so keeping tags immutable is for the repository's tag rulesets to enforce, and this check does not stand in for them. The comparison covers the numbered files only; the trailing untagged file described below exists on builds no tag has sealed yet and is left out of it. + +**What the build does.** `hack/generate-migration-order.sh`, run from the tagged checkout before the image and the chart are packaged. `tags.yaml`, which builds a tag, already checks out with `fetch-depth: 0` and `fetch-tags: true` in its `Checkout code` steps, so the history it needs is present: + +1. Walk this line's tags in release order and `git ls-tree` the two tier directories at each. A line's tags are the ones reachable from the checked-out commit. A pre-release tag is accepted only at the tip of `main` or `release-X.Y` (`tags.yaml` fails on an empty `base_ref`), and a stable tag is cut at the merge commit of the promote PR into the line branch, so every tag of a line is an ancestor of that line's branch. `v1.6.3`, for one, is reachable from `release-1.6` and not from `main`, which is why a backport tagged on a release branch does not place anything on `main`'s line. +2. A slug's batch is the first tag whose tree contains it, except for the slugs listed in `legacy-map` (§10), which are not derived: they belong to batch `00000` and are left out of the tag walk. Slugs sharing a batch are sorted alphabetically, then reordered to satisfy any `requires` edges among them, keeping alphabetical order everywhere those edges say nothing. +3. A slug present in an earlier tag's tree and absent from this one is emitted with `retired` (§11). So is a slug of `legacy-map` whose script is gone from the tree; its `legacy-map` line stays, because batch `00000` and `T` derive from that file, and CI rejects a change to a released line of it. +4. A slug on disk that no walked tag's tree contains is *untagged*. Untagged slugs go into one trailing file, `order.d/99999-untagged`, ordered like any other batch. The file is defined as the slugs on disk minus every slug of every walked tag and minus the slugs of `legacy-map`, so a slug is in exactly one file, and its zero-padded name sorts after every numbered batch, so the runner's glob walk needs no change. +5. Fail on a cycle, on a `requires` naming a slug that does not exist, on a `requires` pointing at a *later* batch, and on a `pre-apply` `requires` naming a `background` slug. Fail on a slug that no stable tag of any line carries and the tree no longer has, unless `revoked` lists it (this check reads all tags, not only the walk): no stable release shipped it, so the floor check of §11 would refuse clusters upgrading from a stable release for a migration they were never offered, and the message says to add the slug to `revoked`. Fail on a shallow clone (`git rev-parse --is-shallow-repository`) and on a clone with no tags, and fail the prefix check of the release build when the published artifact of the previous release in the walk cannot be fetched, rather than skipping the comparison; an artifact whose tree has no `pre-apply/` or `background/` directory and no `legacy-map` predates the generator, and the comparison is skipped for that one release, while one whose tree has any of them and no `order.d/` fails the build. A walk that sees too few tags does not put slugs in the untagged file, since a tag's own tree always carries its slugs; it places every slug in the newest batch, and a release must not ship that. +6. Write `order.d/` into the tree before the migrations image and the packages artifact are built, so both carry it. `order.d/` is gitignored, absent from any commit, and CI rejects a PR that adds one, so it does not travel between the three checkouts `tags.yaml` builds in: `build-arm64` builds its images in its own, `build-amd64` hands `prepare-release` a patch made with `git add --all` that leaves an ignored directory out, and `prepare-release` republishes the packages artifact from a fresh checkout plus that patch. The generator therefore runs in each of the three, after its checkout and before its builds, and each needs the full history and tags it reads. + +**Stable promotion.** `promote-rc.yaml` does not run the steps of `tags.yaml`. It checks out the RC staging branch, which has no `order.d/` because the directory is gitignored, `hack/promote-packages-artifact.sh` re-serializes the packages tree from that checkout, and `hack/verify-promoted-packages.sh` then requires the artifact's file list to equal the release tree's. Left alone, the stable packages artifact would carry no manifest and its chart would fail the render of §7. So the promote job runs the generator on its checkout before `promote-packages-artifact.sh`, and the verifier runs it on the release tree it compares against, which puts the same generated `order.d/` on both sides and keeps the file-list comparison meaningful. The promote checkout already has full history and tags (`fetch-depth: 0`, `fetch-tags: true`); a verifier job that checks out shallow has to fetch them first, since the generator fails on a shallow clone. The migrations image is the RC's, pinned by digest and unchanged by promotion, so it keeps the manifest the RC build generated. `verify-promoted-packages.sh` also requires `order.d/` in the artifact whenever the tree has `pre-apply/` or `background/` content or a `legacy-map`, because with the directory missing from both sides the file-list comparison passes and nothing else renders the promoted chart. + +**Untagged builds.** `build-main.yaml` on every merge, `build-release.yaml` on every backport and the PR e2e all build slugs that no tag carries yet, and all of them fetch the full history and tags; a local render from a checkout gets its manifest by running the same generator first, and the chart's unit tests bring a fixture tree instead (§7). Without step 4 the first PR that adds a migration would fail its own e2e under the fatal condition of §6, and the platform chart on `main` would stop rendering until the next cut. With it those builds get a manifest whose last file holds the not-yet-released slugs; the moment a tag is cut, the tag's own tree carries them and they move to a numbered batch, at the end, which is where they already were. Two limits follow and are stated rather than hidden. The file is regenerated on every build, so a slug added later can sort into its middle, and the agreement argument above holds for tagged releases only: two clusters running different untagged builds of `main` can execute a pair in different orders, and no upgrade path between untagged builds is promised. Nightlies are in that group: `nightly.yaml` republishes what `build-main` already built. And a ledger entry written from an untagged build keeps `99999-untagged` in its `batch` field after the slug has a numbered batch; the field is informational (§4) and nothing reads it. + +**Why generated and not committed.** A committed manifest needs a release-cut slot that lands on the branch the *next* cut is taken from, and no such slot exists. `tags.yaml` commits its release artifacts in the `Commit release artifacts` step, and the only push carrying them, in `Create release branch`, targets `release-${GITHUB_REF#refs/tags/v}` — a per-version staging branch the step's own comment calls mutable and force-updates. Nothing in that file pushes to `main` or to `release-X.Y`; content reaches a line branch only through the stable promote PR. So `v1.7.0-rc.1` would seal a batch onto `release-1.7.0-rc.1`, which neither `main` nor `release-1.7` ever contains, since `cut-prerelease.yaml` creates `release-1.7` at the tagged commit. A migration would then merge to `main`, and the next cut from either line, `v1.7.0-rc.2` from `release-1.7` or `v1.8.0-rc.1` from `main`, would find no batch file for rc.1, seal its own under the same number with a different set, and insert an entry into the middle of a batch already shipped in rc.1. A rule comparing batch files cannot catch it, because the earlier file is absent from the branch rather than different. + +Deriving from tags removes the slot instead of building one. Tags are immutable and the release workflows treat them that way, so a batch assignment cannot be edited, lost, or raced, and the rc case resolves correctly: rc.1's tree fixes the batch of everything it carries, and rc.2 assigns a later batch only to what rc.1 did not. + +**`requires`: who writes it, when, and what it can say.** It is written by the migration's author, in that migration's own header, at the time the migration is written, and it may only name a slug that **already exists in the tree** — CI rejects a dangling target. That bounds what it can express: a dependency on something already merged, never on something not yet written. Which is exactly why the scenario above cannot be fixed by declaring anything. + +What the edge does depends on where its target already sits. If the target shipped in an earlier release, the dependency is satisfied before the runner starts and `requires` acts purely as a check. If neither has shipped yet they land in the same batch, and the generator uses the edge to order them within it. Either way it is resolved when the manifest is generated, so the runner parses no headers and evaluates no dependencies. What remains is the value worth keeping: a declared dependency CI can verify, which fails loudly instead of running in the wrong order. It orders and does not guard: if the target fails, under `on-error=warn` or after a background retry cap, the dependent still runs, because the runner never evaluates `requires`, so a migration whose precondition can fail checks it itself. One direction is refused: a `pre-apply` migration may not `require` a `background` one, because the hook runs before the operator and the target may still be pending when it does, so "satisfied before the runner starts" does not hold for that pair. The other direction needs nothing, since a background migration always runs after the hook. + +**What is guaranteed, and what is not.** Every cluster upgrading along one release line runs any two migrations *of the same tier* in the same relative order, for first attempts: a background migration that failed is retried when its backoff elapses (§7) and may land after later slugs, which the rule below already tolerates. Across tiers nothing is promised, because the tiers run at different moments: `pre-apply` runs in the hook before the chart applies and `background` runs afterwards under the operator. A 1.7 background slug therefore runs after 1.8's `pre-apply` slugs on a cluster that goes 1.6 → 1.8 in one hop, and before them on one that went through 1.7 and let the operator finish. That is the cost of the tier boundary of §2, and the rule there is what pays for it: a background migration must stay correct whichever side it falls on. Cross-line, order may differ for exactly one class, and it is not a rare one: a migration backported to a maintenance branch, which is assigned a batch on that branch and a later batch on `main`. `cut-prerelease.yaml` creates `release-X.Y` at the minor's first rc and every later fix reaches it by cherry-pick, so the class covers every fix that lands after a minor's first rc. Trying to reconcile the two would mean splicing an entry into the middle of a batch `main` already shipped, which is precisely what the prefix invariant forbids — so it is not reconciled. That divergence is safe by the same reasoning that justified the backport: it is the maintainer's assertion that the migration is correct on a branch missing everything in between. Nothing here reports the affected pairs, and the proposal does not promise it. Today the same divergence exists and is invisible — [#3534](https://github.com/cozystack/cozystack/issues/3534) was found by hand. + +**A backport, worked through.** `kubeadm-keep-policy` is authored on `main` and is needed on the 1.5 line too, which by then carries Phase 1 (Rollout). + +1. The PR merges to `main`. No release has been cut since, so no batch names it on either branch. +2. `.github/workflows/backport.yaml` cherry-picks it onto `release-1.5`: same path, same slug, same bytes. **The author does nothing beyond the cherry-pick** — nothing to renumber, nothing to rename, no manifest to edit, because no tag carries the slug yet and there is no manifest in the tree to edit in any case. Until a tag is cut, both branches build it into their `99999-untagged` file. +3. `v1.5.4-rc.1` is cut from `release-1.5`. That branch's manifest gains a batch containing `kubeadm-keep-policy`. +4. `v1.7.0-rc.1` is cut from `main`. Main's manifest gains its own batch, also containing `kubeadm-keep-policy`, because that is the first tag on `main` whose tree carries it. + +Two clusters then differ in order and agree in effect: + +``` +1.5.3 → 1.5.4 → 1.7.0 mariadb-operator-secrets, kubeadm-keep-policy, monitoring-move +1.5.3 → 1.7.0 mariadb-operator-secrets, monitoring-move, kubeadm-keep-policy +``` + +`kubeadm-keep-policy` executes **exactly once** on both paths, because the ledger key is the slug and the slug is identical on both branches. That is [#3534](https://github.com/cozystack/cozystack/issues/3534) eliminated: the situation that today stamps a cluster past a migration it never ran. + +What differs is its position relative to `monitoring-move`, a 1.6 migration the 1.5 line never carried. Reconciling that would mean splicing the slug into `main`'s manifest at its 1.5.4 position — inside batches already shipped, which the prefix invariant forbids. So it is not reconciled, and the divergence is the backport's own premise: cherry-picking to `release-1.5` asserted the migration is correct on a branch holding none of 1.6. + +### 4. Ledger + +The existing ConfigMap is extended rather than replaced by a new kind. One key per applied migration, `m.`, whose value carries timestamp, script sha256, and outcome: + +```yaml +data: + version: "57" # legacy scalar, still advanced by integer migrations + m.redis-failover-group-label: | + {"state":"ok","at":"2026-08-12T10:04:00Z","sha256":"abcd…", + "batch":"00001-v1.6.0","image":"ghcr.io/…@sha256:…","attempts":1} + m.clickhouse-keeper-pvc-labels: | + {"state":"failed","at":"2026-08-14T09:11:00Z","sha256":"ef01…", + "batch":"00002-v1.7.0-rc.1","image":"ghcr.io/…@sha256:…","attempts":1, + "reason":"ScriptError","message":"3 of 812 PVCs rejected the label patch"} +``` + +The key is the bare slug. It carries no tier and no path, so moving a migration between `pre-apply/` and `background/` while its PR is open does not mint a second identity for the same work. + +**The value is structured, not a sentence.** Five things parse it — the runner, the operator, the render gate, the E2E assertion and the retention floor check (§11), which tests for a key and reads its `state` — and a positional string is how five slightly different parsers come into existence. It is JSON: `jq` is installed in the migrations image, so the runner can both read and build it without string surgery. `state` is one of `ok`, `legacy`, `revoked`, `skipped-fresh-install`, `failed`; `batch` records the manifest file §3 placed the migration in, by its full name, which includes the tag. It is informational: it names a file of one line's manifest, and nothing reads it back; `image` records what actually executed, which matters because a background migration pending since 1.7 runs under 1.8's image; `attempts` counts executions of that migration on this cluster and has two sources, the operator's capped background retry and the manual re-run, both in §7 — an entry is written with `1` and grows from there, which makes "this has already been tried three times" visible before anyone tries a fourth and is what the background cap is counted against. A failure additionally carries `reason` and a human `message`. The exact field set will move during implementation and that is fine — what is settled here is that the value is parsed, not read. + +One mechanical consequence worth stating because it is easy to get wrong: the value is a JSON document inside a YAML string inside a `kubectl patch --type merge` payload, so it must be built with `jq -n` and passed as data, never assembled by concatenating shell variables. + +**Nothing stored on the cluster is a batch number.** Whether a migration ran is decided by its slug and nothing else, and slugs are unique across every line, because §5 checks uniqueness against the whole tag history. A batch number, by contrast, is assigned per line: `release-1.6` and `main` number their batches independently, so `00005` can name different sets of slugs on the two, and every minor upgrade crosses that boundary. The ledger therefore holds no watermark and no counter, and no code compares a value read from the cluster with a batch number from the incoming manifest. A batch number means something only inside the manifest it came from, where it fixes an order; the `batch` field records a file name for a human to read. + +**Keys are never deleted.** Not on upgrade, not on downgrade, not when a migration is retired. An absent key then always means one thing, that this migration was never recorded on this cluster, and two later parts of the design lean on it: the retention floor check (§11) and rollback. A ledger entry is on the order of 250 bytes, a ConfigMap holds 1 MiB, and there are 59 migrations today, so growth is not a problem the design needs to solve now. If it becomes one, the fix is to fold the keys of retired slugs into a single list key. That rewrites them rather than removing them, so both the floor check and rollback keep working. A slug that exists only on another line's branch never appears in this line's manifest, and its key stays; it is harmless. + +A `failed` entry is a record like any other: it stays, it is never rewritten except by a retry or a re-run (§7), and it is visible for as long as the cluster lives. + +**Who owns it.** There is one ledger, and it belongs to whatever assembles the platform rather than to any package being assembled. Today that is the platform chart, where `templates/cozystack-version.yaml` creates the ConfigMap; if assembly later moves to `packages/core/installer` or whatever succeeds it, the ledger moves with it and nothing else in this proposal changes. Per-package ledgers are not adopted, and §9 is the reason: migrations stay platform-level, so there is no second owner to hand one to. + +Two properties make a ConfigMap the right container: + +- The render gate reads it with Helm `lookup`. `lookup` against a not-yet-Established CRD returns empty, which would resolve the current version to `0` and silently re-render the whole migration Job. A ConfigMap has no such bootstrap edge. +- `hack/update-codegen.sh` ends with a catch-all `mv` that would misroute a new `api/v1alpha1` CRD into the cozystack-controller chart, and both `internal/crdinstall/install_test.go` and `hack/e2e-install-cozystack.bats` hardcode the current two-CRD set. Downstream consumers (`terraform-provider-cozystack`, `ansible-cozystack`) hand-type this API group. + +**Write mechanics.** Ledger keys are written with `kubectl patch --type merge`, which is atomic and, because the body carries the `resourceVersion` of the read the decision was made on, conditional: the API server rejects it if the ConfigMap changed since, and the writer re-reads and decides again. Every record write has that form, since what a record may do depends on what the key already holds (§8). The existing `stamp_cozystack_version` in `migrations/lib/cozystack-version.sh` uses `kubectl apply` and must keep touching **only** `version`. Apply's three-way merge prunes keys that were in last-applied-configuration but absent from the new manifest; patch-written `m.*` keys were never in last-applied, so they survive — subtle enough to deserve a dedicated test. It is the same class of bug migration `42` was written to fix for the `no-delete` label. + +### 5. The migration contract + +Identity is only half the problem. A migration today is an unconstrained shell script: each author decides what failure means, which shell dialect to use, whether to stamp, and whether to test. Most of the fifty-nine have no test at all, though the six newest each shipped with a bats suite. `48`, `49` and `51` each independently hand-rolled "best-effort" by scattering `|| true`; `47` and `50` each hand-rolled fail-closed. None of that is machine-readable or checkable. + +Shell stays. The contract around it is formalised. + +**The runner owns the lifecycle; the migration owns only the change.** Migrations no longer stamp anything. They do not know their own slug, and there is no "next version number" to get wrong. + +**Declared metadata, not prose.** A header block the runner and the linter both parse: + +```sh +# cozystack-migration: tier=pre-apply # pre-apply | background +# cozystack-migration: on-error=abort # abort | warn +# cozystack-migration: requires=etcd-crds-precreate # optional; must already exist (§3) +``` + +`on-error=warn` turns the "best-effort by design" paragraph into something the runner enforces. The script is then written plainly fail-fast, and the runner decides what a non-zero exit means — instead of `|| true` per command, which also swallows the failures the author wanted to see. + +**A shared library, extended.** `migrations/lib/` already holds `cozystack-version.sh` and `seaweedfs-db-adopt.sh`, so the precedent exists. It grows helpers for operations that keep being re-implemented: a `kubectl` wrapper with retry on transient apiserver errors (what the `|| true` sites are really reaching for), a list helper that does not SIGPIPE under `pipefail` (migration `44` documents that trap in a comment), and the Helm-ownership adopt / `resource-policy: keep` pattern shared by `31`, `35`, `43`, `45`, `53`, `56` and `58` — by far the most repeated operation in the tree. It also grows a fleet-iteration helper: given an app kind, walk every instance of it and apply the same operation to each, carrying the errexit handling `lib/seaweedfs-db-adopt.sh` documents so one instance failing neither aborts the sweep silently nor passes unnoticed. That helper is the direct answer to the `43` class (§9) and is the one worth writing first. A released file in `lib/` is append-only, for the reason a released migration is immutable (§8): a migration sources its helpers, so changing a helper changes what an already-recorded migration does on every cluster that has not run it yet, with the script's own sha256 unchanged, which is the `43`/`53` split by another route (`43` delegates its work to `lib/seaweedfs-db-adopt.sh`). A function may be added, and a released one may not be changed or removed; behaviour that has to change goes into a new function that the migration needing it calls. The guard of §8 checks it: against the merge-base, a released `lib/` file may only gain lines at its end, and CI rejects a second definition of a function name that `lib/` already defines, since a later definition replaces the earlier one when the file is sourced. + +**A linter**, `hack/lint-migrations.sh`, wired into `make unit-tests`: filename matches the slug grammar of §1; header present, parseable, and declaring `tier` and `on-error`, with `tier` equal to the directory the file is in, since the directory is what the runner reads; one shell dialect (`#!/bin/bash`: the image is `alpine` with `bash` explicitly installed, so both dialects run today and the set is split 47 to 12 between `sh` and bash; the legacy files are frozen and cannot be restated, so the rule binds new migrations only, and bash is the choice because one pinned dialect takes ash's divergences off the author's list: they are caught in CI only by a suite that runs the script under the image's own busybox ash, as `hack/migration-54-redis-adopt.bats` does, and nothing makes every suite do that); `shellcheck` clean; `requires` targets exist, name no later batch and are not `background` slugs when the requirer is `pre-apply`; every `revoked` entry names a slug that is on disk or in the tag history; no direct writes to the ledger; and no tracked `order.d/` (checked against `git ls-files`, because the generator writes one into a working checkout), which is the rule that keeps §3's guarantee structural rather than cultural. Slug uniqueness is checked against the tag history rather than against the working tree, so a slug retired and deleted years ago cannot be taken by a new migration and silently skipped as already applied on every cluster that ran the original. A slug already in the tag history is accepted only with the bytes a tag carries for it, which is what a backport is, and only if this line does not flag it `retired`; any other bytes under a known slug fail the check. The release build repeats the comparison against the newest tags of the other lines, which catches two lines that added the same slug independently with different bytes, a case neither PR can see. The check includes the slugs of `legacy-map`, so a new migration cannot take the name of a converted one. The same comparison runs over the function definitions in `lib/` by name, since two lines can each append a function of one name with different bodies and leave every migration's own bytes identical. The rule against direct ledger writes is where the architectural guard currently in `hack/cozystack-version-stamp.bats` moves to: that guard keeps stamps going through the helper, which is what this rule does for ledger writes. + +**Every migration must be safe to run twice.** This is a contract rule rather than a tier: since a failed background migration is retried up to its cap and an operator may re-run any migration by hand (§7), re-running has to be safe for all of them, not for a declared subset. The precedent is already in the tree — `packages/core/platform/values.yaml` says of migration 50 that it "is idempotent, so re-running after the fix is the safe path". No linter can check this, so the mandatory test suite carries it: every migration's suite includes a case that runs it twice against the same fixtures and asserts the second run is a no-op. + +**Tests become mandatory.** Every new migration ships a bats suite driving it against fake `kubectl` fixtures, and the linter fails a migration that has none. The pattern is established: `hack/migration-50-etcd-adopt.bats` drives the real script against `hack/testdata/migration-50/` with 29 cases. This is the highest-leverage rule in the set — migration `43`'s hardcoded release name is caught by a single test case using a non-default instance name. + +**A scaffold**, `hack/new-migration.sh `, generating the script and test stub with the header filled in and today's date. Conventions that require reading a document get followed unevenly; conventions the tooling hands you get followed. + +### 6. Runner + +`run-migrations.sh` gains a second pass, so in-flight integer migrations merge unchanged: + +1. **Legacy pass** — unchanged `seq CURRENT (TARGET-1)` over `migrations/[0-9]+`, gated on `version`. Frozen: CI rejects any *new* integer file. +2. **Slug pass** — a single walk over the manifest, in batch-file order: + +```sh +preflight_manifest || exit 1; for batch in order.d/*; do # zero-padded, so plain glob order is correct + while read -r slug flag; do case $slug in ''|'#'*) continue ;; esac # blank and comment lines of a batch file + is_revoked "$slug" && { record "$slug" revoked-unless-done; continue; } # revoked outranks everything; recorded unless the key is done (§8) + ledger_has "$slug" && { warn_if_sha_differs "$slug"; continue; } # already recorded on this cluster + [ "$flag" = retired ] && continue # deleted below the floor (§11) + script=$(find_script "$slug") || fail "$slug is in $batch but not in this image" + [ "$(tier_of "$script")" = pre-apply ] || continue; if run_migration "$script"; then record "$slug" ok; else on_error "$slug"; fi # the hook runs pre-apply only, background is the operator's (§7) + done < "$batch" +done +``` + +There is no sorting, no dependency resolution and no version comparison at runtime: §3 settled all three at build time. That is what keeps the runner to a flat walk over a file, and it is why `requires` never has to be parsed here. + +Three conditions are fatal and must stay fatal. A slug present on disk but in no file of `order.d/` — neither a numbered batch nor `99999-untagged` — means the manifest was not generated for this build, and a slug in a batch but absent from the image means the packaging is wrong, and a slug in `EXPECTED` (the list the render gate computed, §7) that the image's manifest does not list, or lists without `retired` and has no script for, means the chart is newer than the image; all three abort before anything executes, which is what `preflight_manifest` in the sketch stands for: it checks the whole manifest against the image, `pre-apply/` and `background/` before the first slug runs. `on_error` is the `on-error` header of §5: `abort` exits and records nothing, except in the `ONLY` mode of a background run, where it records `failed` first, and `warn` records `failed` and goes on. `warn_if_sha_differs` compares the recorded `sha256` with the script's and only logs. `find_script` looks in `pre-apply/` and `background/` and, for a slug of `legacy-map`, at the top level of `migrations/`, and `tier_of` answers `pre-apply` for the last. Downgrading either to a warning reintroduces the silent skip. + +Legacy runs first. `background/` is not executed by the hook at all. + +### 7. Execution + +**Pre-apply** stays the render-gated hook. The gate generalises from a scalar compare to a set difference: the chart already ships the whole migrations directory (there is no `.helmignore` in `packages/core/platform`), so `.Files.Glob "images/migrations/migrations/order.d/*"` yields the manifest at render and `.Files.Glob "images/migrations/migrations/pre-apply/*"` the names of the blocking slugs, and the pending list is built from the two with no script reads: a manifest entry that is not flagged `retired`, whose script is in `pre-apply/` or which is a converted slug of `legacy-map`, and which has no ledger key; a `retired` entry that is not in `migrations/revoked`, with no key or with a `failed` one; and a slug that is both in `migrations/revoked` and listed by the manifest, with no key or a `failed` one. The runner skips any slug that has a key (§6), so a `failed` entry of the blocking tier is not pending again, but the other two kinds are work the runner still has to do: a `retired` entry has no script and so no tier, and without it in the list a cluster below the floor would render no Job and skip the check of §11, and a revoked slug has to be recorded (§8) even when nothing else is pending. The Job is only created when the list is non-empty or, until Phase 5, while the stamped `version` is below `targetVersion`, as it is today, and its env carries the list as `EXPECTED`. The runner aborts before running anything if a slug in `EXPECTED` is absent from the image's own manifest, or is listed there without `retired` and has no script in the image. The Job also carries the chart's `migrations/revoked` as `REVOKED`, which the runner adds to the image's own list, so a revocation made in a checkout reaches an image that predates it. At a release the chart and the image come from one tree and cannot disagree, but a checkout can: the in-tree image pin lags `targetVersion` by design (`values.yaml`), and a runner that knew only its own manifest would exit 0 over a slug the chart counts as pending and render the Job again on every reconcile, where today a missing file fails loudly. The check cannot fire on an image that predates it, and the in-tree pin lags the tree by design, so a deploy from a checkout builds the migrations image from the same tree first (`make image-migrations`), as it has to for any new migration today. `templates/sources.yaml` already reads chart files this way, so the idiom is established here. + +Because the manifest is generated rather than committed, a chart built by the release carries it and one rendered straight from a checkout does not. That case has to fail the render rather than quietly produce an empty pending set: when a slug in `pre-apply/` or `background/`, or a converted slug of `legacy-map`, is listed by no file of `order.d/`, the template calls `fail()`: the gate would not see that slug as pending and the index below would not list it. A missing `order.d/` is the plain case, and a manifest generated before the slug was added is the same case. It is the same fatal condition §6 gives the runner, for the same reason. + +A render from a checkout gets its manifest in one of two ways. The platform chart's `Makefile` runs `hack/generate-migration-order.sh` ahead of its `show`, `diff` and `apply` targets, so a local render or a deploy to a dev stand gets the manifest a build would produce, from the history the developer's clone has: a clone with no tags fails like a shallow one (§3), because without the tags the generator cannot know which slugs a release retired, and a manifest built without that would skip the floor check of §11. The chart's unit tests do not run the generator: `make test` renders against a fixture tree, so the tests depend neither on git history nor on which migrations are on disk that day (*Testing*). The gate has no fallback that derives the pending set from the two tier directories when `order.d/` is absent, because a `retired` entry exists only in the manifest, and a cluster below the floor would then render no Job and skip the check of §11. A bare `helm template` of the chart from a checkout that never ran the generator still fails, deliberately. + +**A fresh install records, it does not run.** Today a new cluster runs zero migrations, and that is two behaviours acting together rather than one rule: `templates/cozystack-version.yaml` stamps `targetVersion` straight into the ConfigMap when `lookup` finds none, and `templates/migration-hook.yaml` computes `$shouldRunMigrationHook` only inside `{{- if $configMap }}`, so with no ConfigMap the hook does not render at all. (The bootstrap branch in `run-migrations.sh` is unreachable through Helm for the same reason.) A set difference has no equivalent of that: an absent ConfigMap is an empty ledger, so `pending` would be the entire `pre-apply/` set and the first slug migration to land would change what a fresh install does. + +So the install-time branch of `templates/cozystack-version.yaml` seeds the ledger, recording every slug in the manifest — both tiers, and the ones flagged `retired` too, so that the floor check of §11 passes on a new cluster — with outcome `skipped-fresh-install`. The manifest is already read at render for the gate, so this is one loop on a path taken exactly once in a cluster's life. This belongs to Phase 1 rather than Phase 5: without it Phase 1 is not behaviour-preserving, and the seeding in §10 does not cover it, since that seeds from `legacy-map`, which holds no slug migrations. The hook keeps its existing `{{- if $configMap }}` guard: the seed renders in the same Helm render as the hook, so the hook's `lookup` finds no ConfigMap on install and renders nothing, and the helm-unittest fresh-install case holds that in place. + +**Background** is orchestrated — not executed — by cozystack-operator, because the scripts live in the migrations image and the operator has neither them nor a Helm renderer. The chart therefore writes down the three things the operator cannot derive: which slugs are background and in what order, which slugs are revoked, and which migrations image the current platform release pins. + +```yaml +apiVersion: v1 +kind: ConfigMap +metadata: + name: cozystack-migrations-index + namespace: cozy-system +data: + image: ghcr.io/cozystack/cozystack/platform-migrations:v1.7.0@sha256:… + background: | # manifest order, filtered to the background tier + clickhouse-keeper-pvc-labels + tenant-ancestor-labels +``` + +The index carries a third key, `revoked`, shaped like `background` and left out of the example above; it is empty when `migrations/revoked` is. The ConfigMap ships in Phase 1 with `image` and `revoked` and an empty `background`, because the manual re-run below needs the image before the operator exists. The operator reconciles `pending = background − done − revoked`, where `done` is the slugs whose ledger state is done (§11) and a `failed` entry with `attempts` below the cap counts as pending again once its backoff has elapsed; `revoked` is the index's `revoked` list, the slugs of `migrations/revoked` as the chart read them at render, and it outranks a `failed` key, so a revoked slug is never retried and the operator never has to read the image for it. It walks pending in the listed order, creates one Job per pending slug from `image` with `ONLY=`, and leaves the ledger to the runner in that Job, which records every outcome it observes: `ok`, `failed` under `warn`, and `failed` under `abort` as well, since in the `ONLY` mode of a background run nothing else would. The operator writes `failed` itself only when a Job fails without the runner having written a record, as after an OOM kill or a timeout, increments `attempts` when it does, so a Job killed every time still reaches the cap, and never overwrites a record newer than the Job's start. Because the ConfigMap is re-rendered on every platform upgrade, the image ref and the list cannot drift from the release that shipped them. While `cozystack-version` does not exist the operator creates no Job: on a fresh install the index and the ledger arrive in the same Helm release, and an absent ConfigMap there means the seeding has not been applied yet, not that every background slug is pending. The operator is already `cluster-admin`, so this needs no RBAC change. + +Background Jobs run **serially**, one at a time in manifest order: it matches the current model, keeps failure attribution unambiguous, and avoids two migrations touching the same objects concurrently. + +Two existing patterns carry over to the reconciler: the `Ready`-condition and per-failure-`Reason` vocabulary in `internal/operator/package_reconciler.go`, and the tolerant annotation-bookkeeping idiom in `internal/operator/packagesource_reconciler.go` (`readRecoveryTracking` / `clearRecoveryTracking`), where malformed state degrades to "no prior attempt" instead of wedging the controller. + +Moving backfills off the blocking path also fixes a live operational problem: the migration hook carries `helm.sh/hook-delete-policy: before-hook-creation`, so under Flux's retry loop a failing migration destroys its own logs on the next attempt. + +**A recorded failure is retried only where something is watching.** A `pre-apply` migration that fails under `on-error=warn` is recorded `failed` with its reason, message, image and attempt count, and is not attempted again on the next upgrade. The upgrade proceeds and the record is the report, which is the current behaviour made explicit rather than a new restriction. + +The background tier is different, and treating the two alike was wrong. That tier already has something watching it: the operator reconciles it and reports a `Ready=False` condition with a per-failure reason, so a failure there is surfaced rather than buried in a ConfigMap nobody opens. And on the air-gapped and slow-moving clusters this proposal counts as real when it rejects a kubeadm-style skip ban, "the record is the report" means nobody reads it for months. So the operator retries a failed background migration with exponential backoff up to a small fixed cap, with `attempts` going up by one on each try, written by the runner with the outcome; after the cap the entry stays `failed` and nothing retries it again. What the original objection rejects is unbounded retry of unwatched work, and a capped retry of watched work is not that. §5 already requires every migration to be safe to run twice, so this needs no per-migration opt-in. + +`on-error=abort` is the deliberate exception, and it is an exception because nothing is recorded rather than because a policy says so. The Job fails, the ledger is untouched, the upgrade does not proceed — and the surrounding machinery does retry: the hook Job carries `backoffLimit: 3`, and Flux re-reconciles the failed HelmRelease and creates the hook again. The migration is still pending on every one of those attempts, so it runs again. That is correct for a blocking migration, which must succeed before the upgrade can continue, and it is why the abort path needs no retry policy of its own. + +**Re-running one migration by hand.** Re-running is an operator action, and it needs no new mechanism: it is the mechanism the operator itself already uses. A one-off Job from the pinned migrations image with `ONLY=` runs exactly that migration. In `ONLY` mode the runner deliberately skips the "already recorded" check for the named slug — that is the entire point — executes it, and rewrites the entry with a fresh outcome, the image it ran under, and `attempts` incremented. It still refuses a slug listed in `revoked`, since revocation outranks any operator request, and it still refuses a slug absent from the image. + +The Job is named `migration-rerun-`, takes its image from `kubectl -n cozy-system get cm cozystack-migrations-index -o jsonpath='{.data.image}'`, and runs `run-migrations.sh` with `ONLY=` in its env under the operator's ServiceAccount, which is already cluster-admin. `kubectl create job` takes neither an env var nor a service account, so it is applied as a manifest, the same one the operator builds for a background Job. + +That keeps one execution path rather than two, needs no hand-editing of a JSON value inside a ConfigMap, and cannot corrupt the record it is meant to act on. Because §5 requires every migration to be safe to run twice, it is safe for any of them rather than for a blessed subset. + +**When an operator would do this.** A `failed` entry means the work did not happen and whatever it was fixing is still broken. The record says which migration, why it failed and what it said, so the decision is: remove the cause — a tenant that was mid-delete, a throttling apiserver, an object something else held — and re-run. There is no value in re-running before that, which is why the background retry is capped rather than indefinite and why the blocking tier does not retry a warn failure at all. + +Failures need a surface, since past the cap nothing chases them. For the background tier the operator already reports one, as a `Ready=False` condition with a per-failure reason. For the blocking tier the ledger is the surface: a `failed` entry stays in the ConfigMap for the life of the cluster, and `kubectl get configmap -n cozy-system cozystack-version -o yaml` is the report an operator reads. + +### 8. Revocation + +A migration file is immutable once it reaches any branch a release is cut from — editing `43` in place is what created the `53` situation. A faulty migration has two populations to serve, and both are handled explicitly. + +**Clusters that have not run it yet** must never run it. Its slug goes in `migrations/revoked`, and the runner subtracts that set from pending. It records a `revoked` entry under `m.` **only when no key for that slug exists or the existing one is `failed`**, so the ledger stays a complete account of what was decided without overwriting what executed; a `failed` record does not count as executed, and left in place it would let a release rolled back to before the revocation retry the slug. The write is a read, a check and a merge patch carrying the `resourceVersion` it read, retried on conflict (§4). That binds every record write and not only the `revoked` one: without it, a background Job finishing between the read and the write would have its `ok` overwritten, and a Job that was already running when the revocation landed could overwrite `revoked` with `failed`, so an outcome record that finds `revoked` under its key leaves it. The condition is the whole point: on a cluster that already ran the migration successfully, an unconditional write replaces `ok` with `revoked` and destroys the one fact an operator needs during the incident, which is whether this cluster executed it. The render gate counts a revoked slug with no key as pending (§7), so the runner gets to write that record even when nothing else is pending; otherwise a cluster whose hook never renders would hold no record, and a release rolled back to before the revocation would run the slug. + +**Clusters that already ran it** need repair, which is a new migration with a new slug handling both the never-ran and the ran-broken states. + +``` +# redis-failover-group-label stamped the wrong operator-group value. +# Superseded by redis-failover-group-fix. +redis-failover-group-label +``` + +This turns a silent, dangerous edit into an explicit, reviewable one, and closes the never-ran population automatically rather than by remembering to fold the fix back into the original file. + +**Companion CI guard:** once a migration file exists on any branch releases are cut from, its content may not change (diff against merge-base), and a released file in `lib/` may only gain lines at its end (§5). Scoping the guard to `main` alone would miss the [#3534](https://github.com/cozystack/cozystack/issues/3534) shape precisely: one slug carrying different content on a release branch is the same silent skip in a new spelling, and the guard has to run wherever the divergence can be introduced. The ledger checksum still catches it after the fact on a live cluster, but §11 argues that detection which can be ignored reproduces the failure it is meant to prevent, and that argument applies here too. Revocation is the sanctioned escape hatch, so the guard has somewhere to point. + +### 9. Why migrations stay platform-level + +An earlier revision of this proposal moved migrations that only touch one package's objects into that package, delivered by that package's own pre-upgrade hook, with ledger keys namespaced as `m..`. That does not work, and the reason is the incident this document opens with. + +A namespaced key has a package dimension and no instance dimension. `SeaweedFS` is a user-creatable kind: instance `foo` is owned by release `foo-system`, and a cluster can hold any number of them. One key `m.seaweedfs.db-adopt` covering N instances means the first release to run writes it and every other instance is skipped as already applied — the `43` failure again, now with a ledger entry on top asserting it succeeded. Adding an instance dimension to the key does close that hole, but it makes the ledger unbounded in the tenant's dimension rather than the platform's, and it asks a hook that can only see its own release to reason about a fan-out it cannot enumerate. + +The pattern that works is already in the tree: a platform-level migration that scans the fleet, which is what `lib/seaweedfs-db-adopt.sh` does. What `lib/` was missing is not a scoping mechanism but a helper for that iteration, including the errexit handling that file documents — the guard migration `43` lacked. §5 adds it. + +So migrations stay platform-level. Ownership of the objects being touched is still the right question to ask about a migration; it just does not translate into a delivery location, and three classes make that concrete: + +- **Cross-package.** `26` moves resources between `extra/monitoring` and `system/monitoring`; `20` applies five packages in sequence. Neither has a single owning release. +- **Must precede a *different* package's operator.** `54` had to label every RedisFailover before `cozystack.redis-operator` upgraded. A hook on the redis *app* package fires too late, since those releases `dependsOn` the operator. This class looks package-scoped and is not. +- **Must precede the platform's own artifacts**, which is where the runner already sits. + +Against those, the genuinely package-scoped set is small — `52` deletes a linstor-scheduler Deployment, `27` moves CRDs into a `*-crds` release and so does the CRD part of `22` (the rest of `22` deletes a platform HelmRelease and relabels tenant ones) — and none of them is served worse by running from the platform hook. Scoping bought a cleaner ownership story and cost correctness on the one kind of migration that most needs it. + +### 10. Converting the legacy set + +The dual-mode runner is a transition device, not an end state. Leaving the integer set in place forever means the runner keeps two code paths, `targetVersion` never goes away, and every new contributor has to learn a scheme that is already deprecated. + +The conversion is itself a migration of the migrations. Renaming and ordering are mechanical, and the two content edits the conversion makes are stated after the list: + +1. Rename every integer migration present at the Phase 1 freeze, `migrations/1..N` with `N = T - 1` for the `targetVersion` `T` at that freeze, to slugs describing what each does, so `1` becomes `mariadb-operator-secrets`. The rename is in place: the files stay at the top level of `migrations/` beside `lib/`, because 18 of them (`42` to `59`) source `lib/` relative to their own path, and a move into a tier directory would break that and would put frozen files under the linter. A converted slug is `pre-apply` with `on-error=abort` by definition, which is what the legacy runner does with a non-zero exit. The integer files are under the guard of §8 from Phase 1, and the conversion PR is the one change it lets through, for exactly the edits listed after the steps and no others; from then on the converted files are bound like any other. Order is not encoded in the names and does not need to be: batch `00000` lists all N slugs in their original integer order, which is by construction the order every existing cluster already ran them in. It is the one batch the generator does not derive — the integer set predates the slug scheme, so no tag's tree carries those slugs in that order — and it is supplied instead by the committed `legacy-map` of step 2, which already holds exactly that ordering. That batch sorts before every release batch, so the converted set keeps its position in the global sequence exactly. Its internal order is executed for one population only: a cluster stamped below the freeze's `targetVersion` when Phase 5 lands is seeded for the integers it already ran (step 3) and runs the rest from this batch, in this order, which is the order the integer runner would have taken. Every other cluster has already run the integers or, on a fresh install, has them seeded (§7). +2. Commit the integer → slug mapping as `migrations/legacy-map`, one `i ` pair per line. It is the seeding input for step 3 and the only place the old numbering survives. +3. Before the floor check and the slug pass, the runner seeds the ledger from the map: for every `i ` pair with `i` below `version` and no `m.` key, it records `legacy` (no checksum — those files were not immutable when they ran). The condition is per key, not "no `m.*` keys at all": a cluster that lived through Phases 1–4 already holds `m.*` keys for the slug migrations it ran, and a whole-ledger condition would skip the seeding for exactly those clusters and then run every converted slug again. Idempotent, since a recorded key is never rewritten. A fresh install needs none of it: §7 seeds every slug in the manifest at install time. +4. Delete the legacy pass, `targetVersion`, and `hack/check-migrations-target.sh`. The `version` key stays in the ConfigMap, stamped `T` (one past the largest `i` of `legacy-map`) at install, and on a cluster that crosses Phase 5 from below `T` advanced by the runner to one past the largest `i` such that the `legacy-map` slugs `1..i` are all recorded in a done state, so it reaches `T` with the last of them and never passes a slug that has not run. Nothing in Phase 5 reads it but the seeding of step 3, which it cannot mislead, since it never passes a slug without a done key, and a release rolled back to one that still has the legacy pass. + +**What the conversion edits.** Two things in a legacy script cannot survive the move to the slug runner, and the conversion PR changes exactly those and names each edit. First, every stamp is deleted: `42` onward stamp through `lib/cozystack-version.sh`, and `1` to `41` stamp with a label-less `kubectl apply`, which strips the `platform.cozystack.io/no-delete` label from the ledger ConfigMap. §5 gives the runner the ledger and step 4 gives it `version`, and a stamp left in would let a manual `ONLY` run of a late converted slug raise `version` past slugs that have not run, after which the seeding of step 3 would record them as `legacy`. Second, a script that reports failure by exiting 0 is made to exit non-zero. Migration `46` exits 0 after a partial failure, annotating the `cozystack-version` ConfigMap with the failed tenants, and its comment says the unstamped version is what makes the next upgrade retry it; in the legacy runner that retry survives only if no later migration stamps past it, and the slug runner records `ok` on exit 0, which would turn the partial failure into a recorded success that nothing retries. Converted slugs are `on-error=abort`, so after the edit a partial failure of `46` stops the upgrade instead of annotating and carrying on, which suits a migration the chart's own `fail()` guard depends on (§2). The PR audits every legacy script for an exit 0 that follows a withheld stamp, since the linter does not bind these files. + +Only step 3 touches clusters, and it writes ledger keys rather than running anything. A cluster stamped at the freeze's `targetVersion` ends up with one `legacy` record per converted integer migration and behaves identically. + +### 11. Retention + +Today every migration ever written ships in the image forever, because a dense sequence cannot have holes: deleting file `7` breaks the `seq` loop that walks past it. Under a set model there is no sequence to break, so migrations can be deleted. + +What makes deletion safe is declaring a floor — the oldest platform version this release accepts as an upgrade source, which is already a supported-versions policy decision rather than a new one. Migrations that only ever applied to clusters below the floor are deleted; the runner applies the floor check below and **refuses**, rather than warning, on a cluster beneath it, pointing at a staged upgrade through an intermediate release. Refusing is the only defensible choice: a warning that is ignored produces exactly the silent-skip failure this proposal exists to eliminate. + +Deleting a migration means deleting the script. The `retired` flag is not written by hand and cannot be: §3 derives it, from a slug present in an earlier tag's tree and absent from this one. What the generator emits is: + +``` +# batch 00000, converted from legacy-map +mariadb-operator-secrets retired +flux-tenants-cleanup retired +``` + +The line stays. It costs about forty bytes and it is what lets the runner tell a retired migration from a slug it has never heard of — the second of which is a packaging error that must abort. The invariant holds by construction rather than by lint: **every manifest entry satisfies exactly one of {the script exists, the entry is flagged `retired`}**, because the generator emits an entry only for a slug some tag carried, or that `legacy-map` lists for batch `00000`, and flags exactly the ones this tree no longer has. `retired` is not reversible either, since restoring the slug would collide with the uniqueness check §5 runs against tag history. That is what lets the runner and the floor check below treat a retired slug without reading any script. A slug that no stable release ever carried is not left to reach this state unrevoked, because the generator refuses it (§3), so the floor check only speaks about migrations some stable release shipped. + +**The floor check.** After legacy seeding (§10) and before the slug pass, the runner refuses if any slug the manifest flags `retired` has no `m.` key in the ledger, or has one whose `state` is `failed`, except the slugs listed in `migrations/revoked`. A cluster with no key never recorded a migration this release has since deleted. A cluster with a `failed` key recorded it and the work did not happen, which is where a background migration is left once its retries are spent and where a `pre-apply` failure under `on-error=warn` stays (§7). Either way the work that would have run for the cluster is gone from this release, because the script is no longer in the image and no re-run (§7) can reach it. The refusal names the slug and says whether its key is missing or failed. For a missing key the cluster is below the floor and has to upgrade through an intermediate release that still carries the script. For a failed one the release the cluster is on still carries it, so the operator re-runs it there (§7) and then upgrades, which makes a tolerated `warn` failure a debt cleared before the floor can be crossed. Every other state counts as done: `ok`, `legacy`, `revoked` and `skipped-fresh-install`. A revoked slug is exempt because no work is missing for it: revocation exists so that clusters which never ran the migration never will (§8), and the runner would record it as `revoked` in the slug pass that comes after this check. The check is a set comparison of the ledger against the incoming manifest, so it reads no stored batch number and gives the same answer whichever line the cluster came from. It is exact because keys are never deleted (§4): an absent key means the migration was never recorded here, not that it was cleaned up. Both seeding paths write every key a cluster is entitled to — a fresh install records the whole manifest, retired entries included (§7), and a legacy cluster is seeded from `legacy-map` (§10) — so neither is refused by mistake. + +### 12. What this retires + +`hack/check-migrations-target.sh` and `targetVersion`, the cross-branch slot-alignment check proposed in [#3534](https://github.com/cozystack/cozystack/issues/3534), and the question of which PR owns a number. Backports become safe by construction: a slug applied on `release-1.5` is recorded under that slug, and the 1.6 upgrade skips it and runs its own. + +## User-facing changes + +**Cluster administrators.** No change to the upgrade command or its behaviour. The `cozystack-version` ConfigMap gains one key per applied migration, so `kubectl get configmap -n cozy-system cozystack-version -o yaml` becomes a readable account of what ran and when — today it shows a bare integer. Background migrations become visible as Jobs in `cozy-system` and as conditions on the operator, where today they are invisible steps inside one blocking hook. + +**Tenants.** None. No tenant-facing API, CR shape or dashboard surface changes. + +**Contributors.** The authoring workflow changes materially and is the main documentation deliverable: `hack/new-migration.sh ` scaffolds the script and its test, the header block is mandatory, a bats suite is mandatory, and the linter runs in `make unit-tests`. `docs/agents/contributing.md` gains a migrations section; the "pick the next number and bump `targetVersion`" instruction is removed wherever it appears. + +## Upgrade and rollback compatibility + +**Existing clusters.** The ledger seeds from the existing `version` scalar, so a cluster stamped `T`, the `targetVersion` at the Phase 1 freeze, is treated as having applied legacy migrations `1..T-1` and nothing else. No backfill migration is required — the seeding is runner logic, not a migration, which matters because migrations are the thing being refactored. The seed trusts the scalar to mean what `main`'s integers mean, and that holds only for clusters that ran `main`'s sequence. Maintenance lines cut before Phase 1 have already diverged: as of this writing `release-1.5` stops at `targetVersion` 46, `release-1.6` at 54 and `main` at 60, and slot `45` differs between `release-1.5` and `main`. A cluster on such a line that upgrades to a Phase 1 release carries a scalar that skips integers it never ran, and Phase 5 records those as `legacy`. The slug engine does not repair that divergence, it only stops it growing. + +**Fresh installs.** The ledger is seeded at install time with every shipped slug recorded as `skipped-fresh-install` (§7), so a new cluster runs no migration and starts no pod, exactly as today. + +**Phase 1 is behaviour-preserving.** Every migration is `pre-apply`, the legacy pass is untouched, and fresh install is seeded rather than run, so a cluster upgrading or installing through Phase 1 does exactly what it does today. + +**Rollback.** Rolling the platform back to an earlier release leaves the ledger keys in place. A rolled-back release simply does not know some slugs, so it computes a smaller pending set and skips them; on rolling forward again they are already recorded and are not re-run. Ledger keys are additive and never removed, on downgrade or otherwise (§4). Phase 5 keeps the `version` key for this reason (§10): a release that still has the legacy pass finds `T` there, equal to its own `targetVersion`, and runs nothing, on a cluster installed after Phase 5 as much as on one that crossed it. Without the key it would read `0` and start the sequence at `/migrations/0`, which does not exist, and the rollback would fail. On a cluster still part-way through the converted slugs it finds the boundary the runner had reached and re-runs only what had not finished. + +This is clearer than the scalar rather than safer, and the difference is worth stating because it is easy to overstate. Nothing lowers `version` today: the ConfigMap carries `helm.sh/resource-policy: keep`, its template is guarded by `{{- if not $configMap }}` so it never re-renders once it exists, and after a rollback the hook gate is false anyway because `targetVersion` is now lower than `currentVersion`. Today's rollback runs nothing. What changes is that the new model says so on the object instead of leaving it to be re-derived from an inequality. + +**Irreversibility.** Revocation is not a rollback: it prevents a migration from running on clusters that have not yet run it, and does nothing to clusters that have. Repair of the latter always requires a new migration. This is flagged explicitly because it is the property most likely to be misread under incident pressure. + +**Phase 5** deletes `targetVersion`, which is a breaking change for anything reading it — see Testing for the upgrade-lane assertion that must be replaced first. + +## Security + +No new trust boundary. The migration hook already runs with `cluster-admin` via a per-hook ServiceAccount and ClusterRoleBinding created and deleted around the Job, and cozystack-operator is already bound to `cluster-admin`; the background tier reuses the operator's existing identity rather than introducing another privileged principal. Migrations are, and remain, arbitrary root-equivalent code shipped inside a digest-pinned image. + +The `cozystack-version` ConfigMap carries `platform.cozystack.io/no-delete: "true"` and is guarded by the deletion-protection ValidatingAdmissionPolicy; the added `m.*` keys inherit that protection. No new tenant-supplied input reaches any migration, no new secrets are stored or transmitted, and no new RBAC surface is created. + +One property improves: the ledger records a sha256 of each script as it ran, so a migration whose content changed after being applied is detectable on a live cluster rather than only by reading git history. + +## Failure and edge cases + +- Fresh install → the ledger is seeded with every shipped slug as `skipped-fresh-install`, no Job is created, and nothing executes. +- Migration file missing from the image but pending → runner exits non-zero and refuses to advance, preserving today's behaviour. A missing file indicates a packaging mistake and must fail loudly rather than stall the cluster at a stale state. +- Migration already recorded in the ledger → skipped, not re-run, on every subsequent upgrade. +- Revoked slug, never applied here → skipped, and recorded as `revoked` so the ledger stays complete. +- Revoked slug already applied here → skipped, and the existing `ok` record is left alone. Revocation never rewrites an executed outcome; a `failed` record is superseded by `revoked`, since nothing happened that is worth keeping. +- `on-error=abort` migration fails → the hook Job fails, the platform upgrade does not proceed, and nothing is recorded for that slug. +- `on-error=warn` migration fails → logged, upgrade proceeds, and the slug is recorded as `failed` so it is visible rather than indistinguishable from success. +- Applied migration's file content changed since it ran → the runner logs a WARN naming the slug and both digests, except for a `legacy` record, which carries none. The migration is not re-run; repair is a new slug. +- `requires` names a slug that does not exist, names one in a later batch, or the graph has a cycle → the **build** fails. The runner never evaluates `requires`, because §3 resolved it into the manifest. +- Slug on disk but in no file of `order.d/` → the manifest was not generated for this build. Runner aborts before executing anything, and the render gate fails for the same reason. +- Slug on disk that no tag carries yet → it sits in `99999-untagged`, runs after every numbered batch and moves to a numbered one when a tag is cut. A build that cannot see the tag history fails at generation instead (§3). +- Slug in a batch, not flagged `retired`, and absent from the image → packaging error. Runner aborts. The same abort covers a slug the render gate expects (`EXPECTED`, §7) that the image's manifest does not list, as when the chart is newer than the pinned migrations image. +- Two migrations of the same tier first shipped by the same release with no `requires` → both run in slug order, identically on every cluster, because every cluster sees that batch in one pass off one image. +- Cluster with `version` set and some `legacy-map` slug unrecorded (Phase 5) → those slugs are seeded as `legacy` before anything else runs; running twice seeds nothing further, since a recorded key is not rewritten. +- Background Job fails → recorded `failed` with reason and message, surfaced as a `Ready=False` condition with a distinct reason on the operator, and retried with backoff until `attempts` reaches the cap, after which it is **not** retried again. The upgrade is not blocked at any point. +- Operator re-runs a failed migration → a one-off Job from the pinned image with `ONLY=`; the runner bypasses the already-recorded check for that slug only, and rewrites the entry with a fresh outcome and `attempts` incremented. +- `ONLY=` names a revoked or absent slug → refused. Revocation outranks an operator request, and an absent slug is a packaging error either way. +- Cluster below the retention floor (Phase 5+) → some slug flagged `retired` has no ledger key or a `failed` one; runner refuses, names those slugs, says which of the two each is, and for a missing key names the intermediate release to upgrade through, for a failed one the re-run to do on the current release first. + +## Testing + +Test-first throughout. The runner is a shell script driven against fake `kubectl` fixtures, a format already established by `hack/migration-50-etcd-adopt.bats` (29 cases against `hack/testdata/migration-50/`) and `hack/migration-seaweedfs-db-adopt.bats`. + +**Unit (bats, `hack/cozytest.sh`).** Runner: every case in *Failure and edge cases* above. Ordering: the runner walks batches in filename order and entries in listed order; a slug on disk and in no file of `order.d/` aborts; a slug in a batch and absent from the image aborts; a `retired` entry is skipped without a script. A slug in `EXPECTED` that the image's manifest does not list aborts, and so does one it lists without `retired` and has no script for. A converted slug at the top level of `migrations/` is found, runs as `pre-apply` and finds `lib/`. A recorded key whose `sha256` differs from the script logs a WARN and is not re-run. Seal step: a new batch appends only, `requires` reorders within a batch, a `requires` naming a later batch fails, each file of the previous release's published manifest equals the new release's file of that name apart from the `retired` flag, with the untagged file excluded, so a retirement that only adds the flag passes, a batch that gained or lost a slug fails, and a moved tag that stays among the ancestors is caught because the comparison reads the published copy and not a regeneration. Untagged slugs: a slug on disk that no tag carries lands in `99999-untagged` and in no other file, a slug never appears in two files, `legacy-map` slugs are in `00000` and nowhere else, a `legacy-map` slug whose script is deleted is emitted `retired` in `00000`, a change to a released `legacy-map` line fails CI, a slug that no stable tag of any line carries and that is deleted fails generation unless `revoked` lists it, while one a stable tag of another line carries does not, the untagged file sorts after every numbered batch, and generation fails on a shallow clone, on a clone with no tags and, in a release build, when the previous release's published artifact cannot be fetched. The promotion verifier fails an artifact that has tier directories or a `legacy-map` and no `order.d/`. Ledger durability: `stamp_cozystack_version` after `record_migration` must not prune `m.*` keys, no code path deletes one, and a `revoked` record must not overwrite an existing `ok` but supersedes a `failed` one, and a revoked slug holding a `failed` key is recorded before the already-recorded check can skip it. An outcome record from a Job that was already running when the slug was revoked leaves the `revoked` key in place, and a write carrying a stale `resourceVersion` is rejected and decided again. Cross-line upgrade: a ledger written under the `release-1.6` manifest, run against a `main` manifest whose batch numbers collide with it but hold different slugs, runs every slug that has no key and skips every slug that has one. Floor check: a ledger missing the key of a `retired` slug is refused and the message names the slug; a ledger holding that key with `state` `failed` is refused the same way and the message says it failed; a ledger holding every key in a done state passes; a ledger seeded by a fresh install, retired entries included, passes; a retired slug listed in `revoked` and never run passes; a cluster that already holds slug keys and a `version` scalar is still seeded from `legacy-map` and does not re-run the converted slugs. The runner advances `version` only to one past the largest `i` whose `legacy-map` slugs `1..i` are all done, reaching `T` with the last of them, so a converted slug that failed or was run out of order never leaves the seeding reading a raised scalar, and a rollback finds a scalar no higher than what actually ran. A manifest entry that is neither present on disk nor `retired` is rejected. Idempotence: every migration's own suite runs it twice against the same fixtures and asserts the second run changes nothing. Linter: missing or unparseable header, undeclared `tier`/`on-error`, a shebang other than `#!/bin/bash`, a ledger write from inside a migration, a migration with no bats suite, a dangling `requires`, a slug already in the tag history with different bytes, a slug this line flags `retired` — each rejected, and a backport of a known slug with the tagged bytes accepted. + +**Unit (helm-unittest).** Render gate: nothing pending → zero documents; legacy-only pending; slug-only pending; both; a `retired` entry with no key, and one with a `failed` key → the Job renders; a revoked slug with no key, or with a `failed` one → the Job renders; a `failed` non-retired `pre-apply` slug → no Job; an unrecorded entry of `00000-legacy` → the Job renders; a revoked slug the manifest does not list → no Job; a `retired` entry that is also revoked, with no key → the Job renders and records `revoked`, and with that record → no Job; the Job's `EXPECTED` env equals the pending list and its `REVOKED` env equals `migrations/revoked`. Fresh install (no ConfigMap in `lookup`) → the seeded ledger renders, carrying one `skipped-fresh-install` record per shipped slug, retired ones included, and no Job renders at all; the same holds when the manifest is only `99999-untagged`. The existing `packages/core/platform/tests/migration_hook_skip_backup_test.yaml` asserts the hook's env by positional index, so it is rewritten alongside. The chart's `make test` neither runs the generator nor reads the real `images/migrations/migrations/`. It stages a copy of the chart in a temporary directory, replaces that directory in the copy with a fixture tree from `tests/fixtures/migrations/` (`order.d/` with real content, `pre-apply/` and `background/` with empty files named for slugs, since the gate reads only their names), and runs `helm unittest` against the copy. The default tree carries `00000-legacy`, two numbered batches, a `retired` entry and a `99999-untagged` file, and the ledger comes from `kubernetesProvider` as in the existing suites; the gate cases above and a retired entry with no ledger key (the Job renders, so the runner's floor check of §11 gets to run) use it. The suites that need another shape run against further copies with their own trees: a manifest that is only `99999-untagged`; no `order.d/` at all while `pre-apply/` is non-empty; and an `order.d/` that omits a slug present in `pre-apply/`, `background/` or `legacy-map` (the render must fail in the last two, §7). This is needed beyond the gate suites: every suite of the chart renders the hook template, so a bare `helm unittest .` from a checkout would hit the `fail()` of §7 as soon as the first slug migration lands. The cost is a fixture tree to update by hand when the manifest format changes. + +**Controller (envtest).** Pending computation including revoked (read from the index's `revoked` list, outranking a `failed` key), Job creation, no ledger write by the operator on a successful Job, a failing Job surfacing as `Ready=False` with a distinct reason, being retried only once its backoff has elapsed and **not** after the attempt cap, and an `ONLY=` run bypassing the already-recorded check for that slug alone while still refusing a revoked one. A Job that succeeded after the runner recorded `failed` under `warn` leaves that record in place and is retried by backoff, a Job that failed with no record written gets `failed` from the operator with `attempts` incremented, so repeated kills reach the cap, and the operator never overwrites a record newer than the Job's start. + +**E2E (upgrade lane).** An upgrade lane that asserts the stamp reached `migrations.targetVersion`, as the one proposed in [#3276](https://github.com/cozystack/cozystack/pull/3276) does, cannot survive Phase 5. Its replacement, required before Phase 5 retires `targetVersion`, is an **empty-pending assertion**: every slug in the manifest of the tree under test that is not flagged `retired` must have a matching `m.` ledger key whose state is done — `ok`, `legacy`, `revoked` or `skipped-fresh-install`. Because a fresh install seeds every slug it ships, the lane must additionally assert that the slugs introduced between the base release and the release under test carry `ok` specifically, apart from a slug the tree under test revokes, which carries `revoked`, and the `legacy-map` slugs, which a cluster seeds as `legacy` (§10), or it passes trivially on a clean install. Both sides are cheap — generate the manifest for the tree under test, read the ConfigMap's keys — and it is strictly stronger than the integer compare, because it proves each individual migration reached a done state rather than that a counter moved. The background half is asynchronous, so it polls; any other outcome fails the lane and names the offending slug. + +Existing suites that pin the current shape and are rewritten rather than ported: `hack/cozystack-version-stamp.bats` (its architectural guard hardcodes the `>= 42` numeric-filename convention; it becomes a linter rule), `hack/run-migrations.bats` (it pins the integer runner's resume behaviour — a retried pod starts from the version the failed pod stamped, a pod that finds the target already stamped runs nothing, a missing or malformed `version` falls back — so it is rewritten when Phase 5 deletes the legacy pass), `hack/e2e-install-cozystack.bats` (its deletion-protection case stashes only `version`, deletes the `cozystack-version` ConfigMap and recreates it with that one key, which would drop every `m.` key, so it stashes and restores the whole `data`), and the literal next-version assertions (`51`, `54`) inside `hack/migration-50-etcd-adopt.bats` and `hack/migration-seaweedfs-db-adopt.bats`, along with the `STAMP ` assertions in the suites of `54`, `55`, `57`, `58` and `59` (`hack/migration-59-vmi-hotplug-volumes.bats` asserts `STAMP 60`), all of which disappear with migration-side stamping. `hack/migration-seaweedfs-db-adopt.bats` also `sed`s `FROM alpine:` out of the Dockerfile, so that line's shape must be preserved. + +[#3458](https://github.com/cozystack/cozystack/pull/3458) proposes a second ordered runner (`preflight/`) in the same image at hook weight 0. It would be a co-tenant and is not disturbed by this work. + +## Rollout + +Each phase is independently shippable. + +**Phase 1 — engine and contract.** Dual-mode runner, `record_migration`, set-difference render gate, header parser and `on-error` handling, `hack/lint-migrations.sh`, `hack/new-migration.sh`, `lib/` helpers including the fleet-iteration one, fresh-install ledger seeding (§7), the `cozystack-migrations-index` ConfigMap with `image` and `revoked` (§7), `hack/generate-migration-order.sh`, including the trailing untagged file, wired into the image and chart builds, the stable promotion and its verifier (§3), and the platform chart's `show`, `diff` and `apply` targets, the chart's `make test` reworked to render against fixture trees (*Testing*), and five CI guards (integer freeze, content immutability on every release branch with `lib/` append-only (§5), slug grammar plus uniqueness against tag history, no tracked `order.d/`, and the prefix check against the manifest the previous release published). Every migration is `pre-apply`, so runtime behaviour is unchanged, and the linter rejects `tier=background` until Phase 2 lands, since nothing executes that tier before the operator does. The linter enforces on `pre-apply/` and `background/` only; the frozen integers are grandfathered, and so are the converted slugs that stay beside them after Phase 5 (§10). Phase 1 is backported to every supported maintenance line before the first slug migration merges, and until then `backport.yaml` refuses to cherry-pick a change under `pre-apply/` or `background/` onto a line that lacks `hack/lint-migrations.sh`: the integer runner on such a line would ship the file and never run it, and nothing would say so. + +**Phase 2 — background tier.** The `background` list of the index ConfigMap and the operator reconciler. Applies to new work only; the legacy integers stay frozen where they are, since `44`, `48`, `49` and `51` are already applied essentially everywhere. + +**Phase 3 — withdrawn.** Package scoping was the original Phase 3. §9 records why it is dropped: a package-namespaced ledger key cannot address the individual instances of a user-creatable kind, which is the `43` failure over again. The number is kept rather than reused so that Phase 4 and Phase 5, referenced throughout this document and in its review, keep meaning what they meant. + +**Phase 4 — attrition.** Reduces the rate at which new migrations are needed; retires none of the existing ones. *Admission-time defaulting* for fields on objects Cozystack does not template: migration `51` exists because vm-operator stamps `volumeClaimTemplate` labels only onto PVCs created after the chart change, and the StatefulSet controller never re-labels an existing PVC — Cozystack cannot template those PVCs, a controller creates them. A `MutatingAdmissionPolicy` matching `CREATE` on `persistentvolumeclaims` owned by a cozystack-managed VMCluster stamps the label at creation. It must be capability-gated exactly as `packages/system/cozystack-basics/templates/ingress-hostname-policy.yaml` gates its VAP on `.Capabilities.APIVersions.Has`, because the management cluster floor is 1.33 while `MutatingAdmissionPolicy` is beta in 1.34 and GA in 1.36. It is prevention only — existing PVCs still need the one-shot backfill. *API-layer conversion* for the values-format class: migration `39` rewrites flat `resourcesPreset` names into instance-type names across every App CR, and `pkg/apis/apps/presets/legacy.go` records that the same table is mirrored in four places. Accepting the legacy spelling in the aggregated API's conversion and defaulting layer means the stored CR never has to be rewritten and roughly fifteen migrations of this shape stop existing. Each half warrants its own proposal. + +**Phase 5 — legacy conversion and retention.** Commit `legacy-map`, implement seeding, delete the legacy pass and `targetVersion`, and keep `version` stamped at `T`. Rename the set to slugs, in place beside `lib/`, and supply batch `00000` to the generator from `legacy-map`. `legacy-map` goes under the immutability guard of §8, and the integer-freeze guard becomes a rule that the top level of `migrations/` holds only `lib/`, `revoked`, `legacy-map`, `order.d/`, the two tier directories and the slugs `legacy-map` lists. Depends only on Phase 1 plus at least one shipped release of soak, so the slug path is exercised before the seeding path is added; independent of Phases 2–4. The retention floor is introduced in the first release that actually deletes a migration, not before. + +## Open questions + +- Whether packages ever split into separate repositories is undecided upstream of this proposal, and nothing here depends on the answer: one ledger owned by the assembler (§4) works either way. If they do split, the unit that owns a ledger is worth revisiting — plausibly the bundle rather than the package, since a bundle has cross-package migrations by construction and a single package does not. +- Whether an operator needs a way to acknowledge a `failed` entry that no re-run will fix, so that the retention floor (§11) lets the cluster through. Nothing here provides one: `revoked` is a repo-side list, and editing a ledger value by hand is what §7 avoids. +- What to do about clusters whose scalar comes from a maintenance line cut before Phase 1 (*Upgrade and rollback compatibility*): a final release of each such line that carries `main`'s integers, or an explicit rule that they upgrade through an intermediate release. Nothing here decides it. + +## Alternatives considered + +**Data model — a CRD ledger with operator status.** Rejected for the two reasons in §4: `lookup` against a not-yet-Established CRD fails silently rather than loudly, and `hack/update-codegen.sh`'s catch-all `mv` plus the hardcoded CRD lists in `internal/crdinstall/install_test.go` and `hack/e2e-install-cozystack.bats` make adding a kind to `cozystack.io` a non-local change with downstream consumers. Only the first of those is conditional: the `lookup` hazard exists because a Helm render gate reads the state, so a design that removes the render gate entirely removes the hazard with it. The second cost is unconditional and, as of this writing, unsized. + +**Data model — a stored watermark, with compaction.** An earlier revision kept a `complete-through` batch number in the ledger and compacted the keys it covered. Rejected: batch numbers are per line, so a value stored under `release-1.6` numbering means something else under `main` numbering, and every minor upgrade crosses that boundary. Compaction also deleted the keys of slugs that still ship, which made the runner run them again unless it consulted the number, and it made a cluster that ran a retired migration look like one that never did, which the retention floor check has to tell apart. Per-slug keys that are never deleted need none of it, and cost about 250 bytes each. + +**Data model — keep the scalar, add a cross-branch CI check.** This is the mitigation [#3534](https://github.com/cozystack/cozystack/issues/3534) itself proposes. Rejected as the primary fix: it detects divergence rather than preventing it, requires every supported branch to be checked out at review time, and leaves slot contention and the `targetVersion` lint untouched. + +**Runtime — per-package Helm hooks with no ledger.** Rejected: a hook fires on every upgrade of its release, so one-shot work needs its own state anyway; there is no record of what ran or whether it succeeded; and there is no ordering across charts, which the cross-package migrations require. An earlier revision adopted them as a *delivery* mechanism on top of the ledger; that is withdrawn for an independent reason (§9). + +**Runtime — operator-driven execution for everything.** It does not lose the runs-before-chart-apply guarantee, which is the reason usually given and is wrong: `migration-hook.yaml` is a `pre-upgrade,pre-install` hook on the platform chart, the same chart renders the `Package` CRs, and `internal/operator/package_reconciler.go` builds the HelmReleases from them — so the hook already sits ahead of every package rollout, and the §2 class that must precede a *different* package's operator is covered today. A per-package gate in the reconciler is finer-grained, which buys less blocking rather than more ordering. + +Two things do argue for keeping pre-apply on the hook for now, and a proposal moving it has to answer them first. The controller would need the incoming release's migrations *before* it rolls the bundle that carries them, which the hook gets free from the render; reading the artifact through source-controller, which fetches ahead of any rollout, is the shape to prove there. And the CRD costs in §4 stay unsized. One argument for the move survives both, and this proposal cannot answer it: a ledger entry can only say "done", while some work is an invariant that must hold across several upgrades — `lib/seaweedfs-db-adopt.sh` documents exactly that, noting its `keep` window "is not a one-shot". A controller can hold an invariant. That deserves its own proposal. + +**Tiering — a third, *convergent* tier.** Some migrations are a predicate over current state rather than a one-shot edit: the label backfills (`48`, `49`, `51`) and the defensive cleanup (`44`). Declaring that would let them skip the manifest and the ledger and simply run on every upgrade, the way Flyway's repeatable migrations do — which is why `dpkg` and `rpm` survive version skips so calmly, their conditions being declarative rather than historical. Rejected because the saving is nil and the cost is not: it is four migrations of fifty-nine, a manifest line is forty bytes and the walk is one pass either way, against a fleet-wide sweep on every upgrade forever for work already done. The distinction is also carried already by the two axes that exist — `tier` says whether a migration blocks, `on-error` says what its failure means — and every example in the tree has all three agreeing. The one thing it would genuinely have added, *safe to re-run*, is instead a contract rule in §5 binding every migration, because manual re-runs (§7) need that to hold everywhere rather than in a declared subset. + +**Ordering — a date or sequence in the filename.** Rejected by §3: an authoring-time key is independent of which release first ships the migration, so it agrees with the required order only by luck. The variant that repairs it — a CI rule refusing an ID older than the newest already merged — was rejected separately for its running cost, since it forces a rename on every stale rebase and stale rebases are common. A two-digit intra-day sequence fails for a sharper reason: the author picks it before knowing which migrations land in the same release, so it can express neither intra-day nor intra-release order. + +**Ordering — a declared dependency graph as the order.** Rejected, with the worked reason in §3. Django is the honest version of this design: it disclaims filename ordering outright and never promises a stable order for pairs with no declared edge. `requires` is kept here, but as an assertion resolved when the manifest is generated rather than as the ordering mechanism. + +**Ordering — the release step renames files, `goose fix` style.** goose renames timestamped development files to sequential integers in CI once they are ready to ship. Rejected because the filename is the ledger key here, so a rename would rewrite cluster state and break cross-branch identity. A manifest derived at build time gets the same effect without touching the name. + +**Ordering — a batch stamped into each script's header at release time.** Tempting because a cherry-pick then carries the assignment with the file. Rejected: a migration that reaches `release-1.6` before `main` cuts its next release is stamped on the branch while `main`'s copy is not, so the same slug ends up with two batches — and unlike a central manifest there is no single artifact where the conflict can be detected. It also weakens immutability from "once merged" to "once released" for no gain. + +**Ordering — forbid the skip, as kubeadm does.** `MaximumAllowedMinorVersionUpgradeSkew = 1` with no bypass flag, and Cluster API defers to it; Flyway's `outOfOrder=false` is the same posture. Requiring 1.6 → 1.7 → 1.8 would make the pending set exactly one release's worth on every cluster, satisfying the ordering requirement with no machinery at all. Rejected as a product decision rather than a technical one: the cost lands on air-gapped and slow-moving clusters, which are a real part of the Cozystack population, and it is very hard to relax later once the ordering machinery has been left unbuilt. It also does not close patch-level skips within a line, which is exactly where backports live. + +**Runtime — Kyverno `mutateExisting` for the backfill class.** Rejected: it adds a dependency, and its existing-resource mutation is explicitly asynchronous with variable delay — disqualifying wherever the mutation must land before a Helm prune. + +**Schema — `StorageVersionMigration` (`storagemigration.k8s.io`).** Not applicable: still alpha/beta in the Kubernetes versions Cozystack supports, and App objects are served by an aggregated API server rather than stored as CRDs. + +**Language — rewriting migrations in Go.** Rejected: the operations are `kubectl` calls, and shell keeps them reviewable and testable against fake-binary fixtures without a build step in the image. The problem is the absence of a contract, not the language.