diff --git a/designs/upgrade-workflow.md b/designs/upgrade-workflow.md new file mode 100644 index 000000000..e0313469f --- /dev/null +++ b/designs/upgrade-workflow.md @@ -0,0 +1,362 @@ +# Control plane upgrade workflow + +Status: current behavior as implemented + +This document describes how the Unbounded control plane is upgraded: the +`unbounded-operator` and the components it manages. It covers both the +high-level model and the mechanics behind it. + +## Summary + +The upgrade unit is the operator. + +The operator's compiled-in version is the version of every component it manages. +Upgrading a cluster means replacing one image: the `unbounded-operator` +Deployment. The operator then re-applies every component workload at its own +version, and Kubernetes performs the rollouts. + +Two consequences follow from that and are worth stating up front: + +- **The operator does not upgrade itself.** Something external replaces its + image. That is `kubectl unbounded install` run from a newer plugin, or a + direct apply of the released operator manifest. There is no + `kubectl unbounded upgrade` command. +- **There is no per-component version selection.** The `Site` API exposes no + version, image, tag, or channel field. Components are upgraded as a set, in + lockstep with the operator. + +## Scope + +The operator owns two things: the CRDs, and five component workloads. + +| Component | Kind | Workloads | +|-----------|------|-----------| +| `net` | cluster singleton | `Deployment/unbounded-net-controller`, `DaemonSet/unbounded-net-node` | +| `machina` | cluster singleton | `Deployment/machina-controller` | +| `gantry` | cluster singleton | `DaemonSet/gantry`, `DaemonSet/gantry-containerd-config` | +| `metalman` | per-Site | `Deployment/metalman-controller-` | +| `storage` | per-Site | `DaemonSet/unbounded-storage-supervisor-` | + +Registered in `operator.DefaultRegistry` (`internal/operator/reconciler.go:66-78`). +Cluster components reconcile from the full set of `Site` objects on every pass; +per-Site components reconcile once per `Site`. + +The operator also owns twelve CRDs, six in the `unbounded-cloud.io` group and six +in `net.unbounded-cloud.io` (`internal/operator/bootstrap.go:49-62`). + +**Out of scope for this document.** The host-resident `unbounded-agent` binary +and node OS/kubelet changes are *not* part of this model. They are separate +update planes with their own triggers (`MachineOperation` of kind `AgentUpgrade`, +and `MachineConfiguration` repave respectively) and are not driven by an operator +upgrade. See `designs/agent-upgrade.md` and +`designs/lifecycle-operations.md`. Components deployed outside the operator +(`machine-ops-controller`, inventory, orca, playpen) are also out of scope. + +## High-level workflow + +``` + release tag + | + +--> container images pushed as /azure/: + +--> every Go binary stamped with internal/version.Version = + +--> build/unbounded-operator-.yaml published + | + v + kubectl unbounded install (or kubectl apply -f unbounded-operator-.yaml) + | + | updates ONLY the operator Deployment image + its ConfigMap + v + operator pod restarts (strategy: Recreate) + | + +--> [startup] apply all 12 CRDs, wait for Established + +--> [ready] health server binds, manager starts + | + v + Site reconcile + | + +--> for each component: render manifests, + | overwrite container images with /azure/:, + | stamp config hash, server-side apply + v + Kubernetes rolls each workload per its own update strategy + | + v + Site.status.conditions: NetReady / MachinaReady / GantryReady + MetalmanReady / StorageReady +``` + +Nothing in this flow compares an installed version against a desired version. +The operator stamps its own version unconditionally and lets server-side apply +converge. That makes the reconcile idempotent and means a component that drifts +(manual edit, partial failure, restored backup) is corrected on the next pass +rather than only at upgrade time. + +## Technical detail + +### Where the version comes from + +`cmd/unbounded-operator/main.go:206-207` builds the component config with: + +```go +Config: operator.Config{ + ImageRegistry: cfg.imageRegistry, + ImageTag: version.Version, + APIServerEndpoint: cfg.apiServerEndpoint, +} +``` + +`version.Version` is set at link time by `STAMP_LDFLAGS` (`Makefile:196-198`) +from `VERSION`, which defaults to `git describe --tags --always --dirty` +(`Makefile:186-191`). The operator binary therefore carries the fleet version. + +The registry is separate and resolves from `--image-registry`, then +`$UNBOUNDED_IMAGE_REGISTRY`, then `ghcr.io` (`cmd/unbounded-operator/main.go:76`). + +### Image resolution + +`internal/operator/component/env.go:74-76`: + +```go +func (c Config) Image(repository string) string { + return strings.TrimRight(c.ImageRegistry, "/") + "/azure/" + repository + ":" + c.ImageTag +} +``` + +So registry `ghcr.io`, repository `machina`, version `v0.2.0` yields +`ghcr.io/azure/machina:v0.2.0`. The `azure/` path segment is fixed, and the +registry is operator-wide: every managed component must be published to the same +registry under that path. + +Two stamping helpers exist: + +- `SetPodSpecImages` (`env.go:79-105`) rewrites every init and main container in + a pod template. Used by `net`, `machina`, `metalman`, and `storage`. +- `SetNamedContainerImage` (`env.go:112-152`) rewrites only the named container. + Used by `gantry` (`internal/operator/components/gantry/gantry.go:204`) so its + pinned busybox init container and the busybox-based + `gantry-containerd-config` DaemonSet keep their own images. + +### Embedded manifests + +Most components ship as rendered YAML compiled into the operator binary with +`go:embed` (`deploy/net/embed.go`, `deploy/machina/embed.go`, +`deploy/gantry/embed.go`, `deploy/unbounded-storage-supervisor/embed.go`), and +`unbounded-operator-build` depends on the render targets for all of them +(`Makefile:611`). The operator therefore ships a self-consistent snapshot of +those components' manifests. + +The image tag baked into that embedded YAML is effectively a build artifact: it +is overwritten at reconcile by `Config.Image(...)`. To keep that safe, +`TestEmbeddedManifestsHaveNoLatestImageTags` +(`internal/operator/manifests_guard_test.go`) fails `make test` if any embedded +component manifest pins an image to `:latest`, on the grounds that +operator-managed components must be version-matched to the operator's release. + +`metalman` is the exception: it has no embedded manifest. Its Deployment is +constructed directly in Go (`internal/operator/components/metalman/metalman.go:98-195`) +because it is per-Site and its shape depends on `Site` fields such as +`dhcpAutoInterface` and `replicas`. + +### Apply semantics + +All component objects are applied with server-side apply using field manager +`unbounded-operator` and `ForceOwnership` +(`internal/operator/component/env.go:29-32`, `env.go:234`). Force ownership +means the operator reclaims fields another manager has taken, so manual +`kubectl edit` changes to managed fields are reverted on the next reconcile. + +### CRD lifecycle + +CRDs are handled at operator startup, not in the reconcile loop. Component +reconcilers explicitly skip `CustomResourceDefinition` objects in the embedded +manifests (for example `net.go:117-121`). + +`operator.BootstrapCRDs` (`internal/operator/bootstrap.go:77-79`) server-side +applies every CRD and waits for each to become `Established`. It runs before the +manager starts, because the typed `Site` informer cannot sync until the `Site` +CRD is served. The whole operation is bounded by `CRDBootstrapTimeout`, four +minutes (`bootstrap.go:45`), and the operator's health server only binds after +bootstrap completes. The Deployment `startupProbe` budget is deliberately sized +to exceed that window: 54 failures at 5s, 270s +(`deploy/unbounded-operator/04-deployment.yaml.tmpl:85-86`). + +`CRDMaintainer` (`bootstrap.go:87-137`) then re-applies the CRDs every 60s under +leader election. Maintenance failures are logged and retried rather than +crashing the manager, because established CRDs stay served by the apiserver +regardless of operator liveness. + +The practical effect for upgrades: **CRD schema changes ship with the operator +and land before any component reconciles.** An admin never applies CRDs +separately. `kubectl unbounded install` stopped doing so for exactly this reason +(`cmd/kubectl-unbounded/app/install.go:140-143`). + +### Component configuration is preserved across upgrades + +Component ConfigMaps are create-only. Each component's `ensureConfig` creates the +embedded default only when no ConfigMap exists, and otherwise leaves the existing +object untouched (`net.go:154-187`, `gantry.go:248-282`, `storage.go:111-140`). +The apply mutator then skips that ConfigMap so the manifest apply cannot +overwrite it either (`net.go:123-127`). + +Operator upgrades therefore do not clobber operator-tuned component config. The +trade-off is that new default keys introduced by a release are not merged into an +existing ConfigMap; they have to be added deliberately. + +`machina` is the one exception. It patches `apiServerEndpoint` into the existing +`config.yaml` when the operator-resolved endpoint differs, and only that key +(`machina.go:183-202`). + +### Config changes trigger rollouts + +Each component computes a SHA-256 over the complete ConfigMap payload +(`internal/operator/component/configmap.go:17-31`) and stamps it as a pod +template annotation, for example `unbounded-cloud.io/net-config-hash` +(`net.go:31`, stamped at `net.go:144`). Editing a component ConfigMap changes +the pod template hash, which rolls the workload. + +This is a separate rollout trigger from an image change, and it applies outside +upgrades: a config edit on a stable version rolls the affected workload the same +way. + +### Retention + +No component is auto-deleted when its enabling `Site` goes away. `net`, +`machina`, and `gantry` each check for existing resources and keep reconciling +them rather than uninstalling (`net.go:49-62`, `machina.go:57-69`, +`gantry.go:95-107`). The reasoning is sharpest for `net`: deleting +`unbounded-net-node` would break pod networking cluster-wide. + +The consequence for upgrades is that a retained component continues to be +version-matched to the operator even with no `Site` enabling it. Removal +requires an explicit uninstall flow, which does not exist yet. + +## Rollout behavior per workload + +Once the operator applies the new images, Kubernetes drives each rollout using +that workload's own strategy. These differ meaningfully. + +| Workload | Strategy | Availability during upgrade | +|----------|----------|-----------------------------| +| `unbounded-operator` | `Recreate` | Old pod terminates before the new one starts. Components keep running; only reconciliation pauses. | +| `unbounded-net-controller` | `RollingUpdate`, maxSurge 0, maxUnavailable 1 | Genuine zero-available window. | +| `metalman-controller-` | `RollingUpdate`, maxSurge 0, maxUnavailable 1 | Genuine zero-available window. | +| `unbounded-net-node` | `RollingUpdate`, maxUnavailable 100% | All nodes update simultaneously. | +| `unbounded-storage-supervisor-` | `RollingUpdate`, maxUnavailable 100% | All nodes update simultaneously. | +| `gantry` | `RollingUpdate`, maxUnavailable 10% | Gradual, deliberately limited blast radius. | + +The `maxSurge: 0` settings are not accidental. Both `unbounded-net-controller` +and `metalman` run `hostNetwork` and bind host ports, so a surge pod cannot start +while the old pod holds them. Allowing surge would deadlock the rollout. The +reasoning is recorded at `deploy/net/controller/03-deployment.yaml.tmpl:17-24` +and `internal/operator/components/metalman/metalman.go:138-142`. + +Both hostNetwork Deployments and `unbounded-operator` itself use +`imagePullPolicy: Always`, so a pull failure at the new tag surfaces as a stuck +rollout rather than a silent no-op against a stale cached image. `gantry` uses +`IfNotPresent`. + +## Verification + +### Operator rollout + +`kubectl unbounded install --wait` blocks on the operator rollout, then on CRD +establishment (`install.go:152-160`). + +The rollout check is stricter than usual and intentionally so +(`install.go:521-531`): + +```go +return deploy.Status.ObservedGeneration >= deploy.Generation && + deploy.Status.UpdatedReplicas == desired && + deploy.Status.Replicas == desired && + deploy.Status.AvailableReplicas == desired +``` + +Requiring `Replicas == UpdatedReplicas == AvailableReplicas` matters during an +upgrade specifically. A weaker check can report success while the *old* operator +pod is still the available one, which would let a caller proceed against the +previous version. + +### Component readiness + +The reconciler publishes one condition per component on each `Site`, in registry +order (`internal/operator/reconciler.go:144-192`), so readiness is waitable: + +``` +kubectl wait --for=condition=NetReady site/ +kubectl wait --for=condition=MachinaReady site/ +kubectl wait --for=condition=GantryReady site/ +kubectl wait --for=condition=MetalmanReady site/ +kubectl wait --for=condition=StorageReady site/ +``` + +Conditions belonging to components no longer in the registry are pruned, so the +condition set tracks the operator version rather than accumulating history. + +### Automated gates + +Upgrade is covered by CI, not only by unit tests: + +- `.github/workflows/release-upgrade.yaml` runs after the release build, + verifies the signed release assets, decides `init` vs `upgrade` against the + target cluster, runs `kubectl unbounded install`, waits for every managed + workload's rollout, runs smoke tests, and only then publishes the draft + release. +- `.github/workflows/operator-upgrade-e2e.yaml` and + `hack/operator-upgrade-e2e/e2e.py` install a previously released version and + upgrade to the current tree, asserting migration and CRD-repair behavior. +- `internal/operator/manifests_guard_test.go` blocks `:latest` in embedded + component manifests as part of `make test`. + +## Release artifacts + +A release tag produces the inputs this workflow consumes (`Makefile:1387-1425`): + +- `build/unbounded-operator-.yaml`, the rendered operator manifests + concatenated into one directly appliable file. +- `build/unbounded-manifests-.tar.gz`, per-component rendered manifests + plus a `VERSION` file, for inspection and for installs that do not use the + plugin. +- `build/unbounded-release-bom-.json`, a digest-pinned bill of + materials for the release images. + +## Design characteristics and constraints + +These are properties of the current model, recorded so they are considered +deliberately rather than discovered during an upgrade. + +**Lockstep by design.** `SiteComponentSpec` carries only `Enabled`; the API +comment states it directly: components install at the operator's own version, so +neither namespace nor image is configurable per component +(`api/machina/v1alpha3/site_types.go:141-148`). This buys guaranteed version +consistency across components that share CRDs and wire formats, and it removes a +version-skew matrix. It costs the ability to upgrade or pin one component +independently, including to pick up a single hotfix. + +**No canary or staged rollout across components.** A single reconcile pass +applies every component. Ordering is registry order, cluster components before +per-Site ones, but there is no gating: the operator does not wait for `net` to +become healthy before applying `machina`. Staging has to happen at the cluster +level, by upgrading clusters in sequence. + +**Rollback means deploying an older operator, and is not symmetric.** Component +workloads revert cleanly, since their images are re-stamped from the older +operator's version. CRDs do not: `BootstrapCRDs` applies, it never removes or +reverts fields, so a downgrade leaves the newer CRD schema in place. Downgrades +across a CRD schema change need to be reasoned about individually. + +**Two components have a real zero-available window.** `maxSurge: 0` on +`unbounded-net-controller` and `metalman` is forced by hostNetwork port binding. +For `net` this pauses CIDR allocation, gateway pool, and peering reconciliation +until the new pod is ready; the dataplane (`unbounded-net-node`) is unaffected by +the controller gap, but is itself rolled at `maxUnavailable: 100%`. + +**Config defaults do not migrate.** Because component ConfigMaps are create-only, +a release that adds a new default config key does not add it to clusters that +already have the ConfigMap. New keys must either be optional with a sane in-code +default, or called out in release notes. + +**Single registry.** `Config.Image` hardcodes the `azure/` path segment and uses +one registry for all components. Mirroring for an air-gapped or +restricted-registry install means mirroring every component to a single registry +under that path, and setting `UNBOUNDED_IMAGE_REGISTRY` accordingly.