Talks

Watch the SmallWorlds Lightning Talk.

Refer to the SmallWorlds Architecture Diagram for system topology and data flows.

SmallWorlds Setup Guide

The SmallWorlds dashboard — auto-discovered applications and infrastructure at a glance.

Warning

Prototype — not production-ready. SmallWorlds is still in an early prototyping state. It is intended for experimentation and evaluation only, and is not yet suitable for production environments. Expect breaking changes, incomplete hardening, and no stability or upgrade guarantees. Use at your own risk.

Important

Not Hetzner-only. smallworlds-init.sh supports two deployment targets: hetzner (a Hetzner Cloud VM provisioned by Terraform) and local (an existing Linux machine in your LAN, bootstrapped in place over SSH — no cloud account needed). Hetzner is still required for public DNS — either via Terraform on the hetzner target, or via a free Hetzner DNS zone + API token on an internet-exposed local deployment. A LAN-only local deployment needs no Hetzner account, no domain registration, and no cloud resources at all. See the "Deployment Instructions" section below and doc/local-deployment.md for details.

This document outlines the deployment process for a SmallWorlds GitOps cluster. The architecture relies on an upstream foundation repository and a private, user-controlled configuration repository.

System Components

This project is built upon several foundational open-source technologies, core infrastructure services (installed by default), and optional user applications (selectively installed during initialization):

Infrastructure and Cluster Management

Name	Source URL	Role in this Project
Terraform	terraform.io	Infrastructure as Code tool used on the hetzner target to provision the VM and DNS; on the local target only the (optional) DNS zone is Hetzner-managed, the node itself is bootstrapped over SSH without Terraform.
Kubernetes	kubernetes.io	Core container orchestration system that serves as the foundation for the cluster.
Argo CD	argoproj.github.io/cd	GitOps continuous delivery tool that synchronizes cluster state with the configuration repository (accessible at deploy.<domain>).
Velero	velero.io	Cluster backup and disaster recovery solution.
Grafana	grafana.com	Operational dashboard for cluster monitoring and observability.
CloudNativePG	cloudnative-pg.io	High-availability PostgreSQL database clustering.
Garage	garagehq.deuxfleurs.fr	S3-compatible object storage backend.
Homepage	gethomepage.dev	Application dashboard automatically configured and accessible at dashboard.<domain>.
Keycloak	keycloak.org	Identity Provider (IdP) for Single Sign-On (SSO) and WebAuthn/Passkey management.
Traefik	traefik.io	Ingress routing and reverse proxy for handling incoming requests.
Cert-Manager	cert-manager.io	Automated TLS certificate provisioning and management.
Pod Gateway	tenants/pod-gateway/	Append-only personal data pods (pod.<domain>). Apps deposit each member's source data into a pod they cannot read; the member's home device pulls it and never deletes. Optional. See doc/pod-archive.md.
Operator Console (deactivated)	operator-console/	Not deployed since 2026-08-23. The code, manifests and ADRs are all still here and unchanged, but no cluster runs it — the browser setup wizard (admin-tools/setup-wizard/) takes over the install half of what it did, and the observe half is on hold until that lands. See docs/adr/0051-a-single-use-setup-wizard-replaces-the-bootstrap-launcher.md and doc/tenant-operator-console.md.

End User Applications

Name	Source URL	Role in this Project
Collabora Online	collaboraonline.com	Powerful online office suite for collaborative document editing. Integrated into Nextcloud.
Excalidraw	excalidraw.com	Virtual collaborative whiteboard tool.
Forgejo	forgejo.org	Git repository management and software collaboration.
Immich	immich.app	High performance photo and video backup.
Jitsi Meet	jitsi.org	Secure video conferencing and communication platform.
Nextcloud	nextcloud.com	File synchronization and collaboration.
Plane	plane.so	Open-source project management tool.
Stalwart	stalw.art	Self-hosted mail server with OIDC directory integration. Optional, and not installed by default: mail is an opt-in capability (docs/adr/0049) — onboarding hands out action-token links out of band and alerting pushes over HTTP, so nothing in the cluster needs a mail server. See doc/mail.md.
Bulwark	github.com/bulwarkmail/webmail	JMAP webmail client connected to Stalwart. Select Stalwart too — this is the one app that depends on it.

---

Deployment Instructions

SmallWorlds supports two deployment targets — you are asked which one you want as the first question:

• hetzner — a Hetzner Cloud VM provisioned by Terraform, with public DNS and Let's Encrypt certificates. The internet-facing default.
• local — an existing Linux machine in your LAN (laptop, mini-PC, home server; 32 GB RAM recommended), bootstrapped in place over SSH without Terraform. By default LAN-only: no public DNS (you add router-DNS or /etc/hosts entries). Optionally internet-exposed: with a registered domain, a Hetzner DNS token and router port forwards (80/tcp, 443/tcp, 10000/udp), an in-cluster DDNS job keeps the DNS records pointed at your home IP. See doc/local-deployment.md for requirements and limitations before you start.

There are two ways to install, and they do the same thing — the wizard runs the scripts, so there is one install code path, not two.

The setup wizard (recommended)

git clone https://github.com/stephan271/smallworlds.git
cd smallworlds
./smallworlds-setup

A browser opens on 127.0.0.1. Five panels ask for your domain, your email address, where the cluster should run, a Git access token, and which applications your community gets; everything else is defaulted and sits behind Advanced settings. It creates the private settings repository for you, checks your answers against the real APIs before anything is created, shows you exactly what it will do, then installs and hands you the credentials.

The checks are the part worth having: whether your registrar actually delegates the domain to Hetzner DNS (get this wrong and every certificate stays pending with nothing saying why), whether the tokens work, and whether your data and backup directories are on different physical disks. Each of those otherwise fails minutes into an install.

Prerequisites are the same as the scripted route below, minus the manual steps the wizard does for you (creating the settings repository, creating the DNS zone, uploading the SSH key). You still need to register the domain yourself and point its nameservers at Hetzner — see step 1.

The scripted route

The wizard is a front end for these two scripts; running them directly is fully supported and is what you want for automation. Both also run headless with SMALLWORLDS_NONINTERACTIVE=1, taking their answers from their cache files and exiting 78 naming any required value that is missing.

Steps 1 and 3 below apply to the hetzner target only; steps 2 and 4 apply to both.

1. Domain Registration & DNS Configuration (hetzner target only)

Important

Domain Registration is manual: SmallWorlds does not register or reserve the domain name for you — neither the wizard nor the scripts. You must manually register the domain at a registrar of your choice (e.g., Hetzner Domain service, Namecheap, Cloudflare) and point the domain's Nameservers to Hetzner's DNS servers (e.g., helium.ns.hetzner.de, oxygen.ns.hetzner.com, hydrogen.ns.hetzner.com). Domain registration will incur costs at your registrar. For a LAN-only local deployment none of this applies — the domain never has to be registered, since name resolution happens inside your LAN. For an internet-exposed local deployment, or for Let's Encrypt certificates on a LAN-only one, domain registration and the Hetzner nameservers ARE required (records are then managed by the in-cluster DDNS job instead of Terraform).

The DNS zone and DNS records are automatically managed via the Hetzner API token provided during provisioning (which is free of charge). Subdomains are routed to the provisioned server IP.

A .dev cluster additionally gets its own scoped mail domain: addresses are user@dev.<domain>, with MX/SPF/DKIM/DMARC records placed under the dev subdomain so they can never conflict with production's mail records at the zone apex (see doc/tenant-stalwart.md).

2. Configuration Repository Initialization

A private Git repository is required to store application state and configuration overrides.

Execute the initialization script from the root of this repository:

./admin-tools/prepare-community-repo.sh

This script handles:

• Prompting for your target domain (e.g. smallworlds.network) and environment extension in subdomain syntax (e.g. .dev for a dev cluster, giving hostnames like identity.dev.smallworlds.network; leave empty for production).
• Automatically creating a private GitHub repository using the gh CLI (if installed), or allowing you to provide an empty Git URL manually. (The wizard creates it through the GitHub API instead, so it does not need gh.)
• Interactive selection of optional applications.
• Generation of the corresponding kustomization.yaml overlays, injecting environment-specific patches for the chosen domain.
• Initialization of the local Git repository and automatic push to your remote repository.

3. Infrastructure Provider Setup (hetzner target only)

For the local target, skip this section — instead make sure the target machine meets the requirements in doc/local-deployment.md (systemd Linux, SSH + sudo access, firewalld disabled or configured for k3s, 100 GB+ free disk).

For the hetzner target, these steps are required:

1. Create a Hetzner Cloud account and a new project.
2. Generate an API Token with Read & Write permissions. Save this token.
3. Nothing else. The Primary IP — the community's permanent public address — is created for you: smallworlds-init.sh looks for smallworlds-ip (or smallworlds-ip-dev for a .dev environment — Hetzner resource names always use the dash form, regardless of the DNS syntax), and creates it if it is not there, in a location that can actually build the node. If you would rather create it yourself in the Console, do: it is adopted rather than duplicated, and it then decides where the node goes.

Where the node ends up. Hetzner regularly sells a server size out in one location while the others still have it, so the location is not hard-coded. The installer asks the API which of Nuremberg (nbg1), Falkenstein (fsn1) and Helsinki (hel1) can currently supply the node size, takes the first that can, and creates the Primary IP there. All three are in the EU, so member data stays inside it.

Once the Primary IP exists it pins the choice: it is datacenter-bound, and it is the address every DNS record, the PTR and the tailnet point at. If that location later has no capacity, the installer stops before creating anything and tells you where the hardware is — moving there means deleting the Primary IP in the Console and re-running, which gives you a new address and rewrites every record to match. Rebuilding the node. cloud-init runs only on a node's first boot, and the server ignores later changes to its user-data — so a change to the bootstrap template (infrastructure/cloud-init/k3s-node.yaml.tpl) reaches a running node by exactly one route: replacing it. Run SMALLWORLDS_REPLACE_NODE=1 ./smallworlds-init.sh. Terraform detaches both volumes, deletes the server, creates the new one and re-attaches — doing it by hand in the Console fails, because Hetzner refuses to delete a server while volumes are attached. Both volumes and the Primary IP survive, so the community keeps its data and its address; anything living only on the node's own disk does not, and Argo CD re-deploys the cluster from your overlay. Use this rather than destroy-cluster.sh, which takes the volumes with it.

The node size is remembered. If a node is already running, the installer keeps the size it is — re-running is not a request to resize it — and otherwise reuses the size the last run chose. Only HETZNER_SERVER_TYPE=... in the environment changes it, which is how you ask for a resize deliberately. To pin a location, set HETZNER_LOCATIONS="fsn1 nbg1" the same way. Type names are Hetzner's own and lowercase (cpx42, not CPX42) — the installer folds the case for you and, if the name is one Hetzner does not know at all, says so rather than reporting it as sold out. If the capacity lookup itself cannot be answered, the install continues and lets the Hetzner API give the real answer; only a definite "nowhere" stops it.

DNS records follow your app selection. Only the applications you chose get an A record, alongside the always-installed identity, dashboard, monitoring, deploy and vpn (plus the apex on production). A record for photos.<domain> with no Immich behind it sends members to a certificate error and makes the zone read as though the app were installed. Selecting an app later adds its record on the next run.

4. Cluster Provisioning

Warning

Before your first run against a fresh hetzner node (or a first-time internet-exposed local node), make sure your overlay repo (my-community-config, from step 2) is pinned to the latest smallworlds release tag — see "Managing Updates" below. Some infrastructure-level changes (cloud-init) take effect immediately on a brand-new node, while their corresponding ArgoCD-managed component only appears once your overlay's pinned tag includes it. If those are out of sync at first boot, the result isn't a degraded app or two — it can be a total outage (e.g. every TLS certificate stuck, no app reachable) with nothing obviously pointing at the cause. A newly created overlay may not have caught up via the weekly Renovate PR yet, so check/bump it manually first. The wizard pins the newest release by default, so this applies mainly to the scripted route.

Execute the bootstrap script to provision the server and install Kubernetes/ArgoCD. On the hetzner target it provisions the VM and DNS via Terraform; on the local target it bootstraps your LAN machine over SSH (asking for its SSH target, e.g. root@192.168.1.50, or localhost to install on the machine you are running the script on).

git clone https://github.com/stephan271/smallworlds.git
cd smallworlds
./smallworlds-init.sh

When prompted for Git credentials, provide:

• URL: The HTTPS URL of your private configuration repository (SSH URLs are unsupported).
• Username: Your Git platform username.
• Access Token: A Personal Access Token (PAT) with read-only access to repository contents.

5. Authentication Configuration

By default, registration is invitation-only. To enable self-registration, patch the Keycloak configuration via your kustomization.yaml:

patches:
  - target:
      kind: Job
      name: keycloak-realm-config
      namespace: keycloak
    patch: |-
      - op: replace
        path: /spec/template/spec/containers/0/env/1/value
        value: "self-registration"

6. Custom Application Deployment

To deploy external applications, add standard Kubernetes manifests to your configuration repository and declare them in your kustomization.yaml. ArgoCD will synchronize the state.

---

Maintenance Operations

The rebuild procedures below are for the hetzner target (they drive Terraform). For local deployments, the equivalent lifecycle (uninstall/reinstall preserving data, full wipe) is described in doc/local-deployment.md.

Rebuild (Preserve Data)

This procedure replaces the VM while retaining the persistent volume containing cluster state and data.

cd infrastructure/terraform
terraform destroy -target=hcloud_server.smallworlds_pilot_node
terraform apply

Clean Rebuild (Wipe Data, Preserve TLS Certificates)

This procedure wipes all cluster data but first backs up the TLS certificates to your local machine (~/.smallworlds/cert-backups/<production|dev>/), then re-injects them into the new cluster to avoid Let's Encrypt rate limits. For the dev cluster, prefix each command with ENV_EXT=".dev".

./admin-tools/prepare-fresh-rebuild.sh
cd infrastructure/terraform
terraform destroy -target=hcloud_server.smallworlds_pilot_node
terraform apply
cd ../.. && ./admin-tools/restore-certs-from-laptop.sh

The restore script waits for the new cluster's API, then applies the saved certificate secrets before cert-manager re-issues them; if no local backup exists it exits cleanly and certificates are issued fresh. You can also snapshot certificates at any time without a rebuild via ./admin-tools/backup-certs-to-laptop.sh.

Backups & Storage

Storage layout, the backup chain, restore procedures and scaling paths are all documented in doc/storage-and-backup.md. The chain is Garage-first: databases (CloudNativePG/barman), Velero's cluster-state dumps and nightly PV copies all land in the in-cluster Garage S3, and a single replicator CronJob mirrors every bucket offsite at 04:00.

Important

The offsite leg requires one-time operator setup — an S3 target (recommended: a Backblaze B2 bucket with versioning) and the replicator-config-secret. Until then, backups never leave the node and the nightly replicator job fails (which the monitoring emails about). Follow infrastructure/kubernetes/tenants/backup-replicator/README.md.

Managing Updates — the two-repo model

SmallWorlds runs on two repositories, and understanding their interplay is the key to safe day-2 operations:

Repo	Role	Who changes it
smallworlds (this repo, public)	The upstream base: all app/infra manifests under infrastructure/kubernetes/. Released as semver tags (v1.0.0, v1.1.0, …).	The SmallWorlds project.
my-community-config (yours, private)	The overlay ArgoCD actually deploys from. Each app's kustomization.yaml remote-references the base at a pinned tag (?ref=v1.0.0) plus your local patches.	You, the operator.

Each cluster (production, .dev) has its own private overlay repo, generated the same way by prepare-community-repo.sh with the matching environment extension.

ArgoCD only watches your private overlay. It does not track the base's moving branch. Because the overlay pins the base to an immutable tag, upstream changes never reach your cluster on their own — adopting a new base version is always a deliberate, auditable action in your repo.

Note

This separates two independent concerns. Drift reconciliation (ArgoCD selfHeal) keeps the cluster matching whatever is declared and stays on — it's safe and low-risk. Version adoption (moving to newer upstream code) is the deliberate lever described below. Don't conflate them.

Adopting a new release manually

Bump the pinned tag everywhere in your overlay and commit — ArgoCD (which watches this repo) then syncs the change deterministically:

# in my-community-config, e.g. v1.0.0 -> v1.1.0
grep -rl 'v1.0.0' . | xargs sed -i 's#v1.0.0#v1.1.0#g'
git commit -am "Bump upstream smallworlds base to v1.1.0" && git push

Rollback is just as simple: revert that commit. Because the ref is immutable, what you tested is exactly what deploys.

Tip

prepare-community-repo.sh pins to a release tag by default (it prompts for the version). You can answer HEAD to always track the latest main, but avoid it in production: ArgoCD only re-pulls a floating HEAD non-deterministically (on cache expiry), so you lose reproducibility and can't tell what's actually running.

Automated weekly update proposals (Renovate)

An in-cluster Renovate CronJob is pre-wired to reduce the toil without giving up control. Every Monday it opens one pull request in your private overlay that bumps the pinned base tag to the newest smallworlds release (config in my-community-config/renovate.json). It does not auto-merge — you review the changelog and merge when ready; the merge is the commit that triggers ArgoCD. This gives you a low-effort cadence and a human gate and a full audit trail.

Requirements for the PR automation:

• The private overlay must be listed in the Renovate CronJob's RENOVATE_REPOSITORIES (added via an overlay patch in your kustomization.yaml, so operator-specific config stays out of the public base).
• The Git token Renovate uses (repo-git-creds) must have pull-request / write access to the private overlay repo, not just read.

---

Administrative Tools (admin-tools/)

The admin-tools/ directory contains helper scripts to automate common operations and maintenance tasks:

Cluster Lifecycle & Disaster Recovery

• backup-certs-to-laptop.sh: Downloads Let's Encrypt certificates from the cluster to your local machine to avoid rate limits during rebuilds.
• restore-certs-from-laptop.sh: Injects previously backed-up certificates into a newly built cluster before cert-manager re-issues them.
• prepare-fresh-rebuild.sh: Prepares a cluster for a clean reinstall by backing up certificates and wiping all persistent application data on the server.
• destroy-cluster.sh: Wraps the cluster teardown process. It automatically backs up your certificates and then safely executes terraform destroy to delete the cloud resources.
• build-golden-image.sh: Builds a pre-configured Hetzner snapshot (golden image) to speed up future VM provisioning.

Setup

• setup-wizard/: The browser setup wizard behind ./smallworlds-setup — five panels with live validation, driving the two setup scripts rather than reimplementing them. Python standard library only, no dependencies and no build step. See docs/adr/0051-a-single-use-setup-wizard-replaces-the-bootstrap-launcher.md for why it replaced the Operator Console.
• test-setup-wizard.sh: Tests the wizard's server, defaults and plan, then walks its five panels in a real browser using the Playwright installed for e2e-tests/ (skipped when absent).
• test-noninteractive-install.sh: Asserts that both setup scripts run with no tty under SMALLWORLDS_NONINTERACTIVE=1, that a missing required value exits 78 naming it, and — over a pty — that interactive mode is unchanged.

Repository & Version Management

• prepare-community-repo.sh: Interactive script to initialize a new private configuration repository, select applications, and wire it up to the upstream base. Also runs non-interactively (SMALLWORLDS_NONINTERACTIVE=1), which is how the setup wizard drives it.
• bump-version.sh: Automatically detects the highest release tag, increments the patch version, creates a new Git tag, and pushes it to origin.
• update-community-version.sh: Updates all kustomization.yaml files in your private config repository to pin them to a specific upstream release, then commits and pushes the change.

Utilities

• test-pr-locally.sh: Deploys a local testing environment to validate pull requests.
• generate_domain_patches.py: Automatically generates Kustomize domain patches for all applications when a non-default domain or environment extension (e.g., .dev) is used.
• bulk-invite.py & update_realm_json.py: Python utilities for Keycloak realm manipulation and user invitations.

---

End-to-End Smoke Tests

Browser-based Playwright smoke tests simulate real users logging in via SSO and exercising each application. They live in e2e-tests/tests and run against a live SmallWorlds community.

./e2e-tests/run-smoke-tests.sh <domain> [keycloak-admin-password]

# e.g.
./e2e-tests/run-smoke-tests.sh smallworlds.network

If the Keycloak admin password is omitted, the runner reads it from the cluster via kubectl. The script checks service availability, provisions two test users (sw-test-alice, sw-test-bob), and runs the suite.

OIDC test depth (two-tier)

Full OIDC login roundtrips require the applications to trust the TLS certificate of identity.<domain> for their server-side discovery/token calls. That holds in production (Let's Encrypt) but is structurally impossible on ephemeral staging clusters. The suite therefore runs at one of two depths:

Mode	How to run	What it verifies
Shallow wiring (default)	./e2e-tests/run-smoke-tests.sh <domain>	Each app redirects into Keycloak's authorize endpoint — proving client config, secrets, issuer URL, in-cluster DNS, and OIDC wiring. The deeper login-roundtrip tests are skipped.
Full OIDC	FULL_OIDC=1 ./e2e-tests/run-smoke-tests.sh <domain>	The complete login roundtrips run — auto-login into each app, then asserting the app's authenticated UI loads (Files listing, inbox, timeline, dashboard, etc.). Requires app-trusted certificates, i.e. production.

In shallow mode you'll see the roundtrip tests reported as skipped (with the reason Full OIDC roundtrip needs app-trusted certificates — run with FULL_OIDC=1); this is expected, not a failure.

Other options

These environment variables override the CLI arguments:

Variable	Effect
DOMAIN	Target domain (alternative to the first positional argument).
KC_ADMIN_PASS	Keycloak admin password (alternative to the second positional argument).
FULL_OIDC=1	Run the full OIDC login roundtrips (see above).
HEADED=1	Run in headed browser mode instead of headless.
SLOWMO=500	Slow operations down by the given number of milliseconds — useful when watching a headed run.
SKIP_PROVISION=1	Skip test-user provisioning (reuse existing sw-test-* users).
KUBECONFIG	Path to the kubeconfig used to read the admin password (default: ~/.smallworlds/kubeconfigs/<production|dev>.yaml, matching the configured env_ext).

The HTML report is written to e2e-tests/reports/html; view it with cd e2e-tests && npx playwright show-report reports/html.

---

Developer Guide: Cutting a New Release

Important

While the Operator Console is deactivated (since 2026-08-23), the console and launcher parts of this checklist are optional. Step 1b (the image pin) can be skipped, and publish-operator-console.yml / publish-bootstrap-launcher.yml need not be run — no cluster deploys the console and no operator needs the launcher to install. The 4 bootstrap assets are still required; a release without console artifacts is complete for the wizard-based install. The instructions below are left intact for when the console returns. See docs/adr/0051-a-single-use-setup-wizard-replaces-the-bootstrap-launcher.md.

A SmallWorlds release is a Git tag plus two independently published families of signed release attachments and one container image. Tagging alone does not produce a usable release — the tag only becomes installable once the publishing workflows have run against it.

What a complete release contains

Group	Files	Produced by	Consumed by
Bootstrap assets (4)	smallworlds-bootstrap-<tag>-linux-amd64.tar.gz + .sha256 + .sig, bootstrap-assets.manifest.json	Publish Bootstrap Assets	The launcher, which downloads and verifies them, then installs their contents onto the Cluster Node.
Launcher artifacts (10)	5 native archives (linux_amd64, linux_arm64, darwin_amd64, darwin_arm64, windows_amd64), SHA256SUMS + .sig + .pub, the SPDX SBOM, THIRD-PARTY-NOTICES.txt	Publish Bootstrap Launcher	Operators, who download one archive by hand to get the launcher in the first place.
Operator-console image	ghcr.io/stephan271/smallworlds-operator-console:<tag> (multi-arch amd64+arm64)	Publish Operator Console image	The operator-console Deployment, whose image pin in tenants/operator-console/deployment.yaml must name this tag — see Step 1.

Note

The GitHub UI shows two extra entries — "Source code (zip)" and "Source code (tar.gz)". Those are auto-generated by GitHub and are not real attachments, so a complete release reads as 16 in the browser but 14 via gh release view --json assets.

The operator-console image is not a release attachment: it is published to the GHCR registry, and it reaches a cluster only when the overlay bumps its pinned base tag (like every other manifest). It is counted separately from the 14 release assets.

A tag carrying only the bootstrap assets is installable by an operator who already has a launcher, but offers no way to obtain one. Releases v1.2.25 through v1.2.27 are in exactly that state; v1.2.28 is the first complete release. A tag whose console-image workflow never ran — or whose deployment.yaml pin was never bumped to it — deploys a console that silently reports an older release; v1.2.35 through v1.2.37 were in exactly that state (the pin lagged at v1.2.34), fixed in v1.2.38.

One-time setup

The two release-attachment workflows (assets and launcher) sign with the repository secret SMALLWORLDS_RELEASE_ED25519_PRIVATE_KEY_B64. See docs/releases/github-release-signing-setup.md. The console-image workflow signs nothing — it authenticates to GHCR with the built-in GITHUB_TOKEN.

Important

The signing key must match defaultReleaseSigningPublicKey, compiled into the launcher under operator-console/internal/bootstrapassets/. That compiled key is the only trust anchor a launcher has: assets signed by any other key are rejected outright, with no override. Rotating the release key is therefore a code change plus a launcher rebuild, not a secrets change.

Step 1 — Review and commit the pre-tag changes

Two things must be committed before tagging, because the publishing workflows build from the tagged tree (Step 2's warning) and the console-image pin must reference the tag it ships in:

1a. The input lock. Every release needs a reviewed lock at docs/releases/bootstrap-inputs/<tag>.json pinning the exact K3s installer and Argo CD manifest — version, HTTPS URL, and SHA-256 — that the packager may download. Copy the previous release's lock, update release, and either keep the pins (re-verify the digests against their official sources first) or bump them.

cp docs/releases/bootstrap-inputs/v1.2.28.json docs/releases/bootstrap-inputs/v1.2.29.json
# edit "release", then independently re-verify each digest:
curl -sS https://get.k3s.io | sha256sum

See docs/releases/bootstrap-inputs/README.md.

1b. The operator-console image pin. (Skip while the console is deactivated.) Bump the image tag in infrastructure/kubernetes/tenants/operator-console/deployment.yaml to the release you are about to cut, so the tagged tree pins the console image that Step 4 will build for that tag:

sed -i 's#smallworlds-operator-console:v[0-9.]*#smallworlds-operator-console:v1.2.29#' \
  infrastructure/kubernetes/tenants/operator-console/deployment.yaml

The pin is self-referential — the deployment.yaml at tag vX names image :vX, and Step 4 builds :vX from that same tag (its VERSION build-arg makes the console report vX). Skipping this leaves the console reporting the previous release; that is the v1.2.34 lag described above.

Commit and push 1a and 1b together, before tagging.

Step 2 — Tag the release

./admin-tools/bump-version.sh        # tags main with the next patch version and pushes

Warning

All three workflows build from the tagged tree, not from main. Anything committed after the tag is invisible to them, and a tag can never be packaged with tooling it does not contain. This is precisely why v1.2.25–v1.2.27 cannot be given launcher artifacts retroactively: admin-tools/package-bootstrap-launcher.sh was committed after all three were tagged. The fix is always a new tag, never a re-run.

bump-version.sh only tags — it does not create the input lock or bump the console pin. Do step 1 first.

Step 3 — Validate before publishing

Run all three workflows with publish=false first. The two attachment workflows each build and sign with a throwaway key and upload the result as a downloadable workflow artifact, touching no release; the console workflow builds and tests the image without pushing it. None touches a release or the registry:

gh workflow run publish-bootstrap-assets.yml    -f release=v1.2.29 -f publish=false
gh workflow run publish-bootstrap-launcher.yml  -f release=v1.2.29 -f publish=false
gh workflow run publish-operator-console.yml    -f release=v1.2.29 -f publish=false
gh run list --workflow=publish-bootstrap-launcher.yml

Note

In a validation run the assets/launcher step "Publish GitHub Release attachments" reports skipped and "Upload validation artifact" reports success. The console workflow has no signed-attachment stage: with publish=false it builds and runs go test ./... + the client build but skips the registry login and push. Both are the designed outcome, not a failure.

Step 4 — Publish

gh workflow run publish-bootstrap-assets.yml    -f release=v1.2.29 -f publish=true
gh workflow run publish-bootstrap-launcher.yml  -f release=v1.2.29 -f publish=true
gh workflow run publish-operator-console.yml    -f release=v1.2.29 -f publish=true

Run the assets workflow first: it creates the GitHub Release object if absent (gh release view || gh release create), whereas the launcher workflow only does gh release upload and needs the release to exist.

The launcher job takes roughly five minutes — npm ci, the client build, go test ./..., then five cross-compiles — and uploads all ten files in its final step. An empty-looking release mid-run is normal; check the run, not the release page. Avoid dispatching concurrent publish=true runs against one tag: they race on gh release upload --clobber over identical filenames.

The console workflow is independent of the two release-attachment workflows — it pushes only to GHCR and touches no release object, so it can run in parallel with them. It publishes ghcr.io/stephan271/smallworlds-operator-console:<tag> for amd64+arm64; this must be the tag pinned in deployment.yaml at Step 1b, or the console pod will ImagePullBackOff once an overlay adopts the release.

Step 5 — Verify the published release

gh api repos/stephan271/smallworlds/releases/tags/v1.2.29 -q '.assets | length'   # expect 14

gh release download v1.2.29 -D /tmp/verify -p 'SHA256SUMS*' \
  -p 'smallworlds-bootstrap-launcher_v1.2.29_linux_amd64.tar.gz'
cd /tmp/verify
openssl pkeyutl -verify -rawin -pubin -inkey SHA256SUMS.pub -in SHA256SUMS -sigfile SHA256SUMS.sig
sha256sum -c --ignore-missing SHA256SUMS

# the published key must equal the launcher's compiled trust anchor
openssl pkey -pubin -in SHA256SUMS.pub -outform DER | tail -c 32 | base64 -w0
grep -o 'defaultReleaseSigningPublicKey = "[^"]*"' \
  operator-console/internal/bootstrapassets/*.go

The last check is the one that matters: if the published key and the compiled anchor diverge, every launcher will refuse the release.

Then confirm the console image exists at the pinned tag and that deployment.yaml agrees with it:

docker manifest inspect ghcr.io/stephan271/smallworlds-operator-console:v1.2.29 >/dev/null && echo "image present"
grep -o 'smallworlds-operator-console:v[0-9.]*' \
  infrastructure/kubernetes/tenants/operator-console/deployment.yaml   # must equal the tag above

Step 6 — Curate the release notes

Step 4's assets workflow creates the GitHub Release object but leaves it with GitHub's auto-generated body. Give operators a human-readable summary — what changed, and any upgrade-affecting notes (backup-model changes, newly optional apps, breaking renames):

gh release edit v1.2.29 --title "SmallWorlds v1.2.29" --notes-file <notes.md>   # or --notes "..."

This is the only step whose output faces operators rather than tooling; the deploy works without it, but the Releases page is where they read what a bump will do to them.

Local dry run

To exercise packaging without touching GitHub — worth doing before a first release from a materially changed tree:

cd operator-console/web && npm ci && npm run generate:api && npm run check && npm run build
cd .. && go test ./...
cd .. && SOURCE_DATE_EPOCH="$(git log -1 --format=%ct)" \
  admin-tools/package-bootstrap-launcher.sh --version v1.2.29 --output-directory /tmp/launcher-test
admin-tools/test-build-bootstrap-assets.sh      # structural + reproducibility check for the asset archive

Packaging is deterministic within a toolchain: CI pins Go 1.25.0, so archives built locally on a different Go version will not match CI's checksums. That is expected and not a reproducibility failure. A local GOEXPERIMENT (e.g. nodwarf5) can also cause spurious internal compiler errors during the cross-compiles; clear it with GOEXPERIMENT=none.

Tip

Adding a descriptor for the new release to DefaultCatalog() in operator-console/internal/bootstrapassets/manager.go is optional. A launcher that has never heard of a release falls back to fetching that release's published bootstrap-assets.manifest.json and accepting it solely on the compiled signing key's word — which is what lets an old launcher install a newer release, and what avoids the circularity of needing an archive's digest before the archive exists.

Step 6 — Let operators adopt it

Publishing does not deploy anything. Each operator's private overlay pins an immutable tag, so the new release only reaches a cluster when someone bumps that pin — either manually or by merging the weekly Renovate PR. See "Managing Updates — the two-repo model" above.

---

Developer Guide: Adding a New Application

When adding a new application (tenant) to the SmallWorlds cluster, please ensure you complete all the items on this integration checklist:

• Pin Specific Versions: Always use a specific, stable container image tag (e.g., v2.4.1 or 24.04) rather than latest in your Kubernetes manifests to ensure reproducible deployments.
• Add E2E Tests: Write end-to-end smoke tests (using Playwright) in the e2e-tests/tests directory to verify the application's core functionality and SSO integration.
• Integrate with Dashboard: Add Homepage annotations (e.g., gethomepage.dev/enabled: "true") to the application's Ingress resource so it automatically appears in the user dashboard.
• Make it Selectable: Add the application's identifier to the OPTIONAL_APPS array in the admin-tools/prepare-community-repo.sh script so users can easily toggle its installation.
• Update the README Table: Add the application to the "End User Applications" table in this README file, including a description and its source URL.
• Document the Implementation: Add a descriptive markdown file (or update an existing one) in the doc/ directory detailing the application's YAML manifests, configurations, and architecture.
• Configure DNS Records: Add the application's generic subdomain (e.g., whiteboard, meet, office) to the DNS records array in infrastructure/terraform/main.tf so Terraform provisions the A-record.
• Provision Web Certificates: Ensure the Ingress resource specifies the correct cert-manager.io/cluster-issuer: letsencrypt-prod annotation and the tls hosts block to automate Let's Encrypt SSL certificate generation.