From fc62e36727e68d9fb13ea94a81a8d596f36b0fb0 Mon Sep 17 00:00:00 2001 From: Myasnikov Daniil Date: Tue, 13 Oct 2026 18:07:30 +0500 Subject: [PATCH] docs(user-secrets-api): add the proposal for credentials minted on request and shown once Platform generates passwords of managed application accounts in the Cozystack API instead of at Helm render time. A mint on a per-account Credential generates a password, stores what the engine checks it against and returns the plaintext once, so a deployment that must show a generated credential once can offer managed databases. The proposal covers the Credential view over per-account Secrets, mint and revoke, the credentials block of the ApplicationDefinition, the wave 1 wiring of PostgreSQL, MariaDB and ClickHouse through their operators, platform accounts, audit, and the platform migration with its rollback limits. example.md shows what the definition, the chart and the API write for PostgreSQL, ClickHouse and a kind outside the tree, and follows one PostgreSQL application through the whole cycle. Assisted-By: LLM Signed-off-by: Myasnikov Daniil --- design-proposals/user-secrets-api/README.md | 393 ++++++++++++++ design-proposals/user-secrets-api/example.md | 530 +++++++++++++++++++ 2 files changed, 923 insertions(+) create mode 100644 design-proposals/user-secrets-api/README.md create mode 100644 design-proposals/user-secrets-api/example.md diff --git a/design-proposals/user-secrets-api/README.md b/design-proposals/user-secrets-api/README.md new file mode 100644 index 00000000..f669ed79 --- /dev/null +++ b/design-proposals/user-secrets-api/README.md @@ -0,0 +1,393 @@ +# User secrets API: platform-generated credentials, minted and shown once + +- **Title:** `User secrets API: platform-generated credentials minted on request, shown once, stored as verifiers` +- **Author(s):** `@myasnikovdaniil` +- **Date:** `2026-09-28` +- **Status:** Review + +## Overview + +Cozystack generates the passwords of managed application accounts at Helm render time and keeps them in a `-credentials` Secret that tenants can read again at any time. A deployment whose security policy requires that a generated credential is shown exactly once, at creation or regeneration, with regeneration as the only way back and every issuance audited, cannot be served by that model whatever the dashboard does. + +This proposal moves generation into the Cozystack API. A `mint` call on a per-account `Credential` generates a password, stores only what the engine needs to check it, and returns the plaintext in the response body. Nothing is left to read a second time, and a lost response is answered by minting again, which replaces the password. For PostgreSQL, MariaDB and ClickHouse in the first wave, which store a password verifier, this follows from what is stored. The other engines get the same mint in later waves with the plaintext kept where only the platform can read it, and those that accept a verifier move to one afterwards, as separate work. The platform's own accounts, which share those Secrets today, stop being visible to tenants. Attribution comes from the kube-apiserver audit log that Cozystack already collects, and a small per-account record shows tenants where the live password came from and who changed it recently. + +An application kind counts as converted once its ApplicationDefinition declares its credentials, and each kind converts on its own (§7). [`example.md`](./example.md) shows what a kind's definition, its chart and the API each write, and walks one PostgreSQL application through the whole cycle with the objects at each step. + +## Scope and related proposals + +- [Tenant-supplied secrets by reference](https://github.com/cozystack/community/pull/82) owns the secrets a tenant brings, and this proposal the ones the platform generates. No generated field gets a reference form here, and a later change that adds one keeps it off by default. There is no write path on `tenantsecrets` either: a field that needs a tenant-supplied value stays inline until the reference capability exists. +- It supersedes [managed database password rotation via a controller](https://github.com/cozystack/community/pull/72). Rotation here is the mint call and needs no controller. +- It builds on [cozystack/cozystack#4078](https://github.com/cozystack/cozystack/pull/4078), which removed `users[].password` from the postgres and mariadb charts and took the postgres password out of the init-script. +- It closes [cozystack/cozystack#4164](https://github.com/cozystack/cozystack/issues/4164) for the credentials of converted kinds: the name grants that give every ancestor tenant's ServiceAccount a child's passwords go away. +- It takes nothing from [Unified TLS and PKI](../unified-tls-pki/README.md). Under that proposal's rule no engine's key-bearing Secret is tenant-facing, and the trust anchor it delivers is public. +- Kubernetes access credentials stay at their CA, and VM `cloudInit` is a named gap. Both are classified in §7 and not converted. + +## Decisions + + + +## Context + +Baseline: cozystack/cozystack at `29932ddbb` (2026-09-28), except the postgres chart, read again at `785cbc391` (2026-10-12) after it changed. Paths are in that repository unless stated. + +The postgres, mariadb and clickhouse charts generate passwords at render time with `randAlphaNum 16`, keep them across reconciles with `lookup`, and store them in a chart-rendered `-credentials` Secret (`packages/apps/postgres/templates/init-script.yaml:1-8,99-124`, `packages/apps/mariadb/templates/secret.yaml:88-122`, `packages/apps/clickhouse/templates/clickhouse.yaml:2,21-28`). Since cozystack/cozystack#4078 the postgres init-job reads the password from that Secret at run time (`init-script.yaml:187-195`) instead of carrying it in the init-script. Five routes lead to the password. Two keep a copy of it, two open the Secret that holds it, and one keeps its hashes: + +1. The application spec keeps a copy. cozystack/cozystack#4078 removed `users[].password` from postgres and mariadb, while ClickHouse and OpenSearch still take `users..password` (`packages/apps/clickhouse/values.yaml:97`, `packages/apps/opensearch/values.yaml:215`), which every subject that can get the application reads. +2. `tenantsecrets` opens the Secret. The lineage webhook labels a Secret `internal.cozystack.io/tenantresource=true` when the owning application's ApplicationDefinition selects it (`internal/lineagecontrollerwebhook/webhook.go:182-191`), and `core.cozystack.io/tenantsecrets` serves the whole `data` of such a Secret (`pkg/registry/core/tenantsecret/rest.go:101-120`) to every tier from `use` up (`packages/system/cozystack-basics/templates/clusterroles.yaml:209-214`). All four database definitions select `-credentials`. The webhook decides once, at first admission, and skips the object afterwards (`packages/system/lineage-controller-webhook/templates/mutatingwebhookconfiguration.yaml:43-46`), so editing a definition does not relabel existing Secrets. +3. Name grants open the Secret. Each chart's `-dashboard-resources` Role grants `get` on `-credentials` (e.g. `packages/apps/postgres/templates/dashboard-resourcemap.yaml:16-22`) to the `use` groups and above and to the ServiceAccount of the tenant and every ancestor up to `tenant-root` (`packages/library/cozy-lib/templates/_rbac.tpl:84-96`). This is cozystack/cozystack#4164. +4. Helm history keeps a copy. Every stored revision of the release carries the Secret's plaintext, up to `MaxHistory`, 5 by default (`pkg/registry/apps/application/rest.go:1685,1706`), and a `helm rollback` restores it. +5. The ClickHouse operator keeps hashes in the `chi--common-usersd` ConfigMap in the tenant namespace, on which no tier holds a verb. + +The platform is its own client through the same Secrets. MariaDB `root` shares `-credentials` with tenant users (`secret.yaml:88-90`), and mariadb-operator logs in with it on every SQL reconcile. ClickHouse adds a `backup` account to every instance, backups enabled or not (`clickhouse.yaml:4-5`), and keeps its password there for the backup jobs. OpenSearch's `-credentials` is a copy of its operator's admin account, and Harbor's carries the password of its internal Redis. + +The engines already store verifiers: SCRAM-SHA-256 in PostgreSQL, a `mysql_native_password` hash in MariaDB, SHA-256 in ClickHouse, whose chart renders nothing else into the CHI, and bcrypt in OpenSearch. None of them needs the plaintext once it is set. + +Audit exists outside the Cozystack API, which runs without audit flags (`packages/system/cozystack-api/templates/deployment.yaml:31-33`). On a Talos cluster running a 1.7.0-alpha.1 build, kube-apiserver recorded a request for a subresource of a `tenantsecrets` object that does not exist with the resource, namespace, name and subresource in `objectRef`, next to the user, groups, source IP, response code and authorization decision. monitoring-agents, a default package, ships that log and Kubernetes Events to VictoriaLogs (`packages/system/monitoring-agents/values.yaml:349-358,389-410`), kept for a month by default. The user it names is a person only with OIDC (off by default, `packages/core/platform/values.yaml:471-473`), under which the dashboard passes the user's ID token through (`packages/system/dashboard/templates/gatekeeper.yaml:72-75`). Without OIDC the dashboard signs in with a pasted token, in practice the token of the tenant ServiceAccount. + +### The problem + +- A tenant creates a database and the platform generates its passwords. From then on every `use` member of the tenant and the ServiceAccount of every ancestor tenant can read them at any time, and nothing a tenant can see records who did. +- A leaked password has no platform path to rotation. A change made by hand inside the database is undone on the next run of the init-job or the operator, and a rotation in the chart would keep the old value in Helm history. +- The platform's own accounts (MariaDB `root`, the ClickHouse `backup` user, the OpenSearch admin) are shown to tenants like their own accounts. Tenants should never see them. +- A deployment that must show a generated credential once cannot offer managed databases at all. + +## Goals + +A converted kind is an application kind whose ApplicationDefinition has the `credentials` block (§7). + +- For every account of a converted kind the platform generates the password, with one generator for all engines (§2), and no application field sets one. +- The plaintext is returned once, in the response of the mint that generated it. Afterwards no Cozystack component or Kubernetes object holds it, except, for a kind that stores the plaintext (§7), its account Secret, which no tenant can read. +- A mint replaces the previous password, which the engine rejects once it has applied the new one. +- Every mint and revoke leaves a kube-apiserver audit event naming the caller, the time and the account, and the account's record shows the recent ones. +- A minted value appears in no log, Event, condition, error message or audit record. +- A tenant subject without the mint verb sees account metadata only, and no tenant subject can read a platform account's password in a converted kind. + +### Non-goals + +- Secrets the tenant supplies, which are #82. +- Delivering a minted credential into a tenant's Kubernetes cluster or VM. The caller stores it. +- TLS issuance and trust anchors, and retiring the static admin kubeconfig of managed Kubernetes. +- Closing connections that are already open when a password changes. +- Proving before the response that the new password works (§3 says why). +- Kafka users with topic ACLs, which the Kafka chart does not have yet (§7). +- Moving the wave-2 engines from stored plaintext to verifiers, which follows as separate work per engine (§7). +- Protection from management-cluster administrators, Flux or the operators, which hold every tenant's credentials today. + +## Design + +### 1. The account record + +Each account of a converted application is one `Credential` in the tenant namespace, served by the Cozystack API in `core.cozystack.io/v1alpha1`: + +```yaml +apiVersion: core.cozystack.io/v1alpha1 +kind: Credential +metadata: + name: postgres-orders.web + namespace: tenant-acme + uid: 3f9c2e1a-5b7d-4c0e-9a61-0d2f8e4b7c35 # the account's identity + resourceVersion: "48213" +spec: # derived from the application, read-only + application: {apiGroup: apps.cozystack.io, kind: Postgres, name: orders} + user: web +status: + password: # where the live password came from + origin: Minted # NotIssued | Migrated | Minted | Revoked + at: "2026-10-02T09:14:07Z" + by: jane@example.org + history: # the last ten operations, newest first + - {operation: Mint, at: "2026-10-02T09:14:07Z", by: jane@example.org} + - {operation: Migrate, at: "2026-09-30T02:11:40Z", by: "system:serviceaccount:cozy-system:cozystack-migration-hook"} +``` + +It is a view with no storage of its own. The account Secrets are the record of which accounts exist: the view lists the Secrets of the application, found by their labels, and shows for each the record it keeps. The account Secret: + +- lives in the tenant namespace, named `..account`, where `` is the user name lowercased with every character outside `[a-z0-9-]` replaced by `-`, and only the Cozystack API writes it +- holds the original `username` and, under `password`, the value the engine checks passwords against, in the kind's `format` (§3) +- keeps the record in its annotations +- is owned by the application's HelmRelease and carries its `apps.cozystack.io/application.kind`, `.group` and `.name` labels, which the HelmRelease has, and the labels that the kind's block names for its engine's operator + +The view never shows the Secret's data, no ApplicationDefinition selects the Secret, and no Role grants it. A Secret that no live HelmRelease owns is not listed. The kind's account sources (§7) are asked only when a write needs them, to compute what to seed and what to delete, and on a mint, to check the name, so a read costs one list of Secrets however a kind declares its accounts. + +`status.password.origin` says who can know the live password: + +- `NotIssued`: the account was created with a random password that the API threw away at once. Nobody has ever received a password for it, and nobody can log in until someone mints. +- `Migrated`: the account went through the conversion with the password tenants could read before it. That password still works and should be replaced by a mint. +- `Minted`: the password was shown once, in the response to the mint that `by` made at `at`. Nobody else received it. +- `Revoked`: `by` revoked the password at `at`. The live value is random and was thrown away, so nobody holds a working password until the next mint. + +`status.history` keeps the last ten operations (`Create`, `Migrate`, `Mint`, `Revoke`), so a tenant, who cannot read the audit log, sees who changed the password and when, while the audit log keeps every one. + +The identity of an account is its Secret's UID. It stays the same across reconciles, upgrades, in-place restores and the conversion. When the application is deleted and created again under the same name, it gets new Secrets, and a Secret left over from the deleted one is never reused. + +The Cozystack API keeps account Secrets in step with the accounts of an application whenever it creates or updates one of a converted kind, whichever client sent the request (the dashboard, `kubectl`, an external API consumer): + +1. On create for every account, and on update for every account the write adds, it writes a Secret with the verifier of a random password that it throws away, origin `NotIssued`, replacing one that is not theirs (left from a deleted application of the same name, or holding another `username`). The account then exists in the database, and nobody can log in as it until someone mints. An account that exists already and has no Secret is left alone: the view does not list it, and a mint on it takes no precondition and creates the Secret. That is a state of the conversion (Upgrade), not a steady one: once the conversion is complete every declared account has its Secret, because a create seeds all of them, an update seeds the added ones and a mint recreates a missing one. +2. It writes the HelmRelease. On create, the Secrets from step 1 get the new HelmRelease as their owner right after, and one written on an update carries it from the start. +3. It deletes the account Secrets of accounts that the write removes. + +Two users whose names map to the same object name, or to one that is not a valid DNS name, are refused. + +### 2. Mint and revoke + +Every password is 32 characters drawn uniformly from `[A-Za-z0-9]` by `crypto/rand`, for every engine. That gives about 190 bits of entropy, above the 128-bit security strength NIST SP 800-57 asks of keys used beyond 2030, so even the unsalted hashes ClickHouse and MariaDB store (§3) cannot be brute-forced, and the alphabet needs no escaping in SQL, connection URIs or shells. Today's charts draw 16 characters from the same alphabet. + +``` +POST /apis/core.cozystack.io/v1alpha1/namespaces/tenant-acme/credentials/postgres-orders.web/mint +{"apiVersion": "core.cozystack.io/v1alpha1", "kind": "CredentialRequest", + "spec": {"preconditions": {"resourceVersion": "48213"}}} + +201 Created +{"apiVersion": "core.cozystack.io/v1alpha1", "kind": "CredentialRequest", + "status": {"username": "web", "password": "", "issuedAt": "2026-10-02T09:14:07Z"}} +``` + +This is the pattern of Kubernetes `TokenRequest`: a `create` on a subresource of the object, with a request kind of its own as the body, answered by the same kind with the result in `status` and nothing stored. A mint does four things: + +1. Generates a password. +2. Computes what the kind's `format` stores: the verifier of the password, or for `plaintext` (§7) the password itself. +3. Writes that value and the updated record into the account Secret, in one update that fails if the Secret changed since the caller read it. +4. Returns the plaintext in the response. + +The plaintext exists in the API server's memory for the length of the request and in the response. After a second mint the first password is dead: unlike a token, which is added next to the others, a password is replaced. A mint on a declared user (the sources of §7 are asked here) whose Secret is missing recreates it with a new identity, one on an undeclared user answers NotFound, and one on a user whose name maps to another declared user's answers Conflict. + +`revoke` takes the same body, writes into the account Secret the verifier of a new random password, throws that password away and returns nothing secret, so nobody holds a working password. It retires a leaked password when nobody should get a new one yet. Neither call lets the caller choose the password. + +A lost response (a closed tab, a dropped connection, a crash between the write and the answer) leaves a password nobody holds. The record shows the issuance and its initiator, and the way forward is another mint. + +Two concurrent mints cannot both succeed: the conditional write lets one land, and the other gets `409 Conflict` without a password. A caller can also pass the `resourceVersion` it last read, and the dashboard always does. If someone minted in between, the caller gets a 409 showing the newer issuance instead of silently killing the password another person just received. + +### 3. Where the stored value goes + +An engine here is a database server with its operator. The API knows password formats and no engines. A `format` turns a password into the value the engine stores (`scram-sha-256`, `mysql-native`, `sha256` or `plaintext`), a kind names one in its block (§7), and the account Secret has one layout for all of them, `username` and `password`. The four are functions in one Go package, `pkg/credentials/format`, that the API and the migration's CLI import, so a new format is a change here. CNPG wants one Secret per role, mariadb-operator a key reference per `User`, and the ClickHouse operator watches no Secrets, so each chart wires its engine to the account Secret in its own way: + +| Engine, vendored operator | `format` | Wiring | Stored value | +|---|---|---|---| +| PostgreSQL, CNPG 1.30.0 | `scram-sha-256` | `Cluster.spec.managed.roles[].passwordSecret`, Secret labelled `cnpg.io/reload: "true"` | `SCRAM-SHA-256$:$:` | +| MariaDB, mariadb-operator 25.10.2 | `mysql-native` | `User.spec.passwordHashSecretKeyRef`, Secret labelled `k8s.mariadb.com/watch` | `PASSWORD()` output, `*` and 40 hex digits | +| ClickHouse, clickhouse-operator 0.25.2 | `sha256` | HelmRelease `valuesFrom`, Secret labelled `reconcile.fluxcd.io/watch: Enabled`, rendered by the chart as `password_sha256_hex` | unsalted SHA-256, hex | + +#### PostgreSQL + +Tenant accounts move under CNPG's declarative role management (`spec.managed.roles`) together with the per-database `_admin` and `_readonly` roles and the marker role `cozystack:managed`. CNPG orders the roles by their `inRoles` itself, so the order of the list does not matter. + +The chart renders every role as the init-script creates it today, because CNPG reverts any attribute that differs, revokes parent roles missing from `inRoles` and drops a comment the spec leaves unset, while the init-script finds the roles to remove by that comment and by membership in the marker: + +- the marker gets `login: false`, and a database role also `inherit: false`, the comment `role managed by helm` and the marker in `inRoles`, since the removal of a database role looks for the same comment and membership +- an account gets `login: true`, its `replication`, `createRole` and `bypassRls` flags, the comment `user managed by helm`, `inRoles` from `databases.*.roles` and `cozystack:managed`, and `passwordSecret` naming the account Secret + +The init-script stops setting passwords, attributes and memberships, where it would be a second writer, with one exception. It still creates the database roles, idempotently, because its own `ALTER DATABASE ... OWNER` needs them, and once an account or a database role exists it grants the marker again `WITH INHERIT FALSE, SET FALSE`, and for an account it also sets `createrole_self_grant`, which CNPG has no field for and leaves alone. CNPG grants the marker with the default options, so `set` is true and an account's `inherit` is true, and a privilege given to the marker would reach every account it created. It leaves the narrower options of the re-grant as they are, as it does for a role the init-script created before the conversion. The init-script keeps databases, ownership, privileges, extensions and the removal of undeclared users and roles, which still finds a dropped account by comment and marker, since CNPG leaves a role that left its spec alone. + +The chart refuses to adopt a role it did not mark once any user has held `createRole`, since such a user can plant a role under a name declared later, and CNPG adopts by name and keeps what the planter left, its `ADMIN OPTION` and the role's settings, so the planter can grant itself the role. The init-job is a post-upgrade hook and would find the role already adopted, since CNPG acts within seconds of the spec reaching it. So the check moves into a pre-upgrade hook, a Job that connects as the init-job does, is skipped while the chart renders `bootstrap.recovery`, and runs the init-script's query for every declared user and for the `_admin` and `_readonly` roles of every database. A role with no marker fails the upgrade before the new `Cluster` reaches CNPG, with the init-script's message to inspect and drop the role as the superuser or to declare another name. Roles that CNPG or the init-script created pass, because they carry the marker. The hook reads the flag first and then sets it if the new values grant `createRole`, so the upgrade that first grants it still adopts, since no one has held the attribute before, and the flag is on before CNPG applies it. The init-job keeps setting the flag after an install, where the hook has no cluster to reach. The check fails closed, since no catalog query tells a planted role from an administrator's. + +A changed account Secret reaches the engine without Helm: the operator watches Secrets with the reload label, and the instance manager on the primary runs `ALTER ROLE ... PASSWORD` with the verifier, which PostgreSQL stores as it is. The API has to produce the canonical form exactly, since CNPG hashes anything else again as plaintext and the login then fails with no error. With the account Secret missing, CNPG skips every change to that role, `login: false` included. + +#### MariaDB + +Each `User` swaps `passwordSecretKeyRef` for `passwordHashSecretKeyRef` pointing at the account Secret, which carries the `k8s.mariadb.com/watch` label cozystack/cozystack#4078 already put on `-credentials`. With the label, a change to the Secret makes the operator re-run `ALTER USER`. Without it the change waits for the SQL requeue, ten hours by default. A `User` created without any credential reference is locked for good (`ACCOUNT LOCK PASSWORD EXPIRE`, never unlocked), so every `User` carries the hash reference from its first reconcile. The operator pastes the hash into SQL as it is, which is why only the API computes and writes it. + +#### ClickHouse + +ClickHouse has no route from a Secret to the engine without a Helm run, so a ClickHouse mint costs a release upgrade and waits while the release is suspended. The operator watches no Secrets, `valueFrom.secretKeyRef` needs a pod restart per change, and `k8s_secret_password_sha256_hex` aborts the reconcile from operator 0.27.4. The one path that applies live is a hash inline in the CHI: the operator rewrites the `chi--common-usersd` ConfigMap without a restart, and ClickHouse re-reads `users.d` every two seconds. + +The chart already renders that hash. After conversion it takes it from values that the HelmRelease pulls from the account Secrets, through one `valuesFrom` entry per account, which the `valuesPath` of the block (`_accounts` here) turns on. Both the API (`rest.go:1721-1726`) and the ApplicationDefinition reconciler (`internal/controller/applicationdefinition_helmreconciler.go:148-156`) pin `valuesFrom` to `cozystack-values` today, so both compute the new list with one function, and the reconciler writes it onto existing releases. Each entry reads the Secret's `password` into `_accounts.u`, where `` is the SHA-256 of the user name as `users` lists it, and the chart looks each declared user up by the same digest. A user name fits neither the `targetPath` nor the value (a comma fails the entry), while a hex digest fits both and leaves no order to keep aligned. The chart's own check is on its values: it fails the render when a declared account has no verifier in them, as a HelmRelease written around the API leaves it, because the operator would give such an account the password `default`, and it refuses tenant accounts named `backup`, `default` or `clickhouse_operator`. + +#### A declared account without a Secret + +A chart cannot see at render time whether a Secret exists except through `lookup`, which the Flux digest does not see and which this proposal removes for tenant accounts. The engines refuse on their own: CNPG leaves the role as it is and reports `cannotReconcile`, mariadb-operator sets the `User` to `Ready=False` and leaves the SQL user untouched, and Flux stops a ClickHouse release with `ValuesError`, which keeps its previous revision. + +#### Whether the engine applied a mint + +A `Credential` does not say. The API would have to read the operators' objects, and Cozystack keeps that knowledge in controllers, as the backup strategy controllers (`internal/backupcontroller`) do for CNPG and mariadb-operator. A mint is accepted whatever the engine's state, and the old password keeps working until the engine applies the new one, which PostgreSQL and MariaDB do when the operator sees the Secret change and ClickHouse when the release upgrades. Reading the stored hash back needs a privileged connection into every tenant database, and a login probe needs network access to them and the plaintext, so neither is used. + +### 4. Platform accounts + +The platform's own accounts inside an engine keep their plaintext where their consumer needs it. Every such account stays in the `-credentials` it lives in today, the tenant accounts move out of that Secret, and the Secret stops being visible to tenants: no ApplicationDefinition selects it, no Role grants it, and the chart renders it with `internal.cozystack.io/tenantresource: "false"`. The same rule holds in every engine because MariaDB leaves no other choice: `rootPasswordSecretKeyRef` cannot change once set. These passwords stay chart-generated with `lookup` and keep their plaintext in Helm history, which only management-cluster administrators and platform components read. + +The conversion rotates every platform password tenants could read: + +| Account | Consumer | Rotation | +|---|---|---| +| MariaDB `root` | mariadb-operator, which logs in with it on every SQL reconcile | the Job stores the new value in a platform-only Secret, runs `ALTER USER` with the current root password, then writes it into `-credentials`, and a retry tries both passwords | +| ClickHouse `backup` | the backup CronJob, and the backup sidecar through its environment | the Job writes a new value and forces a release upgrade, which applies its hash and restarts the ClickHouse pods once for the sidecar | +| OpenSearch admin | the operator reads `-admin-credentials`, and `-credentials` holds a copy | with the OpenSearch conversion, in wave 2 | +| Harbor Redis | Flux, through `valuesFrom` | with the Harbor conversion, when its admin password moves out | + +Removing the grants and selector entries closes cozystack/cozystack#4164 for converted kinds. + +### 5. Who may do what + +| Subject | get, list `credentials` | create `credentials/mint`, `credentials/revoke` | +|---|---|---| +| `view` | yes | no | +| `use` | yes | no | +| `admin`, `super-admin` | yes | yes | +| tenant ServiceAccount (`cozy:tenant`) | yes | yes | + +Minting goes with `admin` and above. Cozystack has no fine-grained role management of its own, so the tiers are the only knob, and a mint cuts off every client that holds the old password, so it starts narrow. `use` members, who read `-credentials` today, lose the password and cannot mint a new one, so an administrator mints for them. Letting `use` mint later is a change to one rule. A tier of its own would need a role, a binding, a Keycloak group and a change to the subjects helper, so none is proposed. The tenant ServiceAccount already holds every verb on the tenant's applications, so automation that provisions an application can mint in the same run and store the response itself, which the walkthrough shows together with Terraform ([headless provisioning](./example.md#headless-provisioning-and-terraform)). + +Ancestors inherit, since the tier and ServiceAccount bindings in a tenant namespace include every ancestor (`packages/apps/tenant/templates/tenant.yaml:8-87`). A parent administrator can mint a child's credential, where a parent member at `use` could read it before, and the access is now visible: the mint replaces the password, the child's record shows who issued it, and the audit event names the actor. + +`mint` and `revoke` are separate subresources because the audit log holds no bodies and tells them apart only by subresource, and because a custom role can grant `revoke` alone to an incident responder. By default both go to `admin` and above and the tenant ServiceAccount. `credentials` has no `watch`: the view is built from the account Secrets and the HelmReleases that own them, and a watch would need a `resourceVersion` of its own, while a client reads the record with `get`. It is namespaced with no cluster-wide list, like `tenantsecrets`, and a list reads the account Secrets of the namespace by label from the API's caches. Its `resourceVersion` is the account Secret's, and the only use is the precondition of a mint. + +Without OIDC the dashboard signs in with the tenant ServiceAccount token and the tiers collapse into the ServiceAccount, so a deployment that needs role-based disclosure and per-person attribution enables OIDC. An external API consumer maps its roles onto these tiers and, acting for a person, passes that person's OIDC token or uses Kubernetes impersonation, which the audit event records next to the caller. + +### 6. Audit and service history + +A mint is a `create` on `credentials//mint`, which kube-apiserver audits before proxying it to the Cozystack API, with the account in `objectRef` and the subresource telling a mint from a revoke. kube-apiserver records neither the body nor the response of a request it proxies to an aggregated API, at `Request` and at `RequestResponse` alike (run on kube-apiserver 1.34 and 1.37, see Testing), so the password cannot reach that log. That answers who, when and which account for every issuance, and Cozystack already ships the log to VictoriaLogs. + +The one server that could write the password down is the Cozystack API, if it got an audit policy, and any such policy must keep `credentials` below `RequestResponse`. Distributions that give kube-apiserver no audit policy record nothing, so the installation documentation gets one for them, with the log path monitoring-agents tails. An external SIEM, the log and alerting system a security team runs, has no way in today: the fluent-bit outputs of monitoring-agents are one fixed string (`packages/system/monitoring-agents/values.yaml:377-410`), and overriding it drops the VictoriaLogs outputs. monitoring-agents gets a value for additional outputs. + +A rotation is subscribed to on the audit log, in the SIEM or in a VMAlert rule group over VictoriaLogs that the operator adds, since Cozystack ships no log-based alert rules. Each mint and revoke also emits a Kubernetes Event on the application, naming the account and the initiator and never a value. Events are best-effort, because the client drops and merges repeated events about one object, and the API needs `create` on `events` for them. Only `tenant-root`, whose log store receives the cluster's audit log, can read that log, and every other tenant reads the record's history. + +### 7. Coverage + +A kind joins a wave once its whole path from mint to engine works, not once the engine could accept a verifier. Every wave uses the same `Credential` and mint. In wave 1 the account Secret holds only the verifier, and one-time disclosure follows from storage. In waves 2 and 3 the account Secret holds the plaintext that the engine or its operator reads. No tenant can read that Secret, so one-time disclosure holds on the tenant-facing interface, while platform components and management-cluster administrators can still read it. + +A kind declares its accounts once, in `spec.credentials` of its ApplicationDefinition, which a tenant cannot edit. [`example.md`](./example.md#the-contract-for-a-kind) walks the block field by field, with the chart and the API, for PostgreSQL, for ClickHouse and for a kind outside the tree. `format` is how a password is stored (§3), as `{name: scram-sha-256}`, and `secretLabels` the labels its operator needs on an account Secret. `accounts` is a list of sources, each with a `kind`, and a kind may list several. `ValuesMap` names a values map whose keys are tenant accounts (`path: users`), with the name rules of §1 and no `password` field. `Static` is a fixed account, such as the single credential of a kind with no `users` ([Redis](./example.md#a-kind-with-no-users-redis)), switched on by a values key in `when` and read with its schema default, and `owner: platform` on it marks a platform account (§4): the API does nothing for it, and a `ValuesMap` skips its name. `valuesPath` is for an operator that cannot read Secrets (§3). The API adds no knowledge of kinds or engines, serves no kind with an unknown `format`, and a new format is a new function in it. A kind outside the tree names `plaintext`, which keeps the password in the account Secret as waves 2 and 3 do, or an existing format, and its chart has to read `..account` and render `-credentials` not tenant-visible, or a mint does nothing. Bucket's keys come from the COSI driver, which a format cannot express, so how a mint works there is set in its row of §3 at its conversion. + +`format` and each source are structs with a `kind` from the first release, because a bare string on a CRD field cannot later become an object. Only the built-in formats, `ValuesMap` and `Static` are built. The other kinds are named so that a later change adds them without changing the block: a `Webhook` format, and for accounts `ValuesList` (a list of objects with a `name`), `Expression` (CEL over the application) and `Webhook`, for logic the API does not know, such as asking the COSI driver which keys exist. The API refuses a kind that uses one before it is built, as it refuses an unknown format. A format webhook is called between generating the password and writing it, with the format name, the username and the plaintext, and returns the stored value, and a failure fails the mint with nothing written. Three consequences hold. The plaintext crosses to the webhook, so only a platform administrator declares one, which the ownership of the definition already gives. The migration's CLI has the built-in functions compiled in, so a kind with a webhook format ships its own migration or converts only new applications. A kind whose format webhook is unreachable is not served, as one with an unknown built-in is not. An account source is asked only at write time (§1), so the view does not depend on it. + +| Kind | Engine takes a verifier | What the tenant gets today | Wave | +|---|---|---|---| +| PostgreSQL | yes, SCRAM-SHA-256 | `-credentials` | 1 | +| MariaDB | yes, `mysql_native_password` hash | `-credentials`, `root` included | 1 | +| ClickHouse | yes, SHA-256 | `-credentials`, `backup` included | 1 | +| OpenSearch | yes, bcrypt in `internal_users.yml` | the admin only, since the per-user Secrets the chart renders reach no engine | 2, once users reach the engine | +| NATS | yes, bcrypt in the server config | one Secret with every user, plaintext in the child release values | 2 | +| RabbitMQ | yes, `password_hash` through the HTTP API | per-user Secrets and the administrator `-default-user`, and the topology operator imports a password only when it creates the user | 2 | +| Redis, Valkey | yes, `#` in ACL | `-auth`, one password shared with replication, the operator and the exporter | 2 | +| MongoDB | no, the server computes SCRAM from the plaintext | `-credentials`, the operator's `databaseAdmin` system user | 3 | +| Bucket | no, SigV4 signs with the secret | per-user keys generated by the COSI driver | 3 | +| VPN (Outline) | no, Shadowsocks derives its key from the password | `ss://` links embedding each password | 3 | +| Qdrant | no, the key is compared as is | the single admin API key, also Prometheus's bearer token | 3 | +| Harbor | no, the admin is bootstrapped from plaintext | the admin and Redis passwords | 3 | +| Monitoring | no, Alerta and a job consume the plaintext | `grafana-admin-password` | 3 | +| Kafka | yes, SCRAM credentials | CA certificates only | none yet: no listener authenticates | +| VM instance | cloud-init takes a crypt hash, but `cloudInit` is free-form text | plaintext in the spec, and the chart's own example is `password: ubuntu` (`packages/apps/vm-instance/values.yaml:111-115`) | out of scope | + +Wave 2 engines accept a verifier, but their operator or chart path takes plaintext today, so they convert with the plaintext in the account Secret (`format: plaintext`), like wave 3. Moving each of them to a verifier, the way wave 1 works, is separate work after this proposal, one engine at a time. Wave 3 engines need the plaintext for good. OpenSearch first needs its users to reach the engine, which with plaintext means `OpensearchUser` objects that read the account Secret. Each wave-2 and wave-3 kind gets its own row in §3 at its conversion, shaped by what the table names: a RabbitMQ import that happens only at user creation, keys from the COSI driver, one Qdrant key shared with Prometheus. + +Kafka clients connect without credentials, since no listener declares authentication, the optional external LoadBalancer one included (`packages/apps/kafka/templates/kafka.yaml:32-46`). Users with topic ACLs are a Kafka chart feature, and once they exist they join a wave. Until a kind converts, a deployment whose policy requires one-time disclosure can withhold it: `bundles.disabledPackages` stops the platform rendering its package, and on a running cluster the Package also has to be deleted, because packages carry `helm.sh/resource-policy: keep`. + +Kubernetes access credentials hold no password and stay at their CA. A managed cluster's `-admin-kubeconfig` is a client certificate and key, its `-oidc-kubeconfig` carries neither, and retiring the static certificate is left open. The Talos workers' `-talos-secrets` is not tenant-facing. + +## User-facing changes + +- The credential view of a converted application lists its accounts with the origin of the live password and the recent operations, and offers Regenerate and Revoke to those who may mint. A password appears once, in the dialog that minted it, and creating an application with users, or adding one, chains a mint per new account. +- `use` members stop seeing passwords and cannot mint one, so an administrator does (§5). Tenants stop seeing MariaDB `root`, the ClickHouse `backup` user and the OpenSearch admin, and administer MariaDB through accounts with the admin role on each database. +- A password an account sets for itself inside PostgreSQL is reverted by CNPG. +- API clients get `credentials` with `mint` and `revoke` in `core.cozystack.io/v1alpha1`, and `tenantsecrets` does not change. +- Operators get a numbered migration per converting release, which names the kinds it converts, an audit policy for distributions that ship none, and a monitoring-agents value for an extra output. The migration's second part is a Job with cluster-admin and `pods/exec` that is not a Helm hook and can be rerun by hand, which the release note says. + +## Upgrade and rollback compatibility + +A kind converts in one release, and a release may convert several kinds. The conversion is announced ahead of it, in release notes and on the kind's credential view, so users can store passwords they still need. The API does not track which chart a release renders. It acts on the `credentials` block of the ApplicationDefinition, which arrives in the same platform release as the chart but through a separate HelmRelease, in no fixed order, and it seeds only what §1 says. Until a release switches charts, its old chart keeps serving the old passwords from `-credentials`. A mint or revoke made in that time is recorded, and it takes effect, and the old password dies, at the switch. A user added in that time is `NotIssued`, and the password its old chart generated stops working at the switch. An application created before the block is live gets its Secrets from the first mint. The release note tells operators to hold mints, revokes and new applications of a converting kind until the Job reports done. + +The converting release ships the converted chart and a numbered platform migration in two parts. The first is a pre-upgrade hook of the platform release, before any component chart changes version (`packages/core/platform/templates/migration-hook.yaml:27-60`), in the platform variants that set `migrations.enabled`. The migration names the kinds it converts, because the old ApplicationDefinitions are still live when the hook runs. For each release of those kinds it records the chart version of the HelmRelease and writes the account Secrets with the verifiers of the passwords in `-credentials`, computed by the API's format functions, built into a small CLI that the migration image carries, as it carries `etcd-migrate` for migration 50, origin `Migrated`, so clients keep working. That is additive: nothing an old chart reads changes, so an old chart that renders before the switch breaks nothing. + +Then the charts switch, and a surviving `-credentials` is rendered with `internal.cozystack.io/tenantresource: "false"`. For a Secret that exists already this label is the closure, because the webhook stamped it at first admission and skips the object since. For one created while a definition still selects the name, the lineage webhook learns to keep an explicit `"false"` instead of overwriting it, which is why Rollout step 2 ships first. No field ownership moves, since the API takes over no object Helm owns. + +The second part is a Job that the platform release creates after the charts switch. It is not a Helm hook, so a long wait does not hold up the upgrade, and it can be run again by hand. For each release in the record it first writes any account Secret still missing, as the hook does, then waits until the HelmRelease is Ready on a chart version other than the recorded one, so with the read routes closed, and skips the release after a timeout. A release outside the record was created after the hook and is left alone, and a missing or unchanged chart version counts as not switched. Then it removes any leftover `users.*.password` from the values, deletes the tenant keys from `-credentials` (they stay in its `data` otherwise) and the Secret itself where no platform account remains, as in postgres, and rotates the platform passwords of §4. Rotation suspends the release for the change, so no render writes the old value back, and lifts only a suspension it set. ClickHouse also needs a forced upgrade to apply a new `backup` hash, and MariaDB's `root` a connection to the primary, which the Job has and the API lacks (§3). An annotation on `-credentials` records a rotation, so a rerun does not rotate twice, and the rest of the Job is idempotent. The migration keeps its state in a ConfigMap in `cozy-system` named for it: the recorded chart version of each release, and the releases skipped on the timeout with the time of the run. The Job exits non-zero while the list of skipped releases is not empty, so one place shows what is left. + +The migration degrades per release, never per fleet. A failing migration stops the whole platform upgrade, so a release it cannot convert (its Secret gone, two users mapping to one name, a user named `app` in postgres) is left out and reported in the migration log and with an Event on its HelmRelease. A left-out release keeps its previous revision, old credentials working and exposed as before, except a PostgreSQL release whose Secret was gone: that takes the converted chart, its database keeps the old passwords, and tenants can no longer read them, so a tenant mints by name, since the view lists no account without a Secret ([the cases](./example.md#when-the-migration-leaves-a-release-out)). Once an operator fixes the cause, running the Job again finishes the release, and so does a mint that creates a missing Secret. + +The hook renders only where `migrations.enabled` is set, which the `isp-*` variants do and `default` does not, as migration 59 notes of itself. A platform that runs no migrations runs none of them, so it is not an upgrade target for this proposal, and the release note says that the conversion needs a variant with migrations enabled. An application installed by hand on `default`, which still renders every PackageSource, takes the converted chart with no hook and no Job and ends as a left-out release above, with the tenant keys in the hidden `-credentials` and the platform passwords unrotated. + +No flag keeps passwords viewable. It would be a second credential path in every converted chart and a switch that turns the guarantee off for every tenant of an installation. An installation that needs the old behaviour for longer stays on the release before the conversion, and a tenant that wants the password in a Secret mints through a helper that writes it there, which is later work. Pre-conversion revisions keep the old plaintext in Helm history until `MaxHistory` drops them, and the origin stays `Migrated` until the first issuance or revocation makes that value useless. + +Downgrading the platform past a conversion is not supported, and it is the one irreversible step: the previous charts generate a new password for every account and apply it over the minted one. A manual `helm rollback` of one release is undone at the next reconcile, because the HelmRelease still asks for the converted chart. Until then MariaDB applies the old passwords from the `-credentials` the old revision recreates, and ClickHouse the verifiers of the older revision, even between two converted ones, so a revoked password logs in again. PostgreSQL keeps the minted passwords, since a rollback runs no post-upgrade hook and CNPG leaves roles dropped from its spec alone, but the old Role exposes the dead plaintext. In every case the answer is a mint once the release is back. + +## Security + +Each read route, for converted kinds: + +| Route | Tenant accounts | Platform accounts | +|---|---|---| +| 1. Application spec | no password field (cozystack/cozystack#4078 for postgres and mariadb, the conversion for the others), and the Job removes leftovers | never in the spec | +| 2. `tenantsecrets` | no definition selects an account Secret | `-credentials` is rendered not tenant-visible | +| 3. Name grants | none | none, which closes cozystack/cozystack#4164 for converted kinds | +| 4. Helm history | no tenant password in a converted revision, and older revisions keep the pre-conversion value until `MaxHistory` drops them (hence the `Migrated` origin) | open by decision: chart-rendered plaintext, readable only by management-cluster administrators and platform components | +| 5. Operator-materialised configuration | verifiers only | unchanged | + +The mint response is the one place the plaintext exists by design. It crosses from kube-apiserver to the Cozystack API over TLS and back, and neither server logs or audits it (§6). In waves 2 and 3 the account Secret holds the plaintext as well, which tenants cannot read and platform components can. + +The trust boundary does not move. The Cozystack API already holds `create`, `update`, `patch` and `delete` on Secrets in every namespace (`packages/system/cozystack-api/templates/rbac.yaml:18-20`), and Flux renders every tenant's password today, so generation in the API adds no authority over credentials. The API gains `create` on `events`. The post-upgrade Job needs the rights the migration hook holds today (cluster-admin for the length of the run, `migration-hook.yaml:62-77`) and reaches each MariaDB primary through `pods/exec`, and its database access is no more than mariadb-operator's root login. + +The lineage webhook's `tenantresource` verdict is not a control. A chart may narrow it, since an explicit `"false"` is kept, and a chart-set `"true"` is still overwritten whenever the webhook runs. But the webhook is not called for an object that already carries `internal.cozystack.io/managed-by-cozystack` (`mutatingwebhookconfiguration.yaml:43-46`), so a writer that sets that label gets the `tenantresource` it wrote. A Secret stays off `tenantsecrets` because the converted chart renders `"false"` on those that exist, no definition selects later ones, the webhook keeps the chart's `"false"` in between, and no converted chart labels any `"true"`. + +Verifiers are shown to no one, although they are not treated as secrets. A verifier of a 32-character random password leaves no practical offline attack, which is the only reason ClickHouse's unsalted SHA-256 and MariaDB's double SHA-1 are good enough, and a migrated password keeps the `Migrated` origin anyway. mariadb-operator pastes the hash into SQL unescaped, which is safe because only the platform writes the account Secret. For ClickHouse the verifier arrives under a `_`-prefixed values key, which the Application API hides on read and refuses on write (`pkg/registry/apps/application/rest.go:1269-1304`). + +The mint body carries a precondition and no other tenant input. A Credential lives in its tenant namespace, kube-apiserver authorizes a mint by RBAC in that namespace, and the API resolves the account Secret within it. Without OIDC the tiers collapse into the tenant ServiceAccount (§5). + +## Failure and edge cases + +- The engine's operator is down, or the database is not ready → the mint succeeds, and the old password keeps working until the engine applies the new one. +- A ClickHouse release is suspended or failing → the mint is accepted and takes effect when the release upgrades. +- A user leaves `users` while a mint for it is in flight → NotFound or Conflict, and no password. +- An account Secret deleted by hand → the engine keeps the account as it is (§3) and a ClickHouse release keeps its previous revision, until a mint recreates the Secret with a new password and identity. A ClickHouse user added or renamed around the API → the values check fails the render until the next write through the API rewrites the list. +- A create that fails after the Secrets were written, or a delete with orphan propagation → Secrets no current HelmRelease owns, which the view does not list and the next create under that name replaces instead of reusing. +- Deleting a converted application whose release is suspended → refused, because Flux skips the uninstall of a suspended release and would leave the database running with its last passwords after the account Secrets are gone. This is new Application API behaviour and goes in the release note. +- A PostgreSQL name that makes CNPG's webhook reject the whole Cluster → refused at render, as `postgres` is today. That covers roles CNPG reserves (`streaming_replica`, anything starting with `pg_` or `cnpg_`), a database whose derived `_admin` role would be reserved, an account named like a derived role, and an account named `app`, the owner CNPG creates at initdb and whose password its instance manager resets from `-app` on every start. +- A restore in place → account Secrets and records stay, and the operators apply the current values over the restored catalog. For PostgreSQL CNPG does it once the restored primary is up, replacing the init-job run and the `credentials-pending` annotation of today's restore path. +- A restore into a new application → new records and seeds, and the restored accounts need a mint, as they need new passwords today. +- An application is deleted → its account Secrets go with its HelmRelease, and the audit trail remains. +- The hook or the Job is interrupted → both are idempotent, rotation is skipped where the annotation says it is done, and the next run finishes what is left. +- A `password` left under a key of `users` in a converted kind → dropped with an admission warning pointing at mint, as cozystack/cozystack#4078 did for postgres and mariadb. + +## Testing + +- API unit tests: passwords are 32 characters from the stated alphabet and `crypto/rand`, every verifier checks out against a reference implementation (PostgreSQL's SCRAM, MariaDB's `PASSWORD()`, SHA-256) with the SCRAM form canonical, conflicts return no password, a captured log and event recorder never see the plaintext, and the view lists by label and skips a Secret that no live HelmRelease owns. The API-change gate learns the subresource keys its storage-key pattern skips today. +- Chart unit tests on each converted chart: no tenant password in any rendered object, the engine points at the account Secret, neither account Secrets nor `-credentials` are granted or selected, `-credentials` renders `tenantresource: "false"` and no rendered Secret carries `"true"`, the ClickHouse render fails for a declared account with no verifier in its values, and reserved or colliding names are refused. API and reconciler build the same `valuesFrom` list, and a chart's account-name helper gives the API's name for every accepted user name. +- PostgreSQL roles, on a live cluster: a role planted by a `createRole` holder under the name of a user or of a database role declared later fails the upgrade before CNPG adopts it, and `users.app` still adopts the role CNPG bootstraps. A user and a database that leave the values are dropped by the init-job with their roles. The marker keeps `INHERIT FALSE, SET FALSE` on every account and database role after a mint. +- Writes through `tenantsecrets`, whose code has no tests today: no tenant tier holds a write verb on it, and an update or delete through it on an account Secret or on `-credentials` answers NotFound. +- Lineage webhook: an explicit `"false"` is kept where a definition selects the Secret, and a chart-set `"true"` is overwritten. +- No foreign secret, end to end with a parent and two sibling tenants: `tenantsecrets` never lists an account Secret or `-credentials`, a sibling can neither read nor mint a tenant's credentials, and a parent's mint shows in the child's record. +- No second disclosure, end to end for each wave-1 engine: mint, log in, and find the password on no read path (the Credential, `tenantsecrets`, a direct `get` on the Secret, the application, Events, the audit log, the Helm history of the converted revision). Then mint again and see the first password fail, polling a login until it does. +- Live checks on the pinned versions ran on plain operator objects: CNPG 1.30.0, mariadb-operator 25.10.2, clickhouse-operator 0.25.2 and helm-controller 1.5.0 behave as §3 and §4 state when an account Secret is missing or changes, and kube-apiserver 1.34 and 1.37 record no aggregated call body. For the PostgreSQL roles of §3, CNPG 1.30.0 on PostgreSQL 18.1 reverted nothing in 150 seconds without a spec change, and when a spec change triggered its role reconciler it revoked a marker granted by hand and reverted `createrole` and `bypassrls`. It left role settings and the options of existing memberships alone, took over a role the init-script had created, and took over a role a `createRole` holder had planted, which that holder then granted to itself. The init-script's refusal query, with the flag on, refused only the planted roles, for a user name and for a database role. +- The hook and the Job: clients keep logging in at every point, including while an old chart renders between the hook and the switch, the Job acts only once the recorded chart version has changed, accounts show origin `Migrated`, no `users.*.password` remains, a Job stopped between `ALTER USER` and the Secret write recovers on retry, rotated platform passwords work, and no tenant route reads a converted release's `-credentials`. A release it leaves out does what its cause says in [`example.md`](./example.md#when-the-migration-leaves-a-release-out). +- The window between the block and the chart: a mint, a revoke, an added user and a new application made before the switch, and a switch before the block is live, end as the Upgrade section says. +- Audit: a mint appears in the kube-apiserver log with `objectRef.subresource: mint` and the Credential's name. + +## Rollout + +1. This proposal is accepted. +2. Independent change: the lineage webhook keeps an explicit `tenantresource: "false"` and still overwrites a chart-set `"true"`. +3. The Cozystack API gains `credentials`, `mint`, `revoke`, seeding and the password formats of §3, and the ApplicationDefinition gains the `credentials` block, serving no kind yet, with the Event and its RBAC. The documentation gets the audit policies, and monitoring-agents the output value. +4. Wave 1: PostgreSQL and MariaDB convert once the checks left under Testing pass, and ClickHouse with them or a release later. +5. Wave 2: OpenSearch, NATS, RabbitMQ, Redis and Valkey, each with its §3 row and the plaintext in its account Secret. +6. Wave 3: MongoDB, Bucket, VPN, Qdrant, Harbor and Monitoring, the same way. Kafka joins whichever wave is open once its chart has users. +7. After this proposal, as separate tasks: each wave-2 engine moves from stored plaintext to a verifier. +8. The migration's Job and ConfigMap, which nothing else refers to, are removed once no supported upgrade path starts before the conversion release and no release is left in the skipped list. + +## Open questions + +- Whether a `Credential` should report that the engine applied a mint, and which component would write that signal. A new controller in the pattern of `securitygroup-controller`, or a reconciler in an existing controller, would patch the applied `resourceVersion` into an annotation of the account Secret for the API to show. Neither is built or sized here, and MariaDB reports nothing per account, so it would need a privileged read-back of the stored hash. The block would say whom to ask, as a status source with a `kind`, built in for the operators the platform knows and a webhook for the rest, and the block as proposed leaves room for it. +- Whether platform passwords may stay in Helm history. §4 leaves each live value in the release Secrets for `MaxHistory` revisions, where cluster administrators, Flux and the API's ServiceAccount can read it, and the Job rotates it once, at the conversion. A later rotation, or a password that never enters a chart render, would close that. Neither is designed here. + +## Alternatives considered + +- Keeping the plaintext and guarding a one-time collection window over it, with a durable journal of every access. All of it exists to guard stored plaintext, which verifiers remove. +- Keeping generation in the chart, as today. The plaintext stays in Helm history for `MaxHistory` revisions, so a rotation cannot revoke and a `helm rollback` restores the old value. It stays only for platform accounts, whose history no tenant can read. +- Moving every platform account into a new Secret. MariaDB's `root` reference cannot change once set, so one engine would stay behind in any case. +- A stored `Credential` kind next to the account Secret. CNPG needs a Secret per role anyway, and a record on it cannot drift from the value it describes. +- A subresource on the application, such as `postgreses//mint` with the user in the body. The audit event would name the application and not the account, and RBAC could not address one account. +- Writing the verifier into the live `Cluster` or CHI. Postgres releases are applied client-side, so the patch vanishes at the next render, and a ClickHouse render that lands first gives the account the default password. +- Postgres through the init-job with a forced upgrade per call, which runs the whole chart every time and does nothing on a suspended release. +- ClickHouse through `valueFrom.secretKeyRef`, which restarts the pods on every change, or `k8s_secret_password_sha256_hex`, which operator 0.27.4 removes. + +--- + + diff --git a/design-proposals/user-secrets-api/example.md b/design-proposals/user-secrets-api/example.md new file mode 100644 index 00000000..797d84f5 --- /dev/null +++ b/design-proposals/user-secrets-api/example.md @@ -0,0 +1,530 @@ +# The contract for a kind, and a walkthrough + +This file has two parts. [The contract for a kind](#the-contract-for-a-kind) shows what the author of a kind writes in its definition, what its chart renders and what the API writes, for PostgreSQL, for ClickHouse, whose operator reads no Secret, and for a kind outside the tree that is not a database. The [walkthrough](#walkthrough-one-postgresql-application) follows one PostgreSQL application through the design in [README.md](./README.md), step by step, with the objects each step creates or changes. The README says why things work this way, and this file shows what happens. + +## The contract for a kind + +Three parties each have a fixed part. The author of a kind writes the `credentials` block in its ApplicationDefinition. The chart renders what the engine or the pod needs and nothing secret. The Cozystack API writes the account Secrets and serves the mint, and knows nothing of the kind beyond its block. + +Three kinds follow, and together they show every part of the block: + +| Part of the block | Shown in | +|---|---| +| `ValuesMap` | all three | +| `format` with a built-in function | PostgreSQL (`scram-sha-256`), ClickHouse (`sha256`) | +| `format` the API does not have yet | nginx (`apr1`) | +| `secretLabels` | PostgreSQL, ClickHouse | +| `Static` with `owner: platform`, and the platform's own Secret | ClickHouse | +| `valuesPath` and `valuesFrom` | ClickHouse | +| `Static` with `when`, and `format: plaintext` | [Redis](#a-kind-with-no-users-redis), below | +| the engine reads the account Secret | PostgreSQL | +| the pod reads the account Secret | nginx | +| the chart gets the verifier as a value | ClickHouse | + +### PostgreSQL + +What the author writes in the definition. The block sits next to the `release` and `secrets` fields it interacts with, and each field carries a line saying why it is there. + +```yaml +apiVersion: cozystack.io/v1alpha1 +kind: ApplicationDefinition +metadata: {name: postgres} +spec: + application: + kind: Postgres + openAPISchema: |- # users is a map of flags with no password property; the API drops one with a warning + {"type":"object","properties":{"users":{"type":"object","additionalProperties":{"type":"object","properties":{"replication":{"type":"boolean"},"createRole":{"type":"boolean"},"bypassRls":{"type":"boolean"}}}}}} + release: + prefix: postgres- # the release is postgres-, and the account Secrets are named after it + chartRef: {kind: ExternalArtifact, name: cozystack-postgres-application-default-postgres, namespace: cozy-system} + credentials: + format: {name: scram-sha-256} # the account Secret holds a SCRAM verifier, which CNPG stores as it is + secretLabels: {cnpg.io/reload: "true"} # CNPG notices a changed Secret only with this label + accounts: + - {kind: ValuesMap, path: users} # the keys of spec.users are the tenant accounts + # no Static entry: PostgreSQL has no fixed or platform account + secrets: + include: + - matchLabels: {internal.cozystack.io/tenant-ca: "true"} # the public trust anchor; the entry for -credentials is gone +``` + +What the chart renders, and what it must not. No `lookup`, no `randAlphaNum`, no Secret or init-script holding a password, and no Role or `secrets` entry that names an account Secret. PostgreSQL has no platform account, so after the migration there is no `-credentials` at all. A kind that has one renders that Secret with `internal.cozystack.io/tenantresource: "false"` (README §4). The chart cannot read an account Secret at render time, so it only names it, and CNPG reads it at runtime through `passwordSecret`, which the reload label from the block keeps current. The name is the API's mapping, which the chart computes with a helper, and a chart test compares the helper with the API's function, because CNPG skips a role whose Secret is not there, with no error. [Step 3](#3-the-chart-renders-the-roles) has the whole rendered `managed.roles`, and this is the part that names the Secret: + +```yaml +{{- range $user, $_ := .Values.users }} +- name: {{ $user }} + login: true + passwordSecret: + name: {{ $.Release.Name }}.{{ include "postgres.accountName" $user }}.account # the API's name mapping +{{- end }} +``` + +What the API writes. On create, before the HelmRelease, one Secret per key of `spec.users`. On an update, one more for each key the write adds and a delete for each key it removes. The Secret for `web` is the one in [step 2](#2-the-api-seeds-the-account-secret), and every field of it is fixed: + +- `type: kubernetes.io/basic-auth` +- the name `postgres-orders.web.account`, from the release and the user name lowercased with every character outside `[a-z0-9-]` replaced by `-`, and two names that map to the same object name, or to an invalid one, are refused +- the labels `apps.cozystack.io/application.kind`, `.group` and `.name` of the application, which the view lists by, and `cnpg.io/reload: "true"` from `secretLabels` +- the annotations `credentials.cozystack.io/origin` (`NotIssued`, `Migrated`, `Minted` or `Revoked`) and `credentials.cozystack.io/history` (a JSON list of the last ten `{operation, at, by}`, newest first) +- the data keys `username`, the original name, and `password`, the SCRAM verifier of a random value that the API discarded +- the HelmRelease as owner + +A mint replaces `password` and the two annotations in one update that fails if the Secret changed since the caller read it, and returns the plaintext once ([step 6](#6-the-administrator-mints-a-password)). + +### A kind outside the tree: nginx `auth_basic` + +`MyApp` is a web application behind nginx `auth_basic`, and its kind lives outside the repository. It is not a database, so no operator reads the account Secret and the pod does. `apr1`, the htpasswd hash, is not one of the four formats of README §3. This section takes it as a fifth function in `pkg/credentials/format`, a change to the repository. A kind that needs a format the API lacks names `plaintext` and hashes in the pod, or waits for the format, and a webhook format (README §7) would let it bring its own. + +What the author writes in the definition. + +```yaml +apiVersion: cozystack.io/v1alpha1 +kind: ApplicationDefinition +metadata: {name: myapp} +spec: + application: + kind: MyApp + openAPISchema: |- # users is a map with no password property; the API drops one with a warning + {"type":"object","properties":{"users":{"type":"object","additionalProperties":{"type":"object"}}}} + release: {prefix: myapp-, chartRef: {kind: ExternalArtifact, name: myapp-chart, namespace: cozy-system}} + credentials: + format: {name: apr1} # the account Secret holds an htpasswd hash + accounts: + - {kind: ValuesMap, path: users} # the keys of spec.users are the tenant accounts + # no secretLabels: nothing but the pod reads the Secrets + # no Static entry: no fixed or platform account in this kind + secrets: {include: []} # nothing credential-bearing is tenant-visible +``` + +What the chart renders, and what it must not. The same rules: no `lookup`, no `randAlphaNum`, no Secret with a password in it, and nothing in the dashboard resource map that grants a Secret. The chart cannot read the account Secrets at render time, so the pod reads them at runtime: one Secret volume per declared account, and a sidecar that composes the file nginx reads. `optional: true` also hides a wrong name. If the helper maps a user differently from the API, the volume stays empty and nobody can log in, with no error, which is why the chart test that compares the helper with the API's mapping applies here as well. + +```yaml +volumes: + {{- range $user, $_ := .Values.users }} + - name: account-{{ include "myapp.accountName" $user }} # the API's name mapping + secret: + secretName: {{ $.Release.Name }}.{{ include "myapp.accountName" $user }}.account + optional: true # a Secret written around the API must not block the pod + {{- end }} + - {name: auth, emptyDir: {}} +containers: + - name: nginx # auth_basic_user_file /etc/nginx/auth/htpasswd, re-read per request + volumeMounts: [{name: auth, mountPath: /etc/nginx/auth, readOnly: true}] + - name: htpasswd-sync # writes ":" from the two keys of each mounted account, renames over htpasswd, loops on change +``` + +What the API writes. On create, before the HelmRelease, one Secret per key of `spec.users`. On an update that adds a key, one more, and on one that removes a key, a delete. For `users: {alice: {}, Bob.Smith: {}}` the second is: + +```yaml +apiVersion: v1 +kind: Secret +type: kubernetes.io/basic-auth +metadata: + name: myapp-site.bob-smith.account # lowercased, characters outside [a-z0-9-] replaced by - + namespace: tenant-acme + labels: + apps.cozystack.io/application.group: apps.cozystack.io + apps.cozystack.io/application.kind: MyApp + apps.cozystack.io/application.name: site # the view lists the account Secrets by these + ownerReferences: [{kind: HelmRelease, name: myapp-site}] # apiVersion and uid omitted; set right after the HelmRelease write on create, from the start on an update + annotations: + credentials.cozystack.io/origin: NotIssued + credentials.cozystack.io/history: '[{"operation":"Create","at":"...","by":"jane@example.org"}]' +data: + username: Bob.Smith # the original name + password: $apr1$... # hash of a random value the API discarded +``` + +Nobody can log in until a mint. `POST .../credentials/myapp-site.bob-smith/mint` replaces `password` and the annotations in one conditional write and returns the plaintext once. The mounted file changes on the kubelet's sync period, the sidecar rewrites `htpasswd`, and the new password works. `kubectl get credentials -n tenant-acme` lists both accounts with their origin, `-o yaml` adds the history, and neither shows a password. + +### ClickHouse: an operator that reads no Secret, and a platform account + +ClickHouse differs from the two kinds above in two ways. Its operator watches no Secrets, and the one live path to a user's password is a hash in the `ClickHouseInstallation`, which Helm renders, so the chart gets the verifier as a value and the HelmRelease pulls it from the account Secret. And it has a platform account, `backup`, which the backup jobs use and no tenant may see. + +What the author writes in the definition. + +```yaml +apiVersion: cozystack.io/v1alpha1 +kind: ApplicationDefinition +metadata: {name: clickhouse} +spec: + application: + kind: ClickHouse + openAPISchema: |- # users is a map with a readonly flag and no password property; the API drops one with a warning + {"type":"object","properties":{"users":{"type":"object","additionalProperties":{"type":"object","properties":{"readonly":{"type":"boolean"}}}}}} + release: + prefix: clickhouse- + chartRef: {kind: ExternalArtifact, name: cozystack-clickhouse-application-default-clickhouse, namespace: cozy-system} + credentials: + format: {name: sha256} # ClickHouse stores an unsalted SHA-256 hex, which is the value in the account Secret + secretLabels: {reconcile.fluxcd.io/watch: Enabled} # Flux reconciles the HelmRelease when an account Secret changes + valuesPath: _accounts # the operator cannot read Secrets, so the verifiers reach the chart as values under this key + accounts: + - {kind: ValuesMap, path: users} # the keys of spec.users are the tenant accounts + - {kind: Static, name: backup, owner: platform} # the platform's account: no Credential, no account Secret, no mint, and a users.backup is skipped + secrets: {include: []} # the entry for -credentials is gone, since that Secret now holds only backup +``` + +What the chart renders, and what it must not. For tenant accounts the same rules as before: no `lookup`, no `randAlphaNum`, no password in any rendered object, and no Role or `secrets` entry that names an account Secret. The chart takes the verifier of each declared user from `.Values._accounts`, under the key `u`, where `` is the SHA-256 of the user name as `users` lists it. A user name fits neither a `targetPath` nor a value, since a comma fails the entry, while a hex digest fits both and leaves no order to keep aligned. The render fails when a declared account has no verifier, as a HelmRelease written around the API leaves it, because the operator would give such an account the password `default`. It also refuses tenant accounts named `backup`, `default` or `clickhouse_operator`. `backup` stays as it is today: the chart generates it with `lookup`, keeps it in `-credentials`, which it renders with `internal.cozystack.io/tenantresource: "false"`, and the backup CronJob and the sidecar read it from there. + +```yaml +configuration: + users: + {{- range $name, $u := .Values.users }} + {{- $verifier := get ($.Values._accounts | default dict) (printf "u%s" (sha256sum $name)) | required (printf "no verifier for account %s" $name) }} + {{ $name }}/password_sha256_hex: {{ $verifier }} + {{ $name }}/profile: {{ ternary "readonly" "default" ($u.readonly | default false) }} + {{ $name }}/networks/ip: ["::/0"] + {{- end }} + backup/password_sha256_hex: {{ sha256sum $backupPassword }} # from the lookup of -credentials, as today +``` + +What the API writes. For a release `clickhouse-reports` with `users: {analytics: {readonly: true}}` it writes the account Secret, with the same fields as in the PostgreSQL case, the Flux label in place of the CNPG one, and a SHA-256 hex in `password`: + +```yaml +apiVersion: v1 +kind: Secret +type: kubernetes.io/basic-auth +metadata: + name: clickhouse-reports.analytics.account + namespace: tenant-acme + labels: + apps.cozystack.io/application.group: apps.cozystack.io + apps.cozystack.io/application.kind: ClickHouse + apps.cozystack.io/application.name: reports + reconcile.fluxcd.io/watch: Enabled # from secretLabels + ownerReferences: [{kind: HelmRelease, name: clickhouse-reports}] # apiVersion and uid omitted + annotations: + credentials.cozystack.io/origin: NotIssued + credentials.cozystack.io/history: '[{"operation":"Create","at":"...","by":"jane@example.org"}]' +data: + username: analytics + password: <64 hex digits> # SHA-256 of a random value the API discarded +``` + +It also writes one `valuesFrom` entry per account into the HelmRelease. The API and the ApplicationDefinition reconciler build that list with the same function, and the reconciler writes it onto the releases that exist. The digest below is the SHA-256 of `analytics`, and `backup` gets no entry, no Secret and no Credential. + +```yaml +spec: + valuesFrom: + - {kind: Secret, name: cozystack-values} # as today + - kind: Secret + name: clickhouse-reports.analytics.account + valuesKey: password + targetPath: _accounts.u65f98121a162a56ad8ee919ed9ea394b9eabd8714616acf30636c6092ee350b9 +``` + +A mint changes `password` in that Secret. The label makes Flux reconcile the HelmRelease, helm-controller reads `valuesFrom` again and upgrades the release with the new hash, the operator rewrites the `chi--common-usersd` ConfigMap, and ClickHouse re-reads it within seconds. So a mint costs a release upgrade, and a suspended or failing release holds it until the release recovers. `kubectl get credentials` lists `analytics` and never `backup`. The migration rotates `backup` once: the Job writes a new value into `-credentials` and forces an upgrade, which applies its hash and restarts the ClickHouse pods once for the sidecar (README §4). + +That is the whole contract for a kind. PostgreSQL's page differs from nginx's where CNPG reads `passwordSecret` and the reload label takes the place of the volume and the sidecar, and ClickHouse's where the verifier travels through `valuesFrom` into the chart. + +## Walkthrough: one PostgreSQL application + +This part follows one application through the design, step by step, with the objects each step creates or changes. + +The tenant is `tenant-acme`, the application is a `Postgres` named `orders`, so the release is `postgres-orders`, and it has one user, `web`, with the admin role on the database `shop`. How MariaDB and ClickHouse differ is at the end. + +## 1. A tenant administrator creates the application + +The request reaches the Cozystack API from the dashboard, `kubectl` or any other client. No field carries a password. + +```yaml +apiVersion: apps.cozystack.io/v1alpha1 +kind: Postgres +metadata: + name: orders + namespace: tenant-acme +spec: + users: + web: {} + databases: + shop: + roles: + admin: [web] +``` + +## 2. The API seeds the account Secret + +Before it writes the HelmRelease, the API writes one Secret per user. The password under `password` is the SCRAM verifier of a random value that the API threw away, so the account can exist in the database while nobody can log in as it. + +```yaml +apiVersion: v1 +kind: Secret +type: kubernetes.io/basic-auth +metadata: + name: postgres-orders.web.account + namespace: tenant-acme + labels: + apps.cozystack.io/application.group: apps.cozystack.io + apps.cozystack.io/application.kind: Postgres + apps.cozystack.io/application.name: orders # the view lists the account Secrets by these + cnpg.io/reload: "true" # CNPG watches the Secret + annotations: + credentials.cozystack.io/origin: NotIssued + credentials.cozystack.io/history: '[{"operation":"Create","at":"2026-10-02T09:10:02Z","by":"jane@example.org"}]' +data: + username: web + password: SCRAM-SHA-256$4096:... # base64 in the real object +``` + +Then the API writes the HelmRelease `postgres-orders` and adds it to the Secret as owner. No ApplicationDefinition selects the Secret and no Role grants it, so no tenant can read it. + +## 3. The chart renders the roles + +The chart puts the marker role, the database roles and the user under CNPG's declarative role management. The user gets the flags it has in values, the marker as a parent, and its account Secret. Before Helm applies this `Cluster`, a pre-upgrade hook has checked that no role named `web`, `shop_admin` or `shop_readonly` exists without the marker, which matters once a user has held `createRole` ([below](#a-name-that-a-role-already-holds)). + +```yaml +apiVersion: postgresql.cnpg.io/v1 +kind: Cluster +metadata: + name: postgres-orders + namespace: tenant-acme +spec: + managed: + roles: + - name: cozystack:managed + login: false + - name: shop_admin + login: false + inherit: false + comment: role managed by helm + inRoles: [cozystack:managed] + - name: shop_readonly + login: false + inherit: false + comment: role managed by helm + inRoles: [cozystack:managed] + - name: web + login: true + inherit: true + replication: false + createrole: false + bypassrls: false + comment: user managed by helm + inRoles: [shop_admin, cozystack:managed] + passwordSecret: + name: postgres-orders.web.account +``` + +The init-job creates the database `shop`, hands it to `shop_admin`, sets object privileges and extensions, and sets no password or attribute. Once `web` and the database roles exist it grants the marker to them again `WITH INHERIT FALSE, SET FALSE`, since CNPG grants it with the default options. + +## 4. CNPG creates the role + +The instance manager on the primary creates `web` with the seeded verifier and reports the version of the Secret it applied. + +```yaml +status: + managedRolesStatus: + passwordStatus: + web: + resourceVersion: "48190" # the account Secret's resourceVersion + transactionID: 812 +``` + +## 5. The tenant sees the account, not a password + +Any tenant member at `view` or above can list the accounts. + +``` +$ kubectl get credentials -n tenant-acme +NAME APPLICATION USER ORIGIN +postgres-orders.web Postgres/orders web NotIssued +``` + +## 6. The administrator mints a password + +The dashboard mints right after step 1, so the person who created the application sees the password once, in the creation dialog. Later mints come from the Regenerate button, `kubectl create --raw` or any client. + +``` +POST /apis/core.cozystack.io/v1alpha1/namespaces/tenant-acme/credentials/postgres-orders.web/mint +{"apiVersion": "core.cozystack.io/v1alpha1", "kind": "CredentialRequest", "spec": {"preconditions": {"resourceVersion": "48190"}}} + +201 Created +{"apiVersion": "core.cozystack.io/v1alpha1", "kind": "CredentialRequest", "status": {"username": "web", "password": "<32 characters, shown once>", "issuedAt": "2026-10-02T09:14:07Z"}} +``` + +Inside the mint the API generated the 32 characters, computed their verifier, wrote it into `postgres-orders.web.account` in one update conditioned on `resourceVersion: "48190"`, and put the plaintext into the response. The Secret now reads: + +```yaml +metadata: + name: postgres-orders.web.account + resourceVersion: "48240" + annotations: + credentials.cozystack.io/origin: Minted + credentials.cozystack.io/history: '[{"operation":"Mint","at":"2026-10-02T09:14:07Z","by":"jane@example.org"},{"operation":"Create","at":"2026-10-02T09:10:02Z","by":"jane@example.org"}]' +data: + username: web + password: SCRAM-SHA-256$4096:... # the verifier of the new password +``` + +The plaintext is in no object. If the response is lost, nobody holds this password, and the next call replaces it. + +## 7. CNPG applies the new verifier + +The reload label makes CNPG notice the change. The instance manager runs `ALTER ROLE web PASSWORD 'SCRAM-SHA-256$...'`, and PostgreSQL stores the verifier as it is. CNPG reports `resourceVersion: "48240"` for `web`, and the new password works from then on. The `Credential` shows the record only, and says nothing about the engine: + +```yaml +apiVersion: core.cozystack.io/v1alpha1 +kind: Credential +metadata: + name: postgres-orders.web + namespace: tenant-acme + resourceVersion: "48240" +spec: + application: {apiGroup: apps.cozystack.io, kind: Postgres, name: orders} + user: web +status: + password: {origin: Minted, at: "2026-10-02T09:14:07Z", by: jane@example.org} + history: + - {operation: Mint, at: "2026-10-02T09:14:07Z", by: jane@example.org} + - {operation: Create, at: "2026-10-02T09:10:02Z", by: jane@example.org} +``` + +## 8. What the audit log and the events show + +kube-apiserver records the call before it proxies it to the Cozystack API, at `Metadata` level and without a body: + +```json +{"kind": "Event", "apiVersion": "audit.k8s.io/v1", "level": "Metadata", "stage": "ResponseComplete", + "verb": "create", + "requestURI": "/apis/core.cozystack.io/v1alpha1/namespaces/tenant-acme/credentials/postgres-orders.web/mint", + "user": {"username": "jane@example.org", "groups": ["tenant-acme-admin", "system:authenticated"]}, + "sourceIPs": ["10.244.1.17"], + "objectRef": {"resource": "credentials", "namespace": "tenant-acme", "name": "postgres-orders.web", + "apiGroup": "core.cozystack.io", "apiVersion": "v1alpha1", "subresource": "mint"}, + "responseStatus": {"code": 201}, + "annotations": {"authorization.k8s.io/decision": "allow"}} +``` + +monitoring-agents ships this line to VictoriaLogs, and a SIEM can take it from there and alert on `objectRef.subresource: mint`. The application also gets an Event, which reaches VictoriaLogs the same way: + +``` +$ kubectl get events -n tenant-acme --field-selector involvedObject.name=orders +TYPE REASON MESSAGE +Normal CredentialIssued password of user web issued by jane@example.org +``` + +## 9. A second administrator acts at the same time + +Bob opened the dashboard before step 6, so his Regenerate sends the version he saw: + +``` +POST .../credentials/postgres-orders.web/mint +{"kind": "CredentialRequest", "spec": {"preconditions": {"resourceVersion": "48190"}}} + +409 Conflict +{"kind": "Status", "reason": "Conflict", + "message": "credential postgres-orders.web changed: password issued by jane@example.org at 2026-10-02T09:14:07Z"} +``` + +Bob gets no password, and Jane's keeps working. If Bob regenerates again, knowing this, his mint replaces Jane's password and is recorded under his name. + +## 10. The password leaks: revocation + +``` +POST /apis/core.cozystack.io/v1alpha1/namespaces/tenant-acme/credentials/postgres-orders.web/revoke +{"apiVersion": "core.cozystack.io/v1alpha1", "kind": "CredentialRequest", "spec": {"preconditions": {"resourceVersion": "48240"}}} + +201 Created +``` + +The response carries no password. The API writes into the Secret the verifier of a new random password and throws that password away. CNPG applies it, and `web` can no longer log in with any password anyone holds. The origin becomes `Revoked`, the audit log shows a `create` on the `revoke` subresource, and the Event reads `CredentialRevoked`. Connections that were already open stay open. Getting a working password back is step 6 again. + +## 11. The tenant removes the user + +The tenant updates the application with `users: {}`. The API writes the new HelmRelease, and the chart drops `web` from `managed.roles`, so CNPG stops managing it. The init-job drops the role `web`, because it carries the comment `user managed by helm` and the membership in `cozystack:managed`, and is no longer declared, after reassigning what it owns. Once the HelmRelease write has landed, the API deletes `postgres-orders.web.account`, and `kubectl get credentials` no longer lists the account. + +## 12. The tenant deletes the application + +Helm uninstalls the release, and Kubernetes garbage-collects the account Secrets together with the HelmRelease that owns them. The history of `web` survives in the audit log only. A new `orders` created later under the same name gets new Secrets with new UIDs, and nothing from the deleted one carries over. + +## A name that a role already holds + +A user of the instance that holds `createRole` can create a role under a name that an administrator declares later, and CNPG adopts a role by name, keeping its settings and the creator's `ADMIN OPTION`. So once any user has held `createRole`, the pre-upgrade hook refuses to let that happen. A service account of the application with `createRole` has created a role `reporting`, and later the tenant administrator adds `users.reporting`. The hook finds a role with no marker and fails the upgrade before any resource is updated, so the `Cluster` never lists `reporting` and CNPG never adopts it: + +``` +role "reporting" already exists and this chart did not create it; a user of this instance has held createRole, so the role may be controlled by that user and users.reporting will not adopt it. A platform administrator must inspect the role and drop it as the superuser, or declare the user under another name +``` + +The same check covers `_admin` and `_readonly` for every database. It is off until a user has held `createRole`, which the hook records in a setting of the `postgres` database before CNPG applies the attribute. Once a platform administrator drops the role, or the tenant picks another name, the next retry of the upgrade passes. + +## Converting an application that already exists + +Before the conversion, the chart keeps `web`'s password in plaintext in `postgres-orders-credentials`, readable through `tenantsecrets`, the name grant and Helm history: + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: postgres-orders-credentials + namespace: tenant-acme +stringData: + web: "<16 characters>" +``` + +The release that converts PostgreSQL runs a hook before any chart changes. It records the chart version of the release and writes `postgres-orders.web.account` with the verifier of that same password, origin `Migrated`, and touches nothing else. Then the new chart renders the `managed.roles` of step 3. CNPG takes over `web` and applies the verifier of the same password, so every client keeps working. Once the release is Ready on a chart version other than the recorded one, the Job removes a `users.web.password` if one was left in the HelmRelease values and deletes `postgres-orders-credentials`, since PostgreSQL keeps no platform account there. The record shows `Migrated` until someone mints or revokes, and older Helm revisions keep the old plaintext until `MaxHistory` drops them. + +## When the migration leaves a release out + +A release the migration cannot convert is reported in the migration log and with an Event on its HelmRelease, and the cause decides what it does once the converted charts arrive. + +| Cause | What happens | +|---|---| +| Colliding users, or a postgres user named `app` | The converted chart refuses to render. The release keeps its previous revision, with old credentials working and exposed as before. | +| A MariaDB release without `-credentials` | The converted chart keeps the root guard of today's chart and fails the upgrade, so the release keeps its previous revision. | +| ClickHouse account Secrets that do not exist | The HelmRelease names them in `valuesFrom`, Flux stops the release with `ValuesError`, and it keeps its previous revision. | +| A PostgreSQL release whose Secret was gone | The converted chart renders. CNPG leaves a role without an account Secret untouched, so the old passwords keep working, and tenants can no longer read them. A tenant mints a new one. | + +Once an operator fixes the cause, running the Job again writes the account Secrets that are missing, from the plaintext in `-credentials` while it is there, lets the release switch and finishes it. An account with no Secret and no plaintext is not listed, and a mint by name creates the Secret. + +## How MariaDB and ClickHouse differ + +| Step | MariaDB | ClickHouse | +|---|---|---| +| 2, the account Secret | labelled `k8s.mariadb.com/watch`, `password` holds `PASSWORD()` output (`*` and 40 hex digits) | labelled `reconcile.fluxcd.io/watch: Enabled`, `password` holds the SHA-256 hex, and the HelmRelease gets a `valuesFrom` entry for the Secret, keyed by a digest of the user name | +| 3, the chart | the `User` points `passwordHashSecretKeyRef` at the account Secret | the chart renders `password_sha256_hex` into the CHI from the value that entry supplies | +| 7, applying a new value | the operator re-runs `ALTER USER` | helm-controller upgrades the release, the operator rewrites `chi--common-usersd`, and ClickHouse re-reads it within seconds | +| Conversion | `-credentials` keeps `root`, which the Job rotates with `ALTER USER` | `-credentials` keeps `backup`, which the Job rotates, restarting the ClickHouse pods once | + +## How the kinds declare their accounts + +The block is `spec.credentials` of the ApplicationDefinition of the kind, so a tenant administrator, who can edit the application, cannot change how its passwords are stored. + +| Kind | Block | +|---|---| +| PostgreSQL | `format: {name: scram-sha-256}`, `secretLabels: {cnpg.io/reload: "true"}`, `accounts: [{kind: ValuesMap, path: users}]` | +| MariaDB | `format: {name: mysql-native}`, `secretLabels: {k8s.mariadb.com/watch: ""}`, `accounts: [{kind: ValuesMap, path: users}, {kind: Static, name: root, owner: platform}]` | +| ClickHouse | `format: {name: sha256}`, `secretLabels: {reconcile.fluxcd.io/watch: Enabled}`, `valuesPath: _accounts`, `accounts: [{kind: ValuesMap, path: users}, {kind: Static, name: backup, owner: platform}]` | +| Redis | `format: {name: plaintext}`, `accounts: [{kind: Static, name: default, when: authEnabled}]` | + +`owner` is `tenant` unless the entry says otherwise. MariaDB's `root` and ClickHouse's `backup` are `owner: platform`: they stay in `-credentials`, and the API does nothing for them, so there is no `Credential` and no account Secret for either, and a mint cannot reach them. The entry also keeps a `users.root`, which the MariaDB chart ignores today, from being listed, seeded or migrated as a tenant account. ClickHouse's chart refuses its reserved names itself. + +## A kind with no users: Redis + +Redis has no `users`: one password, in `redis-cache-auth` for an application `cache`, which the chart generates with `lookup` and the operator reads through `auth.secretPath`. Its ApplicationDefinition declares that password as a fixed account, present while `authEnabled` is true: + +```yaml +spec: + credentials: + format: {name: plaintext} # the account Secret holds the password itself, as in wave 2 + accounts: + - kind: Static + name: default # Credential redis-cache.default + when: authEnabled # the values key that switches the account on, true by default +``` + +The chart stops generating the password and points the operator at the account Secret that the API seeds, whose `password` key has the name the operator reads today: + +```yaml + {{- if .Values.authEnabled }} + auth: + secretPath: {{ .Release.Name }}.default.account + {{- end }} +``` + +The account starts as `NotIssued`, and a mint replaces the password in that Secret. How a changed password reaches the replicas and the sentinels is the Redis row of §3 at its conversion. + +## Headless provisioning and Terraform + +A pipeline with no human in it creates the application, and the accounts start as `NotIssued`. It mints from the tenant ServiceAccount, which holds the verb already, and stores the response where the workload reads it, a Secret of the tenant or its own password store. Storing is the caller's, so the platform guarantees that it holds no plaintext, not that the tenant keeps none. A GitOps helper or operator that mints and writes the result into the tenant's external password store can be built on this call, and is later work. + +A Terraform resource that wraps mint keeps the plaintext in state, as `aws_iam_access_key` does. That is the caller's choice and not for the strictest policies. `import` returns the metadata and no password, and a mint made elsewhere shows in `status.password.at` and `by`, which the provider reads as drift.