feat(chart): provision per-experiment DB credentials (RFC-0003 D10, backend#1181)

[LukasWodka]

Summary

The chart half of RFC-0003 D10 (backend#1181) — flips client-runtime#235 from inert to live. All flag-gated on perExperimentDbCreds (default false); default installs render byte-for-byte unchanged.

• perExperimentDbCreds value (schema-typed, own banner in values.yaml) → renders PER_EXPERIMENT_DB_CREDS=1, TB_CREDMGR_USER=tb_credmgr, and TB_CREDMGR_PASSWORD (via secretKeyRef, never plain) onto the jobs-manager container.
• secrets.yaml: generate-once TB_CREDMGR_PASSWORD — same upgrade-stable lookup pattern as POD_TOKEN_SIGNING_SECRET (a regenerated password would needlessly churn the account), emitted only when the flag is on.
• rbac.yaml: the jobs-manager Role's secrets verbs gain delete — needed for the revoke path (_revoke_db_creds deletes the per-job cred Secret; the Secret is otherwise GC'd via its Job ownerReference).
• Chart.yaml 1.9.8 → 1.9.9 (chart publishes on version change).

jobs-manager self-provisions the tb_credmgr account from this Secret on startup (client-runtime#235) — no Helm hook Job needed against the digest-frozen mysql image.

Safety

Default false = today's shared-credential behavior, bit-for-bit. Enablement is per environment (dev first), after the OQ4 image check (all task images already read the injected creds — verified). Companion: client-runtime#235 (the mint/inject/revoke code); merge either order — both are inert until this flag is on.

Coordination

Touches the jobs-manager env, secrets, rbac, and values — not the images.ingestor block, so no collision with the #1360 / #490 ingestor-pin work. Chart bumped to 1.9.9; if #490 lands first at 1.9.9, I'll rebase this to 1.9.10.

Tests

2 new helm-unittest cases (env absent by default; flag on renders the 3 env entries incl. the secretKeyRef). 300 helm tests pass.

Epic: backend#1151 · Design: backend#1181 · Code: tracebloc/client-runtime#235

🤖 Generated with Claude Code

---

Note

Medium Risk
Touches credential material and RBAC for Secret deletion, but behavior stays off by default and is limited to chart templating until operators enable the flag alongside client-runtime mint/revoke code.

Overview
Adds RFC-0003 D10 chart wiring so environments can turn on per-experiment MySQL credentials via perExperimentDbCreds (default false; unchanged installs stay byte-identical). Bumps the chart to 1.9.9.

When the flag is on, the chart exposes PER_EXPERIMENT_DB_CREDS, TB_CREDMGR_USER, and TB_CREDMGR_PASSWORD (from a secretKeyRef) on jobs-manager; materializes upgrade-stable TB_CREDMGR_PASSWORD in the main Secret (generate-once / lookup / optional credmgrPassword pin with alphanumeric validation); and grants jobs-manager delete on Secrets only (ClusterRole and namespace Role) for per-job cred Secret cleanup. Values and schema document the knobs; helm-unittest cases cover default-off behavior and flag-on rendering.

^{Reviewed by Cursor Bugbot for commit 3afec19. Bugbot is set up for automated code reviews on this repo. Configure here.}

[LukasWodka]

👋 Heads-up — Code review queue is at 32 / 30

Above the WIP limit. The team convention is to review existing PRs before opening new work.

Open PRs currently in Code review (oldest first):

• backend#1146 — docs(rfc): platform cost & autoscaling architecture · author: @saadqbal · reviewer: @LukasWodka, @saqlainsyed007, @aptracebloc
• backend#1201 — docs(rfc): Data spaces & federated dataset composition (draft) · author: @LukasWodka · reviewer: @saadqbal
• backend#1217 — docs(rfc): Dataset integrity & score reproducibility (draft) · author: @LukasWodka · reviewer: @saadqbal
• backend#1261 — RFC-BACKEND-0008: board & process reset (decision RFC — for markup) · author: @LukasWodka · reviewer: @saadqbal
• backend#1284 — docs(deploy): backend deploy runbook skeleton (#1263) · author: @LukasWodka · no reviewer assigned
• backend#1332 — feat(check-model): allow KNN/GaussianProcess on a single-edge owned dataset (#923 follow-up) · author: @divyasinghds · no reviewer assigned
• backend#1353 — chore(ci): baseline historical gitleaks findings + wire caller · author: @LukasWodka · no reviewer assigned
• backend#1361 — feat(tasks): bound ExperimentQueue growth for PAUSED experiments (backend#1351) · author: @saqlainsyed007 · no reviewer assigned
• backend#1368 — feat(blob_store): migrate CheckModelClassesView link/model_reform to blob_store (#1162) · author: @saadqbal · no reviewer assigned
• claude-skills#14 — feat(skill): add tracebloc-frontend-app skill · author: @waqaskhanroghani · reviewer: @saadqbal

Pull from review before opening new work. (This is a nudge from the kanban WIP check, not a block.)

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 7522412. Configure here.}

[saqlainsyed007]

Focused, well-documented change — the env/secret material is cleanly flag-gated. The one structural gap: the RBAC grant is not gated on perExperimentDbCreds, so it applies to every install and breaks the "default-off / byte-for-byte unchanged" guarantee this PR is built around. Details inline.

[LukasWodka]

Thanks — all four addressed in 3afec19, plus Bugbot's appVersion:

• #A — the Secret delete verb is now flag-gated on perExperimentDbCreds and scoped to secrets alone (both the ClusterRole and the namespace Role). Default-off is back to ["create", "get"] — byte-for-byte unchanged, no cluster-wide delete-on-all-Secrets shipping to every install.
• #C — TB_CREDMGR_PASSWORD gains the 3rd tier: an explicit .Values.credmgrPassword operator pin (DR / pre-created account / forced rotation), mirroring podTokenSigningSecret, validated alphanumeric with a fail-fast.
• #B — documented the off→on→off→on regen edge in the comment, and that the pin is its fix.
• #D — added helm-unittests for both secrets.yaml (credmgr key absent off / present on / pin flows / non-alnum rejected) and rbac.yaml (delete verb gated + secrets-only). 307 helm tests, lint clean.
• Bugbot — appVersion bumped 1.9.8 → 1.9.9, back in lockstep.

bugbot run

[saqlainsyed007]

Re-reviewed after the push — all five points are addressed cleanly. Thanks for the thorough turnaround.

• RBAC delete gated + secrets-only — now a separate resources: ["secrets"] rule under {{- if .Values.perExperimentDbCreds }} in both the ClusterRole and Role; default installs grant no extra delete, and configmaps stay at create/get. ✅
• Operator pin — .Values.credmgrPassword added as tier 1 of a 3-tier resolution mirroring podTokenSigningSecret, with fail-fast alphanumeric validation, schema, and values docs. ✅
• Toggle rotation — documented as an edge with the pin named as the mitigation. ✅
• Tests — rbac_test.yaml (default-off no delete; flag-on secrets-only for ClusterRole + Role) and secrets_test.yaml (absent by default, present when on, pin flows through, non-alphanumeric pin rejected). documentIndex usage matches the existing convention. ✅

appVersion bump to 1.9.9 alongside the chart version is a fine tidy-up. LGTM.

[LukasWodka]

/fr-pass