Start with the operator files (outside docs/), then every markdown file under docs/.
find docs -type f -name '*.md' is 94. The tables below index those files (including this one). Rows marked historical snapshot are dated captures, not current runbooks.
Paths that live in different directories for the same subsystem (AI, Ceph, network) are grouped together here on purpose. The files themselves were not moved.
These are not under docs/. They are the day-to-day starting points.
| File | What it covers |
|---|---|
AGENTS.md (CLAUDE.md is a symlink to it) |
Day-to-day commands, app layout, secrets, Volsync, networking, agent-facing conventions and anti-patterns |
.claude/skills/ |
JIT-loaded subsystem depth relocated from AGENTS.md NOTES (kopiur-backups, litellm-proxy, authentik-terraform, tdarr-transcoding, intel-gpu, cilium-host-policy, flux-substitution). Each skill's description is the load trigger; tripwires stay in AGENTS.md. |
talos/AGENTS.md |
Talos templates, render/apply/upgrade |
bootstrap/AGENTS.md |
First-time / disaster-recovery bootstrap |
README.md |
Cluster overview |
kubernetes/components/volsync/Readme.md |
Volsync in maintenance (3 dual-engine carve-outs only): schedules, multi-volume pattern, restore. Do not add to a new app. |
kubernetes/apps/base/system/kopiur/README.md |
kopiur Stage 0: operator + ceph/r2 repositories, deletion-protection and credential layout |
kubernetes/components/kopiur/Readme.md |
kopiur backup component (authoritative migration status: Stage 5 COMPLETE - 26 of 29 claims kopiur-only after three waves; 3 dual-engine carve-outs; retirement procedure and the pvc Component, schedules, split-shape overlays, per-namespace r2 hours, credential projection, mover-identity, privileged-mover root gate, SecurityContextCompatible semantics, rollback). |
GPU hardware, the live ai namespace stack (Hermes, ToolHive, AgentGateway, LiteLLM), and the retired-app tombstones. Flat docs/ai-gpu-changelog.md, docs/ai/, and docs/ai-system/ are listed together.
| Path | What it covers |
|---|---|
ai-gpu-changelog.md |
Chronological log of applied ai / B70 GPU changes, evidence, and rollbacks. Use when something in ai or on the B70 broke after a config change. |
ai/gpu-dra-migration-design.md |
Complete DRA migration design and why it is not shipped (vendor driver non-production; cannot share one GPU). Read before retrying DRA; do not apply. |
ai/b70-llm-serving-tuning.md |
Live B70 llama.cpp SYCL serving args, measured A/B matrix, and current ctx/embed settings. Use when changing vLLM/llama.cpp flags or debugging decode speed. |
ai/b70-second-card-decision.md |
Decision memo (2026-06-26; DEFER still stands) on why a second B70 is not authorized. Read before proposing another GPU purchase. |
ai-system/litellm/README.md |
Live LiteLLM governance layer (captain decisions B4/D4): what it is, what it is not, and where manifests live. Start here for any LiteLLM work. |
ai-system/litellm/auto-router.md |
D3 auto complexity-tier router: classifier, thinking-off trap, fail-open, and alerts. Use when changing routing tiers or debugging why everything stays local. |
ai-system/litellm/fallbacks.md |
Phase 5 availability and context-window fallbacks, plus the config-fallback allow-list bypass. Read before adding a cloud fallback or proving failover. |
ai-system/litellm/claude-code-subscription.md |
Claude Code Max/Pro pass-through, owning the natural claude-sonnet-5/claude-opus-5 names since the 2026-08-31 rename: OAuth token forwarding, full $0 pricing incl. cache fields, no cluster Anthropic key, the client-side [1m] context-window fix, plus the metered -metered admin escape hatch. Use when wiring the claude CLI through the proxy. |
ai-system/litellm/request-logs.md |
How to read stored prompts, responses, and cost from Postgres spend logs. Use when investigating what a caller sent or spent. |
ai-system/litellm/pr-reviewer.md |
In-cluster AI PR reviewer: advisory-only comments, zero-priced local alias, standards file and rotation traps. |
ai-system/agentgateway/README.md |
Cluster-specific AgentGateway install (standalone v1.4.1 in ai). Start here; do not follow kgateway docs. |
ai-system/agentgateway/01-quickstart.md |
Live-cluster smoke commands for the three Gateways and the unified /v1 route. |
ai-system/agentgateway/02-installation.md |
Flux install tree (CRDs + chart). Use this instead of helm upgrade -i kgateway. |
ai-system/agentgateway/03-gateway-setup.md |
Three data planes; defers listener/Service inventory to app/gateways/README.md. |
ai-system/agentgateway/04-llm-providers.md |
Model-name routing on unified /v1 and how to add an AgentgatewayBackend. |
ai-system/agentgateway/05-mcp-connectivity.md |
MCP is not federated through AgentGateway here; ToolHive owns it. |
ai-system/agentgateway/06-agent-connectivity.md |
In-cluster clients: kagent is gone; keyless OpenAI endpoint to use instead. |
ai-system/agentgateway/07-security.md |
Authentik extAuth on HTTPS plus API-key Strict on public HTTP. |
ai-system/agentgateway/08-observability.md |
What to scrape (port 15020 PodMonitor) and which kgateway dashboards not to import. |
ai-system/agentgateway/09-advanced-features.md |
Live failover via spec.ai.groups on AgentgatewayBackend. |
ai-system/agentgateway/10-api-reference.md |
CRDs this cluster actually applies (AgentgatewayBackend / Policy / Parameters). |
ai-system/agentgateway/11-cluster-deployment.md |
Live GitOps tree. Older fictional ai-system/ snippets in prior versions of this file must not be applied. |
ai-system/agentgateway/12-troubleshooting.md |
Commands that work in namespace ai (not ai-system / kgateway kinds). |
ai-system/agentgateway/13-function-calling.md |
Gateway-federated MCP tools are not wired; client-side tools still pass through. |
ai-system/agentgateway/14-session-management.md |
No gateway-side sessions; AgentgatewayParameters only sets ADMIN_ADDR. |
ai-system/agentgateway/15-optimization.md |
Live replica, resource, and cost-rule capacity. Ignore generic HPA/price tables from old guides. |
ai-system/agentgateway/GLOSSARY.md |
Cluster-local terms (AgentgatewayBackend, unified /v1, vs Envoy Gateway). |
ai-system/agentgateway/MIGRATION.md |
Actual GitOps chart history (kgateway-dev snapshot to standalone 1.x), not a kgateway upgrade matrix. |
ai-system/agentgateway-testing-report.md |
Historical snapshot (2026-03-31, kgateway-dev v2.3.0-main, namespace ai-system, per-provider URL prefixes). Do not copy its commands; use agentgateway/ instead. |
ai-system/kagent/README.md |
Tombstone: kagent was removed 2026-06-07. Do not install kagent charts or kagent.dev CRDs. |
ai-system/kgateway/README.md |
Tombstone: kgateway is not deployed. Non-AI ingress is Envoy Gateway in network. |
ai-system/kmcp/README.md |
Tombstone: kmcp was removed 2026-06-07. MCP is ToolHive (toolhive.stacklok.dev/v1alpha1). |
ai-system/retired-2026-08-22.md |
What the 2026-08-22 ai retirements kept (Postgres, 1Password, restic repos) and how to revive each app. |
ai-system/agentmemory-retirement-2026-08-31.md |
agentmemory retirement (2026-08-31): RBD destruction on merge, kopiur/VolSync revival identifiers, Hermes holographic memory switch. |
ai-system/hermes-state-db-growth.md |
ai/hermes state.db growth measurement (9.31 GiB, ~79% FTS duplication) and the sessions.retention_days: 30 / vacuum_after_prune: false fix. Read-only measurement method, why VACUUM cannot fit the claim, and the not-fixed items (existing size, boot-path quick_check, per-source retention). Section 11: the 2026-09-20 claim growth (25Gi -> 40Gi) and cache raise (16Gi -> 48Gi), why 16Gi was not inadequate, and the two post-merge operator actions Git alone cannot apply. |
ai-system/comfyui-retirement-2026-09-15.md |
Captain-approved removal of comfyui and its comfyui-mcp ToolHive server (2026-09-15), including deletion of 142Gi of provisioned storage. |
ai-system/embedding-truncation-followup-2026-09-14.md |
TEI truncation/memory sizing follow-up. Update 2026-09-15: the LiteLLM/external-caller traffic this covered moved to ai/embedding-gpu (see ai/embedder-gpu-migration-analysis-2026-09-15.md); remaining live consumer is ToolHive's tool-selection index, still exposed to vmcp's hardcoded Truncate: true. |
ai/embedder-gpu-migration-analysis-2026-09-15.md |
Moving ai/embedding-gpu (Qwen3-Embedding-0.6B) onto the B70. Measured VRAM/throughput tradeoffs with ai/vllm, the host-prompt-cache OOM history, and (section 11) the 2026-09-17 null-vector incident: a pod restart clears the accumulated NaN-vector fault and the liveness probe already recovered it once in production; batch_size=4 is the measured-safe operating point, GPU-side root cause still unattributed. |
ai/vllm-host-prompt-cache.md |
ai/vllm's 2026-09-19 host-RAM prompt-cache leak (no --cache-ram pinned) and its fix: bounded --cache-ram 4096 plus a glibc malloc-trim tunable, and the 2026-09-20 request right-sizing to 12Gi from the measured post-fix steady state. Corrected 2026-09-22: most of that growth was the oneDNN SDPA leak below. |
ai/vllm-onednn-sdpa-leak.md |
ai/vllm's 2026-09-22 memory growth with both #1731 fixes live: llama.cpp's never-evicted cache of compiled oneDNN SDPA kernels (one per new prefill shape). Fixed with GGML_SYCL_FA_ONEDNN: "0" at an expected ~30% prefill cost. Covers the read-only evidence chain, the correction to the #1731 heap diagnosis, the CI gate and the VLLMMemoryRetainedAboveBound alert, the post-merge checks (leak stopped, real prefill slowdown by depth, with a pre-merge baseline), and when to lift the pin. |
| Path | What it covers |
|---|---|
authentik/terraform.md |
Authentik OpenTofu stack: inventory, import vs create, RGW state bucket, apply approval gate, CI read-only plans. Read before any tofu plan / tofu apply. |
authentik/2026-09-26-server-2026.8-upgrade.md |
Planned server/worker/embedded-outpost upgrade 2026.5.6 -> 2026.8.3: release-line and breaking-change research, helm chart diff, pre-upgrade live baseline (CNPG + per-app SSO checks), and why PR #1639's terraform-provider-authentik bump is a separate, dependent change. |
Fleet PVC backup is kopiur on all 29 VolSync-derived claims; 26 are kopiur-only and 3 remain dual-engine after Stage 5 completed across three waves (2026-09-01 / 2026-09-02 / 2026-09-04). Born-kopiur claims (e.g. database/falkordb) sit outside that 29. The VolSync-derived fleet was 30 until downloads/autobrr was removed with its app on 2026-09-02 (kopia snapshots deliberately kept - backups/autobrr-removal-2026-09-02.md). Live claim lists: kubernetes/components/kopiur/Readme.md (authoritative) and kubernetes/components/volsync/Readme.md (3 carve-outs; do not add to a new app). All 30 VolSync-era claims are restore-proven on BOTH ceph and r2 as of 2026-09-01 - authority backups/kopiur-restore-proof-2026-09-01.md. Wave records: pilot backups/kopiur-stage5-pilot-retirement-2026-09-01.md, wave two backups/kopiur-wave-two-retirement-2026-09-02.md (authorised by backups/kopiur-wave-two-reproof-2026-09-02.md part 4), wave three backups/kopiur-wave-three-retirement-2026-09-04.md (last 19 eligible claims; live verification is a post-merge gate). Dual-engine end state: selfhosted/paperless-ngx (permanent), paperless-ngx-media and syncthing-data (not ready). Finding 2 of the fleet proof (r2 needs more kopia cache than ceph) is closed for ai/hermes - r2-proven at 16Gi, raised to 48Gi on 2026-09-20 with the claim (25Gi -> 40Gi) as conservatism against the unpinned eviction plateau, not because 16Gi had become insufficient (backups/kopiur-r2-restore-cache-gate-2026-09-02.md, ai-system/hermes-state-db-growth.md section 11); media/tdarr is r2-proven at 10Gi and standing-populator drift is closed against main (backups/kopiur-populator-drift-2026-09-02.md); media/plex is now also r2-proven at its standing 10Gi, closing the wave-three follow-up (backups/kopiur-plex-r2-restore-proof-2026-09-12.md). Operator/repos: kopiur/README.md.
| Path | What it covers |
|---|---|
backups/corrupt-claim-recreation-runbook.md |
Durable procedure for destroying and rebuilding a claim whose live mount works but whose snapshots will not clone (ext4 resize-inode inconsistency). Owns the empty-latestImage trap and the VolSync restore mode-relaxation warning. |
backups/opencode-volume-recreation-2026-08-31.md |
Historical snapshot / measured evidence for the first execution of the corrupt-claim runbook against ai/opencode on 2026-08-31. |
backups/kopiur-restore-drill-2026-08-30.md |
Verified kopiur restore procedure and Stage 2 gate (ceph + r2, sabnzbd-config byte-identical). Durable procedure plus two findings (empty Stage 1 pilot; mover-identity prerequisite). Timings are a historical snapshot from 2026-08-30. |
backups/kopiur-restore-proof-2026-09-01.md |
Fleet restore-proof evidence table for all 30 kopiur claims on both ceph and r2 (2026-09-01). Authority on restore coverage and the Stage 5 prerequisite (the four-volume retirement it authorised is a separate document); includes CACHEDIR.TAG adjudication. Finding 2 (r2 cache sizing) closed for ai/hermes in the row below. |
backups/kopiur-r2-restore-cache-gate-2026-09-02.md |
Authority on KOPIUR_CACHE_CAPACITY sizing: measured ~6.2 GiB plateau, ai/hermes raised to 16Gi and r2-proven end-to-end, fleet audit that predicted the tdarr/radarr raises. Closes restore-proof finding 2 for the large size class. |
backups/kopiur-populator-drift-2026-09-02.md |
Authority on standing-Restore vs Git drift under ssa: IfNotPresent: 30-row audit, hermes/autobrr recreate, tdarr/radarr 2Gi->10Gi (tdarr r2-proven), post-merge recreate commands, and why deleting a Restore is safe. |
backups/kopiur-stage5-pilot-retirement-2026-09-01.md |
The Stage 5 pilot: VolSync retired from 4 volumes (2026-09-01), why those four, what retiring removes, and the post-retirement re-proof. Carries the dataSourceRef-is-immutable finding and the prune: true claim-deletion hazard that components/kopiur/pvc exists to close. |
backups/kopiur-wave-two-reproof-2026-09-02.md |
Wave-two evidence: near-empty claim re-proof, prowlarr destination-identity closure, retention finding, and per-claim retirement verdicts that authorised the four wave-two retirements. |
backups/kopiur-wave-two-retirement-2026-09-02.md |
Stage 5 wave two: VolSync retired from 4 more volumes (2026-09-02; 8 kopiur-only total at the time). Pure GitOps ship with live verification deferred post-merge (78→66 ReplicationSources). |
backups/kopiur-wave-three-retirement-2026-09-04.md |
Stage 5 wave three / migration complete: VolSync retired from the last 19 eligible claims (2026-09-04; 26 kopiur-only). Pure GitOps; live verification post-merge (66→9 ReplicationSources). Split-shape greenfield deadlock fix and open follow-ups. |
backups/kopiur-plex-r2-restore-proof-2026-09-12.md |
Closes wave three's strongest open follow-up: media/plex restored from r2 at exactly its standing 10Gi cache (24,726 files / 4,948,787,362 bytes, byte-exact against the snapshot, database opens and is usable). Also records that the claim's snapshot grew 4.27→4.61 GiB since the value was sized, with no defensible single growth rate (two measurement windows disagree 2.3x). |
backups/autobrr-removal-2026-09-02.md |
Removing a kopiur-protected app while KEEPING its backups. Authority on what deletion.onPolicyDelete/onScheduleDelete: Retain actually retains - the kopia data, not the Snapshot CRs, which are GC-cascaded via their ownerReference - and why a CR census is the wrong preservation check. Also the post-merge database-drop procedure and the 1Password item left in place. |
backups/falkordb-snapshot-restorability-2026-09-04.md |
Volume-level snapshot restorability for database/falkordb: AOF + appendfsync everysec is load-bearing (91% silent loss on stock RDB-only), measured mover identity 1000:1000, writability proof, and auth/1Password prerequisites. |
backups/recyclarr-config-readable-check-2026-08-31.md |
CSI snapshot-restore readability probe for downloads/recyclarr-config (CronJob claim, no standing pod). Proves mover 2000:2000 can read the claim (2913/2913 files, 607/607 dirs) - readability only, not Stage 5 restore-fidelity (that landed in the fleet proof row above). |
backups/restore-drill-2026-08-23.md |
Verified VolSync restore procedure (Ceph + MinIO destinations, scratch-PVC method). Timings are a historical snapshot from 2026-08-23; the procedure is durable. |
backups/volsync-coverage-2026-08-22.md |
Historical snapshot of a full PVC-vs-VolSync coverage audit from 2026-08-22. Re-measure before trusting any figure. Current pattern: kubernetes/components/volsync/Readme.md. |
backups/kopiur-ceph-index-blob-compaction-2026-09-03.md |
Cause and fix for the ceph ClusterRepository's IndexBlobHealth=False / TooManyIndexBlobs condition (2026-09-01 to 2026-09-03): an epoch-tuning problem, not broken maintenance or lost data. Why takeoverPolicy: Force, raising the threshold, and a flat minDuration are all wrong fixes. |
backups/postgres-offsite-destination-design-2026-09-12.md |
Design for giving the shared postgres-17 CNPG cluster an off-site backup copy (it was the one fleet volume without one), working around CNPG's single-barman-destination-per-Cluster limit via an archive mirror. Shipped suspended pending a 1Password credential. |
backups/volsync-retired-repository-expiry.md |
Expiry policy and reasoning for 48 retired VolSync restic repositories (126.03 GiB) that no --keep-* retention can ever reach zero on a frozen repo. Not in force - no declarative path applies it; the exact-segment-match trap (syncthing is a prefix of syncthing-data) is the reason a naive prefix delete would destroy live data. |
backups/volsync-retired-expiry-apply-plan.md |
The operator runbook that actually applies the expiry policy above, one destination at a time, only on the captain's explicit go-ahead. Merging this document applies nothing. |
Flat docs/ceph-cluster-changelog.md and docs/ceph-performance-review.md are listed with docs/ceph/.
| Path | What it covers |
|---|---|
ceph-cluster-changelog.md |
Authoritative log of applied Ceph changes (CephX rotation, rgw_sigv4_insecure, PG/OSD tunables) and rollback notes. Start here for current Ceph truth. |
ceph-performance-review.md |
Historical snapshot of the 2026-06-01 performance review. Do not re-apply; current tunables live in the changelog. |
ceph/backup-recovery-strategy.md |
Live Ceph metadata CronJob backup (not last-resort DR) and pointer to emergency recovery procedures. |
ceph/lan-isolation-audit-plan.md |
Host-policy gates, rollback, and current status (stage 2 enforcing since 2026-08-25). Read before changing the Ceph CCNP. |
ceph/lan-isolation-audit-results.md |
Historical snapshot of the 25h stage-1 audit window that justified flipping policyAuditMode off. |
ceph/osd-device-path-recovery.md |
Runbook when an OSD pod is stuck Init after reboot (/dev/nvmeXn1 drift). Run task rook:check-osd-device-paths first. |
ceph/osd-store-corruption-recovery.md |
Runbook when an OSD crash-loops in load_pgs (BlueStore OMAP corruption), distinct from device-path drift. |
ceph/pg.md |
Placement-group autoscaler / bulk background. Changelog owns current counts; this is generic PG context. |
ceph/toolbox.md |
How to enable and use the Rook Ceph toolbox for ceph CLI operations. |
| Path | What it covers |
|---|---|
database/falkordb-memory-ceiling.md |
Why the 2026-09-09 database/falkordb OOMKill loop happened (the 8Gi limit was outgrown, not misconfigured) and how the three memory numbers (request / --maxmemory / limit) are derived and must move together. |
| Path | What it covers |
|---|---|
monitoring/exporter-endpoint-repair-2026-09-20.md |
Repairing the plex and n8n Prometheus scrape targets (2026-09-20): both exporters had been silently broken for months, only surfaced once PR #1698 made them scrapeable and TargetDown started firing. Includes the n8n /healthz vs /healthz/readiness probe trap. |
| Path | What it covers |
|---|---|
downloads/sabnzbd-disk-space-runbook.md |
SABnzbd "Too little diskspace" / shared-downloads CephFS full. Use when the movie pipeline is paused. |
media-stack.md |
Architecture of downloads to *arr to transcode to media servers. Use to see how the pipeline fits together. |
tdarr-errored-remuxes.md |
Tdarr safe-transcode traps (librariesToNotProcess Pro no-op, customFunction code vs function, forceConform stream deletion) and recovery path. Authoritative detail; media-stack.md points here. |
tdarr/README.md |
Tdarr rebuild runbook: PVC-only state (library scope, flow nodes, encoder args) that Flux cannot see, and how to restore it deliberately. |
docs/network/ and docs/networking/ are listed together. They were not merged.
| Path | What it covers |
|---|---|
networking/bgp.md |
Current Cilium BGP facts (ASNs, peers, LB pool, L2 off). Start here for BGP. |
network/cmd.md |
Historical snapshot (2025-08-16) of node networking during the 10.10.3.0/24 to 10.10.10.0/24 cutover. Not current node networking. |
network/envoy-gateway-internal-domains-analysis-2026-07.md |
Historical snapshot (2026-07-03/04) of an internal-domain outage analysis. Do not use for current chart versions or hostname counts. |
| Path | What it covers |
|---|---|
pvc/pvc-health-checks.md |
How Flux waits for PVCs to bind before deploying, plus troubleshooting unbound claims. |
| Path | What it covers |
|---|---|
app-structure.md |
App directory shapes: single-component app/ default, plus multi-component family, CRD-split, and parameterized instance; when each applies; authoring traps. Read before adding a new app; scaffold: scripts/add-app/README.md. |
branch-protection.md |
main's GitHub ruleset: what is enforced, why only Labeler is a required check today, and what closes the gap. |
hardware-incidents.md |
Hardware / Talos incident log (root cause, evidence, resolution). Use for node memory, DIMM RMA, and similar failures. |
talosctl.md |
Node IPs (not the VIP) and a short pointer to just talos recipes. |
organisation-services-1password-setup.md |
How to create 1Password items for a new organisation service consumed by ExternalSecret. |
flux-migration-validation-report.md |
Historical snapshot from 2026-03-29 of the pre-base/main Flux layout. Do not treat its FAILs as open work. |
grafana-operator-removal.md |
Why the 10 GrafanaDashboard CRs and the grafana-operator CRD bootstrap were dead, what replaced them, and outstanding live-cluster cleanup. |
system-namespace-consolidation-analysis.md |
Why system-controller and system-upgrade must not merge into system (k8tz webhook exclusion + Talos allowlist blockers). Read before re-proposing; do not apply. |
admission-webhook-create-only-drift.md |
CREATE-only mutating admission webhooks never re-fire on existing objects (k8tz CronJob spec.timeZone class): mechanism, detection, fix, and live sweep. |
k8tz-imagevolume-evaluation.md |
Why k8tz imageVolume injection was evaluated and declined (2026-09-01): QoS gate passes, upstream still recommends initContainer, /etc/localtime drop, 34-pod BestEffort transition. Read before re-proposing; do not apply. |
talos-3-scheduling-truth.md |
talos-3's memory arithmetic: why an undersized ai/vllm request was the real eviction risk, what is genuinely immovable (GPU pin, CNPG anti-affinity, OSDs), what was shed to fit an honest 40Gi request, and the resulting committed percentage and margin. Redo this arithmetic before adding a workload to talos-3 or raising a request there. |
reference.md |
This index. |