From c1a8021b0e2010cfaa2f379847695227aabfa121 Mon Sep 17 00:00:00 2001 From: Konrad Rokicki Date: Tue, 20 Oct 2026 10:16:02 -0400 Subject: [PATCH] docs: production deployment layout (Temporal in k8s, workers under systemd) Where the pieces run in production and why the workers stay on a cluster-attached host as the service account; the code changes needed (systemd backend for mia_ops, Temporal TLS/auth, namespaces as an admin step, agent billing from a file) and the open questions for the HPC team. Tracked in #52. Co-Authored-By: Claude Fable 5.1 --- README.md | 2 +- docs/deployment.md | 80 ++++++++++++++++++++++++++++++++++++++++ notes/autodev-runbook.md | 3 ++ 3 files changed, 84 insertions(+), 1 deletion(-) create mode 100644 docs/deployment.md diff --git a/README.md b/README.md index 681c135..e2ae2b2 100644 --- a/README.md +++ b/README.md @@ -5,7 +5,7 @@ for compute, and GitHub issues as the user interface. Two workflows so far: **au issue into a PR written by Claude Code, revises it on request, and merges it after human approval; **mia-data** turns a dataset in the MIA catalog into verified, published image crops, with a human approving every binding. Setup: [`docs/autodev-local-setup.md`](docs/autodev-local-setup.md). Running it under a bot identity: -[`docs/github-app.md`](docs/github-app.md). +[`docs/github-app.md`](docs/github-app.md). Production layout: [`docs/deployment.md`](docs/deployment.md). ## autodev diff --git a/docs/deployment.md b/docs/deployment.md new file mode 100644 index 0000000..168ea68 --- /dev/null +++ b/docs/deployment.md @@ -0,0 +1,80 @@ +# Deploying to production + +How the services run outside a developer's tmux: Temporal in Kubernetes, the workers and csub brokers on a +cluster-attached host as a service account, under systemd. This page is the plan; the parts marked **todo** are +not built yet. Developer setup is [`autodev-local-setup.md`](autodev-local-setup.md); the service-account checklist +(GitHub App, branch protection, broker policy) is in [`notes/autodev-runbook.md`](../notes/autodev-runbook.md). +Back to the [README](../README.md). + +## The split + +Temporal is only orchestration state and can live anywhere the workers can reach. Everything else has to run +*as the cluster account* on a host that sees the cluster: + +- **Shared filesystem, read-write, as the right uid.** The worker creates job workspaces under `work_root`, syncs a + git mirror in `state_dir`, copies records into workspaces, reads job output back with `read_untrusted`, and + mia-data publishes into `storage_root`. All of it must be visible to compute nodes and owned by the account that + runs the LSF jobs. +- **Claude credentials and the broker policy are the account's.** Jobs run `claude` with the broker account's + credentials; `~/.config/csub/broker.toml`, `~/.csub` and the LSF project belong to that account. +- **`bsub`.** csub has an `ssh` transport (`csub relay`) that would let a worker elsewhere reach a broker on the + submit host, so submission alone is not the blocker; the filesystem is. A worker in Kubernetes would need the + cluster filesystems mounted in the pod with a matching uid, which is a question for clusteradmins and a lot of + coupling for no gain. + +| component | where | notes | +|---|---|---| +| Temporal server, UI, Postgres | Kubernetes | one cluster; one namespace per deployment (`autodev`, `snippets`, `mia-data`); TLS on the frontend; UI behind SSO | +| csub broker + sandboxed worker, one pair per service | cluster-attached host, service account | systemd user units with linger, `Restart=always`, logs in journald | +| agent and harness jobs | LSF via csub | unchanged | +| configs and secrets | the service account's home | `~svc/.config/mia/{mia,autodev,autodev-snippets,mia_data}.yaml`, the App PEM, token files; `$MIA_MANIFEST` names the manifest | +| code | a tagged checkout at a fixed path, with a `current` symlink | the sandbox binds it read-only; deploy = `pixi install` in the new checkout, flip the symlink, `mia restart ` | + +### Which host + +Ask the HPC team for a small VM that is an LSF client with `/groups` and `/nrs` mounted. The submit host is +documented as one core and 16 GB shared by all users, for lightweight orchestration scripts; the worker is +lightweight when idle, but mirror syncs and snapshot extraction are bursts, and a shared host is where the +"unauthenticated dev server" gotcha came from. Fallback if no VM is possible: run each service as a long-running +LSF job (csub's "agent as a csub job" mode provides the sandbox and the broker socket). Temporal makes the 48-hour +queue limit harmless, since a restarted worker reattaches to running jobs, but the billing meter runs the whole +time, so it is a fallback, not the plan. + +### mia.yaml stays + +`mia.yaml` is the per-host manifest: which services run here, where agentic-sandbox is, whether `mia` runs +Temporal. Production uses it with `temporal.managed: false`, so `mia up` only verifies the cluster answers and +never starts or stops it. The service configs keep their shape; only `temporal.address` and the new TLS fields +change. + +## Changes needed (todo) + +None of these touch `workflows.py`, so there is no replay risk. + +1. **systemd backend for `mia_ops`.** A `runner: systemd` key in `mia.yaml` (default `tmux`). `mia systemd + install` writes `~/.config/systemd/user/mia--broker.service` and `-worker.service` from the same + `Stack`/`Service` objects `services.py` uses for tmux; `up`, `down`, `restart`, `ps`, `logs` and `attach` then + dispatch to `systemctl --user` and `journalctl --user`. The broker socket moves from `/scratch/$USER` (node-local, + periodically cleaned) to `$XDG_RUNTIME_DIR`, which exists once `loginctl enable-linger svc` is set. +2. **Temporal TLS and auth.** `TemporalCfg` gains `tls: bool = False` and either `client_cert`/`client_key` paths + (mTLS) or `api_key_file`, all with defaults, so the `Deployment` protocol and every existing config stay valid. + `_client` in `autodev/cli.py` passes them to `Client.connect`; the `temporal` CLI calls in `mia_ops/services.py` + pass the matching `--tls-*` flags. HTTP CONNECT through the sandbox proxy (`via_https_proxy`) works unchanged. +3. **Namespaces are an admin step.** In production `ensure_namespace` checks that the namespace exists and fails + with a message instead of creating it; creation needs an operator credential the worker should not hold. +4. **Agent billing from a file.** Production should not depend on an interactive OAuth login for the service + account. Add `agent.api_key_file`, bind it read-only into the job and set `ANTHROPIC_API_KEY` from it (the + broker policy's `env_allow` must cover it), and drop the `mia up` refusal of `api_key_env`. +5. **State on backed-up storage.** `state_dir` is the durable record on the cluster side (job ids, mirrors, + logs); put it under `/groups`, which is backed up, not under `/nrs`. The Temporal database is the other half; + back up Postgres in Kubernetes. +6. **Release procedure.** Tag, `git clone` into `releases/`, `pixi install`, `pixi run mia exec + check` against production config, flip `current`, `mia restart `. A worker restart mid-round is safe: + job submission is idempotent and `wait` reattaches by job id. Changes to `workflows.py` or `decide.py` still + need `workflow.patched()` for runs in flight ([`development.md`](development.md)). + +## Open questions for the HPC team + +- A cluster-attached VM for the service account, or permission to run lingering user services on the submit host. +- Whether the Kubernetes Temporal frontend is reachable from that host, and who issues its client certificates. +- Rootless bwrap (or podman) for the service account on the VM, since the worker runs inside agentic-sandbox. diff --git a/notes/autodev-runbook.md b/notes/autodev-runbook.md index b8bb703..5da35b7 100644 --- a/notes/autodev-runbook.md +++ b/notes/autodev-runbook.md @@ -63,6 +63,9 @@ round summary count, so a short prefix approved earlier cannot approve a later h ## Run it as the group service +This is the per-account checklist. Where the pieces run in production (Temporal in Kubernetes, workers under +systemd on a cluster-attached host) is [`docs/deployment.md`](../docs/deployment.md). + 1. **Service account** on the cluster with rootless podman / bwrap set up (agentic-sandbox README), a csub broker installed for it (csub `deploy/INSTALL.md`), and an LSF billing group in its broker policy. 2. **Broker policy** (`~svc/.config/csub/broker.toml`): `[agents] claude = ["api.anthropic.com", "claude.ai", "platform.claude.com"]`;