Skip to content

feat(k8s): deliver config.toml via ConfigMap, drop S3 from the k8s deploy path (studio#138) - #139

Merged
brettchien merged 1 commit into
mainfrom
feat/k8s-config-via-configmap-no-s3
Aug 29, 2026
Merged

feat(k8s): deliver config.toml via ConfigMap, drop S3 from the k8s deploy path (studio#138)#139
brettchien merged 1 commit into
mainfrom
feat/k8s-config-via-configmap-no-s3

Conversation

@brettchien

Copy link
Copy Markdown
Contributor

Summary

Brett hit deploy failed ... NoSuchBucket testing a k8s deploy through the New Fleet wizard. Root cause: the k8s path was reusing the exact same S3-backed hooks.pre_seed bundle-carrier the ECS path uses (a deliberate earlier design decision — see the module doc this PR replaces in k8s_driver.rs) — meaning a k8s pod needs AWS credentials just to boot, and a control-plane S3 bucket has to exist at all.

Verified directly against openabdev/openab before writing any of this:

  • Every image's Dockerfile.* bakes CMD ["openab", "run", "-c", "/etc/openab/config.toml"] — a plain local path by default.
  • load_config_raw_from_source (openab-core/src/config.rs) falls through to a local file read for anything that isn't http(s):///s3://.
  • openab's own Helm chart already mounts config.toml via a ConfigMap at /etc/openab and never overrides the command (charts/openab/templates/configmap.yaml + deployment.yaml) — not a new pattern, the upstream project's own existing k8s-native path.

Change

  • studio-cp: new provision_config_k8s_configmap (mirrors provision_acp_auth_k8s_secret's exact shape — Patch::Apply, idempotent) server-side-applies config.toml into a k8s ConfigMap, returns a k8s-configmap://<name>#<key> ref. provision_agent_k8s calls this instead of pushing bundle.zip to S3 / injecting a pre_seed hook / resolving a control-plane bucket at all.
  • oabctl::secrets: new parse_k8s_configmap_uri, same shape as the existing parse_k8s_secret_uri.
  • oabctl::k8s_driver::build_deployment: a k8s-configmap:// configFrom mounts that ConfigMap at /etc/openab and leaves the container command unset (image's own default CMD reads it) — no override, no S3. A legacy s3:///http(s):// configFrom (old stored manifests, and the still-untouched compose-library k8s path) keeps the old override-the-command behavior.
  • Dropped the S3-stored-manifest create-vs-redeploy lookup for the wizard's k8s path entirely — every field build_default_k8s_manifest sets is already resent by the wizard on every call, so there was nothing a stored manifest preserved that this doesn't already provide. K8sDriver::apply() already reports Created-vs-Updated from the live Deployment's own existence.

The AWS/ECS path and the older compose-library k8s path (provision_from_library_k8s) are untouched — this only changes the New Fleet wizard's k8s deploy (provision_agent_k8s). deploy_provision_agent's public signature (MCP tool + Tauri bridge) is unchanged.

Test plan

  • Added build_deployment unit tests: mounts ConfigMap + skips command override for k8s-configmap://, rejects malformed refs, existing s3:// tests untouched and still pass unchanged (backward-compat path verified by construction, not just claim)
  • Added parse_k8s_configmap_uri unit tests mirroring parse_k8s_secret_uri's
  • Could not compile locally — this sandbox reliably OOMs on aws-sdk-ec2 regardless of which crate is targeted. Field types (ConfigMap/Volume/VolumeMount/ConfigMapVolumeSource) were verified by reading the pinned k8s-openapi 0.24 source directly, not guessed. CI (build-test) is the real gate — please don't merge until it's green.
  • Brett to confirm in a live k8s deploy: no more NoSuchBucket, pod boots with zero AWS credentials

🤖 Generated with Claude Code

…ploy path (studio#138)

Brett, testing a k8s deploy: "deploy failed ... NoSuchBucket" — the wizard's
k8s path was pushing a bundle.zip to S3 and wiring openab's `hooks.pre_seed`
to pull it at boot, same as ECS. That's a deliberate reuse of the S3-backed
carrier from an earlier design decision (k8s_driver.rs's old module doc,
sub-slice 3c) — but it means every k8s pod needs AWS credentials just to
boot, and needs a control-plane S3 bucket to exist at all.

Verified directly against openabdev/openab:
- every image's Dockerfile.* bakes `CMD ["openab", "run", "-c",
  "/etc/openab/config.toml"]` — a plain local file path by default.
- `load_config_raw_from_source` (openab-core/src/config.rs) falls through to
  a local file read for any non-http(s)/s3 string.
- openab's own Helm chart already mounts config.toml via a ConfigMap at
  /etc/openab and never overrides the command
  (charts/openab/templates/configmap.yaml + deployment.yaml) — this isn't a
  new pattern, it's the upstream project's own existing k8s-native path.

## Change

- `studio-cp`: new `provision_config_k8s_configmap` (mirrors the existing
  `provision_acp_auth_k8s_secret` shape exactly — Patch::Apply, idempotent)
  server-side-applies config.toml into a k8s ConfigMap and returns a
  `k8s-configmap://<name>#<key>` ref. `provision_agent_k8s` now calls this
  instead of pushing a bundle.zip to S3 / injecting a pre_seed hook /
  resolving a control-plane bucket at all.
- `oabctl::secrets`: new `parse_k8s_configmap_uri`, same shape as the
  existing `parse_k8s_secret_uri`.
- `oabctl::k8s_driver::build_deployment`: a `k8s-configmap://` configFrom
  mounts that ConfigMap at /etc/openab and leaves the container command
  unset (image's own default CMD reads it) — no override, no S3. A legacy
  `s3://`/`http(s)://` configFrom (old stored manifests, and the
  still-untouched compose-library k8s path) keeps the old
  override-the-command behavior for backward compatibility.
- Dropped the S3-stored-manifest create-vs-redeploy lookup for the wizard's
  k8s path entirely — every field `build_default_k8s_manifest` sets is
  already resent by the wizard on every call, so there was nothing a stored
  manifest preserved that this doesn't already provide.
  `K8sDriver::apply()` already reports Created-vs-Updated from the live
  Deployment's own existence, no stored manifest needed for that either.

The AWS/ECS path and the older compose-library k8s path
(`provision_from_library_k8s`) are untouched — this only changes the New
Fleet wizard's k8s deploy (`provision_agent_k8s`).

Verification: could not compile locally (this sandbox reliably OOMs on
aws-sdk-ec2 regardless of which crate is targeted) — field types
(ConfigMap/Volume/VolumeMount/ConfigMapVolumeSource) verified by reading the
pinned k8s-openapi 0.24 source directly rather than guessing. CI is the
real gate here.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@brettchien
brettchien merged commit 38244fa into main Aug 29, 2026
2 checks passed
@brettchien
brettchien deleted the feat/k8s-config-via-configmap-no-s3 branch August 29, 2026 14:56
brettchien added a commit that referenced this pull request Aug 29, 2026
…o#138 follow-up) (#140)

Brett hit "namespaces \"openab-studio\" not found" applying the ConfigMap
right after #139 landed — the New Fleet wizard's "+ Create new namespace…"
field (studio#119) only ever *typed* a name into the deploy request; nothing
actually created the k8s Namespace object, so picking it 404'd the instant
anything (ConfigMap/Secret/Deployment) tried to apply into it.

New `ensure_namespace_k8s` (Patch::Apply, same idempotent shape as
`provision_acp_auth_k8s_secret` — a no-op if the namespace already exists),
called at the top of both k8s provision paths: `provision_agent_k8s` (the
wizard, where Brett hit this) and `provision_from_library_k8s` (the older
compose-library path — same gap, fixed for consistency).

Verification: could not compile locally (this sandbox reliably OOMs on
aws-sdk-ec2). Field types (Namespace/ObjectMeta) verified by reading the
pinned k8s-openapi 0.24 source directly. CI is the real gate.

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant