Skip to content

feat(kubernetes): support corporate proxy CA bundle - #3447

Open
feloy wants to merge 1 commit into
NVIDIA:mainfrom
feloy:feat-3443-k8s-proxy-https
Open

feat(kubernetes): support corporate proxy CA bundle#3447
feloy wants to merge 1 commit into
NVIDIA:mainfrom
feloy:feat-3443-k8s-proxy-https

Conversation

@feloy

@feloy feloy commented Sep 18, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds proxy_ca_bundle to the Kubernetes compute driver, so sandboxes can egress through an https:// corporate proxy with a private CA, or a TLS-intercepting proxy that re-signs upstream certificates. The bundle is read from the gateway Pod's filesystem and staged into the existing per-generation supervisor bootstrap Secret, which needs no new object kind, volume, mount, or RBAC verb. Also fixes the Kubernetes driver demanding proxy_auth_allow_insecure for https:// proxies, where the credential already travels inside a verified TLS session.

Related Issue

Closes #3443.

Changes

Driver

  • crates/openshell-driver-kubernetes/src/config.rs: new proxy_ca_bundle key; validate_upstream_proxy_config now delegates URL / no_proxy / connect_by_hostname / ca_bundle rules to the shared validate_upstream_proxy_settings, keeping the Secret-specific credential block local; new MAX_STAGED_PROXY_CA_BUNDLE_BYTES (256 KiB).
  • crates/openshell-driver-kubernetes/src/sandbox_runtime.rs: UPSTREAM_PROXY_CA_BUNDLE_KEY/_PATH; supervisor_bootstrap_secret takes an optional bundle; supervisor_pod emits --upstream-proxy-ca-bundle. No volume change — the bootstrap volume already mounts the whole Secret at /.openshell/supervisor with no items filter.
  • crates/openshell-driver-kubernetes/src/driver.rs: read_staged_upstream_proxy_ca_bundle helper, a spawn_blocking preflight in new() so a bad path fails gateway startup, and a per-generation read before any Service, Pod, or Secret is created.
  • crates/openshell-driver-kubernetes/src/main.rs: --proxy-ca-bundle / OPENSHELL_UPSTREAM_PROXY_CA_BUNDLE for the standalone driver binary.
  • crates/openshell-core/src/driver_utils.rs: auth_setting_label on UpstreamProxySettings, so the inline-credential diagnostic names the Secret keys instead of proxy_auth_file — a key this driver rejects under deny_unknown_fields. No behaviour change for the Podman or VM drivers.

Helm

upstreamProxy.caBundle.{configMapName,key}, mounted into the gateway Pod through the shared workload template so it covers both Deployment and StatefulSet, rendering proxy_ca_bundle into gateway.toml.

Docs

docs/reference/gateway-config.mdx, docs/reference/sandbox-compute-drivers.mdx (including an OpenShift recipe for copying the cluster proxy's trusted-CA ConfigMap), architecture/sandbox.md, the driver README, and the debug-openshell-cluster skill, which asserted that https:// proxies were unsupported.

Design notes

The bundle is read from the gateway's own filesystem rather than referenced as a ConfigMap in the sandbox namespace. It becomes a trust anchor for every upstream the sandbox reaches, so it must stay in the gateway's trust domain; a ConfigMap in the workload namespace would widen that to anyone with write access there, and would be impossible to pre-provision in managed or operator workspace modes, where the gateway creates namespaces itself. The staging Secret is immutable: true, so the anchor cannot change underneath a running sandbox.

validate_upstream_proxy_settings is adopted for most rules but not for credentials: this driver accepts an explicit proxy_auth_allow_insecure = false without credentials, which the shared rules reject, so a blanket switch would break existing hand-written gateway.toml files. A regression test pins that case.

The 256 KiB staging bound guards the apiserver Secret limit. Note that the sandbox boundary's 1 MiB control frame is the tighter ceiling in practice, so raising the bound requires checking it. Operators should supply only the CA that signs the proxy's certificate, not a full merged trust bundle.

Testing

  • Unit tests added — openshell-core (26 passed), openshell-driver-kubernetes (241 passed)
  • Helm chart tests added — 155 passed, 4 of them new
  • Full test suite green
  • E2E not run — see below

Tests added

  • Unit: config.rs — proxy_ca_bundle TOML round-trip, rejection without https_proxy, empty-value rejection, credentials accepted without acknowledgement for an https:// proxy and still rejected for http://, and the inline-credential diagnostic naming the Secret keys. The existing explicit-false acknowledgement case is kept green as the regression guard for the validator switch. sandbox_runtime.rs — argument present when the bundle is staged and absent when it is not, no volume or mount introduced, and the bootstrap Secret staying immutable with the key present or absent. driver.rs — valid bundle read, fail-closed paths naming proxy_ca_bundle, and oversized-bundle rejection. driver_utils.rs — the new label in both default and overridden form.
  • Integration: gateway_config_test.yaml covers rendering and omitting proxy_ca_bundle; the new gateway_upstream_proxy_ca_test.yaml covers the mount and volume across both workload kinds, a custom ConfigMap key projected onto the fixed mounted filename, and no volume when unconfigured.
  • E2E: an https-ca mode was added to e2e/rust/tests/kubernetes_corporate_proxy.rs and e2e/with-kube-gateway.sh. The wrapper mints a CA and a CA:FALSE listener leaf, publishes the CA as a ConfigMap in the gateway's release namespace, and the test TLS-wraps the proxy listener. It compiles but has not been executed, because no Kubernetes cluster was available. It is the only coverage that proves the full chain — gateway read, bootstrap-Secret key, supervisor mount, --upstream-proxy-ca-bundle argument, TLS handshake — so it should be run before merge:
OPENSHELL_E2E_KUBE_CORPORATE_PROXY=1 \
OPENSHELL_E2E_KUBE_CORPORATE_PROXY_MODE=https-ca \
OPENSHELL_E2E_KUBE_TEST=kubernetes_corporate_proxy \
  mise run e2e:kubernetes

The pre-existing authenticated mode is worth re-running too, since the shared proxy_script() signature changed.

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)
  • Documentation updated

Documentation updated: docs/reference/gateway-config.mdx, docs/reference/sandbox-compute-drivers.mdx, architecture/sandbox.md, crates/openshell-driver-kubernetes/README.md, deploy/helm/openshell/README.md, skills/debug-openshell-cluster/SKILL.md

@feloy
feat(kubernetes): support corporate proxy CA bundle
0215c70 
The Kubernetes driver had no way to supply a CA bundle for the corporate
egress proxy, so an `https://` proxy with a private CA, or a TLS-intercepting
proxy, could not be used. Podman and VM already expose `proxy_ca_bundle`.

Add `proxy_ca_bundle` to `[openshell.drivers.kubernetes]` as a path the
gateway Pod reads. The gateway stages the PEM into the existing
per-generation supervisor bootstrap Secret and passes
`--upstream-proxy-ca-bundle` on the supervisor argv. That Secret is already
immutable, owner-referenced and garbage-collected, and its volume mounts
every key at /.openshell/supervisor with no items filter, so this needs no
new object kind, volume, mount, or RBAC verb, and works in shared, managed
and operator workspace modes.

The bundle is deliberately read from the gateway's filesystem rather than
referenced as an object in the sandbox namespace. It becomes a trust anchor
for every upstream the sandbox reaches, so it must stay in the gateway's
trust domain; the immutable staging Secret also keeps the anchor from
changing underneath a running sandbox.

Bound the staged bundle at 256 KiB. The shared reader's limit is exactly the
apiserver's own Secret limit and the bootstrap Secret carries four other
keys, so a bundle between the two would pass gateway startup and then fail
every sandbox create with an opaque `data: Too long`.

Delegate the URL, no_proxy, connect_by_hostname and ca_bundle rules to the
shared validate_upstream_proxy_settings, keeping the Secret-specific
credential block local: this driver accepts an explicit
`proxy_auth_allow_insecure = false` without credentials, which the shared
rules reject. This also fixes the acknowledgement being demanded for an
`https://` proxy, where the credential travels inside the verified TLS
session. Add auth_setting_label so the inline-credential diagnostic names
the Secret keys instead of proxy_auth_file, which this driver rejects as an
unknown key.

Document that the bundle should carry only the CA that signs the proxy's
certificate, or that an intercepting proxy re-signs upstream certificates
with. Public roots already reach the sandbox through the supervisor image and
its TLS stack, and the bundle is concatenated with that system store into a
single boundary control frame, so a full merged trust bundle spends the frame
budget on duplicated roots. The frame, not the apiserver Secret limit, is the
tighter of the two ceilings in practice; raising the staging bound requires
checking it.

Closes NVIDIA#3443

Signed-off-by: Philippe Martin <phmartin@redhat.com>
@copy-pr-bot

copy-pr-bot Bot commented Sep 18, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@feloy feloy mentioned this pull request Sep 18, 2026
Open
2 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@mrunalp mrunalp Awaiting requested review from mrunalp mrunalp is a code owner
@sjenning sjenning Awaiting requested review from sjenning sjenning is a code owner
@derekwaynecarr derekwaynecarr Awaiting requested review from derekwaynecarr derekwaynecarr is a code owner

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

feat(kubernetes): support corporate proxy CA bundle for https:// and TLS-intercepting proxies

1 participant

@feloy