[FrostGod]

Summary

Adds RFC 0018, which proposes four ways to deploy supervisors: per workspace, per node, per sandbox, and per gateway. It sets out one shared set of safety rules for replacing a failed supervisor (fencing), and a plan to compare the four models and choose one.

Related Issue

Refs #4311. This PR only adds the RFC and changes no code. It also builds on #3661 and #2385.

Changes

• Adds rfc/0018-multi-sandbox-ha-supervisors/README.md with goals, the four proposed models, and the shared rules (fenced recovery, ownership epoch, leases).
• Describes how the models behave on failure and how to compare them, and lists the risks, alternatives, and open questions.

Testing

Docs-only change, so there are no unit or E2E tests. The RFC links, the Mermaid diagrams, and the headings are checked by eye. The full mise run pre-commit has not completed on this branch: the first run failed on a local uv environment error unrelated to the RFC, and the rerun did not start.

• Checks appropriate to the affected code and behavior pass
• Unit tests added/updated (if applicable)
• E2E tests added/updated (if applicable)

Checklist

• Follows Conventional Commits
• Commits are signed off (DCO)
• Architecture docs updated (if applicable)

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[mayawang]

Thanks for opening this. Some input from the Agent Substrate side, mostly on one case the RFC doesn't cover yet: a backend suspending a sandbox on purpose.

Suspended sandboxes

On Agent Substrate an idle sandbox is suspended and uses no compute, and it can resume later, possibly on a different node. The RFC's resume path is about replacing a failed supervisor. For a backend-initiated suspend, we'd hope for the following:

• While a sandbox is suspended, it holds no supervisor session. Otherwise sleeping sandboxes still take supervisor capacity, and reconnect attempts wake them back up (feat(supervisor): hold reconnect while a sandbox is suspended by its backend #4352).
• On resume, the sandbox can be assigned to any healthy replica, with fresh authority. The fenced recovery flow in section 3 looks close: suspend could be a planned release, and resume a new assignment.
• The runtime's reconnect window doesn't count time spent suspended. In our prototype it only started counting after resume, which worked well.
• Binary-identity pins and policy revisions survive a suspend, in line with section 4. On a backend that snapshots memory, they come back with the sandbox.

Placement

Our lean is a pool that isn't tied to nodes: one HA deployment per install, or per gateway group, scaled horizontally, with sessions assignable to any replica. Per node fits a suspend-capable backend poorly, because sandboxes move between nodes on resume and many nodes would mostly hold sleeping sandboxes. Per workspace tends to split capacity.

Capacity

If suspended sandboxes release their sessions, the number to size supervisors for on a backend like ours is active sandboxes, not total sandboxes.

Happy to walk through what we saw in our prototype, and to review as the comparison of the four models takes shape.

[FrostGod]

#4377 (comment)

will take this into consideration and update the RFC, thanks @mayawang

[EmilienM]

The direction makes sense to me. The main gaps are things that exist today and would stop or still authorize the old supervisor (driver reconciler, sandbox bearer, launch Secrets, namespace fence), plus how the lease gets renewed. Details inline.

[FrostGod]

Thanks @mayawang. Suspend isn't supported in OpenShell yet, so I've listed backend-initiated suspend (#4352) as a separate non-goal, but the design leaves scope for it as a planned release plus a fresh assignment on wake. I also noted that per-node fits poorly with backends that move sandboxes, and that capacity should be sized by active sandboxes.

[EmilienM]

Almost there, thanks for the quick turnaround. 9 of the 13 threads are fixed in the text, so go ahead and resolve those.

Two small things before I approve:

• L147, see inline.
• The four items you're leaving for the prototype (per-node binding, per-replica vs per-sandbox lease, launch material delivery, lease renewal path) aren't in the RFC yet. Could you list them under Open questions so they don't get lost?

Also I think this would be a feature behind a k8s feature gate, maybe worth documenting.

[EmilienM]

Another iteration needed, mostly for the new section 5 (cc @dhirajsb). Details inline. L147 and the open questions from my last review still stand.

[FrostGod]

will be addressing the comments today, was not online yesterday

[EmilienM]

Almost there, thanks. Everything from my earlier rounds is addressed. One point inline on L197 (cc @dhirajsb), plus a lint fix: the blank line before ## Motivation (L23) got dropped, so markdownlint fails (MD022/MD032).

[EmilienM]

Section 5 proposal, to help close it out.

The problem: sharing supervisors breaks the 1:1 pod topology that per-sandbox identity rides on today. Every acquisition option discussed so far fails one of three ways: a Workload API socket in the sandbox pod is reachable by the agent (same UID, same mount namespace); the Delegated Identity API is node-scoped; the Broker API, which does per-pod references, is still experimental.

The proposal: the gateway issues per-sandbox identity. The supervisor requests a short-lived, audience-bound JWT with sub = spiffe:///sandbox/ over its existing authenticated gateway session, and injects it post-chain the way token grant injection works today. The gateway mints only when the requesting supervisor currently owns the sandbox (assignment plus ownership epoch, the section 3 state), the audience matches a provider profile attached to that sandbox, and the run is active. sub is bound to the sandbox, run, and epoch at mint and at use, and everything fails closed.

I would write section 5 as those two contracts (what providers verify, what the gateway enforces) with this as the reference acquisition mechanism, so the backend stays swappable: the SPIFFE Broker API is the SPIRE-native acquisition path once it leaves experimental. SPIRE keeps attesting components (gateway and supervisor pods); per-sandbox identity is gateway-derived inside the spiffe://openshell/sandbox/ namespace already reserved in sandbox_jwt.rs. Kata needs nothing special, since no identity material enters the guest. Per-sandbox X.509 mTLS to third parties stays out of scope.

One thing to sanity-check: providers that validate the client assertion against a SPIRE-issued JWKS would repoint at the gateway JWKS. I do not expect that to be a blocker, but it is the interop risk.

@FrostGod @dhirajsb does this work for you as the section 5 shape?

[grs]

the gateway issues per-sandbox identity

I'm not opposed to that necessarily, but it is quite a big change in behaviour. Rather than trusting SPIRE, the token service now needs to trust the gateway and the gateway would need to provide appropriate JWKS endpoint.

A setup that configured Keycloak to accept SPIRE minted SVIDs would need to be re-configured to target the gateway itself instead of SPIRE.

Again, I'm not saying this should invalidate the proposal.

[EmilienM]

the gateway issues per-sandbox identity

Fair, and @dhirajsb says the same in the thread: it changes who providers trust, and that's too big a change to ride along with shared supervisors. I'm dropping it. I've suggested a narrower scope for section 5 in the L179 thread.

[EmilienM]

LGTM

[dhirajsb]

Sandbox-specific workload identity, issued by our trusted SPIFFE provider, is a must-have requirement for NVIDIA AI Factory’s adoption of OpenShell. We also need shared supervisors to overcome the scaling limits of a dedicated supervisor Pod per sandbox; falling back to dedicated supervisors leaves that requirement unmet. A shared-supervisor mode without sandbox-specific workload identity would therefore not be usable for our deployment. #4732, which proposes gateway-managed acquisition of distinct SPIRE-issued sandbox identities, is a possible solution, subject to validating its delegation boundaries and operation at scale.

[FrostGod]

@dhirajsb ,

must-have requirement for NVIDIA AI Factory’s adoption of OpenShell

This requirement will be addressed. For now, this is only an RFC/proposal; the implementation is being tracked in #4732, where the discussion is ongoing.

I suggest we avoid blocking this RFC and amend it later, once the approach is finalized through the PR review process. The reason we’d like to merge it now is that implementation has already started, and prototyping is actively underway.

[FrostGod]

/ok to test 4629c19