feat: consistent sandbox access after the main process exits #4949

User Story

As a user running a command in an OpenShell sandbox, I want to inspect a retained sandbox after its main process exits, with exec, file transfer, port forwarding and exposed services, so that I can collect results and debug failures without restarting the workload.

Problem Statement

After the main process exits, a retained sandbox moves to Completed (exit 0) or Error with MainProcessFailed (nonzero exit). The supervisor deliberately keeps its access plane running. Whether you can still reach the sandbox currently depends on which command you use and which gateway replica receives it:

There are also two related inconsistencies:

• On a multi-replica gateway, the post-exit check only looks at the receiving replica's in-memory session map. A request that lands on a replica that doesn't hold the supervisor session is refused. A Ready sandbox doesn't have this problem, because relay opening already routes to the owning replica. Replica routing at the edge (grpcRoute.replicaRouting.enabled) is off by default, so requests land on any replica.
• The docs and the CLI skill say access stops once a sandbox completes, which isn't true for SSH or file transfer.

Impact / Why This Matters

• Users who run a batch command with --detach can ssh in and download results, but openshell sandbox exec -- cat result.txt fails. For agents driving OpenShell programmatically, exec is the primary interface.
• An exposed service started after the main process exits stops being reachable, even though forward service to the same port works.
• On multi-replica gateways, post-exit access works only some of the time, depending on load-balancer placement.
• Without a stated rule, each new access path decides for itself, and the inconsistencies come back.

Proposed Design

Lifecycle

stateDiagram-v2
    direction LR
    [*] --> Provisioning: create
    Provisioning --> Ready: supervisor connected
    Provisioning --> Error: preparation or admission timed out
    Ready --> Provisioning: supervisor disconnected
    Ready --> Completed: main process exits 0
    Ready --> Error: main process exits nonzero
    Ready --> Stopping: stop
    Error --> Stopping: stop
    Stopping --> Stopped
    Stopped --> Starting: start
    Completed --> Starting: start or restart policy
    Error --> Starting: start or restart policy
    Starting --> Ready: new supervisor connected
    Completed --> Deleting: delete, or --no-keep cleanup
    Error --> Deleting: delete, or --no-keep cleanup
    Stopped --> Deleting: delete
    Ready --> Deleting: delete
    Deleting --> [*]

Error has two causes. MainProcessFailed records a nonzero exit code in status.exit_code. An infrastructure failure, such as ImagePreparationTimedOut, ProvisioningTimedOut or a backend-reported failure, has no main-process result. start accepts Stopped, Completed, an Error with MainProcessFailed, or an Error from a provisioning timeout once cleanup has finished. stop on a Completed sandbox only revokes sessions.

Rules

1. One access rule for every path into a sandbox. Exec (unary and interactive), SSH sessions (SSH config, file transfer, and connect), TCP forwarding (forward start, forward service) and exposed-service HTTP routing all use the same check:

   • Ready: allowed.
   • Completed or Error: allowed while the sandbox's supervisor session is live.
   • Every other phase (Provisioning, Starting, Stopping, Stopped, Deleting): refused with FAILED_PRECONDITION sandbox is not ready, or HTTP 412 sandbox not ready for exposed services.

2. Access follows the supervisor, not the exit code. After the main process exits, the supervisor keeps its access plane, so reachability depends on whether the supervisor is still connected. Exit 0 and nonzero exits behave the same. An infrastructure Error usually has no supervisor left, because timeout cleanup stops compute, so it is refused. If the supervisor survived, access is allowed for debugging.

   sandbox connect passes this check but then fails, because it attaches to the main process and that process has exited. sandbox exec --tty -- /bin/bash -l gives an interactive shell instead.

3. The answer doesn't depend on the replica. "Live" means a session on the receiving replica, or a fresh supervisor owner record that names another replica. Relay opening already routes to that replica, so any replica gives the same answer.

4. The gateway decides. Clients may reject only phases where access can never succeed, and must leave the supervisor check to the gateway. The CLI accepts Ready, Completed and Error and forwards the request.

5. Retention decides whether the sandbox still exists.

   • Retained (the default): after the main process exits, the record and compute stay until you delete, or start a new run. Rule 1 governs access in the meantime.
   • Ephemeral (--no-keep): the gateway deletes the sandbox after the main process exits, whether the exit code is 0 or not, once terminal output has been delivered. Access ends with the delete.

6. A new run revokes old access. Starting a Completed or MainProcessFailed sandbox, manually or by restart policy, invalidates SSH sessions from the previous runtime generation.

Acceptance Criteria

• sandbox exec (unary and interactive) succeeds on a retained Completed or Error/MainProcessFailed sandbox while its supervisor is connected, and is refused once the supervisor has gone.
• Exposed services route to a retained Completed or Error sandbox while its supervisor is connected.
• On a multi-replica gateway, exec, SSH, forwarding and exposed services to a Completed sandbox succeed whichever replica receives the request.
• The CLI no longer rejects exec or service forwards on Completed or Error before asking the gateway.
• A conformance scenario certifies, for exit codes 0 and nonzero: --no-keep sandboxes are deleted after exit, and retained sandboxes record Completed or Error with the exit code and still accept exec.
• Published docs and the CLI skill describe access after exit for retained and ephemeral sandboxes.

Alternatives Considered

• Keep exec limited to Ready. This leaves exec inconsistent with SSH and file transfer, which already work after exit, and agents would have to fall back to SSH to read results.
• Allow access after exit only for Completed and Error/MainProcessFailed, and refuse every infrastructure Error. This is tighter, but it removes the debugging shell in the rare infrastructure failure where the supervisor survives. With rule 2, the supervisor check already refuses the common case, where compute is gone.
• Rely on edge replica routing to reach the owner. This is off by default and documented as a latency optimization. Correctness shouldn't depend on it.

Agent Investigation

• The SSH and forward check (sandbox_relay_reachable, added in fix(sandbox): complete successful main processes #2884) already allowed Completed and Error with a live supervisor. The exec handlers and service_routing.rs were not updated.
• Live checks on Docker and Kubernetes (single replica): after exit, SSH and download worked on both Completed and Error, and exec was refused by the CLI pre-check.

Out of scope, to be tracked separately:

• After a Docker gateway restart, the supervisor container of each Completed or Error sandbox is removed and not recreated, so the sandbox becomes unreachable even though its data is still on disk. Kubernetes is not affected.
• start on such a Docker sandbox fails until it is stopped and started again.