Gateway restart leaves every sandbox in Error (StartFailed); --restart-policy always not honored #4910

Activity

1. github-actions

   added

   state:triage-neededOpened without agent diagnostics and needs triageOpened without agent diagnostics and needs triage

   on Oct 25, 2026
2. addozhang commented on Oct 25, 2026

   

   addozhang

   on Oct 25, 2026

   Author

   More actions

   After filing, we looked into the v0.1.5 source, and part of this behavior is documented design rather than a defect:

   • The workload dying across a gateway restart follows from the one-supervisor-per-sandbox design ("If the supervisor disconnects, the sandbox freezes the agent. Only the same supervisor process can reconnect and resume it. A new supervisor can't take over a running sandbox, even with valid credentials." — architecture docs).
   • The driver also anticipates this path: boundary_startup_termination_marker explicitly recognizes "sandbox boundary received SIGTERM before supervisor confirmation" (crates/openshell-driver-podman/src/watcher.rs, mirrored in the docker driver), and the readiness wait has a dedicated branch for "Docker supervisor exited before becoming ready".

   We're not contesting the non-resumability itself. The parts we'd still like to see addressed — or documented — are:

   1. --restart-policy always is not consulted on this path. The flag is documented as "Restart behavior after the canonical main process exits". A gateway restart does terminate the main process (SIGTERM before supervisor confirmation), yet Restart count stays 0: the gateway-startup restore reports StartFailed without evaluating the restart policy. If that's intentional, the flag/docs should state that the policy does not cover gateway restarts; if not, the restore path could fall back to the policy before parking the sandbox in Error.
   2. sandbox start is blocked from the post-restart Error phase ("sandbox must be Stopped, Completed, or a failed main-process Error to start … stop the sandbox before retrying start"). Either allowing start directly from this state, or documenting the required stop → start recovery, would remove the surprise for operators.

   Happy to reframe this issue to just those two points (or split them) if that helps triage.
3. O96a commented on Oct 25, 2026

   

   O96a

   on Oct 25, 2026

   More actions

   Likely two separate failures stacked here, and the policy one is the easier to explain: restart policy is only ever evaluated by the supervisor loop that owns the sandbox, so if that loop dies with the gateway and nothing re-reads the persisted policy on startup, --restart-policy is dead config no matter what you pass. Worth confirming the policy field actually round-trips into the state store — if it isn't persisted, that alone explains "always not honored".

   The StartFailed is a different bug, and you need to know which of two shapes you have before touching policy code:

   openshell sandbox list -o json
   docker ps -a
   

   Run both right after the restart. If the sandbox record persists but the container is gone, the start path is trying to start something that isn't there — the reconciler needs a "runtime object missing → re-create from spec" branch. If the container is still alive and only the gateway process died, it's an adopt bug: the gateway is re-adopting a handle it no longer owns and failing on the stale handle.

   I haven't run this on macOS with brew services, so treat the second half as what I'd check first rather than a diagnosis.
4. addozhang commented on Oct 26, 2026

   

   addozhang

   on Oct 26, 2026

   Author

   More actions

   Ran both checks on v0.1.5 (macOS, brew services, docker driver).

   Persistence: the policy field does round-trip. Same sandbox (--restart-policy always), immediately before and after brew services restart:

   $ openshell sandbox get lab-h1 -o json | grep -E 'restart_'
   "restart_count": 0
   "restart_policy": "always"        # before
   
   (gateway restart; listener back in ~2s; sandbox → Error)
   
   "restart_count": 0
   "restart_policy": "always"        # after — still present, still not acted on
   

   So "dead config" doesn't explain this one: the restored record carries the policy; the restore path just never consults it.

   Shape: docker ps -a right after the restart:

   openshell-default--lab-h1-<uuid>    Exited (1) 1 second ago    # workload: present, killed during restore
   openshell-default--lab-h2-<uuid>    Exited (1) 1 second ago
   # no supervisor containers remain: the restore-started supervisors exited before becoming healthy and were removed
   

   So it's neither shape exactly: the sandbox record persists, the workload container is present but freshly Exited (1) (the "boundary received SIGTERM before supervisor confirmation" exit), and the supervisor container is gone after failing readiness. sandbox start then refuses purely on the phase gate:

   Error: sandbox must be Stopped, Completed, or a failed main-process Error to start
     (current phase: Error); stop the sandbox before retrying start
   

   —the runtime objects a direct start would need (fresh supervisor + restarted workload) are exactly what the forced stop → start cycle creates anyway.

   That makes it closest to your adopt/state-machine shape: restore attempts adoption → the new supervisor can't resume the frozen workload (by design) → everything unwinds to Error → and the restart policy is never evaluated even though the field survived the restart in the record.