Skip to content

feat(supervisor): apply streamed configuration snapshots - #3265

Open
pimlock wants to merge 3 commits into
1731-config-update-stage-1/pimlockfrom
1731-config-update-stage-2/pimlock
Open

feat(supervisor): apply streamed configuration snapshots#3265
pimlock wants to merge 3 commits into
1731-config-update-stage-1/pimlockfrom
1731-config-update-stage-2/pimlock

Conversation

@pimlock

@pimlock pimlock commented Sep 10, 2026

Copy link
Copy Markdown
Collaborator

Summary

Initialize supervisors and apply live configuration updates through ConnectSupervisor. Protocol revision 2 uses streamed snapshots and requires successful bootstrap before the gateway marks a supervisor initialized. Older protocol revisions retain polling compatibility.

Before-and-after Stage 2 walkthrough

Related Issue

Part of #1731.

Stacked on #3244.

Changes

  • Use streamed snapshots for startup and live policy, settings, middleware, and provider updates in combined and sidecar deployments.
  • Preserve image policy discovery and reconnect for a fresh bootstrap after updating the baseline.
  • Track update ordering and acknowledgements, reject stale results, and reconcile missed delivery.
  • Record update outcomes without persisting credential-bearing snapshots. Update architecture and operational documentation.

Testing

  • mise run pre-commit
  • mise run ci
  • mise run e2e:docker

Known reliability issue observed under load

A focused reproduction with 32 running sandboxes and 32 concurrent exec clients produced 6 false exit-code-1 results across 37,376 exec calls, including commands that explicitly ran exit 0. Diagnostic logs confirmed that the orphan reaper collected a child's exit status 0 before SSH's waiter received ECHILD and substituted exit code 1. No policy changes or gateway restart were needed.

The implicated spawn/register/reap code predates this PR. Existing fix commit c5cf4ec2b is included in #3142.

Checklist

  • Conventional commits with DCO sign-off
  • Architecture and user-facing documentation updated
  • Related public skill reviewed and updated
  • No generated TypeScript sources committed

@github-actions

Copy link
Copy Markdown

@pimlock
pimlock added this pull request to stack #3266 September 10, 2026 23:38
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
@pimlock
pimlock force-pushed the 1731-config-update-stage-2/pimlock branch from e2b8a35 to 797c36e Compare September 11, 2026 02:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant