feat(api): add durable workspace mutation admission and replay

[mrunalp]

Summary

Implement durable at-most-once admission and bounded success replay for ordinary unary mutations. This PR includes phase 3a and the already-merged phase 3b (#3323), including interceptor-aware replay. It is stacked on phase 2 (#3317); exec launch admission follows in #3324.

Related Issue

Part of accepted issue #3051. Base: feat/3051-mutation-outcomes/mrunalp. This does not close the issue; credential capabilities, curated SDK request-ID helpers, and the final mutation audit remain separate work.

Changes

• Add optional UUID request IDs to the six workspace/template mutations and 24 additional ordinary unary mutations. Omitted IDs retain existing behavior.
• Use atomic, quota-checked admission to select one owned executor. Cancellation does not abandon admitted work. Unresolved claims are never expired or taken over; confirmed successes have a 24-hour retention window.
• Scope workspace create/delete request IDs to each target name, retaining platform-admin authorization and same-target payload-mismatch checks. Preserve the reviewed cross-target and name-reuse regressions.
• Store explicit resource references, deletion outcomes, and reviewed scalar/diagnostic receipts—not arbitrary credential-bearing response snapshots. Reauthorize retries and protect original identities, resource versions, and refresh grant epochs.
• Key sensitive payload fingerprints with existing gateway signing material. Fail closed if the key is unavailable or changes; replicas must share that material.
• Preserve original decoded payload identity across interceptors, revalidate effective requests, prevent request-ID mutation, and suppress duplicate post-commit observations on server-marked replay.
• Regenerate Go bindings and update Rust consumers, Python E2E coverage, gateway architecture, and the public API migration guide.
• Refresh the stack onto main's split-runtime architecture. Keep the OIDC test fixture's hardened defaults and box the inner create future to stay below Clippy's future-size limit.

Testing

Latest rebase verification:

• mise run pre-commit.
• Full gateway library suite on this PR: 1,591 passed, 9 opt-in tests ignored.
• Go, Python, and TypeScript protobuf generation.
• mise run docs: 0 errors, 3 warnings.

Earlier implementation verification includes full CI, TypeScript SDK CI, live PostgreSQL admission tests, and Docker/Podman workspace E2E. Those historical attestations remain in the PR comments; they are not claims about rerunning every deployment lane on this new head. Current full-stack verification and GitHub CI status are recorded in the latest update comment.

Checklist

• Follows Conventional Commits; commits are signed off for DCO.
• Architecture and published API docs updated.
• Prior review fixes preserved, including the workspace target-scoping fix.
• Existing skill and workflow behavior remains unchanged.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[mrunalp]

🏗️ build-from-issue-agent

E2E Test Attestation

Local E2E passed against an isolated Docker-backed gateway with fresh gateway,
CLI, and musl supervisor builds. The tested source tree is committed as
5dc3c72940b13ca0da55a0b62ec792d71e50cf17; the build's version text retained the
parent SHA because verification preceded the commit. The commit hook passed and
left no unstaged implementation changes.

Command:

mise run e2e:conformance:build
OPENSHELL_CONFORMANCE_BIN=/home/mrunalp/code/OpenShell/target/debug/openshell-conformance \
OPENSHELL_E2E_DOCKER_TEST=workspace_lifecycle \
e2e/with-docker-gateway.sh bash -lc \
  'e2e/rust/e2e-docker.sh __openshell_run_docker_e2e && uv run pytest -o "python_files=test_*.py *_test.py" e2e/python/test_workspace_api.py'

Results: conformance smoke passed; Rust 2 passed, 0 failed, 0 ignored;
Python 6 passed, no skips.

Tests executed:

• Conformance smoke: status/authentication, create, get-ready, list-visible,
  exec, delete, list-empty.
• workspace_lifecycle::workspace_full_crud_lifecycle.
• workspace_lifecycle::workspace_terminating_rejects_creates.
• test_workspace_api.py::test_workspace_crud.
• test_workspace_api.py::test_workspace_create_with_labels.
• test_workspace_api.py::test_workspace_list_includes_created.
• test_workspace_api.py::test_workspace_delete_nonexistent_raises_not_found.
• test_workspace_api.py::test_workspace_get_nonexistent_raises_not_found.
• test_workspace_api.py::test_workspace_request_id_replays_without_deleting_replacement.

Additional verification passed: mise run pre-commit, clean full mise run ci
(1,532 server unit tests, 9 ignored; 247 Python tests; Go SDK CI),
mise run sdk:ts:ci (108 tests), and strict docs validation (0 errors, 3 warnings).
The opt-in PostgreSQL backend scenario passed against a disposable PostgreSQL
18.6 container, using a fresh test-owned schema; that container and its test data
were removed afterward. The one new normally ignored test is the explicitly
run PostgreSQL scenario; the other eight server ignores predate this PR.

Independent review found no remaining correctness blockers. This evidence covers
phase 3a only, not the deferred interceptor/credential/streaming integrations.

[github-actions]

🌿 Preview your docs: https://nvidia-preview-pr-3321.docs.buildwithfern.com/openshell

[johntmyers]

gator-agent

PR Review Status

Platform administrators who reuse a request UUID for the same workspace mutation in two different workspaces will have the second valid request rejected, because workspace create and delete currently collapse every target into one global admission scope.

Action required: include the requested workspace name in the CreateWorkspace and DeleteWorkspace admission scope and add cross-workspace reuse regressions.

Blocking findings:

• GATOR-5dc3c729-01: CreateWorkspace and DeleteWorkspace omit the requested workspace from their admission keys.

Carried findings:

• None

Non-blocking suggestions:

• None

Gator metadata

• Validation: Project-valid phase 3a of accepted issue #3051.
• Docs: Fern API error documentation and gateway architecture documentation are updated, but the implementation currently violates the published workspace-scoping contract.
• Checks: DCO, vouch, Trivy, and docs preview succeeded; required Branch Checks and Helm Lint have not completed.
• E2E: Runtime gateway behavior requires test:e2e after review feedback is resolved.
• Head SHA: 5dc3c72940b13ca0da55a0b62ec792d71e50cf17
• Base SHA: f24eb09eb40482661fbf3f8bd6c74b7f44b16407
• Merge base SHA: f24eb09eb40482661fbf3f8bd6c74b7f44b16407
• Patch ID: 4f2cea229750ecc1a28fc3f68def569f3cab5c89
• Gator payload: 9
• Review mode: initial
• Previous reviewed SHA: none
• Review budget exhausted: no
• Maintainer decision required: no
• Next state: gator:in-review

[johntmyers]

gator-agent

PR Review Status

Thanks @johntmyers. I treated your approval and merge of stacked PR #3323 as trusted authorization for its added phase 3b scope and did not duplicate that review. On the resulting head, the earlier phase 3a workspace-scoping blocker is still present.

Action required: @mrunalp, include the requested workspace name in both CreateWorkspace and DeleteWorkspace admission scopes, keep workspace_id unset, and add the requested cross-workspace UUID-reuse regressions.

Blocking findings:

• No new blocking findings

Carried findings:

• GATOR-5dc3c729-01: CreateWorkspace and DeleteWorkspace still omit req.name from their admission keys, so the same UUID collides across different workspace targets.

Gator metadata

• Validation: Project-valid phases of accepted issue #3051; maintainer-approved stacked PR #3323 is incorporated.
• Docs: Existing architecture and Fern API-error documentation cover the replay contract, but the implementation still violates its workspace-scoping guarantee.
• Checks: Current-head Branch Checks and Helm Lint are pending; other visible required gates are green.
• E2E: test:e2e remains required after review feedback is resolved; dispatch is deferred while the blocker remains.
• Head SHA: 59a60779d0b83c5f92fec2f5d2a853651384b2a4
• Base SHA: f24eb09eb40482661fbf3f8bd6c74b7f44b16407
• Merge base SHA: f24eb09eb40482661fbf3f8bd6c74b7f44b16407
• Patch ID: cd57cb55aad078ea3529c6953c81cd0cac26145b
• Gator payload: 9
• Review mode: follow_up
• Previous reviewed SHA: 5dc3c72940b13ca0da55a0b62ec792d71e50cf17
• Review budget exhausted: no
• Maintainer decision required: no
• Next state: gator:in-review

[mrunalp]

🏗️ build-from-issue-agent

E2E Test Attestation

Local E2E passed for the source tree committed as 213285668c07989d35f840bb710c99472e1de89b. Verification preceded the commit, so the freshly built binaries' version text retained parent SHA 59a60779d; they included the tested working-tree changes. The commit hook passed and left no unstaged implementation changes.

Gateway mode: isolated Podman-backed gateway with freshly built gateway, CLI, and supervisor. Docker's socket was absent; the Docker harness stopped before executing tests, and the equivalent Podman lane was used. The harness stopped its gateway and cleaned up its temporary resources after success.

mise run --shell 'env RUSTC_WRAPPER= bash -c -o errexit -o pipefail' e2e:conformance:build
OPENSHELL_AUDITABLE=1 RUSTC_WRAPPER= \
OPENSHELL_CONFORMANCE_BIN=/home/mrunalp/code/OpenShell/target/debug/openshell-conformance \
OPENSHELL_E2E_PODMAN_TEST=workspace_lifecycle \
e2e/with-podman-gateway.sh bash -lc \
  'e2e/rust/e2e-podman.sh __openshell_run_podman_e2e && uv run pytest -o "python_files=test_*.py *_test.py" -v e2e/python/test_workspace_api.py'

Results: conformance smoke passed; Rust 2 passed, 0 failed, 0 ignored; Python 7 passed, no skips.

Tests executed:

• Conformance smoke: status/authentication, create, get-ready, list-visible, exec, delete, list-empty.
• workspace_lifecycle::workspace_full_crud_lifecycle.
• workspace_lifecycle::workspace_terminating_rejects_creates.
• test_workspace_api.py::test_workspace_crud.
• test_workspace_api.py::test_workspace_create_with_labels.
• test_workspace_api.py::test_workspace_list_includes_created.
• test_workspace_api.py::test_workspace_delete_nonexistent_raises_not_found.
• test_workspace_api.py::test_workspace_get_nonexistent_raises_not_found.
• test_workspace_api.py::test_workspace_request_ids_are_scoped_to_each_target.
• test_workspace_api.py::test_workspace_request_id_replays_without_deleting_replacement.

Full local CI, pre-commit checks, and docs validation also passed. This follow-up did not rerun the opt-in PostgreSQL integration test or the full deployment E2E suite.

[johntmyers]

gator-agent

PR Review Status

Thanks @mrunalp. I checked the workspace-scoping update and its server and live-gateway regressions: both CreateWorkspace and DeleteWorkspace now include the requested workspace name in the admission scope while leaving workspace_id unset, so the carried blocker is resolved. The bounded follow-up review found no new blocking issues.

Blocking findings:

• No blocking findings remain

Carried findings:

• GATOR-5dc3c729-01: Resolved by the current head; the Gator-owned review thread has been closed.

Gator metadata

• Validation: Project-valid phases of accepted issue #3051, including maintainer-approved stacked scope.
• Docs: Architecture and Fern API-error documentation cover the current replay and identity-aware deletion contracts.
• Checks: Current-head Branch Checks and Helm Lint are pending.
• E2E: Runtime gateway behavior requires test:e2e; authorized dispatch follows this review.
• Head SHA: 213285668c07989d35f840bb710c99472e1de89b
• Base SHA: b1e41c59c06c3428022ac5beb4335ed658abd493
• Merge base SHA: b1e41c59c06c3428022ac5beb4335ed658abd493
• Patch ID: 8cf23f4ae001d219e941960fdaf1fee72d1d43ab
• Gator payload: 9
• Review mode: follow_up
• Previous reviewed SHA: 59a60779d0b83c5f92fec2f5d2a853651384b2a4
• Review budget exhausted: no
• Maintainer decision required: no
• Next state: gator:in-review

[johntmyers]

/ok to test 2132856

[github-actions]

Label test:e2e applied for 2132856. Open the existing run and click Re-run all jobs to execute with the label set. The run will execute the standard E2E suite after building the required gateway and supervisor images once. The matching required CI gate status on this PR will flip green automatically once the run finishes.

[mrunalp]

Restacked after #3113 onto updated #3317 (10c4441f907617c1f2a2b727181e27e60095aea3). New head: 5aba844baf2b31716a254d1f463b172229b50683.

Replay now reads #3113's V2 refresh-state format and uses deletion-time presence, retaining the provider identity and authorization-epoch checks. Added a regression proving that both configure and rotate replay reject a refresh tombstoned at the Unix epoch. Updated the architecture invariant, the two new upstream test request literals, generated Go bindings, and the public schema fingerprint. The reviewed workspace target-scoping commit remains patch-identical.

Pre-commit passed and the gateway library suite passed at the replay integration step (1,612 passed, 9 opt-in tests ignored). The final descendant also passed full Rust/gateway integration checks and focused live workspace/exec tests; see #3324's update. Fresh remote CI and human approval remain required for the new heads.

[johntmyers]

gator-agent

PR Review Status

Thanks @mrunalp. I checked the restack adaptations you described: the V2 refresh-state binding, deletion-time presence check, Unix-epoch tombstone regression, generated schema updates, and the patch-identical workspace target scoping. The critical-only review found no newly introduced Critical defect, and the earlier workspace-scoping finding remains resolved.

Blocking findings:

• No blocking findings remain

Carried findings:

• GATOR-5dc3c729-01: Resolved on the prior reviewed head and preserved by this restack.

Gator metadata

• Validation: Project-valid implementation of accepted issue #3051, including maintainer-approved stacked scope.
• Docs: Architecture and Fern API-error documentation cover the replay contract and migration behavior.
• Checks: Current-head Branch Checks, Helm Lint, and E2E are pending test dispatch; other visible gates are green.
• E2E: test:e2e is present; fresh current-head /ok to test dispatch follows this review.
• Head SHA: 5aba844baf2b31716a254d1f463b172229b50683
• Base SHA: 10c4441f907617c1f2a2b727181e27e60095aea3
• Merge base SHA: 10c4441f907617c1f2a2b727181e27e60095aea3
• Patch ID: 9dec245e7ce33912426b762cde040a33583f1187
• Gator payload: 9
• Review mode: critical_only
• Previous reviewed SHA: 2df9ce812e7596b7999aa52636ef7f14285cf493
• Review budget exhausted: yes
• Maintainer decision required: no
• Next state: gator:blocked
• Blocked reason: test_dispatch_required

[johntmyers]

/ok to test 5aba844

[johntmyers]

/ok to test 6a1c05c

[johntmyers]

gator-agent

Maintainer Approval Needed

Gator validation and PR monitoring are complete. The current head is a patch-equivalent rebase of the last reviewed patch, the earlier workspace-scoping blocker remains resolved, and no Gator review threads are open.

Review: No blocking findings remain; the review ledger suppresses a redundant review of this unchanged effective patch.

Human maintainer approval is now required.

Gator metadata

• Validation: Project-valid implementation of accepted issue feat(api)!: standardize mutation idempotency, results, and structured errors #3051, including maintainer-approved stacked scope.
• Docs: Architecture and Fern API-error documentation cover the replay contract and migration behavior.
• Checks: Current-head Branch Checks, Helm Lint, Trivy Changes, and required E2E gates are green.
• E2E: test:e2e is present and OpenShell / E2E passed on the current head.
• Head SHA: 6a1c05c13b1a8a358f0d369a67464789ee7c4108
• Patch ID: 9dec245e7ce33912426b762cde040a33583f1187
• Gator payload: 9
• Review mode: already_reviewed
• Next state: gator:approval-needed

[johntmyers]

gator-agent

Monitoring Complete

Monitoring is complete because this PR has merged.

Final status: Gator reached merge-ready before the PR was merged.

I removed the active gator:* label because there is nothing left for gator to monitor on this PR.

Gator metadata

• Head SHA: 6a1c05c13b1a8a358f0d369a67464789ee7c4108
• Gator payload: 9
• Final state: merged