refactor(persistence): adopt continuation-token pagination

[drew]

Summary

Replace fixed limit/offset pagination with AIP-158 continuation tokens across OpenShell's public paginated list APIs. The gateway uses stable keyset cursors, while the CLI and internal consumers preserve tokens and exhaust pages safely where a complete collection is required.

Related Issue

Closes #2802

Changes

• Add direct page_size, page_token, and next_page_token fields to eight list RPCs, with a default of 100, a maximum of 1,000, negative-size rejection, and page-size changes allowed between requests.
• Add a private versioned pagination proto and URL-safe token codec that binds tokens to their RPC and non-page-size request parameters without treating tokens as authorization.
• Add SQLite/Postgres keyset page queries and exhaustive record/message helpers; remove fixed-page truncation and offset-shift behavior from affected internal scans.
• Preserve continuation metadata in CLI JSON/YAML response envelopes while keeping table output human-friendly.
• Make TUI full-collection and draft-count refreshes cancellable and non-overlapping, discard stale generation/context results, and bound badge RPC concurrency.
• Validate provider composition one page at a time instead of materializing every sandbox.
• Add SQLite/Postgres indexes for workspace-scoped and platform-wide keyset traversal, and return a client error instead of panicking on an invalid service-list combination.
• Add multi-page, malformed-token, token-misuse, and token-version coverage, and update conformance, contributor guidance, examples, and architecture/user documentation.

The curated SDK pager APIs are intentionally isolated in stacked PR #3256.

Testing

• mise run pre-commit passes
• OPENSHELL_SYSTEM_GATEWAY_DIR=/tmp/openshell-system-config-empty-2802-followup mise run test passes, including 1,450 server tests
• mise run test:python passes (151 tests)
• mise run sdk:ts:ci passes (104 tests)
• mise exec -- go test ./openshell/v1 ./openshell/v1/fake ./openshell/v1/internal/converter ./openshell/v1/types passes
• cargo test -p openshell-conformance passes
• Unit tests added/updated
• E2E callers updated (full gateway E2E not run)

mise run ci reaches the Go test lane after all earlier checks and the complete Rust suite pass, then fails in existing gateway-discovery tests because this host has /etc/openshell/gateways/default; those tests expect the system gateway directory to be empty. The Go format, lint, build, proto, and docs checks pass, as do the affected Go SDK packages listed above.

Checklist

• Follows Conventional Commits
• Commits are signed off (DCO)
• Architecture docs updated

[gmenher]

@drew I ran the targeted tests and did manual CLI validation and everything works as described. One small question: would it be useful to add multi-page tests for ListProviderProfiles and ListSandboxPolicies, since they use custom cursor ordering? Anyways it LGTM.

[github-actions]

🌿 Preview your docs: https://nvidia-preview-pr-3249.docs.buildwithfern.com/openshell

[johntmyers]

gator-agent

PR Review Status

This pagination refactor is project-valid through maintainer-authored issue #2802, but one published Go SDK example no longer compiles after the option-field migration.

Action required: @drew, update the remaining Limit field in the all-workspaces sandbox-template example to PageSize.

Blocking findings:

• GATOR-bd7da2eb-01: the published ListAll example initializes a removed Go field.

Carried findings:

• None

Gator metadata

• Validation: Maintainer-authored PR implements the linked pagination audit in #2802.
• Docs: Updated broadly, with one materially stale Go SDK example identified by this review.
• Checks: Current-head Branch Checks are still in progress; DCO, Trivy, and applicable completed gates are passing.
• E2E: test:e2e is required for the gateway and sandbox-list behavior changes, but dispatch is deferred until blocking review feedback is resolved.
• Head SHA: bd7da2eba89328f9b113dd57732ec4608ee5a3d7
• Base SHA: 3c0f58872ea956816abf82cdf50fa94601329cc5
• Merge base SHA: 3c0f58872ea956816abf82cdf50fa94601329cc5
• Patch ID: d9ebc606cdbecbad1738259c7b92a6960fd080be
• Gator payload: 9
• Review mode: initial
• Previous reviewed SHA: none
• Review budget exhausted: no
• Maintainer decision required: no
• Next state: gator:in-review

[github-actions]

Label test:e2e applied for fefe01e. Open the existing run and click Re-run all jobs to execute with the label set. The run will execute the standard E2E suite after building the required gateway and supervisor images once. The matching required CI gate status on this PR will flip green automatically once the run finishes.

[johntmyers]

gator-agent

Re-check After Author Update

Thanks @drew. I checked your current-head change from the removed Limit field to PageSize in the published Go SDK sandbox-template example, and the prior GATOR-bd7da2eb-01 obligation is resolved. The bounded follow-up review found no new blocking issues.

What I checked: the author-only delta after the rebase, the durable review-feedback ledger, and the corrected ListAll example at sdk/go/docs/src/api/sandbox-templates.md.

Disposition: resolved; no blocking items remain. The required test:e2e label is applied, and the current-head Branch E2E workflow rerun is queued.

Blocking findings:

• None

Carried findings:

• None

Gator metadata

• Validation: Implements the linked maintainer-authored pagination audit in #2802.
• Docs: Updated, including the corrected Go SDK example.
• Checks: Branch Checks are still running; DCO, Trivy, Helm, and applicable completed gates are passing.
• E2E: test:e2e applied; Branch E2E run 34543505398 rerun queued for the current head.
• Head SHA: fefe01e07f5f167413939b4e24589a4309f8624b
• Base SHA: ce25acca5a57220ddc89795f946f3df666ec78bd
• Merge base SHA: ce25acca5a57220ddc89795f946f3df666ec78bd
• Patch ID: 4ea987b452fe3c6bbbb0bebf47f2ba3940b67499
• Gator payload: 9
• Review mode: follow_up
• Previous reviewed SHA: bd7da2eba89328f9b113dd57732ec4608ee5a3d7
• Review budget exhausted: no
• Maintainer decision required: no
• Next state: gator:watch-pipeline

[johntmyers]

gator-agent

Monitoring Complete

Monitoring is complete because this PR has merged.

Final status: The pagination follow-up review had no remaining blocking findings, maintainer approval was present, and the required E2E run completed with one failing core E2E job before the PR was merged.

I removed the active gator:* label because there is nothing left for gator to monitor on this PR.