refactor(persistence): adopt continuation-token pagination

.agents/skills/create-github-pr/SKILL.md

@@ -187,17 +187,17 @@ gh pr create \
   --body "$(cat <<'EOF'
 ## Summary
 
-Add `--limit` and `--offset` flags to `openshell sandbox list` for pagination.
+Add `--page-size` and `--page-token` flags to `openshell sandbox list` for continuation-token pagination.
 
 ## Related Issue
 
 Closes #456
 
 ## Changes
 
-- Added `offset` and `limit` query parameters to the sandbox list API call
-- Default limit is 20, max is 100
-- Response includes `total_count` field
+- Added `page_size` and `page_token` fields to the sandbox list API call
+- Default page size is 100, max is 1,000
+- Structured responses include `next_page_token`
 
 ## Testing
 

.agents/skills/tui-development/SKILL.md

@@ -170,13 +170,15 @@ Phase 1: GetSandboxLogs  →  500 initial lines  →  send via Event::LogLines
 Phase 2: WatchSandbox(follow_logs: true)  →  live tail  →  send via Event::LogLines
 ```
 
-**Sandboxes**: Fetched via `ListSandboxes` on a 2-second tick, scoped to the current workspace (or all workspaces).
+**Sandboxes**: Fetched via `ListSandboxes` in a background collection-refresh task scheduled from the 2-second tick, scoped to the current workspace (or all workspaces). Follow `next_page_token` until empty so the dashboard reflects the complete collection.
 
-**Providers**: Fetched via `ListProviders` on each tick. Provider profiles are fetched per-workspace via `ListProviderProfiles` and cached in a `ProviderProfileCache` keyed by `(workspace, profile_id)`.
+**Providers**: Fetched via `ListProviders` in the background collection-refresh task. Provider profiles are fetched per-workspace via `ListProviderProfiles` and cached in a `ProviderProfileCache` keyed by `(workspace, profile_id)`. Follow each list RPC's `next_page_token` until empty.
 
 **Settings**: Global settings are fetched via `GetGatewayConfig` on each tick. Sandbox settings are fetched alongside the sandbox policy via `GetSandboxConfig` and refreshed on each tick when viewing a sandbox.
 
-**Workspaces**: The workspace list is fetched via `ListWorkspaces` on each tick.
+**Workspaces**: The workspace list is fetched via `ListWorkspaces` in the background collection-refresh task, following `next_page_token` until empty.
+
+Only one collection-refresh task may run at a time. Workspace and gateway changes abort the active task, and refresh results carry their gateway/workspace context so stale results are discarded.
 
 ### Never block the event loop
 
@@ -411,9 +413,9 @@ All actions are accessible via keyboard shortcuts displayed in the nav bar. The
 | File | Purpose |
 | --- | --- |
 | `crates/openshell-tui/Cargo.toml` | Crate manifest — dependencies on `openshell-core`, `openshell-bootstrap`, `ratatui`, `crossterm`, `tonic`, `tokio` |
-| `crates/openshell-tui/src/lib.rs` | Entry point. Event loop, gRPC calls (`refresh_data`, `refresh_providers`, `refresh_global_settings`, `refresh_workspaces`, `refresh_sandboxes`, `spawn_log_stream`, `handle_sandbox_delete`), gateway switching, mTLS channel building, provider CRUD spawners, settings CRUD spawners, draft approval spawners |
+| `crates/openshell-tui/src/lib.rs` | Entry point. Event loop, background collection refresh (`spawn_list_refresh`), gRPC calls (`refresh_global_settings`, `spawn_log_stream`, `handle_sandbox_delete`), gateway switching, mTLS channel building, provider CRUD spawners, settings CRUD spawners, draft approval spawners |
 | `crates/openshell-tui/src/app.rs` | `App` state struct, `Screen`/`Focus`/`InputMode`/`LogSourceFilter`/`MiddlePaneTab`/`SandboxPolicyTab` enums, `LogLine`/`GatewayEntry`/`GlobalSettingEntry`/`SandboxSettingEntry`/`ProviderListEntry`/`ProviderDetailView` structs, create sandbox/provider form state, all key handling logic |
-| `crates/openshell-tui/src/event.rs` | `Event` enum (`Key`, `Mouse`, `Tick`, `Redraw`, `Resize`, `LogLines`, `CreateResult`, `ProviderCreateResult`, `ProviderDetailFetched`, `ProviderUpdateResult`, `ProviderDeleteResult`, `DraftActionResult`, `GlobalSettingsFetched`, `GlobalSettingSetResult`, `GlobalSettingDeleteResult`, `SandboxSettingSetResult`, `SandboxSettingDeleteResult`, `ForwardWarnings`), `EventHandler` with mpsc channels and crossterm polling |
+| `crates/openshell-tui/src/event.rs` | `Event` enum (`Key`, `Mouse`, `Tick`, `Redraw`, `Resize`, `LogLines`, `ListRefreshCompleted`, `CreateResult`, `ProviderCreateResult`, `ProviderDetailFetched`, `ProviderUpdateResult`, `ProviderDeleteResult`, `DraftActionResult`, `GlobalSettingsFetched`, `GlobalSettingSetResult`, `GlobalSettingDeleteResult`, `SandboxSettingSetResult`, `SandboxSettingDeleteResult`, `ForwardWarnings`), `EventHandler` with mpsc channels and crossterm polling |
 | `crates/openshell-tui/src/theme.rs` | `colors` module (NVIDIA_GREEN, EVERGLADE, BG, FG) and `styles` module (all `Style` constants) |
 | `crates/openshell-tui/src/clipboard.rs` | Clipboard copy support for log lines |
 | `crates/openshell-tui/src/ui/mod.rs` | Top-level `draw()` dispatcher, `draw_title_bar` (with workspace display), `draw_nav_bar`, `draw_command_bar`, screen routing, shared setting-edit overlay, modal helpers |
@@ -502,12 +504,15 @@ use openshell_core::proto::{
   `Some(all_workspaces_selector())`; do not use that marker on other requests.
 - `GetSandboxLogsRequest` fields: `sandbox_id`, `lines` (u32), `since_ms` (i64),
   `sources` (Vec<String>), `min_level` (String), `workspace_scope`.
-- `ListSandboxesRequest` fields: `limit` (u32), `offset` (u32),
+- `ListSandboxesRequest` fields: `page_size` (i32), `page_token` (String),
   `label_selector` (String), `workspace_scope`.
-- `ListProvidersRequest` fields: `limit` (u32), `offset` (u32),
+- `ListProvidersRequest` fields: `page_size` (i32), `page_token` (String),
   `workspace_scope`.
-- `ListWorkspacesRequest` fields: `limit` (u32), `offset` (u32),
+- `ListWorkspacesRequest` fields: `page_size` (i32), `page_token` (String),
   `label_selector` (String).
+- Paginated list responses return `next_page_token`. Continue with the same
+  request parameters and that token until it is empty; changing filters or
+  scope invalidates the token.
 - `UpdateConfigRequest` fields include `name` (String, sandbox name or empty for
   global), `setting_key`, `setting_value`, `delete_setting` (bool), `global`
   (bool), and `workspace_scope`. Sandbox-scoped updates require a named selector;
@@ -543,21 +548,21 @@ The connect timeout for gateway switching is 10 seconds with HTTP/2 keepalive at
 4. On success:
    - `app.client` is replaced with a new intercepted client
    - `reset_sandbox_state()` clears all sandbox/log/draft/policy data
-   - `refresh_data()` runs the full capability refresh sequence: `refresh_health` → `refresh_global_settings` → `refresh_workspaces` → `refresh_providers` → `refresh_sandboxes`
+   - health and global settings are refreshed, then `spawn_list_refresh()` starts the cancellable workspace/provider/sandbox refresh task
 5. On failure: `status_text` shows the error
 
 ### Initial startup lifecycle
 
 On launch, before the event loop starts:
 
 1. `refresh_gateway_list()` — discover gateways from disk
-2. `refresh_data()` — full refresh (health, global settings, workspaces, providers, sandboxes)
+2. Refresh health and global settings, then start `spawn_list_refresh()` for workspaces, providers, and sandboxes
 
 ### Workspace switching lifecycle
 
 1. User presses `[w]` on the providers or sandboxes panel → `cycle_workspace()` advances through discovered workspace names, then "all"
 2. `pending_workspace_refresh = true` is set, cursor indices are reset
-3. Event loop calls `refresh_providers()` and `refresh_sandboxes()` with the new workspace scope
+3. Event loop cancels any in-flight collection refresh and starts `spawn_list_refresh()` with the new workspace scope
 
 ### Settings CRUD lifecycle (global and sandbox)
 

architecture/gateway.md

@@ -329,7 +329,7 @@ Compute-driver, credential-driver, gateway-interceptor, and
 supervisor-middleware services are compiled contracts for internal extension
 boundaries, not public gateway RPCs. The current public inventory has 74
 methods, 278 messages, and 12 enums
-(`c95ae90962c10fb28747db2b645adf4044562a3d208d84dfe4699d677e4364ee`).
+(`0f14943574349d02bdc61076c8c5a59a98b627325564ef1a6d21d7941825dc46`).
 
 Storage-only messages live in the private, versioned
 `openshell.storage.v1` package under `crates/openshell-server/proto`. The server
@@ -529,11 +529,33 @@ modes:
   `UpdateProvider`, `UpdateProviderProfiles`, and `UpdateConfig` (policy
   backfill and sandbox annotation updates).
 
-**Lists.** The `list_messages` and `list_messages_with_selector` helpers decode
-protobuf payloads from list results and hydrate `resource_version` from the
-authoritative database column into each decoded message, mirroring the
-`get_message` pattern. This ensures list responses carry correct versions
-without requiring callers to manually hydrate each record.
+**Lists.** Public list RPCs follow AIP-158: requests carry direct `page_size`
+and `page_token` fields, and responses carry `next_page_token`. The gateway
+clamps page sizes to 1,000 and returns opaque base64url continuation tokens.
+Tokens bind the RPC and every request parameter except `page_size`, contain no
+authorization grant, and use immutable keyset cursors rather than database
+offsets. Each page repeats normal authentication and authorization. Pagination
+is weakly consistent under concurrent writes and deletes; it does not provide a
+historical snapshot.
+
+The token wire format is a private shared protobuf used only by the gateway.
+Public request and response messages repeat the standard AIP fields directly
+instead of wrapping them in a shared pagination message.
+
+The CLI returns paginated JSON and YAML as response-shaped envelopes containing
+the resource collection and `next_page_token`; table output reports a non-empty
+token on stderr. The TUI traverses complete workspace, provider, profile, and
+sandbox collections in one cancellable background refresh task, never overlaps
+periodic list refreshes, and discards results after a gateway or workspace
+change.
+
+Persistence distinguishes one-page operations from exhaustive scans.
+`list_object_page` and `list_message_page` return one keyset page and its next
+cursor. `collect_records` and `collect_messages` exhaust those pages, fail on
+database or protobuf decode errors, and hydrate `resource_version` from the
+authoritative database column. Internal callers that require every matching
+record use the exhaustive helpers; bounded lookups continue to use page-level
+methods.
 
 **Deletes.** Delete operations are not yet CAS-protected -- the delete request
 protos do not carry `expected_resource_version`. A `delete_if` primitive exists

crates/openshell-cli/src/commands/provider.rs

@@ -298,18 +298,18 @@ pub async fn ensure_required_providers(
     let mut known_names: HashSet<String> = HashSet::new();
     let mut type_to_name: HashMap<String, String> = HashMap::new();
     {
-        let mut offset = 0_u32;
-        let limit = 100_u32;
+        let mut page_token = String::new();
         loop {
             let response = client
                 .list_providers(ListProvidersRequest {
-                    limit,
-                    offset,
+                    page_size: 100,
+                    page_token,
                     workspace_scope: Some(openshell_core::proto::workspace_selector(workspace)),
                 })
                 .await
                 .into_diagnostic()?;
-            let providers = response.into_inner().providers;
+            let response = response.into_inner();
+            let providers = response.providers;
             for provider in &providers {
                 known_names.insert(provider.object_name().to_string());
                 if !provider.r#type.is_empty() {
@@ -319,10 +319,10 @@ pub async fn ensure_required_providers(
                         .or_insert_with(|| provider.object_name().to_string());
                 }
             }
-            if providers.len() < limit as usize {
+            if response.next_page_token.is_empty() {
                 break;
             }
-            offset = offset.saturating_add(limit);
+            page_token = response.next_page_token;
         }
     }
 
@@ -1325,8 +1325,8 @@ fn provider_credential_keys(provider: &Provider) -> Vec<String> {
 #[allow(clippy::too_many_arguments)]
 pub async fn provider_list(
     server: &str,
-    limit: u32,
-    offset: u32,
+    page_size: i32,
+    page_token: &str,
     names_only: bool,
     output: &str,
     workspace: &str,
@@ -1336,8 +1336,8 @@ pub async fn provider_list(
     let mut client = grpc_client(server, tls).await?;
     let response = client
         .list_providers(ListProvidersRequest {
-            limit,
-            offset,
+            page_size,
+            page_token: page_token.to_string(),
             workspace_scope: Some(if all_workspaces {
                 openshell_core::proto::all_workspaces_selector()
             } else {
@@ -1346,12 +1346,21 @@ pub async fn provider_list(
         })
         .await
         .into_diagnostic()?;
-    let providers = response.into_inner().providers;
+    let response = response.into_inner();
+    let next_page_token = response.next_page_token;
+    let providers = response.providers;
 
     // Handle structured output formats (json, yaml)
-    if crate::output::print_output_collection(output, &providers, provider_to_json)? {
+    if crate::output::print_paginated_output_collection(
+        output,
+        "providers",
+        &providers,
+        &next_page_token,
+        provider_to_json,
+    )? {
         return Ok(());
     }
+    crate::output::print_next_page_token(&next_page_token);
 
     if providers.is_empty() {
         if !names_only {
@@ -1444,15 +1453,24 @@ pub async fn provider_list_profiles(
     tls: &TlsOptions,
 ) -> Result<()> {
     let mut client = grpc_client(server, tls).await?;
-    let response = client
-        .list_provider_profiles(ListProviderProfilesRequest {
-            limit: 100,
-            offset: 0,
-            workspace: workspace.to_string(),
-        })
-        .await
-        .into_diagnostic()?;
-    let mut profiles = response.into_inner().profiles;
+    let mut page_token = String::new();
+    let mut profiles = Vec::new();
+    loop {
+        let response = client
+            .list_provider_profiles(ListProviderProfilesRequest {
+                page_size: 100,
+                page_token,
+                workspace: workspace.to_string(),
+            })
+            .await
+            .into_diagnostic()?
+            .into_inner();
+        profiles.extend(response.profiles);
+        if response.next_page_token.is_empty() {
+            break;
+        }
+        page_token = response.next_page_token;
+    }
     profiles.sort_by(|left, right| {
         left.category
             .cmp(&right.category)

crates/openshell-cli/src/completers.rs

@@ -36,8 +36,8 @@ pub fn complete_sandbox_names(_prefix: &OsStr) -> Vec<CompletionCandidate> {
         let mut client = completion_grpc_client(&endpoint, &gateway_name).await?;
         let response = client
             .list_sandboxes(ListSandboxesRequest {
-                limit: 200,
-                offset: 0,
+                page_size: 200,
+                page_token: String::new(),
                 label_selector: String::new(),
                 workspace_scope: Some(openshell_core::proto::workspace_selector(
                     workspace_from_args(),
@@ -63,8 +63,8 @@ pub fn complete_provider_names(_prefix: &OsStr) -> Vec<CompletionCandidate> {
         let mut client = completion_grpc_client(&endpoint, &gateway_name).await?;
         let response = client
             .list_providers(ListProvidersRequest {
-                limit: 200,
-                offset: 0,
+                page_size: 200,
+                page_token: String::new(),
                 workspace_scope: Some(openshell_core::proto::workspace_selector(
                     workspace_from_args(),
                 )),
@@ -89,8 +89,8 @@ pub fn complete_workspace_names(_prefix: &OsStr) -> Vec<CompletionCandidate> {
         let mut client = completion_grpc_client(&endpoint, &gateway_name).await?;
         let response = client
             .list_workspaces(ListWorkspacesRequest {
-                limit: 200,
-                offset: 0,
+                page_size: 200,
+                page_token: String::new(),
                 label_selector: String::new(),
             })
             .await