Expose OpenAPI permission requirements through analyzer, API metadata, and CLI help

[sdairs]

Problem and outcome

ClickHouse Cloud now publishes the API-key permissions required by operations. Surface that information in the Rust HTTP library and in relevant CLI leaf-command help, with the OpenAPI analyzer detecting additions, removals, and changes so the published information stays current.

An agent running clickhousectl cloud service get --help should see the precise permission ID without credentials or a network request. CLI help should consume API-library metadata; it should not maintain another copy of permission strings.

This issue records a proposed design. Implement the analyzer/library layer first, followed by a separate CLI PR.

Verified upstream representation

Sources: live OpenAPI document, create-service documentation.

In the document inspected for this design:

• Permissions are strings in an operation's security[].basicAuth array, not prose to extract from the documentation page.
• instanceGet declares control-plane:service:view.
• instanceCreate declares control-plane:organization:create-service.
• postgresInstanceRestore and postgresInstanceCreateReadReplica each declare two permissions: control-plane:postgres-service:manage and control-plane:organization:create-service. Both are required.
• components.securitySchemes.basicAuth.x-permission-scopes explicitly documents the semantics: these strings are Cloud API-key permission IDs; all listed permissions are required; an empty list requires a valid key without additional named permissions; alternatives are not currently expressed.
• Root security is [{"basicAuth": []}]. Five operations omit operation-level security and inherit it: organizationGetList, organizationPrometheusGet, organizationPrometheusDiscoveryGet, instanceGetList, and instanceStateUpdate.
• The current live document has 157 operations, with operation-level security on 152. The checked-in snapshot at a65a9ca6 has it on 72. There are 80 changed operation-security declarations. Some service permissions already exist in the checked-in snapshot; the analyzer currently does not consume them.

Treat these as upstream-declared requirements. Do not infer permission IDs from HTTP verbs, command names, role names, or the CLI's existing read/write classification. For example, some GET operations require a manage-* permission.

Existing structure

• crates/clickhouse-cloud-api/src/client/services.rs: Client::instance_get(&self, organization_id: &str, service_id: &str) returns Result<ApiResponse<Service>, Error>. Other operations are similar typed async methods. Parameters and types are represented by Rust signatures and request/response models.
• crates/clickhouse-cloud-api/src/meta.rs: existing public beta/deprecation metadata provides a natural home for operation descriptors.
• crates/clickhouse-openapi-analyzer/src/openapi.rs: OperationInfo currently retains operation ID, method, path, summary, and pointer, but no security/permissions.
• rust_inventory.rs: reads public async client signatures and metadata from the meta.rs module tree. This inventory is private tooling, not a runtime reflection API.
• compare.rs, report.rs, and scripts/check-openapi-drift.py: already provide the comparison, stable JSON/text report, and GitHub issue pipeline.
• CLI domain files own command definitions and handlers. main.rs builds Cli::command() before parsing, so static metadata can enrich help before credentials or dispatch are involved.

Verification: a clean build of the analyzer at a65a9ca6 (report schema 8), run against a temporary copy of the snapshot with only live operation/root security and security-scheme definitions applied, reports zero findings. This isolates the permission blind spot from unrelated live-spec drift. No repository source was modified for this investigation.

Proposed API-library design

Add a small generated operation catalog beneath meta, reusing its existing module-tree support. An illustrative public interface:

use clickhouse_cloud_api::meta::operations;

let endpoint = &operations::INSTANCE_GET;
assert_eq!(endpoint.operation_id, "instanceGet");
assert_eq!(endpoint.rust_method, "instance_get");
assert_eq!(endpoint.required_permissions, &["control-plane:service:view"]);

// Existing execution API remains available.
let response = client.instance_get(org_id, service_id).await?;

The descriptor initially needs only operation_id, rust_method, HTTP method, path template, and required_permissions: &'static [&'static str]. The list means all of. Use exact upstream permission strings, allowing new permission IDs without extending a Rust permission enum. Keep metadata independent of Service, request bodies, and other wire models.

Generate one literal descriptor per OpenAPI operation, plus an iterable catalog referencing those descriptors. An optional lookup by exact operation ID should return Option<&'static OperationMetadata>: an unknown operation must not turn into an empty permission list. Keep the fields/API extensible rather than promising exhaustive struct construction by callers.

Generate committed Rust literals using the analyzer's OpenAPI inventory. Do not parse the OpenAPI document at application startup or include syn/analyzer dependencies in published runtime dependencies. Extend the Rust metadata inventory to read the literal descriptors and validate their association with actual client methods.

This supplies the requested “endpoint and its properties” access without changing how callers execute requests. Rich parameter/schema reflection can be added to the descriptor later if needed; it is not necessary for permission help, and Rust method items themselves do not provide that property interface.

Analyzer behavior

1. Resolve effective security per operation: explicit operation security overrides root security; missing operation security inherits root security. Keep source pointers/provenance for useful diagnostics.
2. Support the documented single basicAuth conjunction and normalize its permission IDs as a set for comparison and deterministic generation. Ignore ordering and duplicates.
3. An inherited or explicit basicAuth: [] means a valid API key with no additional named permissions. Missing/unresolved metadata, a missing catalog entry, malformed values, or unsupported security shapes must never silently become that state. Explicit anonymous security (security: [] / {} alternatives) is distinct from a valid-key requirement.
4. Keep the first implementation small: report an actionable unsupported-security finding if upstream introduces alternative requirement objects, another scheme, anonymous security, or another shape not represented by the descriptor. Do not flatten OR into AND. Supporting those shapes can extend the model deliberately later.
5. Compare target spec versus snapshot for effective permission additions/removals/replacements, including changes inherited from root security.
6. Compare target spec versus library metadata for missing/extra descriptors and permission mismatches. Validate operation ID, method-name association, HTTP method, and path if these are public descriptor fields. This catches stale library metadata even after somebody refreshes the snapshot.
7. Report before/after requirements and added/removed permission sets, operation identity, exact spec pointer, and Rust metadata location. Use typed finding kinds and stable report fields, rather than requiring Python to rediscover the meaning.
8. Bump the report schema version current at implementation time and update the Python renderer/validator and fixtures together. Python retains only fetching, rendering, and issue orchestration; permission parsing, normalization, and generation live in Rust.

Generate from a fully refreshed upstream snapshot rather than hand-editing it. The fetched live document also includes unrelated changes; coordinate those with ordinary drift remediation rather than claiming the refresh contains permissions alone.

CLI consumption and help

Maintain command-to-operation bindings beside each owning domain's clap definitions. Compose static help from these bindings and the library descriptors through a shared renderer. Use the same enriched command tree in the executable and help-structure tests; avoid a main-only decoration path that tests bypass.

For a simple leaf:

clickhousectl cloud service get --help

CONTEXT FOR AGENTS:
  Cloud API key permission required: control-plane:service:view

Append within the existing CONTEXT FOR AGENTS: block, creating the block if needed. Keep one header, the current eight-content-line budget, and existing leaf-specific facts. Parent group help and local commands do not gain endpoint-permission lines. Covered leaves with an empty declared list can state that a valid API key is required with no additional named permissions.

Commands may invoke more than one API operation. Their bindings should distinguish unconditional operations from conditional ones, then union/deduplicate permissions within each execution case:

• service get: INSTANCE_GET; resource-name resolution also invokes INSTANCE_GET_LIST when --name is supplied (currently no extra named permission).
• service delete --force: fetching/polling adds INSTANCE_GET; stopping invokes INSTANCE_STATE_UPDATE, in addition to deletion. Managed query-key cleanup can add API-key operations depending on local state.
• Name/email/source-name selectors, organization resolution, read-before-write paths, polling, and Query API auto-setup need the same review. Attach extra permissions to the option/condition that triggers the extra call; do not present the union of every optional path as universally required.
• Non-OpenAPI execution paths, such as SQL query privileges, need an explicit classification; do not invent Cloud API-key permission IDs for database authorization.

The CLI owns workflow mapping and conditions; the library owns endpoint permission strings. Keep the existing OAuth read-only/write-key gate separate: these descriptors document API-key permissions, not a new local authorization decision or a test of the user's actual access.

Delivery and acceptance

PR 1: analyzer and API library

• Inventory security inheritance and permission conjunctions; add typed drift findings and deterministic report rendering.
• Generate/export the small operation catalog; check descriptor coverage, identities, values, and stale entries against client methods and spec.
• Test inherited/overridden requirements, one/two/zero named permissions, add/remove/replace, root changes, reordering/duplicates, malformed/unsupported forms, and unknown lookup behavior.
• Updating the snapshot alone must not silence a library permission mismatch. Permission-only upstream changes must produce actionable drift.
• Keep parser/tooling dependencies outside the published runtime dependency graph; add a compile-tested consumer example.
• Update API-library README, analyzer/Python report tests, and both path classifiers for added files. Run required library/analyzer checks.

PR 2: CLI

• Bind relevant cloud leaf commands to library operations, auditing compound and conditional calls; explicitly classify unmapped/non-OpenAPI leaves.
• Render short API-key permission facts from library metadata in offline, credential-free help.
• Cover command-binding completeness, correct sets and conditional associations, preservation of existing context, and absence on parent/local help. Assert structure/metadata, not whole help screens or prose wording.
• Update root README and CLI help-policy guidance as needed; run all required CLI checks, including telemetry-disabled builds/linting.

This issue is about discoverability and keeping declared permission requirements accurate. Broader parameter reflection and permission-granting workflows can build on the catalog separately.

Implementation plan agreed with Al

• The previous Reject mixed service tag updates #980–Allow BYOC-only service-profile lookup in the library and CLI #992 stack and external PR fix(install): surface clickhouse --version stderr on failure #993 have merged. The permission commits were rebased cleanly onto main at df81e46cfec0bb58ea981d17cc6fd2de24442e7d, preserving both patches unchanged.
• GitHub stack #1001, registered with gh stack, has this order: main → Expose typed endpoint permissions and detect permission drift #999 (codex/998-permission-metadata) → Generate mandatory Cloud permission help from operation declarations #1000 (codex/998-permission-help).
• Merge Expose typed endpoint permissions and detect permission drift #999 first, then Generate mandatory Cloud permission help from operation declarations #1000. This permissions series is independent of the completed earlier stack.
• Every Cloud leaf must declare an operation contract, including an explicit reason for commands that use no Cloud OpenAPI operation. Domain-local typed operation arrays feed the same helper that derives and deduplicates help permissions. Missing, duplicate, stale, or unknown declarations must fail the shared command-tree validation and CI.
• Conditional operation groups describe name lookup, polling, setup, cleanup, and flag-dependent calls. An independent handler audit and focused call-path tests verify those declarations: complete help coverage by itself does not prove an arbitrary Rust call graph is complete.

Implementation and validation

• #999 — Expose typed endpoint permissions and detect permission drift: 157 generated HTTP-library operation descriptors; permission inheritance and set deltas; missing/stale/unsupported metadata findings; report schema 10 and issue rendering.
• #1000 — Generate mandatory Cloud permission help from operation declarations: 175 executable Cloud command declarations, mandatory shared help/parser validation, typed conditional flags, and deduplicated permission help.

Both PRs remain drafts. Their current heads after restacking are 5770830fea8b992a7329c95848ac00bcc8007694 (metadata) and 04063c49d88064a128087df7274d9838a71c52a3 (CLI). PR descriptions and base branches now reflect the registered stack.

Required local formatting, library/analyzer and CLI lint/test gates, telemetry-disabled checks, and all 96 Python tests passed. The declaration audit covers current handlers, including setup, cleanup, rollback and lookups. Focused real-binary tests compare advertised permissions against actual mock API calls for force deletion with query-key cleanup and query-endpoint read-before-write. Live Cloud CI has been requested on both PRs because classifier edits require it.

After the rebase, formatting and 13 focused tests pass, including structural declaration coverage, offline/multiple-permission help, compound request coverage, and the pagination behavior newly inherited from main. The earlier live runs failed on a query-key propagation assertion (#999) and a ClickPipe timeout with server errors (#1000); new exact-head live checks have been requested for both rebased PRs.