[jhjaggars]

🏗️ build-from-issue-agent

Summary

Adds an optional service_hostname_format = "hashed" gateway setting that disambiguates exposed-service URLs when multiple gateways share a Kubernetes cluster, ingress, and wildcard certificate. In hashed mode, service hostnames become a deterministic 16-character lowercase base32 SHA-256 label of (installation name, workspace, sandbox, service) under the same base domain — exactly one label deep, so a single shared *.domain wildcard certificate keeps working, and each gateway's URLs are globally unique. The discriminator is the existing [openshell.gateway].name installation identity (operator- or platform-assigned, defaulting to the Helm release name), so no second per-gateway name is introduced; the platform deploying shared-domain gateways is responsible for keeping those names unique, and gateway_jwt.gateway_id stays separate with its security and Kubernetes-ownership semantics. The default (readable) preserves the existing hostname format byte-for-byte.

Related Issue

Closes #4840

Changes

• crates/openshell-core/src/config.rs: ServiceHostnameFormat enum (readable default / hashed), ServiceRoutingConfig.service_hostname_format field, with_service_hostname_format builder, Config::service_routing_discriminator() (the installation name, only in hashed mode), DNS-1123-label validation of the installation name when hashed, and config tests.
• crates/openshell-server/src/config_file.rs: [openshell.gateway] service_hostname_format TOML parsing/serialization.
• crates/openshell-server/src/cli.rs: --service-hostname-format / OPENSHELL_SERVICE_HOSTNAME_FORMAT with existing CLI > env > TOML precedence, startup validation of the installation name, and precedence tests.
• crates/openshell-server/src/service_routing.rs: SHA-256/NUL-delimited tuple hashing with a hand-rolled fixed-size lowercase base32 encoder (no new dependency), hashed endpoint_host output, ServiceRouteIndex (register/reserve/remove/lookup/rebuild), async state-aware resolve_service_host supporting both hashed and legacy hosts, and routing tests with fixed vectors.
• crates/openshell-server/src/lib.rs: route index in ServerState, hydrated from persisted ServiceEndpoint records at startup before listeners bind; ambiguous persisted hashes fail startup.
• crates/openshell-server/src/http.rs: HTTP middleware and loopback plaintext router use the async state-aware service-host classification.
• crates/openshell-server/src/grpc/service.rs: hash route reserved before endpoint persistence (AlreadyExists on collision, no write), rolled back on persistence failure, removed after successful delete; collision test.
• crates/openshell-server/src/compute/mod.rs: sandbox cleanup removes route index entries for deleted endpoints.
• deploy/helm/openshell/: server.serviceRoutingGatewayName value rendered into gateway TOML via _toml.tpl (empty stays unset; explicit gatewayConfig takes precedence), chart tests, regenerated README.
• Docs: docs/how-it-works/gateways/configuration.mdx (field reference), docs/how-it-works/gateways/overview.mdx (hostname grammar — also fixes the pre-existing omission of the workspace segment), docs/how-it-works/sandboxes/overview.mdx, docs/kubernetes/ingress.mdx (operator-managed exact-host ingress guidance for multi-gateway shared-cert deployments), docs/kubernetes/setup.mdx.

Deviations from Plan

None functionally. The route reservation API internally tracks whether the current call inserted the route so persistence-failure rollback cannot remove a pre-existing idempotent reservation from a concurrent request (an implementation detail within the plan's intent).

Testing

• mise run pre-commit passes (format, license headers, Helm lint) — with one pre-existing, unrelated failure: rust:lockfiles:check rejects examples/supervisor-middleware-content-guard/Cargo.lock because clap 4.6.7 was yanked from crates.io; that lockfile is untouched by this PR and fails on unmodified main as well.
• Unit tests added/updated
• E2E tests: N/A — no changes to sandbox/supervisor protocols, persistence schemas, or service relay transport; no files under e2e/ were modified.

Tests added:

• Unit:
  • openshell-core config: format defaults, discriminator derivation and validation (DNS-label installation name required when hashed), with_server_sans preserves the format.
  • openshell-server service_routing: fixed hash vectors (enh47uqyj7bpyttz, eiud2z5apv3yeucg), hashed URL construction, index idempotency/removal/collision/rebuild-from-store, configured-hash host resolution for Host header and HTTP/2 authority, legacy-host acceptance when configured, unknown-hash fallthrough.
  • openshell-server grpc::service: expose_service_rejects_route_hash_collision_before_persistence.
  • openshell-server config_file/cli: TOML parse/serialize, file-populated value, CLI/env override precedence, effective-config reach.
  • openshell-cli: service_url_for_gateway preserves the hashed label while normalizing the port.
• Integration: index rehydration from a shared store (restart path) and cross-state miss-triggered refresh, exercised in openshell-server service_routing tests.
• E2E: N/A (see above).

Verification runs (local):

• cargo fmt --check, git diff --check — pass
• cargo clippy --workspace --all-targets -- -D warnings — pass (includes a small fix for main-landed sandbox_runtime.rs doc lints that predate the last Branch Checks run on main)
• cargo test -p openshell-core — 607 unit + 1 doctest pass
• cargo test -p openshell-server — 2388 pass (21 Postgres-dependent ignored as expected), including new tests for the miss-triggered rebuild debounce and hashed-host cross-origin OCSF emission
• cargo test -p openshell-cli service_url — pass
• mise run helm:test — 361 chart tests pass

Checklist

• Follows Conventional Commits
• Commits are signed off (DCO)

Documentation updated:

• docs/how-it-works/gateways/configuration.mdx: new TOML field reference (default, validation, precedence, hash URL example, distinction from gateway_jwt.gateway_id)
• docs/how-it-works/gateways/overview.mdx: corrected hostname grammar (workspace segment) + legacy vs hashed formats
• docs/how-it-works/sandboxes/overview.mdx: consume printed URLs, not constructed hostnames
• docs/kubernetes/ingress.mdx: exact-host ingress guidance for multi-gateway shared-wildcard-cert deployments
• docs/kubernetes/setup.mdx: chart-values table entry
• deploy/helm/openshell/README.md: regenerated chart values docs

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[TaylorMutch]

/ok to test 3f1fd80

[jhjaggars]

🏗️ build-from-issue-agent

Cluster Verification Report

Verified end-to-end on a real multi-tenant OpenShift (ROSA) cluster with two gateways sharing one base domain and wildcard certificate — the exact ambiguous scenario from #4840.

Environment

Item	Value
Cluster	ROSA, mixed-arch (arm64 + amd64), OpenShift Routes as the shared ingress
Gateways	openshell-a (name gateway-a) and openshell-b (name gateway-b), separate namespaces, images built from PR head 3f1fd809bda8
Shared base domain	operator-owned zone (shown here as apps.example.com) via server_sans = ["*.apps.example.com"] on both
Wildcard certificate	Single Let's Encrypt *.apps.example.com (cert-manager DNS-01), presented by both gateways through passthrough routes
Auth	OIDC via existing Keycloak realm (device grant)
Sandboxes	Identical workspace=default, sandbox=demo, service=web (HTTP servers returning distinct content) exposed on both gateways

Results

#	Test	Result
T1	Distinct hashed hostnames for identical workspace/sandbox/service across gateways	✅ pyldrduzj554prit vs 2scisywrf3egxnmz; both match independently computed SHA-256/NUL-joined/10-byte/lowercase-base32 values
T2	Correct routing through shared ingress — the reported bug scenario	✅ hash-a → served-by-gateway-a, hash-b → served-by-gateway-b; no cross-talk; full TLS verification (LE wildcard presented, ssl_verify_result=0)
T3	Legacy readable hostnames (default--demo--web.<base-domain>) still accepted on a hash-configured gateway	✅ routed correctly (transition compatibility)
T4	Gateway pod restart → route index rehydration from persisted endpoints	✅ hashed URL routes correctly after restart
T5	URL regeneration	✅ service list regenerates identical hashed URLs on both gateways
T6	Unknown hash-shaped hostname falls through to normal routing	✅ HTTP 404, not proxied to any sandbox

Operator-managed ingress was used per the PR's documented model: exact-host passthrough Routes created per emitted hashed hostname, each targeting its own gateway's Service.

Also exercised during setup: Helm rendering of server.serviceRoutingGatewayName into gateway.toml (service_routing_gateway_name = "gateway-a"), image multi-arch (arm64+amd64 manifest lists), and chart install on OpenShift with cert-manager TLS and podSecurityContext/securityContext UID overrides for restricted-v2 SCC (note: fixed runAsUser: 1000/fsGroup: 1000 defaults required nulling on OpenShift — possibly worth a follow-up for OpenShift users).

[jhjaggars]

Rebased onto the latest main (9a85667d0) — new head is 28551bd8. The only conflict resolution was in crates/openshell-server/src/compute/mod.rs, where the new supervisor_host field from #4702 landed alongside the service_route_index field added here; both were kept. Full local verification passed: clippy, fmt, openshell-core (581 tests), openshell-server suite (2,443 tests), CLI service-URL tests, and helm:test including the config-parser preflight across all CI overlays.

@TaylorMutch the branch was force-pushed for the rebase, so the earlier /ok to test (pinned to 3f1fd809b) no longer covers the head — could you re-approve for 28551bd8 when you have a moment?

[mrunalp]

/ok to test 28551bd

[sjenning]

/ok to test a868504

[sjenning]

/ok to test 17fd4cd

[jhjaggars]

Dev-cluster verification

Verified the dedicated-controller refactor live on an OpenShift dev cluster (two gateways, openshell-a with kind=route and openshell-b with kind=httproute, sharing one service domain behind a single router, mixed amd64/arm64 nodes), using multi-arch images built from this branch (pr4842-722016eea).

Architecture in action

• Gateway feed publisher runs in-process and writes the hostname set to the feed ConfigMap every 5s (format-versioned, unchanged-content skips).
• The chart-deployed service-ingress controller Deployment (1 replica, dedicated ServiceAccount) reconciles from the feed. Exposure → feed → Route creation completed within 5s end-to-end; the controller's own logs confirm it (not the gateway) creates the objects.
• Controller self-heal: manually deleting a reconciled Route was repaired within 5s.

Kind handling

• The chart's httproute spelling renders through to the gateway config and boots cleanly (the pre-alias build crashlooped the gateway on config validation — this is what motivated 233d927).
• Cross-kind pruning: gateway-b had an orphaned Route left over from an earlier kind switch (8h old). The new controller deleted it within 5s of starting while keeping the HTTPRoute for the still-exposed hostname.

End-to-end lifecycle (gateway-a, kind=route)

• sandbox create --from python:3.12-slim --expose 8080 → hashed hostname minted under the shared service domain
• Controller created the exact-host Route (TLS passthrough → gateway Service)
• external-dns created the Route53 record set (CNAME + TXT ownership)
• curl over public DNS returned HTTP 200 from the sandbox workload
• Sandbox deleted → feed emptied → Route removed → external-dns deleted all three records

RBAC separation confirmed on-cluster: the gateway ServiceAccount holds only the feed-publisher Role (pinned to the one ConfigMap); the controller ServiceAccount holds the ingress-object Role. No cluster-scoped permissions involved.

Verification also surfaced one fix in this branch: the controller subcommand initially ran without a tracing subscriber (722016e).

[jhjaggars]

Per review, this PR is now scoped to just the hashed service-routing hostnames (service_routing_gateway_name). The in-tree service-ingress reconciliation (controller, feed ConfigMap, chart RBAC) has been removed from this branch.

The follow-up plan is a discovery API on the gateway so ingress provisioning and reconciliation can be built as a controller alongside the gateway rather than inside it — the removed work is preserved on a side branch as a starting point for that design.

[TaylorMutch]

Thanks for separating the Kubernetes reconciliation work from this PR. The one-label hashed hostname fits the shared wildcard certificate and exact-host routing use case, and this functionality can land independently of RFC 0017.

I think we should resolve the routing identity model and two request-path issues before merging:

1. OpenShell already has [openshell.gateway].name, an operator-assigned installation identity shared by its replicas. This introduces a second value described as a gateway name. If URL identity intentionally needs an independent lifecycle, could we name it something like service_routing_discriminator and document that distinction? Otherwise, I would prefer an explicit hashed-hostname mode that derives its discriminator from the existing installation name. gateway_jwt.gateway_id should remain separate because it has security and Kubernetes ownership semantics.

2. A syntactically valid unknown hash currently causes a full all-workspaces service endpoint scan. Because this runs on the public request path before authentication, repeated random Host headers can amplify into repeated database scans.

3. Cross-origin rejections for hashed service hosts no longer emit the existing OCSF security event because the logging path still uses the readable-host parser.

The repository instructions also require updating skills/debug-openshell-cluster/SKILL.md when gateway TOML or Helm rendering changes.

[jhjaggars]

Thanks for the thorough review — all three points are addressed in 2a442b5.

Identity model: Took your preferred direction. service_routing_gateway_name is gone, replaced by an explicit service_hostname_format = "readable" | "hashed" mode whose discriminator is the existing [openshell.gateway].name installation identity (operator/platform-assigned, defaulting to the Helm release name). We kept the derive-from-name shape rather than a separate discriminator field because the name users pick for gateways is only ever surfaced in the platform layer — end users never touch gateway.toml or see this field — so the platform can enforce cross-gateway uniqueness as part of gateway creation. The config reference documents the distinction from gateway_jwt.gateway_id (which stays untouched), the DNS-label requirement, the platform-owned uniqueness contract, and the rename-changes-every-URL consequence.

Request-path amplification: Miss-triggered index rebuilds are now debounced to one attempt per five seconds (the timestamp stamps the attempt, so a failing store can't be retried more than once per window either). A stream of random well-formed labels now costs in-memory lookups plus at most one full-store scan per window. Startup hydration and explicit rebuilds bypass the debounce, so legitimate cold-start misses still recover immediately.

OCSF regression: The cross-origin rejection emitter now resolves hashed hosts through the in-memory route index and emits the security event with the resolved sandbox/service identity; unknown hash-shaped labels still emit, attributed to the raw label. It deliberately does not call the miss-triggered rebuild path, so the emitter itself cannot be used for amplification.

The debug-openshell-cluster skill is also updated for the new gateway TOML fields, and the chart value is now server.serviceHostnameFormat (with server.name as the discriminator).

On the follow-up: we'd like the discovery API to include cross-gateway hostname-claim validation — with the discriminator being platform-assigned this should be structurally prevented at creation time, but a domain-scoped check in the controller would catch platform bugs and cross-cluster conflicts in a shared zone.

[jhjaggars]

Dev-cluster verification of 2a442b5

Deployed the review-feedback commit to both dev-cluster gateways (multi-arch image built from the branch), migrating values to the new surface: server.serviceHostnameFormat: hashed with the installation name left at its release-name default, and the removed serviceRoutingGatewayName value nulled.

Discriminator derivation — verified exactly: the label minted for a fresh exposure equals the expected base32 SHA-256 of (installation name, workspace, sandbox, service), and differs from the label the removed per-gateway-name field would have produced for the same exposure. The hash input is provably the installation name.

Manual ingress path (external automation owns ingress, per this PR's scope):

• sandbox create --from python:3.12-slim --expose 8080 → hashed hostname minted
• Hand-created exact-host Route (TLS passthrough → gateway Service) admitted by the router
• external-dns created the DNS record set
• curl over public DNS returned HTTP 200 from the sandbox workload
• Sandbox deleted → Route removed → external-dns deleted all records

Rebuild debounce — fired a burst of concurrent requests carrying random well-formed hash labels: all rejected quickly, zero index-rebuild or error log lines in the gateway, health endpoint responsive throughout, and an in-flight live exposure kept serving 200s. Worst case is one store scan per five-second window as designed.

OCSF emitter — the cross-origin rejection path serves the loopback-bound plaintext browser listener, which a Kubernetes gateway (bound non-loopback) correctly refuses plaintext on, so this path is not reachable in an ingress topology; it is covered by the new unit tests exercising the emitter with the OCSF capture harness, including resolved hashed hosts, unknown labels, and non-service hosts.

[TaylorMutch]

Non-blocking docs follow-up: the implementation now uses the existing gateway installation name plus service_hostname_format = "hashed", but two passages still describe the removed “service routing name” concept:

• docs/kubernetes/ingress.mdx (“unique routing name”, “With a routing name set”, and “routing-name field”)
• docs/how-it-works/sandboxes/overview.mdx (“configured with a service routing name”)

Could those be updated to say that each gateway needs a unique installation name (server.name) and hashed hostname mode enabled? The configuration examples themselves look correct.

[TaylorMutch]

/ok to test 2a442b5

[TaylorMutch]

The substantive review concerns are resolved:

• hashed hostname mode now uses the existing gateway installation name instead of introducing a second gateway identity;
• miss-triggered route-index rebuilds are bounded by a per-replica debounce; and
• hashed-host cross-origin rejections emit the expected OCSF event.

The updated cluster debugging guidance and successful CI/live validation provide good coverage. The remaining stale “routing name” wording is a non-blocking documentation follow-up.