Eai 7502 verify openbao apikey write credential - #786
Draft
johnl-amd wants to merge 14 commits into
Draft
Conversation
Add a gateway-scoped SecurityPolicy (authorization.defaultAction: Deny) targeting the ai-gateway Gateway, gated behind aiGateway.nativeAuth.enabled. When that switch is on, any model route without a per-model apiKeyAuth policy (from ai-gateway-discovery) is refused instead of falling through unauthenticated, and the cluster-auth ext_authz targetRef is dropped from ai-gateway (it stays on https) so the two gateway-scoped policies never target the same Gateway. Off by default; flipping the switch is the ai-gateway half of the cluster-auth cutover. The https workloads-route deny is route-scoped and lives in aim-engine (EAI-7302); this covers only the ai-gateway side.
cluster-auth used to inject x-aim-service-id (the aim_service_id metric label) from the cluster-auth/aim-service-id route annotation. With cluster-auth off ai-gateway after the native apiKeyAuth cutover, nothing set it, so per-model usage graphs went empty (key-level totals still worked). Have the existing body->header Lua also set x-aim-service-id from the backend UUID (same value as x-ai-eg-backend and the aim-keys-<workload-id> Secret), so attribution survives without cluster-auth. ai-gateway-scoped only; the https/legacy path is untouched.
Per-model metric/access-log attribution (aim_service_id) came from x-aim-service-id, which cluster-auth injected server-side. With cluster-auth off the ai-gateway that header is never set, so the per-model usage graphs went blank (key-level totals were unaffected). The ext_proc snapshots request-header attributes at the header phase, so a body-reading Lua sets the header too late (confirmed on app-dev: the metric had no aim_service_id label at all). Instead capture the client-sent x-ai-eg-backend header directly — it's present at header phase (like x-api-key-id), is the deployment/workload-id UUID AIWB already indexes on, and drives routing too so it can't disagree with where the request went. Map it to aim_service_id in the ai-gateway controller args and the two access-log configs; drop the inert Lua x-aim-service-id line. Caveat: only requests that send x-ai-eg-backend as a header are attributed per-model (the AIMConnect snippet does); body-only callers still count in key totals but not per-model.
Back out the x-aim-service-id Lua stamp (1bcb00b) and the x-ai-eg-backend attribution swap (c1abd02). Attributing by the client-sent x-ai-eg-backend surfaced raw deployment UUIDs (ugly, and showed stale/duplicate deployments) in the AIWB usage UI, and the per-model graphs still didn't render. Restore the pristine chart default (x-aim-service-id:aim_service_id) and the original body->header Lua. The default-deny backstop (bb0d448) is kept. Per-model attribution needs a different approach (human-readable id and/or UI resolution) — tracked separately.
Capture the client-sent x-ai-eg-backend header (the deployment/workload-id
UUID) as the aim_service_id metric/access-log label, so per-model token
usage keys on the same id the rest of the product already uses — GPU/vLLM
metrics, the /inference/{id} API, and AIWB URLs all key on workload_id.
The AIWB usage UI is updated in lockstep (getAimServiceId -> workload-id
label) so the graphs still render the friendly deployment name.
Chosen over a per-model EnvoyExtensionPolicy injecting <ns>-<isname>:
same result for header-sending clients (the AIMConnect snippet sends the
header), far less machinery, and consistent with platform-wide workload_id
attribution. Caveat: body-only callers land in key totals but not per-model.
Drop the aiGateway.nativeAuth.enabled switch: enabling ai-gateway now IS the cutover. The gateway-scoped default-deny backstop renders whenever ai-gateway is enabled, and cluster-auth ext_authz targets https only (never ai-gateway). Enable ai-gateway only once every model on it is natively enforced and keys are populated, or keyed traffic is denied.
Provision a write credential for AIWB's ADR-0002 Scenario 2 key store, scoped to secrets/apikeys/* instead of reusing the root token: - secret-manager (reconciling, every 5m + init) idempotently ensures apikeys-write-policy and mints a scoped token at secrets/aiwb-openbao-token, re-minting only when the stored one is missing or invalid. No renewal machinery (nothing else in the stack renews); the token carries the default TTL and rotates on expiry. - aiwb-external-secrets: an ExternalSecret materializes that token into the aiwb namespace as aiwb-openbao-token, which the AIWB backend already consumes as OPENBAO_TOKEN. ESO reads it via the existing readonly store. Policy capabilities match the AIWB client exactly: data create/read/update and metadata create/update/delete (delete_version_after + revocation). The OCI publish of aiwb-external-secrets and the root repoVersion bump are handled by the external release pipeline, not here.
…edential' into EAI-7502-verify-openbao-apikey-write-credential
…d requests hit apiKeyAuth Envoy selects the route before this Lua sets x-ai-eg-backend/x-ai-eg-model from the request body, so a body-only OpenAI request (model/backend in JSON, no headers) matched the ai-gateway-injected catch-all rule instead of the header-matched rules that carry the per-model apiKeyAuth SecurityPolicy — so key auth was bypassed and the request served regardless of the key. Clear the route cache after setting the headers so the request re-matches the enforced rule. Latent under cluster-auth (ext_authz only reads the header, route-agnostic); native route-scoped apiKeyAuth needs the route re-matched to enforce.
…dy-routed requests hit apiKeyAuth" This reverts commit 10a6738.
Temporary logWarn to determine whether request_handle:body() is nil (unbuffered) at this filter stage, and what backend/model it extracts. Diagnosing why body-only requests bypass per-model apiKeyAuth. Revert after capture.
This reverts commit a7fb4d2.
… + clearRouteCache Body-only requests bypass per-model apiKeyAuth because request_handle:body() in the set-model-header Lua returns a truncated first chunk (ext_proc consumes the body first), so x-ai-eg-model is never set and the request stays on the auth-less catch-all. Order the Lua before ext_proc so it buffers the full body, then set the routing headers and clearRouteCache so the request re-matches the apiKeyAuth rule. Temporary SETMODEL logWarn to confirm the body is now full. Revert if it regresses.
…traction + clearRouteCache" This reverts commit df47584.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.