[anatolykoyfman]

Summary

AuthBridge's outbound leg switches from token-exchange (one route per target, in authproxy-routes) to token-exchange-aiac (no routes), from rossoctl/cortex#1496. The plugin derives the audience from the destination host and asks Keycloak, so a new tool needs no route and no agent restart.

 agent ──► AuthBridge outbound
-            token-exchange       # route per host, scope from the route
+            token-exchange-aiac  # audience = spiffe://<td>/ns/<ns>/sa/<host>, no scope
+                                 # <td> from the chart, <ns> from the host or the pod's own namespace
+                                 # no bearer: 401 to a target (no_token_policy: deny)
+                                 # no unknown-audience cache (unknown_audience_ttl: "0")
             opa
           ──► github-tool inbound (jwt-validation, mcp-parser, opa)

With no scope parameter, only a default client scope of the agent's client grants a target. That changes the grant and the failure shapes:

	Before	After
Wiring a tool	route in authproxy-routes + agent restart	none (Keycloak only; no restart, even for a caller that called the tool before its client existed)
Agent's agent-<ns>-github-tool-aud	optional scope	default scope (an optional link is moved to default)
Target not granted	route-dependent	403 policy.forbidden, plugin: token-exchange-aiac
Target's client disabled (quarantine)	—	403 policy.forbidden (target_disabled)
Keycloak error or outage	503	503 upstream.token-exchange-failed
No bearer, to a target	401	401 auth.unauthorized (no_token_policy: deny; the workload's own token is never sent)

k8s/
├── opa-kind-enable.sh            # overlay wires token-exchange-aiac, then opa, on the outbound leg
├── opa-kind-verify.sh            # Verify 1 = authbridge-pipeline-check.py
├── authbridge-pipeline-check.py  # the one wiring rule: verify.sh, the demo driver, the skip gate
├── authbridge-rejection.py       # the one outbound classifier: the harness and the demo's probes
└── opa-kind-runbook.md           # outcome table, error shapes, the default-scope rule
demo/                             # no authproxy-routes; WIRE grants the default scope
docs/                      # PRD, specs, testing docs, two analysis notes
test/system/               # harness follows the plugin (below)

The system harness:

• Grant. grant_exchange_scope links the tool's audience scope as default, moving an optional link first (the same move as _link_default_scope of the IdP Configuration Service, copied rather than imported). Keycloak silently skips a default link for a scope that is already optional. Any Keycloak error raises.
• No restart. The harness and the demo never restart the agent: the overlay sets unknown_audience_ttl: "0", so an audience the plugin saw as unknown before the tool's client existed is asked again on the next call. Each rung asserts the bundle the live onboarding produced.
• Skip gate. require_pipeline skips cleanly unless the runtime config has opa once on each leg, exactly one token-exchange-aiac and no token-exchange on the outbound leg, a spiffe://<td>/...{service}... audience template, no_token_policy: deny, and unknown_audience_ttl: "0". It parses the config. The rule is k8s/authbridge-pipeline-check.py: opa-kind-verify.sh, the demo driver and teardown.sh run that file, and the gate loads it, so they cannot disagree. The gate returns the trust domain from the same read, so a bad template skips before any cluster change.
• Classifier. One file, k8s/authbridge-rejection.py (standard library only), loaded by the harness and sent into the agent pod by the demo's probes. An OPA deny is a policy.forbidden verdict only. An exchange refusal or failure, OPA with no bundle yet, an OPA evaluation error (OPA decision error: …), another plugin's rejection, and AuthBridge's fallback frame are never a deny, and never an allow. The demo's ENFORCE passes test-user only on github-tool's own OPA deny. The rollback test counts a refusal as blocked only when the tool's client is disabled; a refusal for a target not granted, and a failed exchange, still fail it.
• Trust domain. The harness exchanges for the trust domain of the live token-exchange-aiac template, which the skip gate returns.
• Removed. ensure_github_tool_route, restart_agent and its knob AIAC_AGENT_DEPLOYMENT are gone.

Decisions

Each design decision is recorded in the spec for its feature (docs/agents/domain.md):

• Routing and grants: PRD D36 — no routes; only default client scopes; no_token_policy: deny; no unknown-audience cache (unknown_audience_ttl: "0"), so adding a target needs no restart; Service = ServiceAccount; no default github-full-access; the realm-default widening (Task: the operator's realm default audience scopes grant every later client every earlier target #248); the rejected options.
• Test harness: uc1-onboarding-pipeline.md → Testing Decisions — what an OPA verdict is (an evaluation error is not one); one copy of each shared k8s/ rule; the wiring order; no agent restart; exact-name teardown; the harness's own link-move copy.
• Demo: demo.md → Step 5, How ENFORCE waits, and why — the bundle wait; no cache to wait out, so no restart; no 401 retry.

Evidence

At e6188c9 (after the sixth review; the next commit changes only docs and a comment):

• Unit: pytest (the default lane) passes: 2197 passed. Pure tests cover the classifier and its edge frames (OPA with no bundle, an OPA evaluation error in both shapes, AuthBridge's fallback frame, another plugin's rejection, a tool's own JSON-RPC error), the inbound classifier, the refusal rules, and the skip gate on parsed configs: half-migrated, inbound-only, duplicated, commented-out, quoted and bare-string entries, a non-list plugins, opa before the plugins it reads, a template without {namespace} or {service} or with them in the wrong order, no_token_policy other than deny, and unknown_audience_ttl that is unset, non-zero, an int, or not a Go duration. I checked the zero-duration matcher against Go's time.ParseDuration. Also tested: the gate skips when a shared k8s/ file is missing or broken, or PyYAML is missing, and importing the classifiers needs no k8s/. test/unit/k8s/test_shared_scripts.py runs both k8s/ scripts as the shell scripts run them, on the same configs.
• System: pytest -m system passes on a Kind cluster re-enabled with this branch's overlay (no_token_policy: deny, unknown_audience_ttl: "0"), with the AuthBridge sidecar built from cortex main at ef5ac5f8 (#1496): 87 passed, 0 failed, 14 skipped (the test/system/disabled/ modules, skipped on every run). No agent is restarted. The run was on this commit's code.
• Demo: make e2e, ./driver.sh --only-enforce and make restore in demo/use-cases/onboarding/ pass, with no agent restart, and restore leaves team1 empty. With the agent's own sidecar config set back to the 5 m default, ENFORCE stops and says to re-run k8s/opa-kind-enable.sh.
• Live checks on that cluster:
  • A throwaway target got its Keycloak client created after its caller's first call. The next call, one second later and with no restart, went through the exchange (403 audience_not_granted).
  • ensure_default_audience_scope moved an optional-only link to default. The previous code left it optional and still reported success.
• Platform: k8s/opa-kind-verify.sh and k8s/keycloak-spi-verify.sh pass all checks on that cluster.

Merge Danger

Door: two-way

Revert, then run k8s/opa-kind-enable.sh from a cortex checkout without the plugin to get the route-based pipeline back. Nothing in AIAC's own services changes.

Blast Radius: platform

• Needs a cortex with token-exchange-aiac: main at or after ef5ac5f8 (Feat: Add token-exchange-aiac, an outbound token exchange with no routes cortex#1496, merged). An older cortex fails at sidecar start: with an unknown-plugin error, or, before 1049bd07, by rejecting the namespace_file key.
• k8s/opa-kind-verify.sh, the demo driver and teardown.sh now need python3 with PyYAML.
• A cluster enabled from an earlier head of this branch must re-run k8s/opa-kind-enable.sh. Earlier overlays set no_token_policy: client-credentials, or left unknown_audience_ttl at the plugin's 5 minutes; the wiring check now requires deny and "0", so verify.sh and the demo preflight fail, and the system lane skips, until it is re-run.
• Accepted risk, by decision (message match): an OPA evaluation error is told from a deny only by its message prefix (OPA decision error:), copied from cortex core/plugins/opa/plugin.go into one constant. If cortex rewords it, an evaluation error reads as a deny again, as it did before this PR. We accept this because the check can only remove false passes, never add one; a rewording falls back to the earlier behaviour; and every allow expectation still fails on a Rego that does not evaluate. Rejected: matching cortex's "policy denied" instead (it pins another string, breaks on any Rego that sets a reason, and stalls the convergence polls); waiting for a machine-readable reason (token-exchange-aiac: put the refusal reason in the rejection, not only the message cortex#1570, closed as deferred). Revisit when cortex adds a reason code. Recorded in uc1-onboarding-pipeline.md, Testing Decisions.
• unknown_audience_ttl: "0" costs one Keycloak token request per outbound call to an in-cluster host that has no client (today, the agent's calls to the unregistered github-tool-mcp, Task: exchange the demo github-agent's calls to the production github-tool-mcp under token-exchange-aiac #247). Hosts outside the cluster are never asked about.
• A cluster still on the old pipeline makes the system lane skip with "run k8s/opa-kind-enable.sh". It does not fail.
• The demo no longer ships authproxy-routes. On the stock chart pipeline its tool calls go out unexchanged (INSTALL.md says so). teardown.sh warns about a route left by an older demo and does not delete it.
• Known limitation (creation order): the rossoctl operator registers each agent-<ns>-<workload>-aud scope as a realm default scope, and Keycloak links realm defaults to every client created later. So an agent created after a tool already holds that tool's audience with no grant step, and its exchange is never refused (403). The tool's own inbound OPA still decides each call. The runbook says so; follow-up: Task: the operator's realm default audience scopes grant every later client every earlier target #248.
• Known limitation: the demo agent's live GitHub calls to the production tool (github-tool-mcp) are not exchanged yet. The AIAC side is wired: MCP_URL derives …/sa/github-tool-mcp, and WIRE links that tool's audience scope as a default scope of github-agent when it exists. The cortex demo must still run the tool under a ServiceAccount github-tool-mcp and enroll it (github-issue demo: make the production github-tool-mcp reachable by token-exchange-aiac cortex#1540, which cortex closed as deferred on 2026-10-29: the Kubernetes path is not their current priority). github-full-access is never granted: as a default scope it would give every user the privileged PAT, so the tool uses its public PAT. The UC-1 demo and the system lane call the stub github-tool and are not affected. Follow-up: Task: exchange the demo github-agent's calls to the production github-tool-mcp under token-exchange-aiac #247.
• The wiring check (k8s/authbridge-pipeline-check.py) is stricter: opa must run after jwt-validation and mcp-parser inbound and after token-exchange-aiac outbound, and the audience template must carry {namespace} before {service}. A cluster enabled by k8s/opa-kind-enable.sh meets it; a custom overlay that does not now fails Verify 1 and skips the system lane.

🤖 Generated with Claude Code

[oblinder]

Automated review (Claude Code). 10 findings; 9 are inline.

Not inline (the line is outside the diff): demo/use-cases/onboarding/driver.sh:360. The driver preflight still checks only that name: opa == 2. It does not check that the outbound leg runs token-exchange-aiac, but the demo now ships no routes and depends on that plugin. On the stock chart pipeline (token-exchange with routes), which INSTALL.md now says to avoid, preflight passes and WIRE grants the default scope. Then ENFORCE gets un-exchanged calls that github-tool rejects (401 or deny), and these are reported as policy failures. The PR adds a clean-skip gate only to the Python harness (exchange_plugin_reason), not to the driver.

The two most severe findings are the namespace in opa-kind-enable.sh and the removed github-tool-mcp route.

🤖 Generated with Claude Code

[oblinder]

Wrong namespace for agents outside AGENT_NAMESPACE. The overlay sets namespace: "${AGENT_NAMESPACE}" (team1) for short host names. But the chart renders this one authBridge.pipeline into the authbridge-runtime-config of every namespace in agentNamespaces (team1 and team2 by default).

Scenario: an agent in team2 calls http://foo:8080. The plugin derives spiffe://localtest.me/ns/team1/sa/foo, not .../ns/team2/sa/foo. No such client exists, so the call passes through un-exchanged and the callee's jwt-validation rejects it. Or, if a team1 client foo exists, the plugin asks for a token for another namespace's service.

[anatolykoyfman]

Fixed in 452341d. The plugin now has a namespace_file key (rossoctl/cortex#1496, 1049bd07): when namespace is empty it reads the namespace once at start, and a missing, blank or invalid file stops the sidecar. The overlay points it at /var/run/secrets/kubernetes.io/serviceaccount/namespace, so a short name called from team2 derives a team2 audience. ${POD_NAMESPACE} was not an option: the webhook injects no such env, and the config loader leaves an unset ${...} unexpanded. I checked that the injected authbridge-proxy container mounts that path.

[oblinder]

Tool calls on github-tool-mcp are no longer exchanged. The PR deletes the authproxy-routes route (github-tool-mcp → github-tool, token_scopes: openid github-tool-aud github-full-access) but keeps MCP_URL=http://github-tool-mcp:9090/mcp. No step replaces the route's work: no Service=SA alignment and no default github-full-access grant.

Scenario: every agent tool call now derives an audience that has no Keycloak client. The call passes through with the user's un-narrowed token, and the tool's jwt-validation rejects it (401). Also, no token carries github-full-access now, so PAT selection breaks.

[anatolykoyfman]

Agreed. The UC-1 driver isn't affected (it calls the stub github-tool directly), but the production tool's live calls are: they go out un-exchanged, and no token carries github-full-access. In 452341d this is a known limitation in INSTALL.md and the PR's Merge Danger. The fix (Service = ServiceAccount alignment, or an MCP_URL change, plus github-full-access as a default scope of github-agent) is tracked in #247.

[oblinder]

This grep also matches token-exchange-aiac. grep -c 'name: token-exchange' counts name: token-exchange-aiac too, so a correctly wired cluster prints 1, not the expected 0. An operator can conclude that the legacy plugin is still wired. opa-kind-verify.sh uses an anchored \s*$ pattern, so the runbook and the script disagree.

[anatolykoyfman]

Fixed in 452341d. Check 1b now runs k8s/outbound-exchange-plugins.py, the helper verify.sh uses. It expects token-exchange-aiac=1 token-exchange=0.

[oblinder]

The grep gate and the Python gate can disagree. This greps the whole config.yaml with an unquoted, line-anchored pattern. The Python gate (pipeline_plugins / exchange_plugin_reason) parses YAML and looks at the outbound leg only.

• - name: "token-exchange-aiac" (quoted, which the new unit test accepts) gives DX_COUNT=0, so verify.sh fails.
• A commented-out # - name: token-exchange gives TX_COUNT=1, a false failure.
• A legacy token-exchange on the inbound leg fails verify.sh but passes the system-lane gate.

Consider parsing the YAML here too (for example with yq), so both gates give one verdict.

[anatolykoyfman]

Fixed in 452341d. verify.sh now parses the YAML with k8s/outbound-exchange-plugins.py (python3 + PyYAML) and counts whole plugin names on the outbound leg only. A unit test runs the helper on every config the harness gate is tested with, including the quoted, commented-out, inbound-only and half-migrated cases, and checks that the two give the same verdict. The demo driver's preflight uses the same helper.

[oblinder]

Out-of-date docstring. This (and exchanged_subject in uc1_onboard.py) says AuthBridge uses the client-secret identity that k8s/opa-kind-enable.sh sets. The PR removes identity: type: client-secret from the overlay, and the overlay comment now says that key is rejected. The harness reasoning that "the exchange must succeed" rests on a setting the script no longer applies.

[anatolykoyfman]

Fixed in 452341d. Both docstrings now say why the exchange must succeed: token-exchange-aiac authenticates with the client secret the operator writes to /shared/client-secret.txt (its default client_secret_file) and has no other way to authenticate. They no longer mention the client-secret identity key.

[oblinder]

A third, unlinked source of the trust domain. TRUST_DOMAIN defaults to localtest.me, separate from the chart's spireTrustDomain and the harness's AIAC_TRUST_DOMAIN.

Scenario: a cluster installed with spireTrustDomain: example.org and enable.sh run without TRUST_DOMAIN. The plugin derives spiffe://localtest.me/..., which names no client, so every exchange silently passes through un-exchanged. Reading the value from the chart values would remove the drift.

[anatolykoyfman]

Fixed in 452341d. TRUST_DOMAIN is gone. The audience template reads the chart's operator-chart.signatureVerification.spireTrustDomain through the chart's tpl(), the value the operator names SPIFFE clients with. helm template with another value puts it in every rendered copy.

[oblinder]

Duplicate of an existing helper. The optional→default move here (get_client_optional_client_scopes / delete_client_optional_client_scope / add_client_default_client_scope) already exists as _link_default_scope in src/aiac/idp/service/configuration/keycloak/main.py:326, and driver.sh repeats it in curl. The "Keycloak skips a default link of a scope already linked as optional" workaround now lives in three places, so a fix to one will not reach the others.

[anatolykoyfman]

Fixed in 452341d. grant_exchange_scope now calls _link_default_scope, so the harness and onboarding share the workaround. driver.sh is bash, so it keeps its curl version, with a comment pointing at the helper.

[anatolykoyfman]

Correction: I reverted the import in 94f237c. Importing main.py configures the root logger and builds its FastAPI app inside the pytest process, so the harness changes logging for the whole session. grant_exchange_scope keeps its own 3-line copy again, with a comment saying it mirrors _link_default_scope and why it is not imported. driver.sh still keeps its curl copy, pointing at the same helper.

[oblinder]

Third copy of the JSON-RPC error-frame parsing. outbound_outcome and deny_origin already unwrap "an error frame at HTTP 200 to error.data, else the plain 403 body" separately. A shared helper would keep the three classifiers in agreement if the frame shape changes.

[anatolykoyfman]

Fixed in 452341d. A new rejection_doc unwraps both shapes (error.data of a frame at 200, else a plain 403 body) once. outbound_outcome, exchange_refusal, deny_origin and notification_outcome all use it, and it has its own unit tests.

[oblinder]

Risky default for body. The function now needs the body to classify a deny, but body defaults to "". A caller with the old call shape, notification_outcome(code), silently gets "error" for a real OPA 403 deny. Making body required would catch this at the call site.

[anatolykoyfman]

Fixed in 452341d. body is now required.

[anatolykoyfman]

Re the finding outside the diff (demo/use-cases/onboarding/driver.sh:360): fixed in 452341d. The driver preflight now checks that the outbound leg runs token-exchange-aiac and not token-exchange, with the same parsed rule as k8s/opa-kind-verify.sh (k8s/outbound-exchange-plugins.py). So on the stock pipeline it stops with "run k8s/opa-kind-enable.sh first" instead of reporting un-exchanged calls as policy failures.

[oblinder]

Review of 58cdcd6. These findings come from reading the diff plus the local operator and cortex checkouts; they are not verified on a live cluster, and I could not read the token-exchange-aiac plugin source (feat/token-exchange-aiac is not in my cortex checkout), so the plugin-behaviour points (realm default scope, audience_unknown cache, passthrough on unknown audience) are worth a check against the plugin.

Highest priority: the runbook's grant boundary (realm default-default scope, creation order), no_token_policy: client-credentials, and the tool-less-rung rationale.

Points on lines outside the diff:

1. demo/use-cases/onboarding/driver.sh:662 — probe_tool does not tell a token-exchange refusal from an OPA deny. Every 403/503 is DENIED_HTTP and every 200 error frame is DENIED_JSONRPC; the body's plugin is not checked. demo.md now says a body naming token-exchange-aiac means the WIRE grant is missing, but the driver does not act on it. If the default-scope link is lost or the exchange fails (403 audience_not_granted, disabled target, 503), the test-user -> source-read check (!= ALLOWED_RESULT) still passes as "correctly denied" and the verdict table shows OPA-style denials for all probes, though OPA was never consulted. Suggest classifying on plugin and failing (or a distinct verdict) when it is token-exchange-aiac.
2. test/unit/agent/uc/onboarding/test_uc1_subject_scope.py:607 — stale docstring. It says k8s/opa-kind-enable.sh sets "the AuthBridge token-exchange identity to client-secret"; this PR removed that identity: client-secret block and the overlay now wires token-exchange-aiac.
3. test/system/test_uc1_onboard_agent_only.py:15 and :137 — same "short-circuits before OPA" claim as the inline comment on uc1_onboard.py:2005.

[oblinder]

The "default scope only, else 403" boundary may depend on client creation order. The operator's EnsureAudienceScope (operator/internal/keycloak/audience.go:82) registers each workload's agent-<ns>-<wl>-aud scope as a realm default-default scope — the case this paragraph says "would grant every target to every caller".

Scenario: deploy github-tool first, then any agent. Keycloak attaches agent-team1-github-tool-aud to the new agent client as a default scope at creation. token-exchange-aiac requests no scope, so it mints a github-tool token for that agent with no WIRE grant and no audience_not_granted 403. So the "not granted -> 403" row holds for agent-then-tool but not for tool-then-agent.

[anatolykoyfman]

Confirmed, on the Kind cluster. Keycloak links a realm default-default scope to every client created later: a test scope registered as one, then a new client, had it in its default-client-scopes. So with the tool deployed first, the agent holds the tool's audience with no grant step and is never refused. The tool's own inbound OPA still decides each call.

In 8d83c4c the runbook states this under "Known limitation: creation order": the "not granted -> 403" row holds only for a caller created before the target. The fix belongs in the operator (EnsureAudienceScope makes the scope a realm default) or in AIAC onboarding; #248 lists both options and tracks it.

[oblinder]

"no restart" conflicts with the plugin's negative cache. token-exchange-aiac caches audience_unknown for 5 minutes, and that is why WIRE (driver.sh) and the harness's restart_agent restart the agent.

Scenario: the agent calls the tool before the tool's client exists, so the audience is cached as unknown. The operator adds the Keycloak grant per this runbook and does not restart. For up to 5 minutes calls pass through un-exchanged and github-tool's jwt-validation rejects them with 401. Suggest documenting the restart (or the cache TTL).

[anatolykoyfman]

Fixed in 8d83c4c. The runbook no longer says "no restart" without a condition. It now says that the plugin remembers an audience with no Keycloak client as unknown for 5 minutes (unknown_audience_ttl), so a caller that called the target before the target's client existed keeps passing through un-exchanged (and the target answers 401) for up to 5 minutes. A restart of the caller ends that at once, as WIRE and the harness do. The table row says the same, and so does docs/analysis/token-exchange-vs-token-exchange-aiac.md.

[oblinder]

no_token_policy: client-credentials widens what a bearer-less outbound call can do. Before, a request with no Authorization header went out unauthenticated and the tool's jwt-validation rejected it. Now the sidecar exchanges the workload's own client-credentials token, so any process in the agent pod (app code or a library) that calls http://github-tool:9090/mcp with no header gets an agent-audienced token. Only the tool's rego user gate stands between it and the tool; where a policy keys on client_id only, the call is allowed. Is this intended? If so, it is worth a line in the runbook's security notes.

[anatolykoyfman]

Not intended, and nothing used it: every probe of the harness and of the demo sends a bearer, and the agent forwards the inbound one. Fixed in 8d83c4c: the overlay sets no_token_policy: deny (the plugin default, which the route-based pipeline also left in place). A bearer-less request to a target now gets 401 auth.unauthorized; the workload's own token is never sent in its place. A destination Keycloak has no client for still passes through.

The wiring check now also requires deny (or unset), so a cluster still on the old overlay fails opa-kind-verify.sh and the demo preflight, and the system lane skips, with "run k8s/opa-kind-enable.sh". The runbook, the PRD (D31 limits), the IdP Configuration Service spec and both analysis notes say the same. The system lane passed on a cluster re-enabled with deny.

[oblinder]

The optional-scope DELETE likely also removes the existing default link on a re-run. Keycloak's removeOptionalClientScope calls the same client.removeClientScope as the default path, so if the scope is already a default scope (second make enforce), this DELETE drops it and the PUT re-adds it. In between, in-flight exchanges get 403 audience_not_granted. The comment ("drop any optional link") also describes the call as optional-only. Option: GET optional-client-scopes first and DELETE only when the scope is listed there. The same applies to _link_default_scope and the harness's grant_exchange_scope.

[anatolykoyfman]

Confirmed on the Kind cluster: DELETE .../optional-client-scopes/{id} on a scope linked as default removed the default link. Fixed in 8d83c4c: WIRE reads the optional scopes first (scope_linked ... optional <scope>) and deletes only when the scope is listed there, so a re-run never drops the live grant.

_link_default_scope and the harness's grant_exchange_scope already read the optional list first (get_client_optional_client_scopes), so they were not affected; the comment in driver.sh now says why the check matters.

[oblinder]

grep -c 'name: opa' on L362 still counts OPA by substring, while the exchange check below now parses the outbound leg. A commented # - name: opa or a duplicate entry on one leg gives 2, so the preflight passes where the harness's per-leg pipeline_unwired_reason skips. The configmap is also fetched twice. Same in k8s/opa-kind-verify.sh Verify 1. Suggest extending outbound-exchange-plugins.py (or a sibling) to report OPA per leg too, and fetching the config once.

[anatolykoyfman]

Fixed in 8d83c4c. One parsed check now covers OPA and the exchange: k8s/authbridge-pipeline-check.py (it replaces outbound-exchange-plugins.py) requires opa exactly once on each leg, one token-exchange-aiac and no token-exchange outbound, a spiffe://<td>/...{service}... audience template, and no_token_policy: deny. With --namespace NS it reads the ConfigMap itself, once, with kubectl's stderr kept out of the YAML. The driver's preflight and its ENFORCE re-check call it; so do opa-kind-verify.sh (Verify 1), teardown.sh (--opa-legs) and the restore hints. A commented # - name: opa, or opa twice on one leg, now fails the same way in all of them.

[oblinder]

Second fetch of authbridge-runtime-config — the OPA count on L118-119 already fetched it, and still counts with grep -c 'name: opa' (substring, not per leg). See the comment on driver.sh:367.

[anatolykoyfman]

Fixed in 8d83c4c, with the driver (see the reply on driver.sh:367). Verify 1 is now one call to k8s/authbridge-pipeline-check.py --namespace "$NS": one read of the ConfigMap, OPA counted per leg (not by substring), the exchange plugin, the trust domain and no_token_policy. Verify 4 uses the trust domain that check returns, unless TRUST_DOMAIN is set.

[oblinder]

This re-implements the YAML walk and count rule of launcher._leg_plugins / exchange_plugin_reason, and a unit test now has to keep the two copies in agreement. A change to one copy only (e.g. also checking inbound, or parsing OPA) makes verify.sh/driver and the system skip gate disagree, caught only for the configs the test lists. Could the harness and this script share one implementation (e.g. the script imports the launcher helper, or the launcher shells out to the script)?

[anatolykoyfman]

Fixed in 8d83c4c: there is one implementation. k8s/authbridge-pipeline-check.py (renamed from this file) holds the rule. opa-kind-verify.sh, the demo driver and teardown.sh run it as a script, and the harness's skip gate loads the same file (pipeline_check in test/system/launcher.py) instead of keeping its own YAML walk. So the system gate, verify.sh and the driver cannot disagree. A unit test still runs the script on every config the gate is tested with, now to check the script's exit code and output rather than a second copy.

The same goes for the outbound classifier: k8s/authbridge-rejection.py (standard library only) is loaded by the harness and sent into the agent pod by the demo's probes, so the demo and the harness tell an exchange refusal from an OPA deny by one rule.

[oblinder]

Rationale may be wrong for the tool-less rung. With no github-tool client, the derived audience is unknown, so token-exchange-aiac passes the call through (skip / audience_unknown) — it does not settle it before OPA, and OPA is consulted. An outbound "deny" on rung 1 would be an OPA verdict on an un-exchanged request, not a Keycloak refusal. The same claim is in test/system/test_uc1_onboard_agent_only.py (module docstring L15 and L137: "token-exchange short-circuits ... before OPA is ever consulted"). The decision to not probe rung 1 still stands; only the stated reason needs a fix.

[anatolykoyfman]

Right, thanks. With no github-tool client the audience is unknown, so token-exchange-aiac passes the call through (audience_unknown) and OPA is consulted. Fixed in 8d83c4c: the reason now says the probe would see the agent's pass-through outbound OPA on a request that reaches no tool, not the tool check. It is corrected in all three places (here, and test_uc1_onboard_agent_only.py at the module docstring and the note at L137) and in docs/testing/uc1-onboarding-pipeline.md. The decision not to probe rung 1 is unchanged.

[oblinder]

_exchange_trust_domain() runs at ctx build time — after deploy, onboarding, grant_exchange_scope and restart_agent — and on every rung, including agent-only, which never uses it. It is not part of the require_pipeline skip gate. If the live audience_template is not spiffe://<td>/..., require_pipeline passes (it counts plugin names only), the run mutates and restarts the cluster, then fails with RuntimeError instead of a clean skip up front. Suggest reading it in the skip gate.

Also, bundle_service_probe (L953) still builds the SPIFFE ID from the env TRUST_DOMAIN, so the diagnosis can probe a different trust domain from the one the exchange uses.

[anatolykoyfman]

Fixed in 8d83c4c. The trust domain is now part of the skip gate: require_pipeline checks the token-exchange-aiac audience template (it must be spiffe://<td>/...{service}...) and returns its trust domain from the same read of the config it approved. A run with a bad template skips before any cluster change, and the stack reads the config once, not twice. _exchange_trust_domain is gone.

bundle_service_probe now uses that trust domain too. AIAC_TRUST_DOMAIN stays only as an explicit override of that probe, the way TRUST_DOMAIN works in opa-kind-verify.sh.

[anatolykoyfman]

Re the three points outside the diff in the review of 58cdcd6, all fixed in 8d83c4c:

1. driver.sh probe_tool did not tell an exchange refusal from an OPA deny. Both probes now classify an answer with k8s/authbridge-rejection.py, the rule the harness uses (verdict). An exchange refusal or failure gets its own verdict, EXCHANGE_REFUSED <code> <error>. ENFORCE's test-user check passes only DENIED_HTTP 403, github-tool's own OPA deny; an exchange refusal, an OPA deny on the agent's pass-through outbound (DENIED_JSONRPC, D24), OPA with no bundle yet, or any other answer fails it, each with its cause. A missing login token is reported as such, not as an exchange failure.
2. test_uc1_subject_scope.py:607, stale docstring. It now says the agent client authenticates with the client secret, because token-exchange-aiac authenticates only with the secret the operator writes to /shared/client-secret.txt.
3. test_uc1_onboard_agent_only.py:15 and :137. Corrected with uc1_onboard.py:2005 (see that thread).

Evidence for 8d83c4c:

• Unit lane: 2114 passed.
• System lane, on a Kind cluster re-enabled with no_token_policy: deny, sidecar from Feat: Add token-exchange-aiac, an outbound token exchange with no routes cortex#1496 at 7122cd65: 87 passed, 0 failed, 14 skipped (the test/system/disabled/ modules).
• k8s/opa-kind-verify.sh: all checks pass.

[oblinder]

Code review (high effort): 8 findings, posted inline. The changed classifier, skip-gate and subject-scope unit tests pass (229 passed). Most important: plugin order not checked (pipeline-check), broken clean-skip path (uc1_onboard imports), and the demo agent tool path regression (deployment.yaml).

🤖 Generated with Claude Code

[oblinder]

Wiring check does not check plugin order. The check counts plugins on each leg but never checks their order. A pipeline with opa before token-exchange-aiac on the outbound leg (or before jwt-validation on the inbound leg) passes.

Failure scenario: a values override lists outbound plugins as [opa, token-exchange-aiac]. check() returns ok trust_domain=..., so verify.sh Verify 1, the demo preflight and the require_pipeline skip gate all pass. OPA then runs with no input.delegation or outbound input.identity — the opposite of what the overlay comment requires ("opa runs AFTER token-exchange-aiac").

[anatolykoyfman]

Fixed in b43bd33. The check now also checks the order on each leg: opa must run after the plugins that set what AIAC's Rego reads there. Inbound that is jwt-validation (input.identity) and mcp-parser (input.mcp, read by a tool's inbound package); outbound, token-exchange-aiac (input.identity, input.delegation). A leg without one of them is reported as missing, not as an order problem. The other parsers, and mcp-parser on the outbound leg, may run anywhere: AIAC's outbound package is a pass-through (D24), and no AIAC policy reads input.a2a or input.inference. I checked which inputs the generated Rego reads (src/aiac/pdp/service/policy/opa/rego.py) to keep the rule to those. The check also accepts the bare-string entry form (- opa), which cortex's config.PluginEntry reads as {name: opa}.

[oblinder]

Promised clean skip for a missing/broken shared file never happens. uc1_onboard imports is_opa_violation, outbound_outcome and target_disabled_refusal from launcher at import time. That goes through launcher's module __getattr__, which loads k8s/authbridge-rejection.py during import.

Failure scenario: k8s/authbridge-rejection.py is missing or has a syntax error. Every system test module that imports uc1_onboard fails at collection with ImportError, so the marker-selected lane errors out. pipeline_check() was written to turn this into a skip reason, but never gets to run.

[oblinder]

Reproduced in the second review: with k8s/authbridge-rejection.py deleted (in a worktree), import test.system.uc1_onboard fails with ImportError: .../k8s/authbridge-rejection.py is missing. It does not skip.

This also breaks the offline unit lane, not only the system lane: unit tests that import uc1_onboard (for example test_uc1_subject_scope and test_uc1_staged_gates) fail at collection.

[anatolykoyfman]

Right: names bound through the module __getattr__ loaded the file at import time. Fixed in b43bd33: the classifiers in launcher.py are now plain wrapper functions that load k8s/authbridge-rejection.py on their first call, so from test.system.launcher import outbound_outcome (as uc1_onboard does) loads nothing. pipeline_check loads the file in the skip gate, so a missing or broken file skips the live suite up front with the file named. A file that does not load (a syntax error) is now an ImportError naming it, so the gate catches that too. Unit tests import the launcher from a copy of test/ with no k8s/, and check the missing-file and broken-file skips.

[anatolykoyfman]

Thanks for reproducing it. I checked again at 0d0d46e, in a worktree with k8s/authbridge-rejection.py deleted:

• import test.system.uc1_onboard now succeeds. The classifiers are wrapper functions that load the file on their first call, so importing them needs no k8s/.
• Two unit tests still fail there, both in test_uc1_staged_gates.py (test_an_opa_403_points_at_the_cr and test_a_403_that_is_not_from_opa_says_so), because they call the classifier, and the classifier is that file. test_uc1_subject_scope.py passes.

I'm keeping that as intended. A unit test of the classifier needs the file it tests, the same way the existing test/unit/k8s/ tests on main need the manifests they read. In a real checkout a missing file is a packaging bug, and a failing test names it; a skip would hide it. What must skip cleanly is the live suite, and it does: pipeline_check loads both shared files in the skip gate, so a missing or broken one is a skip reason, with the file named.

[oblinder]

Demo agent's tool calls are no longer exchanged. This PR deletes the authproxy-routes route for github-tool-mcp but leaves MCP_URL at http://github-tool-mcp:9090/mcp.

Failure scenario: on a cluster wired with opa-kind-enable.sh, the derived audience spiffe://<td>/ns/team1/sa/github-tool-mcp matches no client, so the plugin forwards the user's login token unchanged and the tool's inbound jwt-validation rejects every call with 401. The old route exchanged this call for the github-tool audience with github-full-access. The PR body lists this as known limitation #247, but it is a working path this PR breaks.

[anatolykoyfman]

Agreed it is a working path this PR breaks. The fix needs changes on both sides, so I split it:

• cortex (github-issue demo: make the production github-tool-mcp reachable by token-exchange-aiac cortex#1540): the production tool's demo manifest gives the tool a ServiceAccount github-tool-mcp, equal to its Service, and enrolls it, so the operator registers spiffe://<td>/ns/team1/sa/github-tool-mcp. Today it runs under the default ServiceAccount with a plain client github-tool that setup_keycloak.py creates by hand, and it has no sidecar. The name github-tool cannot be reused: AIAC's UC-1 stub is Service = ServiceAccount = github-tool in team1.
• AIAC (b43bd33): MCP_URL stays at github-tool-mcp, which already derives that audience. The onboarding demo's WIRE phase links agent-<ns>-github-tool-mcp-aud as a default scope of github-agent when the scope exists, best effort, and skips it otherwise. restore.sh removes the link. The deployment comment, the agent README and spec, INSTALL.md and the runbook describe this wiring.

github-full-access is deliberately not granted. token-exchange-aiac requests no scope, so the only way to get it into a token is a default scope of github-agent, and then every token the agent exchanges carries it, whoever the user is: every user would get the privileged PAT. So the tool uses its public PAT for now. A per-user grant is a design question for the cortex demo, recorded in #1540.

The UC-1 demo and the system lane are not affected: they call the stub github-tool. Both pass at b43bd33 (see the top-level comment).

[oblinder]

Exact match on a human-readable cortex message. target_disabled_refusal tells "target disabled" from "not granted" only by matching the cortex message text. A copy of that string is the only link between the two repos.

Failure scenario: cortex rewords "the destination's client is disabled". _outbound_block then sees error, not refused, on every post-quarantine probe, and test_failed_tool_outbound_call_denied fails although the quarantine works. Suggest cortex emit a machine-readable reason (e.g. details.outcome=target_disabled) and key on that.

[anatolykoyfman]

Agreed it is fragile, and a machine-readable reason would be better. token-exchange-aiac's deny() sets no Details today, so there is nothing else to key on; I opened rossoctl/cortex#1570 to emit the reason (target_disabled / audience_not_granted) in the rejection's details. Until then the match fails closed: a reworded message makes target_disabled_refusal false, so test_failed_tool_outbound_call_denied fails as error. It cannot pass wrongly. The comment above TARGET_DISABLED_MESSAGE says so, and points at the cortex line that sets the message.

[oblinder]

pristine_stack and onboarded_stack now yield different ctx shapes. Here onboarded_stack keeps the trust domain from require_pipeline and puts it in ctx["trust_domain"]. pristine_stack (line 3236) calls require_pipeline(...) and drops the result, so its ctx has no trust_domain key.

Failure scenario: a rung-5 or failure-path test on pristine_stack calls uc1.exchanged_subject(ctx, user) or builds a diagnosis from its ctx and gets KeyError: 'trust_domain'.

[anatolykoyfman]

Fixed in b43bd33. pristine_stack now keeps the trust domain require_pipeline returns and yields it as ctx["trust_domain"], so both stacks yield the same keys for what needs no deploy.

[oblinder]

{namespace} not required in the template path. trust_domain() requires {service} but not {namespace}, so spiffe://td/sa/{service} or a hard-coded namespace passes.

Failure scenario: a shared values file pins audience_template: spiffe://localtest.me/ns/team1/sa/{service}. The check passes, and every non-team1 agent namespace derives a team1 audience — exchanges are refused or sent unexchanged, while verify.sh and the skip gate report the cluster as correctly wired.

[anatolykoyfman]

Fixed in b43bd33. The template must now be spiffe://<td>/...{namespace}...{service}...: {namespace} and {service} both in the path, {namespace} first (the operator names a workload .../ns/<ns>/sa/<sa>). A pinned namespace, a missing {namespace}, or {service} before {namespace} fails the check, and so verify.sh, the demo preflight and the skip gate.

[oblinder]

AIAC_TRUST_DOMAIN silently changed meaning. It used to set the exchange audience; now it only changes the bundle-service probe in the diagnosis dump, with no warning when it disagrees with the live template.

Failure scenario: an environment still exports AIAC_TRUST_DOMAIN=example.org for a cluster whose template says localtest.me. The exchange uses localtest.me (correct), but the diagnosis probes /bundles?spiffe=example.org/..., giving a misleading 404/403 in every gate-failure message.

[anatolykoyfman]

Fixed in b43bd33. The gate-failure dump names the trust domain the bundle probe asked for, and when AIAC_TRUST_DOMAIN differs from the live token-exchange-aiac trust domain it says so: "(AIAC_TRUST_DOMAIN=example.org differs from the live token-exchange-aiac trust domain localtest.me)". So a 404 from the probe is not taken for the cluster's. The env table in docs/testing/ says the variable changes only that probe.

[oblinder]

Third copy of the "optional → default" scope rule. grant_exchange_scope copies _link_default_scope (src/aiac/idp/service/configuration/keycloak/main.py:333-335) line for line, and driver.sh phase WIRE (line 590) has a bash copy.

If the IdP service rule changes, the harness and the demo silently diverge from production. A small shared helper with no FastAPI import (or a lazy import) would let the harness call the real function.

[anatolykoyfman]

I kept the copies, on purpose, and made each say what it mirrors:

• The harness can't call the real function. _link_default_scope lives in the IdP Configuration Service's main.py, which configures the root logger and builds its FastAPI app when it is imported. Importing it changed logging for the whole pytest session, which is why 94f237c went back to the copy. Moving the helper into its own module would change the service image: its Dockerfile copies only main.py and requirements.txt.
• driver.sh is bash. It is now one function, grant_default_scope, used for every default link the demo makes.
• The Keycloak quirk is documented next to each copy. A DELETE of an optional link also drops a default link, which I checked on the Kind cluster. All three copies read the optional list first and delete only when the scope is there.

If you would rather have one source, the next step is a small keycloak_scopes.py next to main.py that the image also copies, imported by the service and the harness. I'd do that as a follow-up, since it touches the service build.

[oblinder]

Second code review (high effort): 5 new findings, posted inline. Four other findings repeat the first review and are not posted again; for one of them (import-time load of k8s/authbridge-rejection.py), the reviewer reproduced the failure, see the reply in that thread. The 229 unit tests that touch the changed code pass.

🤖 Generated with Claude Code

[oblinder]

Leftover-route check misses the old route's host. The check for a stale authproxy-routes matches only host: "github-tool-mcp". But the old WIRE phase replaced routes.yaml with a host: "github-tool" route, and the old harness ensure_github_tool_route appended the same route.

Failure scenario: a cluster where the old make enforce (WIRE) or the old system lane ran has authproxy-routes with only - host: "github-tool". teardown.sh finds no match and gives no warning, so the stale route stays in team1 and the operator still mounts it in every pod.

[anatolykoyfman]

Right, the old WIRE phase and ensure_github_tool_route wrote a github-tool route, which the check missed. Fixed locally, in the next commit (not pushed yet): teardown.sh now reports a leftover route for either host, github-tool-mcp or github-tool, quoted or not. As before, it reports and never deletes: the route carries no owner label, so it cannot be told from a platform-seeded one. I'll reply here with the commit once it is pushed.

[anatolykoyfman]

Pushed in f2b6d0f. teardown.sh now reports a leftover authproxy-routes route for either host, github-tool-mcp or github-tool: quoted or not, with or without a trailing comment, matching only the host key (not, say, upstream_host). It still reports and never deletes.

Evidence at f2b6d0f: unit lane 2141 passed; system lane 87 passed, 14 skipped (the test/system/disabled/ modules); the onboarding demo (make e2e, ./driver.sh --only-enforce, make restore) passes, and restore leaves team1 empty.

[oblinder]

enable.sh now fails late on a host with no PyYAML. This PR adds the python3 + PyYAML requirement here. opa-kind-enable.sh runs this script as Step 6, but its header (line 34) still says "python3 not needed here", and it has no matching preflight.

Failure scenario: on a host with no PyYAML, enable.sh builds and loads both images, runs the helm upgrade, applies the combiner and restarts every agent and tool pod. Only then does Step 6 die with "python3 with PyYAML not found". The script exits non-zero after it has already changed the cluster. Suggest the same preflight at the top of enable.sh and an updated header.

[anatolykoyfman]

Fixed in 0d0d46e: opa-kind-enable.sh checks for python3 with PyYAML at the top, before the image builds and the helm upgrade, and its header now lists the requirement instead of "python3 not needed here".

[oblinder]

A policy.forbidden body with no plugin name is counted as an OPA verdict. The cortex pipeline always fills in Violation.PluginName (pipeline.go:340), so a body that names no plugin cannot be an AuthBridge OPA rejection.

Failure scenario: a 403 {"error":"policy.forbidden","message":...} from a component that is not AuthBridge (the tool app, another proxy or middleware) is classified as deny / DENIED_HTTP 403. A deny expectation then passes in the harness and in the demo ENFORCE phase although OPA did not decide.

[anatolykoyfman]

Agreed. Fixed in 0d0d46e: is_opa_violation now requires policy.forbidden and plugin == "opa", so a body with no plugin name is not an OPA verdict (the same point as the later comment on this file; details there). I also checked the live cluster: every OPA deny it returns names "plugin":"opa".

[oblinder]

The harness and the demo driver handle HTTP 409 differently. grant_exchange_scope raises RuntimeError on any KeycloakError. The copy of the same grant in driver.sh treats 409 "already assigned" as idempotent and continues.

Failure scenario: Keycloak answers 409 to the default-scope PUT (for example, the realm-default scope already linked it in the tool-then-agent order). The harness stops the rung with RuntimeError, but driver.sh only warns and continues. Either the harness gives a false failure, or the 409 branch in the driver is dead code. (Related to the three-copies comment at line 1234.)

[anatolykoyfman]

The harness was wrong. Fixed in 0d0d46e: grant_exchange_scope takes a 409 as an existing default link, as driver.sh does, and any other Keycloak error still raises. A unit test covers both (same point as the later comment on line 1262).

[oblinder]

Unused scope parameter. exchange_token keeps a new optional scope parameter that no production caller uses. Its only caller (exchanged_subject) now sends no scope, and the only user of the parameter is the new unit test test_exchange_token_sends_an_explicit_scope.

This adds code and a test to maintain, and it makes a scoped exchange look like a supported path, but the plugin under test never sends one. Suggest removing the parameter and its test.

[anatolykoyfman]

Removed in 0d0d46e: the scope parameter, its branch, and test_exchange_token_sends_an_explicit_scope.

[anatolykoyfman]

Replies to the review of 8d83c4c are inline; all 8 are addressed in b43bd33. Two cortex issues came out of it:

• github-issue demo: make the production github-tool-mcp reachable by token-exchange-aiac cortex#1540 — the github-issue demo's production tool needs a ServiceAccount github-tool-mcp and an enrollment, so token-exchange-aiac can exchange the AIAC agent's calls to it. Per-user github-full-access is a design question there.
• token-exchange-aiac: put the refusal reason in the rejection, not only the message cortex#1570 — put the refusal reason (target_disabled / audience_not_granted) in the rejection, so the harness can stop matching the message text.

One more fix that the demo run turned up: ENFORCE checked dev-user's allow once, right after WIRE restarts the agent. github-tool's and the agent's OPAs poll bundle-service every 10-120 s, so a bundle could land after that check and fail the run. The same check is on main. It now waits one bundle cycle (BUNDLE_WAIT_SECS, default 180), before the per-tool matrix, so the matrix shows the enforced policy.

Evidence:

• Unit lane at b43bd33: 2137 passed.
• System lane: 87 passed, 0 failed, 14 skipped (the test/system/disabled/ modules), on a Kind cluster with no_token_policy: deny and the sidecar from cortex main at ef5ac5f8 (#1496, head 7122cd65). The last run was on the code before the final demo-script changes; those change only demo/use-cases/onboarding/ and docs, which the system lane does not run.
• AIAC onboarding demo (make e2e, then make restore) at b43bd33: passes end to end, and restore leaves team1 empty. WIRE skips the github-tool-mcp scope (not registered on this cluster) and grants the stub's.
• k8s/opa-kind-verify.sh and python3 k8s/authbridge-pipeline-check.py --namespace team1|team2: pass.

[oblinder]

Automated code review (/code-review): 10 findings, none a crash on the main path. Nine are inline. One is here, because its line is outside the diff:

test/system/uc1_onboard.py:1294 — the optional-to-default link move is now in three places: _link_default_scope in src/aiac/idp/service/configuration/keycloak/main.py, a copy in grant_exchange_scope, and a bash copy (grant_default_scope in driver.sh). A future fix to the move must go into all three, and they already differ on 409 handling (see the inline comment on line 1262). Moving _link_default_scope into a side-effect-free module would remove the "importing configures the root logger" reason for the copy.

Note: no verifier pass ran, and the cortex token-exchange-aiac plugin source was not checked.

🤖 Generated with Claude Code

[oblinder]

Harness and demo driver disagree on Keycloak 409. grant_exchange_scope now raises RuntimeError on every KeycloakError, including a 409 on the default-scope PUT. The driver's copy of the same move (grant_default_scope in driver.sh) treats HTTP 409 as an idempotent success.

Failure scenario: on a Keycloak version that answers 409 for a scope that is already linked as default, a re-run of the system lane (or phase 3 of the rollback test after the tool is already linked) raises and aborts the whole fixture. Before this PR the harness logged it as benign. Either the harness is wrong, or the driver's 409 branch is dead code.

[anatolykoyfman]

The harness was wrong. Fixed in 0d0d46e: grant_exchange_scope takes a 409 on the default-scope PUT as an existing link and returns, as driver.sh's grant_default_scope does. A 409 is a likely answer here: the operator's realm default scope can link the tool's audience when the agent's client is created (#248). Any other Keycloak error still raises. A unit test covers both.

[oblinder]

A rejection with no plugin name is counted as an OPA verdict. cortex's pipeline always stamps PluginName on a reject (pipeline.stampPluginName), so a policy.forbidden with no plugin never comes from AuthBridge's OPA.

Failure scenario: a plain HTTP 403 JSON body {"error":"policy.forbidden",...} from a non-AuthBridge hop (a gateway, or the tool's own handler), or a tool JSON-RPC error frame with data: {"error":"policy.forbidden"}, is classified deny / DENIED_HTTP 403. ENFORCE then passes on a deny that did not come from github-tool's OPA, and deny_origin reports TOOL_INBOUND.

[anatolykoyfman]

Agreed, checked in cortex: pipeline.stampPluginName names the plugin on every rejection. Fixed in 0d0d46e: is_opa_violation now requires policy.forbidden and plugin == "opa". A plain 403 {"error":"policy.forbidden"} with no plugin, from a gateway or the tool's own handler, is no longer deny and no longer DENIED_HTTP 403; ENFORCE fails on it.

The JSON-RPC case had a second hole: _outcome checked for an OPA verdict before it checked that the frame was AuthBridge's at all. A tool's own error frame (its own code, data: {"error":"policy.forbidden"}) read as an OPA deny. It now first asks whether the frame is AuthBridge's (code -32000, or data naming a plugin); anything else is the tool's own error, and the call was let through. deny_origin now follows outbound_outcome, so it returns nothing for that frame either. Unit tests cover both.

[oblinder]

Quarantine refusal is found by an exact match on human-readable prose. target_disabled_refusal matches "the destination's client is disabled".

Failure scenario: a small wording change in cortex (punctuation, capitals, added context) makes test_failed_tool_outbound_call_denied classify the real quarantine refusal as error, and the rollback test fails. The real fix is a machine-readable reason code (cortex#1570); until then, consider at least a TODO that links it.

[anatolykoyfman]

Agreed. The machine-readable reason is rossoctl/cortex#1570, opened from the previous round. The comment above TARGET_DISABLED_MESSAGE now links it. Until then the match fails closed: a reworded message makes the rollback test fail as error, it never passes wrongly.

[oblinder]

Bundle-wait retry matches on a truncated body. The loop greps upstream.unreachable in the verdict text, which carries only body[:200], not the parsed error code.

Failure scenario: Go marshals map keys sorted, so description/details come before error. With a long description, upstream.unreachable falls after character 200, the match fails, and the loop breaks at once, so the demo dies on a transient state it should wait out. verdict() could emit the parsed error code instead.

[anatolykoyfman]

Right, Go marshals the keys sorted. Fixed in 0d0d46e: verdict() now writes the parsed error code right after the status, UNEXPECTED <code> <error> <body[:200]> (- when there is none). The bundle-wait loop matches that field ("UNEXPECTED 200 upstream.unreachable "*), so a long description cannot push it out. A unit test builds a frame with a 300-character description.

[oblinder]

Teardown no longer sweeps all client-credentials Secrets. It now deletes only the two names recomputed from scn.AGENT_WORKLOAD / scn.TOOL_WORKLOAD with a copied operator hash.

Failure scenario: the agent runs under an AIAC_AGENT_DEPLOYMENT that is not scn.AGENT_WORKLOAD, or the operator changes the hash/key format of KeycloakClientCredentialsSecretName. The real Secret is not deleted (--ignore-not-found hides it), and the full-to-pristine teardown leaves stale credentials for the next run.

[anatolykoyfman]

Partly changed in 0d0d46e. The teardown still deletes by exact name, on purpose: the namespace also holds other workloads' credentials, for example the production github-tool-mcp's, which a prefix sweep would delete. Two changes cover the cases you name:

• The name now comes from one place, k8s/operator-names.py, which both the harness and restore.sh use. A unit test pins it against the operator's Go formula.
• After the delete, the teardown lists the namespace's credentials Secrets and logs a warning for any whose owner reference (the operator sets the workload's Deployment as owner) is one of the two workloads. A changed hash, or a Secret left under another name, is reported instead of passing in silence.

AIAC_AGENT_DEPLOYMENT names only the Deployment restart_agent rolls. The client and its Secret are named from scn.AGENT_WORKLOAD, which that Deployment's manifest also uses.

[oblinder]

scope parameter is now unused in production. After this change no production caller passes scope to exchange_token; only a unit test (_exchange(scope="openid extra")) uses it. Consider removing the parameter, its branch, and that test case.

[anatolykoyfman]

Removed in 0d0d46e: the scope parameter, its branch and test_exchange_token_sends_an_explicit_scope. exchange_token now always sends what token-exchange-aiac sends: audience only, no scope.

[oblinder]

Redundant alias. _tool_aud_scope() is now only workload_aud_scope(scn.TOOL_WORKLOAD). Call that directly at the single call site in grant_exchange_scope and delete _tool_aud_scope.

[anatolykoyfman]

Done in 0d0d46e: _tool_aud_scope is gone, and grant_exchange_scope calls workload_aud_scope(scn.TOOL_WORKLOAD).

[oblinder]

Second copy of the operator Secret-name hash. client_credentials_secret_name copies the formula (sha256 of ns\0workload\0..., first 16 hex chars) that demo/use-cases/onboarding/restore.sh already has inline. Both must change together when rossoctl/operator changes. A shared helper under k8s/ (as this PR does for the classifier) would keep one copy.

[anatolykoyfman]

Done in 0d0d46e: k8s/operator-names.py (standard library only) holds the operator's names, the credentials Secret and the audience scope. restore.sh runs it and the harness loads it, the same way as the other two shared k8s/ files. A unit test checks it against the names the operator's Go code gives.

[oblinder]

Test module imports private names from another test module. It imports _CONFIG, _PASSING, _SKIPPING, _reason from test.unit.test_launcher_outcome and edits sys.path to do so. A rename in that file breaks collection here, and its module-level setup runs twice. The shared configs belong in a conftest or a small fixtures module.

[anatolykoyfman]

Fixed in 0d0d46e. The shared configs moved into test/unit/pipeline_configs.py, a plain module with no tests. Both test files import it by its public names, so neither reaches into the other's private names or runs its setup.

[anatolykoyfman]

Replies to the review of b43bd33 are inline; all are addressed in 0d0d46e. On the point outside the diff (the optional-to-default move in three places): the harness copy now agrees with the driver on 409, which was the drift you found. The reason it cannot import _link_default_scope is unchanged. Importing main.py sets up the service's logging and app, and the service image ships only main.py. The shared-module follow-up I offered in the previous round still stands.

Evidence at 0d0d46e:

• Unit lane: 2141 passed.
• System lane: 87 passed, 0 failed, 14 skipped (the test/system/disabled/ modules), on the Kind cluster with no_token_policy: deny. The teardown left no credentials Secret of the demo workloads.
• AIAC onboarding demo (make e2e, then make restore): passes end to end, and restore leaves team1 empty.

[anatolykoyfman]

Decisions in this PR, and where they are recorded

Every design decision behind this PR now has a written reason in the repo, in the spec for its feature (the repo's convention, docs/agents/domain.md):

Routing and grants: PRD D36 (new, under Key architectural decisions)

• The outbound leg runs token-exchange-aiac with no routes; a target is granted only by a default client scope of the caller.
• no_token_policy: deny, not client-credentials, which would let any process in the agent pod reach a target as the agent.
• Service name must equal ServiceAccount name. The production github-tool-mcp doesn't yet: github-issue demo: make the production github-tool-mcp reachable by token-exchange-aiac cortex#1540, which cortex closed as deferred; tracked here in Task: exchange the demo github-agent's calls to the production github-tool-mcp under token-exchange-aiac #247.
• github-full-access is not granted as a default scope: every user would get the privileged PAT.
• The operator's realm default audience scopes widen the grant (Task: the operator's realm default audience scopes grant every later client every earlier target #248). Each callee's inbound OPA still decides (D34).
• What was rejected, and why.

Test harness: uc1-onboarding-pipeline.md, Testing Decisions

• Only a policy.forbidden from a named opa is an OPA verdict. An exchange refusal, OPA with no bundle, and a tool's own error are not.
• The rules shared with scripts live once each in k8s/: the wiring check, the classifier, and the operator's names.
• A missing shared file is a clean skip for the live suite, and a failing unit test for the classifier's own tests.
• The wiring check requires what AIAC's Rego reads, in order.
• Teardown deletes by exact name, so other workloads' objects survive, and warns about a demo Secret that survives.
• Why the harness keeps its own copy of the optional-to-default link move.

Demo: demo.md, Step 5 How ENFORCE waits, and why

• ENFORCE waits one bundle cycle before the matrix, because the OPAs poll their bundles.
• It restarts the agent instead of waiting out the unknown-audience cache.
• It doesn't retry a 401.

This commit (db37855)

A delta review of 0d0d46e...f2b6d0f found 8 points. Fixed:

• make enforce restarted the agent twice. It runs WIRE and ENFORCE as two processes. ENFORCE now skips its restart when the agent restarted in the last 2 minutes.
• restore.sh could abort before deleting anything if k8s/operator-names.py failed. It now warns, skips only the audience scopes, and removes the clients, CRs, workloads and Secrets as before. Step 3 names each component's scope from that component, not a two-way ternary.
• An ENFORCE comment overstated its read-only checks. It said a failing cluster is "left as it was"; the comment now names the one write before them, the idempotent ROPC default-scope link.
• Small cleanups: the demo's loader for operator-names.py is cached and names a missing file; one import moved to the top block; the shared-configs module's docstring says it holds a helper.

Not changed: removing the 401 retry. With the cache cleared by the restart, a 401 is a real audience refusal. No demo run here has shown a transient one. demo.md records this.

Evidence at db37855:

• Unit lane: 2141 passed.
• System lane: 87 passed, 0 failed, 14 skipped (the test/system/disabled/ modules).
• Demo: make e2e, make enforce and ./driver.sh --only-enforce all pass, then make restore leaves team1 empty. make enforce now restarts the agent once: its ENFORCE step logged "restarted 33s ago by WIRE … no second restart". --only-enforce, run 150 s later, restarted it itself.

[oblinder]

Automated code review (Claude Code). 9 findings, checked against head db37855, most severe first. Eight are inline comments. The most severe finding is on a line outside the diff, so it is here:

1. Correctness: ensure_default_audience_scope does not promote an existing OPTIONAL link (demo/use-cases/onboarding/lib/setup_keycloak.py:75)

ensure_default_audience_scope is now called with aud_scope_name from 05-onboard-tool.py and run(). It never moves an existing OPTIONAL link to default before add_client_default_client_scope. The _link_default_scope move that this PR copies into driver.sh and grant_exchange_scope does this, but this function does not. Keycloak skips the default link without an error, so the function reports "added" but no default grant exists.

Failure scenario: a cluster where an older driver WIRE linked agent-team1-github-tool-aud as an optional scope. 05-onboard-tool.py logs "added ... as a default scope", but the scope is still optional only. token-exchange-aiac requests no scope, so every agent->github-tool call is refused with 403 policy.forbidden (plugin token-exchange-aiac), while the demo onboarding reports success.

No doc in the diff gives pytest a path, and I found no clear repo CLAUDE.md violation.

🤖 Generated with Claude Code

[oblinder]

Correctness: an OPA evaluation error is classified as a deny. is_opa_violation treats every OPA policy.forbidden as a policy verdict. But cortex's OPA plugin also returns Deny("policy.forbidden", "OPA decision error: ...") on an evaluation error (core/plugins/opa/plugin.go:449/501). So a Rego runtime or decision error is classified as deny / DENIED_HTTP 403.

Failure scenario: a generated bundle whose Rego fails at evaluation. github-tool's inbound answers 403 {error: policy.forbidden, plugin: opa, message: 'OPA decision error: ...'}. outbound_outcome returns deny, and the driver's test-user check says "correctly denied by github-tool's inbound". Deny expectations false-pass. The PR says the classifier now prevents this ("an OPA deny is a policy.forbidden verdict only").

[anatolykoyfman]

Fixed in e6188c9, on both legs.

Root cause. cortex's OPA plugin answers an evaluation error with policy.forbidden too: "OPA decision error: ..." and "OPA response decision error: ..." (cortex core/plugins/opa/plugin.go:449, :501). Only the message tells it from a verdict. A plain 403 has the message at the top of the body; a frame at HTTP 200 has it in error.message, outside error.data.

Change

• k8s/authbridge-rejection.py:70: is_opa_violation(doc, message) now requires the message. A default could not read it from a frame's data, so leaving it out would bring the bug back.
• k8s/authbridge-rejection.py:86: opa_verdict(code, body) and opa_decision_error(code, body) read the message from the right place in both shapes. outbound_outcome, verdict, deny_origin, notification_outcome and probe_hint all go through them.
• Inbound. My self-review found the same bug on the inbound leg; it was not in your comment. inbound_outcome (test/system/launcher.py:699) counted any 403 as a deny. It now needs an OPA verdict in the body. The demo driver's probe_agent makes the same check on a wanted 403 (demo/use-cases/onboarding/driver.sh:697). A live inbound deny body was {"error":"policy.forbidden","message":"policy denied","plugin":"opa"}, so a real deny still passes.

Result for your scenario. An evaluation error is now error, never deny. The driver gives UNEXPECTED 403 policy.forbidden …, so its test-user check fails instead of passing, and deny_origin returns None.

Tests (test/unit/test_launcher_outcome.py): both shapes; the response variant; a deny whose reason only mentions the phrase (still a deny); opa_verdict / opa_decision_error on raw answers; inbound_outcome.

Recorded in: uc1-onboarding-pipeline.md, Testing Decisions ("only a named opa deny is one"), and the runbook's Reading a decision table (k8s/opa-kind-runbook.md:590).

Residual risk. The two prefixes are copied from cortex, and this match does not fail closed. If cortex rewords them, an evaluation error reads as a deny again, as before this fix. A machine-readable reason in the rejection would remove the copy. rossoctl/cortex#1570 asked for one for the exchange plugin's refusals (target_disabled), and cortex closed it as deferred. The prefixes sit in one constant, OPA_DECISION_ERROR_PREFIXES (k8s/authbridge-rejection.py), next to the cortex lines they copy.

[anatolykoyfman]

Follow-up in e677824: the residual risk in my reply above is now recorded as a deliberate, accepted risk, with its rationale, not left as a loose limitation.

Where: docs/testing/uc1-onboarding-pipeline.md:802, Testing Decisions ("Accepted risk: the evaluation error is told apart by its message text"). The constant OPA_DECISION_ERROR_PREFIXES in k8s/authbridge-rejection.py points there. The PR body's Merge Danger section lists it as an accepted risk.

Why it is accepted

• Without the check, every evaluation error reads as a deny. So the check can only remove false passes, never add one.
• If cortex rewords the prefix, detection falls back to the behaviour before this PR, not to something worse.
• The allow side does not depend on it: an evaluation error never reads as an allow. So every allow expectation still fails on a Rego that does not evaluate.

Rejected alternatives

• Matching cortex's exact deny message, "policy denied" (its text for allow: false with no reason; AIAC's Rego sets none). It would fail closed, but it pins a different cortex string and breaks on any Rego that sets a reason. Each such deny would read as error, which the convergence polls retry until they time out.
• Waiting for a machine-readable reason code. token-exchange-aiac: put the refusal reason in the rejection, not only the message cortex#1570 asked for that for the exchange plugin's refusals, and cortex closed it as deferred.

Revisit when cortex puts a reason code in its rejections: key on it, and drop the prefixes.

[oblinder]

Correctness: the 120 s rule can skip the restart that clears the cache. ENFORCE skips its cache-clearing agent restart when the newest github-agent pod started less than 120 s ago. It does this whatever caused the start, not only when WIRE restarted it.

Failure scenario: the agent pod restarts (a crash, the operator's final rollout, a manual delete) before github-tool's Keycloak client exists. Then --only-enforce runs within 120 s. No restart happens, and token-exchange-aiac still holds the github-tool audience as unknown (5 min TTL). The dev-user call passes through without an exchange, and github-tool's jwt-validation answers 401, giving "UNEXPECTED 401". The bundle-wait loop treats that as non-retryable, so the driver stops at once.

[anatolykoyfman]

Fixed in e6188c9, by removing the rule together with the restart it guarded (see the opa-kind-enable.sh thread).

Change. The overlay sets unknown_audience_ttl: "0" (k8s/opa-kind-enable.sh:466), so no audience is ever cached as unknown and ENFORCE has nothing to clear. agent_age, agent_restarted_within and restart_agent_for_exchange are deleted from driver.sh.

Result for your scenario. A pod that restarted for any reason within 120 s of --only-enforce no longer matters. Nothing is cached, so the dev-user call is exchanged, not passed through as a 401.

The case this leaves open, and its guard. The running agent could still have an old sidecar config. The webhook rebuilds authbridge-config-github-agent from the namespace ConfigMap only when it admits the pod. So ENFORCE checks that per-agent config (check_agent_wiring, driver.sh:249) and stops with "run k8s/opa-kind-enable.sh again". I checked this on the cluster by setting that ConfigMap back to the 5 m default.

Evidence. make e2e and ./driver.sh --only-enforce pass with no agent restart.

[oblinder]

Stale comment / missed case. The leftover-Secret warning matches owner references only against the scenario workload names (scn.AGENT_WORKLOAD / TOOL_WORKLOAD). So the case that the comment names ("the agent ran under another Deployment", that is AIAC_AGENT_DEPLOYMENT) is never reported. The old prefix sweep that deleted those Secrets is gone.

Failure scenario: AIAC_AGENT_DEPLOYMENT=my-agent. The operator's Secret is owned by my-agent, which is not in {github-agent, github-tool}. The exact-name delete misses it and the warning does not fire. A stale client-credentials Secret stays in team1 across runs with no log line, although the comment says it is reported.

[anatolykoyfman]

Fixed in e6188c9, by making the comment true rather than widening the match.

Why not widen the match. AIAC_AGENT_DEPLOYMENT only named the Deployment that restart_agent restarted. It never changed what the harness deploys: deploy_workload always applies the demo manifest, so the Deployment is always github-agent. The operator's controller reference on the Secret (clientregistration_controller.go:600, SetControllerReference) always names it too. So the case in your scenario cannot occur through the harness.

Change

• restart_agent is gone (see the opa-kind-enable.sh thread), so AIAC_AGENT_DEPLOYMENT and its two entries in the testing docs are gone too.
• The comment (test/system/uc1_onboard.py:1152) names the only case left: an operator naming change that the exact-name delete misses. The owner check catches that.

Recorded in: uc1-onboarding-pipeline.md, Testing Decisions ("Teardown removes the demo workloads' own objects by exact name").

[oblinder]

Altitude: fix the cache TTL once, not with restarts. Three places work around the 5-minute unknown_audience_ttl cache with agent restarts: the driver WIRE, the ENFORCE restart with its agent_restarted_within age heuristic, and the harness restart_agent. If this dev overlay sets the plugin's unknown_audience_ttl to a few seconds, the root cause goes away.

Each new caller of a new target needs its own restart logic, and the age heuristic is already fragile (see the comment on driver.sh). A runbook user who adds a tool still sees un-exchanged calls and 401s for up to 5 minutes.

[anatolykoyfman]

Agreed, done in e6188c9. Your suggestion is now the design.

Change

• k8s/opa-kind-enable.sh:466 sets unknown_audience_ttl: "0". At a TTL of 0 or less the plugin's cache does nothing (cortex tokenexchangeaiac/cache.go, has/add).
• All three workarounds are removed: WIRE's restart, ENFORCE's restart with the age rule, and the harness's restart_agent (with its call in the rollback test, and AIAC_AGENT_DEPLOYMENT). No rung now asserts a bundle that a restart re-fetched. The convergence polls already wait for each OPA's next bundle poll.

So it cannot regress silently. The wiring check requires a zero Go duration, as a string (k8s/authbridge-pipeline-check.py:161). I checked the matcher (:66) against Go's time.ParseDuration: "0", "0s", "-0s" and ".0s" pass. Unset (the plugin's 5 m), "5m", "0.5s", "00" and a bare YAML 0 fail; the plugin's string field refuses a bare 0 at sidecar start. The driver preflight, opa-kind-verify.sh and the harness skip gate run that check. A cluster on an older overlay is told to re-run k8s/opa-kind-enable.sh instead of failing later with a 401.

The running agent's own config. A sidecar runs authbridge-config-<agent>, which the webhook rebuilds only when it admits the pod. So WIRE and ENFORCE also check authbridge-config-github-agent (demo/use-cases/onboarding/driver.sh:249). The harness needs no such check: onboarded_stack deploys the agent after its skip gate passed.

Cost. One Keycloak token request per call to an in-cluster host that has no client; today, the agent's calls to the unregistered github-tool-mcp (#247). Hosts outside the cluster are never asked about.

Rejected alternatives: keeping the restarts; waiting out 5 minutes; a short non-zero TTL, which still lets a new target's first calls pass through un-exchanged.

Recorded in: PRD D36, the new bullet "No unknown-audience cache" (docs/specs/PRD.md:466) and its rejected list; uc1-onboarding-pipeline.md, Testing Decisions (:817: required value, cost, the per-agent check, rejected options); the runbook's Outbound token exchange section; demo.md Step 5.

Evidence

• System lane: 87 passed, 0 failed, 14 skipped, with no agent restart.
• make e2e passes.
• A live check with a throwaway target. A call to an in-cluster host with no client passed through. I created its Keycloak client, and the next call, one second later with no restart, went through the exchange: 403 audience_not_granted. With the 5 m TTL it would have passed through again.

[oblinder]

Cleanup: dead variable. AGENT_AUD_SCOPE is computed (and can set NAMES_OK=0) but nothing reads it. Step 3 calls aud_scope again for each component at line 150 instead of using the names computed at the top.

Result: a failure of the github-agent name sets the "audience scopes are not removed" warning, although step 1 never uses that name. Step 3 also runs operator-names.py again with different error handling (|| true, empty AUD_NAME). The same names have two sources.

[anatolykoyfman]

Fixed in e6188c9. Each name now has one source: the three assignments at the top of restore.sh, and AGENT_AUD_SCOPE is read.

Change

• Step 3 takes each workload's own name from those variables (demo/use-cases/onboarding/restore.sh:134: github-agent → AGENT_AUD_SCOPE, github-tool → TOOL_AUD_SCOPE). It no longer calls aud_scope again with || true.
• An empty name (operator-names.py failed; the warning is printed) matches no scope, so the prefixed github-*. scopes are still deleted. Before, an empty AUD_NAME was compared to the scope names as it was.
• The comment above the assignments says which step reads which name, so the warning's claim is accurate.

Evidence. make restore leaves team1 with no demo objects.

[oblinder]

Reuse: duplicated helpers. Step 1 inlines the same json_eval expression as the new scope_id_in in driver.sh (next(... "absent") if isinstance(d, list) else ""). Also, aud_scope_name in lib/_lib.py re-implements the load-by-path cache of launcher._load_k8s_module for k8s/operator-names.py.

The "absent" vs "" contract and the k8s/ module loader each exist in two copies. A change to one (for example a new sentinel, or a clearer ImportError message for a broken file, as the launcher has) can silently diverge from the other. This goes against the PR's own "one copy of each shared rule" aim.

[anatolykoyfman]

Fixed in e6188c9, both duplications.

Shell. json_eval, aud_scope, realm_client_scopes and scope_id_in now live once, in the new demo/use-cases/onboarding/lib/keycloak.sh, and driver.sh and restore.sh both source it. The "absent" / "" contract is written once (lib/keycloak.sh:35). Restore step 1 calls scope_id_in, and steps 1 and 3 call realm_client_scopes.

Python. aud_scope_name in lib/_lib.py has no loader or cache of its own now. It reads k8s/operator-names.py with runpy.run_path (lib/_lib.py:41), which touches no sys.modules. A missing file is a FileNotFoundError that names it.

Why one Python loader cannot be shared. docs/testing/testing-strategy.md (Dependency direction) forbids the harness depending on demo/, and lib/_lib.py is meant to run without the test/ tree. So _load_k8s_module stays in the harness, and the demo reads the same k8s/ file directly. The rule has one copy, in k8s/operator-names.py.

Evidence. make e2e and make restore pass with the shared file.

[oblinder]

Cleanup: unused public API. The new harness wrappers exchange_refusal, is_exchange_refusal and target_disabled_message (and the public classify in authbridge-rejection.py) have no system-lane caller. Only unit tests use them.

This adds harness API surface and docstrings to maintain that no live test uses. The unit tests can call the k8s/authbridge-rejection.py module directly, as test_launcher_outcome already does for classify through launcher._rejection().

[anatolykoyfman]

Fixed in e6188c9.

Change

• exchange_refusal, is_exchange_refusal and target_disabled_message are removed from test/system/launcher.py. The wrappers left are the ones the harness calls (outbound_outcome, rejection_doc, opa_verdict, opa_decision_error, target_disabled_refusal at :770), and the comment above them says so.
• Their unit tests call k8s/authbridge-rejection.py through launcher._rejection(), as test_launcher_outcome already did.
• The public classify is gone too. Its docstring moved to _classify, which verdict already used.

Evidence. Unit lane: 2197 passed.

[oblinder]

Cleanup: misleading error without python3. The top-level AGENT/TOOL/MCP_AUD_SCOPE assignments run python3 k8s/operator-names.py before the Preflight step that checks that python3 is on PATH (line 389).

Failure scenario: on a host without python3, the driver exits 1 with "k8s/operator-names.py failed" before Preflight runs. It does not give the clear "missing required command on PATH: python3" message that Preflight exists to give.

[anatolykoyfman]

Fixed in e6188c9.

Change. The three *_AUD_SCOPE assignments now come right after Preflight's command -v loop (demo/use-cases/onboarding/driver.sh:379). A failure goes through die "k8s/operator-names.py failed", so --collect-logs captures it too. The helpers they call come from lib/keycloak.sh, which defines functions only and runs nothing when sourced.

Result for your scenario. With no python3 on PATH, the driver stops at Preflight with "missing required command on PATH: python3".

[anatolykoyfman]

Replies to the review of db37855 are inline. All 9 findings are addressed in e6188c9, and each reply gives the change with file:line, the reason, the test, and where the decision is recorded.

The finding outside the diff (ensure_default_audience_scope did not promote an existing OPTIONAL link): fixed. It now makes the same move as _link_default_scope (demo/use-cases/onboarding/lib/setup_keycloak.py:83): if the scope is linked as optional, it removes that link, and only then links it as default. Removing the optional link only when one exists matters, because Keycloak's DELETE of an optional link also drops a default one. I reproduced your scenario on the Kind cluster: agent-team1-github-tool-aud linked to github-agent as optional only. The old code reported "added" and left it optional; the new code leaves it default. uc1-onboarding-pipeline.md, Testing Decisions, now lists this copy beside the driver's and the harness's.

The largest change follows your opa-kind-enable.sh comment. The overlay sets unknown_audience_ttl: "0", the wiring check requires it, and every agent restart is gone from the driver and the harness. PRD D36 records why, the cost, and the rejected options.

Two more fixes from my own review, both in the same commit:

• The inbound classifiers also counted an OPA evaluation error (or any 403) as a deny.
• With no restart, an agent admitted before the overlay changed would keep the old cache in its own sidecar config. WIRE and ENFORCE now check that config.

Evidence at e6188c9

• Unit lane: 2197 passed.
• System lane: 87 passed, 0 failed, 14 skipped (the test/system/disabled/ modules), on the Kind cluster re-enabled with the new overlay, with no agent restart.
• AIAC onboarding demo (make e2e, then make restore): passes end to end, and restore leaves team1 empty.
• k8s/opa-kind-verify.sh and k8s/keycloak-spi-verify.sh: all checks pass.