fix(api): gate response-example completeness and document the SAML users response

[ericfitz]

Follow-up to #654. The campaign that PR existed to justify has now run, and this closes the gap it
exposed.

The campaign: 542 → 162

Run 20260731T200650Z, 107,608 requests, 38m. Valid run — transport errors 0.07% (limit 1%),
non-FP 401s 0.08% (limit 5%), all 6 seeded fixtures survived.

category	baseline	predicted	actual
Not matching response schema	428	~4	72
Unexpected 401 (/saml/providers/{idp}/users)	19	0	0
Unexpected 403 (/me/groups/{id}/members)	12	0	0
Unexpected 200 (CheckDeletedResourcesNotAvailable)	2	0	0
Unexpected 500	1	0	0
everything else	~79	~79	~99

Every #654 remediation landed. Zero 500s across the whole run.
/saml/providers/{idp}/users went from 69/69 × 401 to 58 × 200 — genuinely reachable for the
first time — and immediately produced 18 new findings, all resolved here.

Why 72 and not 4

The narrow prediction held exactly: 4 findings on GET .../diagrams/{id}/model, the
unsatisfiable MinimalDiagramModel.metadata residual. The other 68 were the same #637 mechanism on
operations the previous run never got a 2xx from.

#654 built examples from observed response bodies — right for value accuracy, but a property
that was null or absent in the fixture never entered the example. Absent-in-fixture is not
absent-in-schema. PATCH /teams/{team_id} produced 54 findings because its example lacked
email_address and uri.

Coverage shifts, so this recurs indefinitely unless it becomes a gate.

Changes

make check-response-examples (new, wired into make validate-openapi) fails when a 2xx
response example omits a property its schema declares. It is a ratchet in both directions: a new
gap fails the build, and a baselined gap that no longer reproduces must be retired, so the file
cannot quietly re-accumulate debt. Verified by deleting email_address from the teams example — the
exact regression behind the 54 findings — and confirming the build fails.

24 example properties added — every gap the server demonstrably returns, cross-checked against
all 107,608 responses. Values come from the schema, never the corpus: an earlier pass pulled a
fuzzer payload (criticality="SONXBHYIJTGAEK...") and a real seeded user's UUID into the spec,
which is precisely the contamination #654 warned about. Value selection is now schema-first and
never takes a string from response data.

GET /saml/providers/{idp}/users gets a real response schema. It was declared
{"type": "object"} — shape entirely undocumented — and its example listed an id field the server
has never returned, while omitting internal_uuid, idp, total and last_login. Shape taken
from the handler (api/saml_user_handlers.go:96-123), not inferred from fuzzed responses. Kills 8
findings. Nobody had noticed because the endpoint returned 401 to everyone until #650.

SAML_USERS_CROSS_PROVIDER_403 false-positive rule. RandomResources expects 404 for a
nonexistent idp; the handler answers 403 before any lookup, deliberately refusing to distinguish a
foreign provider from a nonexistent one — 404 would make the endpoint a provider-existence oracle.
403 is already documented for the operation. Kills 10. Dry-run confirmed 10 matches, zero against a
5xx. At 5.8% of remaining true positives it is marginally over the skill's 5% warn threshold, but it
is pinned to an exact path, method, status and error string, and it can only ever suppress a
denial — unauthorized cross-provider access would be a 2xx, which the rule cannot match.

What is deliberately not fixed

611 gaps across 93 operations are baselined, not filled. None fires today — each was checked
against all 107k responses and never observed. They are two different bugs: some operations are
merely unexercised, but some mean the schema declares more than the server returns.
Project.team is declared a full Team while the server returns {id, name}; team.created_at
appears in 0 of 107,608 responses. Filling those would document fields the endpoint never emits
— worse than the gap, and it would reverse a deliberate #654 decision. Split out as #659 with a
concrete proposal (TeamSummary), since narrowing the schema changes generated types.

RollbackResponse.restored_entity is left alone for the same reason as
MinimalDiagramModel.metadata: an untyped object whose keys depend on which entity was rolled back.
Enumerating every restorable entity's properties would satisfy CATS and produce a nonsense document.

Verification

• make validate-openapi — 0 errors
• make generate-api (oapi-codegen v2.7.1) — embedded spec only, no type or signature changes
• make build-server, make lint — 0 issues
• make test-unit — 2451 passed, 0 failed
• make test-dev-scripts — 177 passed (13 new, pinning the walker)
• make test-integration — 82 passed, 0 failed, 9 skipped
• Security review — no HIGH or MEDIUM findings; final spec diff scanned clean of fuzzer, PII and
  fixture strings

The new tests are not decoration. The walker's first draft seeded its cycle guard with a schema's
own $ref before dereferencing it, so every $ref'd schema resolved to {} and the check
reported 3 gaps instead of 651 — passing green while blind. A gate that under-reports is worse than
no gate, because the green build gets taken as proof.

Issues filed

• chore(cats): track upstream Endava/cats#206 — CATS validates 2xx responses against example, not schema #657 — upstream example-vs-schema defect (Response validated against the declared 'example' instead of the 'schema': valid responses reported as "Not matching response schema" Endava/cats#206)
• chore(cats): CATS comma-joins array query params despite explode: true, causing 10 spurious findings #658 — CATS comma-joins array query params despite explode: true; 10 findings, 3 from
  HappyPath. Also means the severity/priority/threat_type/threat_ids filters have never
  been fuzzed with valid input — the same hidden coverage loss as chore(cats): anchor-path decoys erase ~97% of the anchor's own coverage (zero 2xx on PATCH/PUT for /threat_models/{id}) #651
• fix(api): response schemas declare properties the server never returns (611 baselined example gaps) #659 — schema/server shape divergence and the baseline burn-down
• fix(auth): transient Redis timeout on the token-blacklist check returns 500 to the client #660 — Redis i/o timeout on the token-blacklist check returns 500. Surfaced as a cascading
  TestDiagramCRUD failure mid-verification; re-run passed 82/82, and the server log confirmed the
  transient timeout, so it is not a regression from this change

HANDOFF.md is refreshed — it still said "DO NOW: merge the PR".

Refs #637, #657, #658, #659

🤖 Generated with Claude Code

https://claude.ai/code/session_01WnrVyySuyJFqcMxB8q3pNc