feat(dbtool): emit a CATS fixture manifest for run-integrity gating - #625
Merged
Conversation
A CATS campaign can DELETE the seeded fixtures its own refData points at. Because CATS walks paths in lexical order, an anchor path is usually fuzzed before everything nested under it, so from that moment on every nested test runs against a 404 -- and those 404s get reported as findings. The refData committed before this change already pointed at a project that no longer existed, so the damage outlives the run that caused it. Anchor decoys (#608) reduce this but nothing measured whether they worked. Write cats-fixtures.json alongside the refData it describes: every seeded id the campaign substitutes, a GET that returns 200 iff it still exists (parent ids resolved inline), the anchor path whose DELETE would consume it, and whether that anchor already has a decoy -- so a fixture that dies anyway is reported as "the decoy is not working" rather than "add a decoy". Fixtures that cannot be checked are named in an `unverifiable` map rather than omitted (credential_id has no GET, addon_id has no spec path). A gate that silently skips part of the fixture set reads as "all fixtures survived" when it really means "the ones I could see survived". The CATS runner consumes this via `cats.fixtures` to gate a run twice: after seeding (refuse to fuzz against absent fixtures) and after the campaign (a fixture the campaign destroyed makes the run invalid, so it never becomes latest.db). Verified: all 18 fixtures resolve against the live API; the manifest URLs were fault-injected to confirm the tests catch a wrong one. Refs #608 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WnrVyySuyJFqcMxB8q3pNc
This was referenced Jul 30, 2026
ericfitz
added a commit
that referenced
this pull request
Jul 30, 2026
…lidation bypass (#635) * fix(deps): point the Go client replace at v1_6_0 tmi-clients regenerated its Go clients to 1.3.0/1.6.0 and removed the v1_5_0 directory this module's `replace` pointed at, so `go build ./cmd/dbtool` failed with "replacement directory ... does not exist". That broke `make cats-seed` and therefore every CATS campaign. v1_6_0 is a drop-in: only the import path changes, no call sites move, and the version now matches the server it is generated from. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WnrVyySuyJFqcMxB8q3pNc * fix(test): tidy the nested integration module test/integration is its own module and `replace`s the parent to ../.., so the root module's dependency bumps raise its minimum versions too. Its go.mod had drifted behind, and Go refused to run the suite at all: "updates to go.mod needed; to update it: go mod tidy". The failure was quiet in the worst way. `make test-integration` still reported "22 passed, 0 failed" because the api/ package ran fine; the workflows package -- 60 tests across workflows, tier2_features and tier3_edge_cases -- never compiled. Only the skip-is-a-failure gate added in 41abfe5 turned that into a non-zero exit. Versions only, no code changes. With this the full suite runs: 82 passed, 0 failed, 9 skipped. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WnrVyySuyJFqcMxB8q3pNc * test(api): tolerate clock skew in the modified_at assertion The assertion compared the server's modified_at against the test runner's own clock with no tolerance: pm.expect(new Date(responseData.modified_at)).to.be.at.most(new Date()) That holds only while the server shares a clock with the runner. Against CLUSTER=k3s the server is a different host, and the check failed deterministically by ~2ms (…27.065Z vs …27.063Z) on every run -- normal NTP drift, not a server fault. Allow 5s of skew. Still catches a timestamp that is genuinely wrong (unset, epoch, or far-future) while surviving a remote server. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WnrVyySuyJFqcMxB8q3pNc * fix(dbtool): decoy the team-note and webhook fixtures so CATS stops eating them Run 20260730T153736Z destroyed 15 of its own seeded fixtures and the #625 integrity gate correctly marked it INVALID. Two of the losses were leaves with no decoy at all: an ordinary DELETE on /teams/{team_id}/notes/{team_note_id} and /admin/webhooks/subscriptions/{webhook_id} consumed the only seeded record, and every test nested under them then ran against a 404. Decoys existed only for anchors, on the theory that leaves are not fuzzed destructively. They are. Seed a throwaway note and webhook, and point each leaf path at them. The note hangs off the REAL team on purpose: only the /teams/{team_id} anchor is overridden, so the nested path still substitutes the real team_id and a note under the throwaway team would just 404. Both globals had to be pinned by name as well. teamNoteID and webhookID resolved via findRefByKind, which ranges over a map -- with a second record seeded it could return either one, and returning the decoy would leave the real fixture unfuzzed while the decoy absorbed both roles. This is the hazard the survey-response:0 lookup already guards against. Adds teamNoteRef/teamNotePrefix so the ref prefix and the lookup cannot drift; transformTeams now builds its prefix from the same helper. TestFixtureManifestMarksDecoyedAnchors asserted these two were undecoyed, which is exactly what this inverts. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WnrVyySuyJFqcMxB8q3pNc * feat(scripts): add a CLI for reading and writing system settings Drives GET/PUT/DELETE on /admin/settings/{key} against a running server, authenticating through a browser PKCE flow. Interactive auth is required by design: service-account tokens are denied on /admin/* (#399). Secrets go in via --value-file rather than argv, and the tool never echoes a value back after a write. Note what it cannot do. Settings supplied through the environment or a config file outrank the database, and the server returns 409 for a PUT to any such key. On Kubernetes deployments that inject config via envFrom that includes every OAuth provider key, so this is not a way to rotate provider credentials -- see #631. The 409 handler says so explicitly rather than leaving the caller to guess. Not yet exercised end to end; see the follow-up issue for reworking it onto the generated Python client. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WnrVyySuyJFqcMxB8q3pNc * fix(api): document the status codes the bulk and webhook-test handlers return CATS run 20260730T174620Z flagged 63 undocumented-response-code true positives across three operations. In every case the handler is right and the spec was incomplete, so document the behaviour rather than change it (per the documented-status-code policy). POST /admin/webhooks/subscriptions/{webhook_id}/test -> 202 (39 hits) webhook_handlers.go:358 returns StatusAccepted: the delivery is created and queued, and its outcome is reported separately via the deliveries endpoints. 202 is the honest code for that. PUT /threat_models/{threat_model_id}/assets/bulk -> 200 (23) PUT /threat_models/{threat_model_id}/repositories/bulk -> 200 (1) Both bulk upsert handlers return StatusOK unconditionally (asset_sub_resource_handlers.go:647, repository_sub_resource_handlers.go:640). Only 201 was documented. 201 is left in place on the bulk PUTs. Those handlers never actually emit it, so it looks like dead documentation, but removing a documented code is a contract change for clients rather than a spec correction -- worth a separate look. Version 1.6.0 -> 1.6.1 (manual bump, #627). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WnrVyySuyJFqcMxB8q3pNc * test(cats): classify reflected team/project uri payloads as false positives SSRFInUrlFields sets the uri field to an internal address and flags the value coming back in the response body. For teams and projects that echo is not evidence of a request the server made: the field is persisted and serialized (team_store_gorm.go:954) and nothing dereferences it. Scoped deliberately to the four team/project contract paths. URIs are genuinely fetched elsewhere -- Timmy pulls document and diagram content over HTTP (timmy_content_provider_http.go) through SafeHTTPClient and URIValidator -- and an SSRF finding there would be real, so the rule must not reach those paths. Suppresses 172 findings, 5.46% of the run's true positives. That is above the 5% review threshold, confirmed with the maintainer before adding; the share reflects how many tests the fuzzer aimed at these four paths, not a loose condition. Verified against run 20260730T174620Z: dry run and live reclassification both moved exactly 172 (3152 -> 2980 true positives), and the rule matched no 5xx. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WnrVyySuyJFqcMxB8q3pNc * fix(api): scope negotiated media types to the routes that produce them AcceptHeaderValidation kept a coarse server-wide allowlist that included four representations only two operations actually produce: text/csv and application/x-ndjson (system-audit export), application/yaml and application/graphml+xml (diagram model). The comment acknowledged the gate was blunt and assumed per-endpoint negotiation would make the precise choice -- but endpoints that do no negotiation of their own never make that choice. They passed the gate and answered 200 with JSON to a client that asked for CSV. RFC 9110 calls for 406 there, and the spec documents 406 on those operations, so this was a contract violation as well as a correctness one. CATS run 20260730T174620Z hit it 93 times, all with `Accept: text/csv`, across 83 paths that have nothing to do with CSV. Scope each type to the routes that produce it. The two handlers that call negotiateContentType still return 406 themselves for a type they do not offer, so this gate only has to stop the type being waved through everywhere else. Verified by hand before the change: application/xml, text/plain, text/html, image/png and nonsense/type were already 406, while text/csv passed the gate alongside application/json -- which is what singled out the negotiated types as the cause. Four existing subtests asserted the old server-wide behaviour on an arbitrary route. They now assert 406 there, which is the behaviour change stated as a test. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WnrVyySuyJFqcMxB8q3pNc * fix(api): close the JSON-escape bypass of Unicode validation UnicodeNormalizationMiddleware scanned the request body as raw text. In raw text a \uXXXX escape is six ASCII bytes, so every check it performs -- dangerous zero-width, BiDi overrides, Hangul fillers, fullwidth structural characters, control characters, excessive combining marks -- could be evaded simply by escaping the character instead of emitting it literally. Confirmed against a running server. POST /teams with a literal U+200B in the name is rejected 400 by this middleware. The byte-identical character sent as returns 201, and reading the team back shows the zero-width space stored in the database. Re-run the same checks on the decoded strings, walking object keys as well as values. Keys matter independently: a zero-width character inside a field name makes the field unrecognized, so it is silently dropped and the request still succeeds -- which is how CATS's ZeroWidthCharsInNamesFields got 96 2xx responses in run 20260730T174620Z by sending "responsible_parties". The raw-text scan stays: it catches literal characters, and it still runs first. The new pass only adds rejections. A body that does not parse is left to ordinary request validation rather than being rejected here. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WnrVyySuyJFqcMxB8q3pNc * test(cats): classify verified-correct 4xx rejections as false positives Ten rules covering findings from run 20260730T174620Z where the server did the right thing and CATS expected otherwise. Security-flavoured findings, all disproved by evidence rather than assumption: AUTH_ERROR_CODE_IS_NOT_A_DETAILS_LEAK (39) -- the "leaked" keyword is "unauthorized", the RFC 6749 error code, in the standard 401 body. SSTI_PAYLOAD_REFLECTED_NOT_EVALUATED (33) -- 0 of 33 responses contained 8051, the evaluated form of the 83*97 payload; all 33 contained the payload verbatim. CATS's "engine" keyword matched our own seeded member role "engineering_lead". Requiring the unevaluated payload keeps a real evaluation surfacing. SSRF_URI_VALIDATION_REJECTS_UNRESOLVABLE_400 (117) -- this is the SSRF control reporting itself. ssrf_validator.go resolves the host and checks every IP against the blocklist, and safe_http_client.go dials the validated IP instead of re-resolving, so the lookup cannot be raced by a DNS rebind. Correct rejections of mangled payloads (600 + 46 + 18) and genuine state conflicts (230 + 87 + 15): nested schema violations, undecodable base64, corrupted pagination cursors, duplicate metadata keys, restoring a non-deleted entity, and a second threat model from one survey response. Each is keyed on the error the server actually returned rather than on the fuzzer name, so an unrelated 400/409 on the same path still surfaces. Together with the MassAssignment rule this takes the run from 2,980 true positives to 1,134. Nothing matched a 5xx. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WnrVyySuyJFqcMxB8q3pNc * test(cats): classify SSE responses validated as JSON schema as false positives POST /threat_models/{threat_model_id}/chat/sessions streams text/event-stream, the only content type its 200 response declares. CATS wraps the non-JSON body in a notAJson envelope and then reports that it "does not match the corresponding schema" -- there is no application/json schema for it to match, which is correct and intentional. Found while diagnosing the "Not matching response schema" bucket: this was the one operation of 23 with no application/json schema declared, and its own evidence settles it (response_content_type is text/event-stream, exactly as documented). 12 findings. The remaining 409 in that bucket do not reproduce as schema violations and are deliberately left unsuppressed pending a fresh run -- see the analysis notes on the branch. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WnrVyySuyJFqcMxB8q3pNc * fix(scripts): bind the OAuth callback to this flow and keep tokens out of argv Security review of the new settings CLI found three problems in its browser login. All are in code added earlier on this branch. State was effectively optional. The check read if result.get("state") not in (None, state): so a callback carrying no state at all satisfied it -- the parameter was only validated when the sender chose to supply it. Worse, the branch below returned any access_token found in the callback without the state having constrained anything. Together that meant a page open in the operator's browser could issue a cross-origin GET to the listener and hand the tool an identity of its choosing for the rest of the run. The ephemeral port raises the cost of finding the listener but does not prevent it, and the window stays open for the full five-minute timeout. Now: - The handler compares state with secrets.compare_digest before it looks at anything else, so a callback that is not bound to this flow is 400'd and never even captured. A missing state fails exactly like a wrong one. - authenticate() re-checks rather than trusting a check made far away in the file. - A token delivered straight to the callback is refused. It is not bound to the PKCE verifier this process generated, so accepting it would defeat the point of using PKCE. Only a token the server mints in exchange for our code and verifier is used. - The listener binds 127.0.0.1 explicitly instead of the "localhost" name. Verified by driving the handler directly: forged callbacks with no state, with a token and no state, and with a wrong state all return 400 and capture nothing; the correctly-stated callback returns 200 and is captured. Separately, --token is replaced by --token-file. A bearer token in argv is readable by any local process via ps and lands in shell history, which is the same reason setting values already use --value-file. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WnrVyySuyJFqcMxB8q3pNc --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
A CATS campaign can DELETE the seeded fixtures its own refData points at. CATS walks paths in lexical order, so an anchor path (
/projects/{project_id}) is normally fuzzed before everything nested under it — from that point on every nested test runs against a 404, and those 404s are reported as findings.This is not hypothetical or confined to a single run: the refData committed before this change pointed at a project that no longer existed, so the damage outlived the run that caused it.
Anchor decoys (#608) already mitigate this for six anchors, but nothing measured whether they worked. That's the gap this closes.
Change
cmd/dbtoolnow writescats-fixtures.jsonalongside the refData it describes, so the two can't disagree about which ids a campaign depends on. Per fixture:Fixtures that can't be checked are named in an
unverifiablemap rather than omitted (credential_idhas no GET;addon_idhas no spec path). A gate that silently skips part of the fixture set reads as "all fixtures survived" when it really means "the ones I could see survived".How it's consumed
The CATS runner reads this via
cats.fixturesand gates a run twice:latest.dbnot repointed, pruning skipped.This is the third run-validity gate, alongside the existing transport and credential gates. Unlike those it isn't a percentage — one dead anchor silently invalidates every path beneath it.
Runner-side changes live in the
catsplugin repo; this PR is the manifest producer.Verification
latest.dbuntouched/admin/users/→/admin/quotas/users/), confirmed the test catches it, revertedmake lint0 issues,make build-serverOK,make test-unit2444 passed / 0 failedGate 2 firing on a real campaign is unit-verified only — it needs a full run, which is the next step.
Notes
Refs #608rather thanFixes #608: the validating full-campaign re-run hasn't happened yet, and aFixestrailer would auto-close the issue on merge to main.The enabling
cats.fixtureskey lives in.local/cats/config.yaml, which is gitignored by design — a fresh clone needs it re-added./cats:init's rendered template now documents it.🤖 Generated with Claude Code
https://claude.ai/code/session_01WnrVyySuyJFqcMxB8q3pNc