feat: agent gates — deploy-gate + auth audit (DSE-1257, DSE-1258) - #91
Conversation
|
CI note — the four red jobs here are pre-existing breakage on Root cause: Evidence it isn't this PR:
This PR's own verification, on the build server under exact CI conditions ( Sequencing: DSE-1261 should land first to restore a green baseline; this PR should not carry a dependency cap, since mixing a supply-chain pin change into a security-classified diff is exactly the kind of scope-mixing |
Two fail-closed CI gates extending warden past MCP-surface integrity into the
adjacent controls agent deployments lack.
deploy-gate (DSE-1257) verifies a deploy's evidence against a declared gate
policy: required eval suites met their thresholds, required guardrails are
active, a budget/quota is declared, and a human-approval receipt is present
when required. The gate adjudicates evidence rather than running evals, which
keeps verdicts reproducible from two JSON files, free of any eval framework's
dependency tree, and makes missing or malformed evidence an unambiguous
failure instead of a silent skip. Nine WRD-GATE-* rules.
auth audit (DSE-1258) audits MCP client/server config for remote endpoints
declaring no authentication, cleartext http:// transport, and credential
literals committed into config, reusing the existing vendor secret patterns.
Static only — no server spawn, no DNS, no network — which keeps it safe in CI
and immune to churn in the MCP auth spec. Runtime capability brokering stays
out of scope (DSE-725). Deliberately does not flag loopback servers, ${VAR}
secret references, or local stdio servers; credential literals are redacted in
findings, snippets, and SARIF.
Both reuse the check exit-code contract (0 clean / 1 finding / 2 fail closed)
and the shared SARIF + JSONL emitters, so an existing code-scanning pipeline
needs no new plumbing.
Verified on the build server under CI conditions (dev+sigstore extras,
COVERAGE_PROCESS_START): 834 passed, coverage 86.14% (floor 80), ruff clean.
All four new modules at 100% coverage.
Self-review caught a real leak in the module's own reporting path: the WRD-AUTH-PLAINTEXT-HTTP finding echoed the raw URL, so an endpoint configured as https://user:token@host would have written that credential into the finding snippet and the SARIF report — the audit widening exposure of the very thing it reports. - _host_of now drops any user:pass@ userinfo, so authority parsing cannot mistake the userinfo for the host (which also fixed a locality-detection hole: user@127.0.0.1-style values no longer confuse the loopback check). - _safe_url renders scheme://host for snippets, never the credential. - New WRD-AUTH-URL-CREDENTIAL (high): a URL-embedded credential is itself a config finding, not just something to redact. 836 passed, auth_audit.py at 100% coverage, ruff clean.
Adds examples/agent-gates/ — a six-server MCP config and pass/fail deploy
evidence — plus a regression suite that asserts the examples produce exactly
the verdicts their README claims.
The demo config deliberately includes three servers that must NOT be flagged
(local stdio, loopback, and a ${VAR} secret reference). A gate that cries wolf
on correct configuration gets switched off, so the non-flags are pinned as
tightly as the findings.
test_agent_gates_examples.py also asserts the two planted fake credentials
never appear in any finding field, keeping the redaction guarantee honest
against the real shipped artifact rather than a synthetic fixture.
843 passed, coverage 86.13%, ruff clean.
…eport DSE-1256. The loudest complaint about agents in production is that they are insecure by default and nothing stops a bad config or a regressed deploy from shipping. Plenty of tools report; few return a non-zero exit a pipeline must answer for. Reframes the layer table and 'who it's for' around the common thread across all four commands: check blocks on surface drift, auth audit blocks on weak MCP auth posture, deploy-gate blocks a deploy below its declared safety bar, and all of them fail closed on missing or unreadable input.
c09be6b to
d78a14c
Compare
Closes DSE-1257, DSE-1258. Repo-side half of DSE-1256.
Two fail-closed CI gates that extend warden past MCP-surface integrity into the two adjacent controls agent deployments actually lack: did the deploy meet its declared safety bar, and is the MCP auth posture sound.
deploy-gate(DSE-1257)Verifies a deploy's evidence against a declared gate policy — required eval suites met their thresholds, required guardrails active, a budget/quota declared, and a human-approval receipt present when required. Nine
WRD-GATE-*rules.The design decision worth reviewing: the gate does not run evals, it adjudicates evidence. Running evals is the pipeline's job and is framework-specific; adjudicating them is deterministic and portable. This keeps the gate free of every eval framework's dependency tree, makes a verdict reproducible from two JSON files, and makes missing or malformed evidence an unambiguous failure rather than a silent skip.
Scope honesty, documented in
docs/AGENT_GATES.md: it does not verify the evidence is truthful. A pipeline that fabricates a score passes. Bind evidence to a trusted producer when that matters — the signed-decision path stays inPOLICY_ENFORCEMENT.md.auth audit(DSE-1258)Audits MCP client/server config for remote endpoints declaring no auth, cleartext
http://, and credential literals committed into config, reusing the existing vendor secret patterns fromcheck.Static only — no server spawn, no DNS, no network. That is what keeps it safe to run against any config in CI and, more importantly, immune to churn in the MCP auth spec. Runtime capability brokering is deliberately out of scope (DSE-725).
Precision choices (a gate that blocks CI must not cry wolf) — it deliberately does not flag loopback servers,
${VAR}/$VAR/{{ secret }}references, or local stdio servers. Every credential literal is redacted in findings, snippets, and SARIF.Contract
Both reuse the
checkexit-code contract —0clean /1finding /2unreadable input (fail closed) — and the shared SARIF + JSONL emitters, so an existing code-scanning pipeline needs no new plumbing.Verification
Run on the build server under exact CI conditions (
dev+sigstoreextras,COVERAGE_PROCESS_START=pyproject.toml):ruff check src testscleanNote for reviewers: running
pytest --covwithout the sigstore extra andCOVERAGE_PROCESS_STARTunder-reports to ~79.8% — the workflow comments already warn about this, and it is a measurement artifact, not a regression.Review note
release_control.pyclassifies this security-specific (auth_audit.pyis a security-boundary path). Merge requires one authenticated human approval perrules/release-control.md.