feat: add MCP012/AI013/AI014 detection rules, fix a real MCP011 FP - #35
Merged
Merged
Conversation
…peed up regression
MCP012 flags an MCP server config launched via a raw shell interpreter
(bash/sh/cmd/powershell). npx/uvx-style launchers at least require a
package-registry fetch to weaponize; a bare shell interpreter has no such
gate — the "args" field IS the payload. This closes the gap the
MCPoison/CVE-2025-54136 rug-pull pattern (and the 2026 Miasma worm
campaign) exploits: editing an already-trusted committed config's launcher
after approval. Added to advisories.ts as well: @lanyer640/mcp-runcommand-server,
a documented npm backdoor (reverse shell on install) missing from the
curated malicious-package list.
AI013 flags a schema-validated LLM result (generateObject/streamObject)
whose field is executed as a shell command or spliced into a second LLM
prompt with no content check. A schema proves shape ("command" is a
string), not content — a model can still put a shell command or an
injection payload into a schema-conformant field. This is the structured-
output analogue of AI012 (which flags output that skipped validation
entirely): AI013 flags output that passed validation and, precisely
because of that, gets treated as trusted downstream.
AI014 (Python-only; new python-scanner.ts checker + typesafe_sdk import
gate) flags request-controlled input flowing into a TypeSafe-style
system_one() decision call whose confidence/noul score directly gates a
dangerous execution sink, with no independent check on the underlying
input. A confidence score measures the model's certainty about its own
answer, not whether the input that produced it was safe — the same
"typed does not mean sanitized" blind spot as AI013, for the emerging
typed-decision-model category.
Also fixed a real false positive found while re-validating the regression
gate: isFetchLikeCall (shared by AI010/MCP011) matched "request(" as a
raw substring, so `extra.sendRequest(...)` — an MCP protocol call to the
connected client (sampling/elicitation), not an HTTP fetch — false-
positived on modelcontextprotocol/servers. Rewrote it to match callee
identifier/property names via the AST instead.
scripts/regression-scan.js: sparse-checkout llama_index and litellm
(core+vector_stores, proxy/ only) instead of full clones — same rule
coverage, full regression run in minutes instead of timing out.
Verified against all 10 regression repos: MCP012/AI013/AI014 all clean
(zero findings, so zero false positives) across openai-node, both
Anthropic SDKs, the MCP typescript-sdk and servers repos, vercel/ai,
llama_index, anthropic/skills, cisco-ai-defense/skill-scanner, and
litellm. servers' false positive is gone; ai's genuine pre-existing
MCP011 finding and llama_index's sparse-scope baseline changes are
reviewed and recorded in regression-baseline.json. npm test: 158/158.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
@lanyer640/mcp-runcommand-server(documented npm backdoor) toadvisories.ts.generateObject/streamObject) whose field is executed as a shell command or spliced into a second LLM prompt with no content check. Schema proves shape, not content — the structured-output sibling of AI012.typesafe_sdkimport gate): flags request-controlled input flowing into a TypeSafe-stylesystem_one()decision call whose confidence/noul score directly gates a dangerous execution sink, with no independent check. Verified live against the real Jev model via OpenRouter: a benign ticket correctly escalated at confidence 1.00, an adversarial one hit 0.99 confidence to auto-run a remediation command.isFetchLikeCall(shared by AI010/MCP011): a substring match on"request("matchedextra.sendRequest(...)(an MCP protocol call, not an HTTP fetch), found re-validating the regression gate againstmodelcontextprotocol/servers.scripts/regression-scan.js: sparse-checkoutllama_indexandlitellminstead of full clones — same rule coverage, full regression run finishes in minutes instead of timing out.Test plan
npm run build && npm test— 158/158 passingnpm run regression— clean across all 10 repos (openai-node, both Anthropic SDKs, MCP typescript-sdk + servers, vercel/ai, llama_index, anthropic/skills, cisco-ai-defense/skill-scanner, litellm); baseline updated with reviewed entries onlytypesafe_sdkexample repo (rajivkuriakose/typesafe-jev-examples) — 0 findings even at--paranoid🤖 Generated with Claude Code