diff --git a/docs/index.md b/docs/index.md index 4dc7629..8c4179d 100644 --- a/docs/index.md +++ b/docs/index.md @@ -1,7 +1,7 @@ --- layout: default title: AI PatchLab Scans -description: "98 curated security scans of open-source AI agents, MCP servers and LLM apps - 23 confirmed fixes, run local-first with Semgrep, Gitleaks, Trivy and pip-audit." +description: "99 curated security scans of open-source AI agents, MCP servers and LLM apps - 23 confirmed fixes, run local-first with Semgrep, Gitleaks, Trivy and pip-audit." --- # AI PatchLab Scans @@ -20,7 +20,7 @@ remediation and confidence rules to normalize the findings. > **Want this run privately against your own codebase?** I do independent > security review of AI agents, MCP servers, and LLM apps — -> [**work with me →**]({{ '/work-with-me' | relative_url }}). 98 scans, 23 confirmed fixes, methodology in the open. +> [**work with me →**]({{ '/work-with-me' | relative_url }}). 99 scans, 23 confirmed fixes, methodology in the open. > **OpenAI just launched [Daybreak](https://openai.com/index/daybreak-securing-the-world/) and Patch the Planet.** > Same remediation loop, opposite trade-off: their path is a cloud frontier model; @@ -94,12 +94,13 @@ login and static assets. Fifty-two flagged, none reported. ## All scans -98 scans, newest first. **Findings** is the raw count the tools produced; +99 scans, newest first. **Findings** is the raw count the tools produced; **Real** is what survived curation. The gap between those two columns is the entire job. | Date | Repository | Findings | Real | Outcome | |---|---|---:|---|---| +| 2026-09-09 | [mims-harvard/ToolUniverse](scans/mims-harvard-tooluniverse.html) | 284 | 1 real — withheld | private | | 2026-09-08 | [chigwell/telegram-mcp](scans/chigwell-telegram-mcp.html) | 38 | 0 real | — | | 2026-09-07 | [realiti4/claude-swap](scans/realiti4-claude-swap.html) | 22 | 0 real | — | | 2026-09-06 | [ApodexAI/FrontierAgent](scans/apodexai-frontieragent.html) | 58 | 1 real — withheld | private | diff --git a/docs/scan-log.md b/docs/scan-log.md index cd8caca..60dfcf6 100644 --- a/docs/scan-log.md +++ b/docs/scan-log.md @@ -6,8 +6,9 @@ description: "The complete AI PatchLab scan log: every public repository scanned # Full scan log -Every scan in the series, newest first, with the summary written on the day of the scan. 98 scans. For the compact index, see the [scan log home]({{ '/' | relative_url }}). +Every scan in the series, newest first, with the summary written on the day of the scan. 99 scans. For the compact index, see the [scan log home]({{ '/' | relative_url }}). +- **2026-09-09** — [mims-harvard/ToolUniverse](scans/mims-harvard-tooluniverse.html) — 284 findings, **1 real — withheld** — a **tool-calling platform for "AI scientists"** (1.7k★, Apache-2.0, Harvard MIMS lab, ~600 wrapped scientific and biomedical APIs, 10 distinct PR authors in 60 days). The class is an **intra-repo guard differential**: a genuinely well-reasoned shared security module documents two independent controls for the project’s network servers — an opt-in bearer token and a request-time Host/Origin check — and its own docstring says the second is what protects a loopback bind from DNS rebinding *when no token is set*, which is the shipped default. Every network server it fronts is consistent except two, which install one control and not the other. **No rule found this** — it came from reading the module’s docstring as a specification and tabulating which implementations satisfy it, the [contract-versus-artifact](scans/ginlix-ai-langalpha.html) move again. Reported with a runnable differential built from the project’s own helpers imported verbatim, with a positive control (the legitimate local client still gets `200`, so the fix breaks nothing) and a negative control (the project’s **own sibling surface** rejects the identical request with `421`). **The two dependency tools disagreed and both were right**: pip-audit read `pyproject.toml` and reported **0** across 156 packages, Trivy read `uv.lock` and reported **97** — open floors describe the user install (`uv pip install`), the lockfile describes the *contributor* install the developer guide prescribes (`uv sync`), and both paths are real and documented. My own `dependency-scan-unaudited-lockfile` meta-finding flagged the gap before I noticed it — twice in three days now. **82 Gitleaks hits, 0 secrets**: a repo wrapping 600 scientific databases is *made* of high-entropy strings (accession IDs like `4DNEXHVF8WA9`, dataset UUIDs), and the one real hardcoded key is the vendor’s **documented public demo key**, with the signup URL on the line above — the clearest case yet that [finding count scales with surface richness, not risk](scans/whiteguo233-openbiliclaw.html). The `exec`/`pickle`/dynamic-import tier is **product surface** for a tool platform, and the one endpoint that deserializes a caller-supplied pickle path *is* behind the guard; all four workflow shell-injection hits need write access already (`workflow_dispatch`/tag-push/schedule, no `pull_request_target` anywhere); the `render_template_string` hit is a static literal; and the [parameterized-SQL identifier FP](scans/doobidoo-mcp-memory-service.html) took its **tenth** appearance. On a codebase this careful the generic checklist is exhausted immediately — what is left is finding where the project is inconsistent **with itself**. PVR enabled → private route (`GHSA-mv53-jxjr-hp8g`, in triage) · reported privately, post-only - **2026-09-08** — [chigwell/telegram-mcp](scans/chigwell-telegram-mcp.html) — 38 findings, **zero real after curation; thirty-second clean scan** — an **MCP server that hands an agent a whole Telegram account** (1.6k★, Apache-2.0, 80+ tools over Telethon, 37 merged PRs from 26 authors in 60 days). **The entire `Critical`+`High` tier is one dependency file no install path touches**: Trivy read `poetry.lock` (`h11 0.14.0`, `mcp 1.6.0`, `starlette 0.46.2`, `rsa 4.2`, `pyasn1` …) and reported twenty advisories — but the `pyproject.toml` has **no `[tool.poetry]` section**, the README installs with `uv sync` (`uv.lock` pins `mcp 1.29.0` / `starlette 1.6.0` / `h11 0.16.0` / `rsa 4.9.1`, all patched), the Dockerfile installs from `requirements.txt` open floors that resolve to current releases, and the Poetry lines in the Dockerfile are **commented out**. A committed lockfile that nothing generates and no installer reads, driving the whole scary tier — the inverse of [OpenBiliClaw](scans/whiteguo233-openbiliclaw.html), where two lockfiles were *both* real install paths. **The scariest line, "DNS Rebinding Protection Disabled by Default in MCP", is inert**: the shipped `mcp 1.29.0` has it **on** by default — verified by executing the SDK, `allowed_hosts` pinned to `127.0.0.1:*`/`localhost:*`/`[::1]:*` and passed into the streamable-HTTP session manager — and the compose binds `127.0.0.1` only. The rest is placeholder Gitleaks (`.env.example` API-hash, a README proxy-secret example, two test fixtures), ten GHA mutable-tag hardening hits, a SHA-1 that names a lock file (not auth), and an [`os.chmod(dir, 0o700)`](scans/realiti4-claude-swap.html) the rule wants to *loosen* — one scan after five of the same. **The defenses a scanner cannot credit are the story**: file-path tools are **deny-all** unless the MCP client advertises Roots (server roots need an explicit opt-in), prompt-injection is handled by a structural JSON boundary rather than a keyword denylist, the session string is never logged, and an `install_guard` refuses to run if the installed distribution points at the PyPI name-squat. The one honest tool signal was the pip-audit coverage note on `requirements.txt`. Not strict-norm (no `SECURITY.md`, PVR off) · post-only, nothing filed - **2026-09-07** — [realiti4/claude-swap](scans/realiti4-claude-swap.html) — 22 findings (20 above the medium floor), **zero real after curation; thirty-first clean scan** — a **multi-account credential switcher for Claude Code** (2.4k★, MIT, eight months old, 52 merged PRs from 27 authors and 40 closed issues from 31 authors in 60 days): a tool whose entire job is reading, copying, refreshing, exporting and handing off OAuth refresh tokens, chosen as the stress test for the credential-handling question set four days after the [davinci-resolve-mcp](scans/samuelgursky-davinci-resolve-mcp.html) token-in-log finding. **Full coverage for once** — Semgrep 55 files, 0 skipped, `errors: []`; Trivy on `uv.lock` and pip-audit on `pyproject.toml` both empty and agreeing. **Every one of the 22 collapsed**, and five of them are the clearest [active-harm false positive](scans/roflcoopter-viseron.html) yet: `insecure-file-permissions` fired five-for-five on `os.chmod(dir, 0o700)`, the project *tightening* its directories, with a remediation that would loosen them. Four `dynamic-urllib` hits on four constant URLs (Anthropic's token/profile/usage endpoints and PyPI); the lone High is Gitleaks on Claude Code's own *public* OAuth client id (13th placeholder-tier vote); the logger rule matched a parameter name in scope. **The hand sweep is a defence inventory, and several entries close classes this series has filed elsewhere**: `/usr/bin/security` pinned by absolute path with secrets passed over stdin so they never reach argv (the davinci class, anticipated); every credential writer on `mkstemp` with a docstring explaining why write-then-chmod is wrong; a salvage copy that uses `copy` not `copy2` because a prior cut *measured* a `primaryApiKey` landing world-readable; session launch scrubbing five auth-override env vars then `execvpe`; import validating email and slot before either reaches a filename; export that strips device-bound tokens and machine identity and refuses an envelope claiming to be encrypted; OAuth error classification by the RFC 6749 `error` member rather than a substring. **One hardening note, published in full because it is a race with a one-line fix, not a vulnerability**: `_write_json` is the single writer of eight that still does write-then-chmod, and four of its call sites write the *live* `~/.claude.json` — a temp file in `$HOME`, a directory the project does not own, at umask mode for the duration of the write; the file is secret-adjacent (MCP server `env`/`headers`), verified against a current install, and the fix is the project's own `atomic_write_json`, already used at seven sibling sites. Graded Low, three preconditions, not filed — the [Nth-implementation-that-differs](scans/zilliztech-memsearch.html) shape on file writers. Not strict-norm (no `SECURITY.md`, PVR off) · post-only, nothing filed - **2026-09-06** — [ApodexAI/FrontierAgent](scans/apodexai-frontieragent.html) — 58 findings (58 above the medium floor), **1 real — withheld** — an **agent runtime, terminal product and evaluation suite** (1.8k★ two weeks after going public, Apache-2.0, Apodex AI): ReAct and coordinator-plus-sub-agents workflows over a task-scoped sandbox, a fifty-module plugin tool tree, and an optional Gradio demo for a Hugging Face Space, with a real `SECURITY.md` (named mailbox, 48h ack, public vuln issues forbidden) and private vulnerability reporting enabled. **Every one of the 58 collapsed on inspection** — loader imports keyed on a runtime class, an execution epoch logged as a "token" (seven-for-seven false), `sk-test-…` placeholders inside the project's own *leak-guard test suite*, operator-configured endpoints behind the `urllib` rule, the sandbox shell itself behind `shell=True`, a tar extractor that both resolves members and passes `filter="data"`. **The real finding is the composite class no rule sees:** a download allow-list one directory too wide, in a component whose README records — as *verified* — that the sibling directory it exposes is never served to a browser; the [docstring-as-oracle](scans/hkuds-openopc.html) diff between that paragraph and one launch argument is the whole finding. Verified by execution both ways: the artifact served, then the one-line remedy keeping deliverables downloadable while the artifact returns 403. Same-session only, filed Low; class only here · reported privately via GitHub PVR, post-only diff --git a/docs/scans/mims-harvard-tooluniverse.md b/docs/scans/mims-harvard-tooluniverse.md new file mode 100644 index 0000000..27afbce --- /dev/null +++ b/docs/scans/mims-harvard-tooluniverse.md @@ -0,0 +1,142 @@ +--- +layout: default +title: "mims-harvard/ToolUniverse: security scan" +date: 2026-09-09 +--- + +# mims-harvard/ToolUniverse — security scan + +**Repository:** [mims-harvard/ToolUniverse](https://github.com/mims-harvard/ToolUniverse) +**Commit scanned:** `7a0ceb2` +**Scan date:** 2026-09-09 +**Disclosure status:** reported privately — detail withheld pending maintainer response + +## Summary + +| Severity | Count | +| --- | ---: | +| Critical | 1 | +| High | 140 | +| Medium | 140 | +| Low | 0 | +| Info | 3 | + +**Total findings:** 284 (1 real after curation, reported privately) + +## Top findings + +**Withheld.** ToolUniverse has GitHub private vulnerability reporting enabled, which +is a signalled preference for a private channel, so the report went there +(GHSA-mv53-jxjr-hp8g, filed 2026-09-09, currently in triage) rather than into a public +issue. The specifics stay out of this page until the maintainers have had a chance to +respond. What can be said without handing anyone a recipe: + +- **One Medium finding**, reported privately. The class is an **intra-repo guard + differential**: the project has a well-reasoned shared security module that documents + two independent controls for its network servers, and applies both consistently across + every network server it fronts — except for two, which install one control and not + the other. In the shipped default configuration the missing control is the only one that + would have applied. +- Two smaller supporting items were included in the same report: a dependency-pinning + issue described below, and three calls that disable TLS certificate verification. + +The thing worth saying publicly is *how* it was found, because no rule found it. No +scanner emitted this finding. It came out of taking the security module's own docstring +as a specification and checking it against every implementation — the same +contract-versus-artifact move that has produced most of the real findings in this +series. The module's docstring states plainly which control is the one that protects the +default deployment. Once that sentence is read as a requirement rather than as prose, the +question becomes mechanical: *which surfaces satisfy it?* Building that table is what +surfaced the two that do not. + +The report shipped with a runnable differential: two apps built from the project's own +security helpers, imported verbatim, differing only in the guard wiring each surface +actually installs. It has a positive control (the legitimate local client still gets +`200`, so the proposed fix does not break normal use) and a negative control (the +project's own sibling surface rejects the attack request). A finding that can show the +maintainer their own code both accepting and rejecting the same request is much harder to +misread than a prose description of a threat. + +## Patterns observed + +**284 findings, one of them real, and the ratio is not an indictment of anything.** +ToolUniverse wraps several hundred scientific and biomedical APIs. That means the +repository is full of accession identifiers (`4DNEXHVF8WA9`), dataset UUIDs, and species +strings — which is to say, full of high-entropy tokens. Gitleaks produced 82 hits. Every +one of them is a biomedical identifier, a test fixture, or documentation, except a single +hardcoded key that the line above it documents as the vendor's *public demo key*, with +the URL where anyone can get their own. This is the clearest example yet in the series of +a rule I keep re-learning: **finding count scales with surface richness, not with risk.** +A repo that integrates 600 scientific databases will out-score a genuinely dangerous +500-line service on every secret scanner, forever. + +**The two dependency tools disagreed, and both were right.** pip-audit reported +**0 vulnerabilities** across 156 packages. Trivy reported **97**, including the only +Critical. The instinct is to decide which one is broken. Neither is: they opened +different files. pip-audit resolved `pyproject.toml`, where every declared floor is open +(`flask>=2.0.0`), so it described the install path a user gets — current, patched +versions. Trivy read `uv.lock`, which pins specific older versions, and described the +path a *contributor* gets, because the project's own developer guide tells contributors +to run `uv sync`. Both install paths are real, both are documented, and they resolve to +materially different dependency trees. My own scanner flagged this as a coverage gap +before I noticed it — the `dependency-scan-unaudited-lockfile` meta-finding fired, +saying pip-audit had not read the shipped lockfile. That check has now paid for itself +twice in three days. + +**Where the noise came from is itself a finding about scanning tool platforms.** The +`exec()`, `pickle`, and dynamic-import hits are not defects here; running +caller-supplied tool definitions is what ToolUniverse *is*. For projects whose purpose is +executing code, those primitives are product surface, and the useful question is not "is +there an `exec`?" but "is there an *unintended* path to it, and is the one real trust +boundary guarded?" The answer here was reassuring: the one endpoint that deserializes a +caller-supplied pickle path is behind the request guard, and its docstring explains +exactly why. Similarly, four workflow shell-injection hits all sit in workflows triggered +by `workflow_dispatch`, tag push, or schedule — an attacker needs write access before any +of them matter. There is no `pull_request_target` anywhere in the repo. + +**A note on what "well-built" changes about a review.** This is a codebase where the +security module anticipates DNS rebinding, refuses to bind off-loopback without a token, +fails closed when its auth backend cannot initialize, and uses `hmac.compare_digest` for +token comparison. On code like this the generic checklist is exhausted almost +immediately — everything on it is already done. What is left is looking for places where +the project is inconsistent with *itself*, which is the only kind of finding that +survives when the maintainers are already better than the rulebook. That is also why the +report leads by quoting their own docstring back to them: the ask is self-consistency, +not compliance with an outside standard. + +## Notes on the tool + +- **The lockfile coverage meta-finding worked.** `dependency-scan-unaudited-lockfile` + correctly identified that pip-audit had resolved `pyproject.toml` and never opened + `uv.lock`. Without it the honest-looking headline would have been "pip-audit: 0 + vulnerabilities", which describes one install path and hides the other. +- **Semgrep partial coverage, 18 rule timeouts.** All in the `insecure-transport` family, + across biomedical tool modules. Low-value rules in this case, but the meta-finding + named each `rule -> file` pair, which is the point: absence of a result is not + evidence of absence. +- **The secret scanner has no notion of domain identifiers.** 82 hits, 0 secrets. A + future improvement worth considering: a confidence penalty for high-entropy strings + that appear inside `data/*.json` fixture files or match known accession-ID shapes. +- **`render_template_string` produced a false positive** on a static template literal + passed with bound variables. The rule fires on the call, not on whether the template + itself was built from formatting. +- **The parameterized-SQL identifier false positive appeared again** — its tenth + appearance in this series. Both hits f-string only a generated `?,?,?` placeholder + list and bind every value. This is now far and away the most repeated false positive + in the corpus and deserves a dedicated suppression heuristic rather than a manual + read every time. + +## Disclosure timeline + +- 2026-09-09 — scan run +- 2026-09-09 — reported privately via GitHub private vulnerability reporting + (GHSA-mv53-jxjr-hp8g), including a runnable differential repro; state: triage +- *pending* — maintainer response +- *pending* — public detail, once fixed or after a reasonable window + +## Reproduce + +```bash +git clone https://github.com/mims-harvard/ToolUniverse /tmp/scan-target +python scanner/run_scan.py --repo /tmp/scan-target --reports-dir ./reports/mims-harvard-tooluniverse --min-severity medium --ignore-samples +```