Fix SSRF allow-list bypass via URL parser differential

[Pino de Candia (pinodeca)]

Summary

• parse each outbound request URL once with reqwest's URL parser
• use that same canonical URL for scheme enforcement, endpoint checks, and transport
• apply the validation path consistently to regular and multipart requests
• retain restricted-build HTTPS enforcement
• update the security documentation and unreleased changelog
• add unit and end-to-end regressions for backslash authority termination, exact domains, suffixes, and IP literals

Security impact

The previous allow-list extracted a host with a hand-written parser, while reqwest interpreted the URL using WHATWG rules. A backslash could therefore terminate the authority for reqwest but remain part of the authority seen by the allow-list, allowing validation of a different host from the request target.

Validation

• cargo fmt -p pg_durable -- --check
• cargo check --features pg17,http-allow-test-domains
• cargo clippy --features pg17,http-allow-test-domains -- -D warnings
• cargo test --features pg17,http-allow-test-domains ssrf::tests (61 passed)
• ./scripts/test-e2e-local.sh --verbose 06_http_and_ssrf (passed)

[Copilot]

Pull request overview

This PR hardens df.http() SSRF protections by eliminating URL-parser differentials: it parses request URLs once using the same WHATWG-based parser that reqwest uses, validates that canonical Url (scheme + allow-list), and then sends that exact value. It also applies the same validation path to multipart requests and documents the updated security model.

Changes:

• Introduce canonical URL parsing (parse_request_url) and refactor SSRF validation to operate on Url (not raw strings) for scheme + allow-list enforcement.
• Update HTTP and multipart activities to validate and send the same parsed URL; disable proxy discovery in restricted builds.
• Add unit + E2E regressions for backslash authority termination, percent-decoding/canonicalization, exact/suffix domains, and IP literal bypass attempts; update docs and changelog.

Reviewed changes

Copilot reviewed 8 out of 9 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
tests/e2e/sql/06_http_and_ssrf.sql	Adds E2E regression covering backslash authority-termination SSRF bypass vectors.
src/ssrf.rs	Refactors SSRF scheme/allow-list validation to consume a single canonical Url; removes hand-rolled host parsing.
src/dsl.rs	Switches DSL-time scheme validation to precheck_url_scheme (advisory precheck).
src/activities/execute_multipart.rs	Parses URL once, validates canonical URL, and uses it for the multipart request.
src/activities/execute_http.rs	Parses URL once, validates canonical URL, uses it for transport; disables proxies in restricted builds; adds proxy-hardening test.
docs/http-security.md	Documents “one parse, one URL” model and proxy disabling in restricted builds.
CHANGELOG.md	Notes SSRF hardening change in unreleased security section.
Cargo.toml	Adds direct url dependency to align parsing/types with reqwest URL handling.
Cargo.lock	Records url as a direct dependency of the crate.

Suppressed comments (1)

src/activities/execute_http.rs:312

• Lock the new ENV_MUTEX for the duration of the test before setting/unsetting proxy environment variables. Without this, concurrent tests can observe partially-modified proxy env state and behave unpredictably.

        let _http_proxy_upper = EnvGuard::set("HTTP_PROXY", &proxy_url);
        let _http_proxy_lower = EnvGuard::set("http_proxy", &proxy_url);
        let _no_proxy_upper = EnvGuard::remove("NO_PROXY");
        let _no_proxy_lower = EnvGuard::remove("no_proxy");

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.