diff --git a/CHANGELOG.md b/CHANGELOG.md index 778c7be..d5d301b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,130 @@ # Changelog +## 1.0.0 + +Breaking: a request that names no profile now sends `modern`, which is 11 +suites and TLS 1.2 or 1.3. A server too old for that is still reached, by +the ladder falling back to `compatibility` on the second handshake. + +- **Connection profiles.** Three of them. `compatibility` offers every suite + the build provides and every protocol version back to SSLv3. `modern` offers + 11 current suites and claims nothing. `chrome131` imitates Chrome's TLS, + HTTP/2 settings and headers together. Select one with `profile=`; an unknown + name is now an error rather than silently falling back +- **Automatic shifting between them.** A handshake failure that suggests the + peer could not negotiate widens the offer; a refusal from a recognised + protection product changes what the client claims to be. Per redirect hop, + since a redirect can land on a differently protected host, and without + spending the caller's retry budget, which is a different question. A profile + that gets through is remembered per host so a scan does not re-walk the + ladder on every request. Naming a `profile`, `cipher_string`, + `min_tls_version` or `max_tls_version` pins the configuration and the ladder + does not move +- The ladder only moves on positive evidence that a product intervened, not on + any refusal. Most 403s are ordinary authorization failures, and retrying each + one with different TLS settings would double the request count of any scan + that touches one +- A host that refuses every rung is remembered too, for ten minutes: during + that time a refusal from it is taken as the answer after one request, rather + than walking the ladder again. Only success used to be remembered, so a host + that refused everything cost two requests for every one a scan sent it, on + exactly the hosts most likely to rate-limit or ban. Getting through clears + it, and it expires because a refusal is a judgement that changes +- **`Response.protection`** reports what a bot-management product did: + `(outcome, vendor)` where outcome is ok, present, challenge or blocked. A + 404 or a 500 is ok: the server answered and no product intervened +- **`Response.conclusion`** reduces that to one verdict: reached, blocked, + needs_browser, needs_legacy_tls or unreachable. `needs_browser` means a + JavaScript challenge, which no HTTP client passes, so the URL should go to a + real browser rather than be retried. A `blocked` with vendor `-` is an + ordinary refusal from the origin, not a product +- **`Response.attempts`** lists what the ladder tried, in order +- Imperva's `_Incapsula_Resource` loader script no longer counts as a + challenge. Imperva appends it to pages it let through, so ordinary pages and + the origin's own error pages were judged `needs_browser`. Imperva's incident + page still counts +- **The anti-bot rules are data.** What `protection` recognises now lives in + `antibot_rules.json` instead of code, and `blasthttp.set_antibot_rules(path)` + (`antibot::set_rules` in Rust) replaces them at runtime, so a rule can change + without a rebuild. `Vendor` is a name rather than a fixed list, so a rules + file can add a product +- **TLS failures now say why.** The reason used to be destroyed in transit: + hyper-util's error displays as the literal "client error (Connect)" for every + connect-time failure, so a cipher mismatch, a rejected certificate, a DNS + failure and a refused connection were indistinguishable. Failures now raise + `TransportError` carrying `kind`, `tls_failure`, `retryable`, `conclusion` + and `attempts`. `TransportError` subclasses `RuntimeError`, so existing + handlers keep working +- Fixes a consequence of that: TLS failures were being retried, because they + were misclassified as connection errors, contradicting the rule that says + they should not be +- Connect failures before TLS say why too. A refused connection, a DNS failure + or a proxy that refuses the tunnel now includes the cause, e.g. "tcp connect + error: Connection refused (os error 111)", not only "client error + (Connect)", and a connect timeout is reported as a timeout instead of a + retryable connection error +- Batch failures carry the same detail. `BatchResult.transport_error` is the + `TransportError` that `request()` would have raised, with `kind`, + `retryable`, `conclusion` and the rest. `error` is still the message string +- `blasthttp.mock` raises `TransportError` where the real client would: for a + URL with no mock response (`kind="connection"`) and for too many redirects + (`kind="too_many_redirects"`). A mocked batch request for an unmocked URL + gets a `transport_error` too +- Asking for a pre-TLS1.2 version without naming a profile now selects + `compatibility`, rather than failing with "no ciphers available" because the + default offers nothing that exists in SSLv3 +- `BLASTHTTP_BISECT=tls,headers,http2` disables profile layers independently, + for finding out which one a detector is reacting to +- Batches no longer have every request in an opening burst work out the same + host's profile separately. A request whose host is already being discovered + stands aside and lets a request for a *different* host take its slot, then + goes once there is an answer. Six requests to a host needing a shift went + from twelve handshakes to seven, with nothing idling: standing aside only + happens when there is undispatched work elsewhere, so when a batch is all + one host every request goes immediately and discovers it again, which is the + cheaper mistake. A slow request still never holds up faster ones behind it. + Measured over a grid of batch shapes, the duplicate handshakes go to zero in + every one: a scan of 2000 URLs across 20 hosts drops 62 handshakes, 500 URLs + across 10 hosts drops 139, and wall time is unchanged. A batch no deeper than + its own concurrency limit pays one extra round trip for it, since there the + discovery cannot be overlapped with anything +- `raw_connect`, and requests using `resolve_ip` or `request_target`, widen + their cipher offer and retry on a handshake the peer could not negotiate, + the same as an ordinary request. They stop at that rung: the other one + reacts to a refusal by changing what the client claims to be, and these are the callers who asked for exact + control over one request +- `download()` takes `profile`, `cipher_string`, `min_tls_version`, + `max_tls_version` and `redirect_cookies`. It hardcoded no profile with no + way to pass one, so a file behind a host that only answers a browser was + unreachable through `download` while the same URL through `request` was fine +- `blasthttp.mock` forwards every kwarg to the real client on a passthrough + request. It named six and dropped the rest, so a URL excluded from mocking + was dialled with different TLS, a different profile and no timeout from what + the caller asked for +- A request that names a profile no longer writes it into the per-host memory. + Reads already skipped the memory when the caller pinned something; writes + did not, so one deliberate `profile="chrome"` request turned every later + request to that host into a browser claim, and made the outcome depend on + the order two unrelated requests ran in +- Legacy ciphers are reachable, which is what the custom OpenSSL build has always been for. `compatibility` offers every suite the build provides, so a server speaking only RC4, RC4-MD5, 3DES, SEED, Camellia or anonymous DH is reachable without passing `cipher_string` yourself +- Null-encryption suites are the deliberate exception. They stay reachable through an explicit `cipher_string` but are never offered by any profile, since negotiating one by accident returns a connection that looks like TLS and encrypts nothing +- The profile's cipher list applies on both TLS paths, the pooled client and the `resolve_ip` / `request_target` / `raw_connect` path, which build their SSL contexts separately +- SSLv3 works. `min_tls_version` / `max_tls_version` / `--min-tls` / `--max-tls` accept `3.0` (also `ssl3`, `sslv3`), and an SSLv3-only server is reachable without asking for it +- The OpenSSL build cache is keyed on the feature flags and version as well as the target, so changing what the build asks for triggers a rebuild. The first run after upgrading rebuilds once +- Export ciphers are not supported. OpenSSL removed them outright in 1.1.0, so no build flag brings them back +- A TLS failure is reported only for the request that hit it. A later error to the same host, such as a refused connection, is classified on its own and retried as usual +- A `cipher_string` with no `profile` named selects `compatibility`, the same as asking for a pre-1.2 TLS version, so no TLS 1.2 floor applies and `cipher_string="RC4-MD5"` reaches a TLS 1.0 server. Naming a profile as well keeps that profile's floor +- `Content-Encoding: zstd` is decoded, including bodies of several frames. `chrome131` advertises it, as Chrome does, and Cloudflare answers with it. A frame asking for a window over 8 MiB (RFC 9659's limit) is refused and the body returned undecoded, so a response can't make the client allocate an arbitrary window +- With `verify_certs` on, anonymous (aNULL) suites are never offered, including from an explicit `cipher_string`. They send no certificate, so verification would report success while checking nothing. With verification off, nothing is excluded +- A refusal that sends the ladder to another profile is kept. If that next attempt gets no response at all (a failed handshake, a reset, a refused connection), the caller gets the refusal, with the failed attempt listed in `attempts`, rather than the error. The same holds when the timeout runs out during that next attempt: `timeout` still bounds the whole request, redirects and ladder included, but a refusal already received is returned instead of a timeout error +- Akamai's block rule only fires on a refusal status. edgesuite.net also serves customer assets, so a 200 page loading one was classified as blocked and re-sent +- A product's cookie or injected script on a refusal no longer names it as what refused. Bot-management products set their cookies and Cloudflare injects its JavaScript detections script on every response, the origin's own 401s and 403s included, so an ordinary auth failure behind a CDN walked the ladder and could report `needs_browser`. Cloudflare's own error page (a firewall-rule block such as 1020) is recognised as a Cloudflare block instead +- A handshake failure concludes `needs_legacy_tls` only when `compatibility` wasn't tried, such as when a pin stopped the ladder. Once the widest offer has failed too, as with a server wanting a client certificate, the conclusion is `unreachable` +- A connection reset during the TLS handshake is a `connection` error and retried, as before 1.0, rather than a non-retryable `tls` one. `tls_failure` still reads `reset` +- A batch's rate limit is taken when a request is sent rather than when it's spawned, so requests that stood aside for a host's first request go at the configured rate when it finishes instead of all at once +- A batch request with its TLS pinned (`profile`, `cipher_string` or a TLS version) skips host discovery. It never reads or writes the per-host memory, so it no longer stands aside for a host's first request, and never makes unpinned requests wait on it +- When the ladder switches to `chrome131` after a denial, Chrome's User-Agent and client hints replace the caller's, so the request doesn't claim Chrome 131 in its handshake and hints and another browser in its User-Agent. First contact keeps the caller's User-Agent, every other header and cookie the caller added stays, and naming the profile yourself keeps your headers as before + ## 0.10.1 - Python 3.15 support: PyO3 0.30, and 3.15 wheels are published diff --git a/Cargo.lock b/Cargo.lock index 70e07ed..22de470 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -87,12 +87,13 @@ checksum = "c4512299f36f043ab09a583e57bceb5a5aab7a73db1805848e8fef3c9e8c78b3" [[package]] name = "blasthttp" -version = "0.10.2" +version = "1.0.0" dependencies = [ "brotli", "bytes", "clap", "flate2", + "foreign-types-shared", "futures", "http", "http-body-util", @@ -100,9 +101,11 @@ dependencies = [ "hyper-util", "murmur3", "openssl", + "openssl-sys", "psl", "pyo3", "pyo3-async-runtimes", + "ruzstd", "serde", "serde_json", "thiserror", @@ -704,6 +707,15 @@ dependencies = [ "proc-macro2", ] +[[package]] +name = "ruzstd" +version = "0.9.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "69e4ce4a039f42bf25920b95667e4d789e98749ab11bbb8d2c2936f788421bb5" +dependencies = [ + "twox-hash", +] + [[package]] name = "serde" version = "1.0.229" @@ -938,6 +950,12 @@ version = "0.2.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "e421abadd41a4225275504ea4d6566923418b7f05506fbc9c0fe86ba7396114b" +[[package]] +name = "twox-hash" +version = "2.1.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "86a801b3cea342a06d468c8710662aa29e5e05e4f5c0d62f00bbb7f2ad7941c2" + [[package]] name = "unicode-ident" version = "1.0.24" diff --git a/Cargo.toml b/Cargo.toml index e9e7ba5..9434ece 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "blasthttp" -version = "0.10.2" +version = "1.0.0" edition = "2024" description = "Offensive-first HTTP library with Python bindings" license = "AGPL-3.0" @@ -45,6 +45,7 @@ http = "1" # Decompression flate2 = "1" brotli = "9" +ruzstd = "0.9.1" # Public Suffix List — decides where a cookie's `Domain` stops being allowed # to widen. Without it `Domain=com` or `Domain=github.io` is accepted and the @@ -70,5 +71,15 @@ pyo3-async-runtimes = { version = "0.30", features = ["attributes", "tokio-runti [dev-dependencies] # TLS test server — uses our custom OpenSSL to serve with weak ciphers tokio-test = "0.4" + +# Raw SSL_client_hello_get* bindings, for reading back the ClientHello we sent. +# The safe `openssl` wrapper exposes ciphers and versions but not the extension +# list, and the extension list is most of a fingerprint. Both of these are +# already in the tree as transitive dependencies of `openssl`, so they pull +# nothing new in; `foreign-types-shared` is only here for the `as_ptr()` that +# hands those bindings an `SSL *`. +openssl-sys = "0.9" +foreign-types-shared = "0.1" + # h2 test server, for checking what the client puts on the wire over HTTP/2 hyper = { version = "1", features = ["server"] } diff --git a/README.md b/README.md index 8452f99..6af36f9 100644 --- a/README.md +++ b/README.md @@ -25,7 +25,8 @@ cargo add blasthttp - **Batch connection reuse** — connections are pooled and reused within a batch, dramatically reducing overhead when scanning many URLs on the same hosts - **Rust performance** — async I/O, zero-copy where possible, and native concurrency give significant speed improvements over pure Python HTTP clients - **SSL cert info on every request** — extracts CN, SANs, emails, issuer, validity dates, and fingerprint during the TLS handshake that's already happening, eliminating the need for a separate connection to gather this information -- **All TLS ciphers available by default** — custom-compiled OpenSSL 3.3.2 with legacy provider baked in (RC4, 3DES, export ciphers, SSLv3) so you can connect to anything +- **Connection profiles, and automatic shifting between them**: `modern` by default (11 cipher suites, TLS 1.2+), `compatibility` for servers that speak only RC4, 3DES, SEED, Camellia, anonymous DH or SSLv3, and `chrome` to imitate a browser's TLS, HTTP/2 and headers together. A handshake the peer cannot negotiate widens the offer; a refusal from a recognised protection product changes what the client claims to be. When it switches to `chrome` that way, Chrome's User-Agent and client hints replace any you set, so the request claims one browser; your other headers and cookies stay. What works is remembered per host +- **Says why it failed**: a blocked request reports which product blocked it, and a JavaScript challenge reports `needs_browser`, because no HTTP client passes that tier and retrying will not change it. The products and how to recognise them are rules in [`antibot_rules.json`](antibot_rules.json), and `blasthttp.set_antibot_rules(path)` swaps in your own at runtime - **No cert validation by default** — offensive-first: connects to self-signed, expired, and misconfigured TLS without extra config - **HTTP/2 support** — automatic via ALPN negotiation, falls back to HTTP/1.1 - **Low-level primitives** — `RawConnection` for byte-level TCP/TLS I/O (bypasses HTTP framing entirely) and `blasthttp.h2` for manual H2 frame construction (includes a permissive HPACK encoder that emits bytes a strict encoder refuses, the building block for H2 smuggling / CRLF-injection tooling) @@ -115,9 +116,10 @@ Output is JSON (one object per response), including status, headers, redirect ch | `--verify` | Enable TLS cert validation | off | | `-x, --proxy` | HTTP/SOCKS proxy URL | | | `--no-proxy` | Host that bypasses the proxy: `host`, `*.suffix`, IP, CIDR, or `*` (repeatable) | | -| `--ciphers` | OpenSSL cipher string | all | -| `--min-tls` | Minimum TLS version (1.0–1.3) | | -| `--max-tls` | Maximum TLS version (1.0–1.3) | | +| `--profile` | Connection profile: `modern`, `compatibility`, `chrome` | `modern` | +| `--ciphers` | OpenSSL cipher string (`:!aNULL` is appended under `--verify`) | from the profile | +| `--min-tls` | Minimum TLS version (`3.0` for SSLv3, or 1.0–1.3) | | +| `--max-tls` | Maximum TLS version (`3.0` for SSLv3, or 1.0–1.3) | | | `-v, --verbose` | Verbose output (-vv includes body) | | ## Python API @@ -242,9 +244,10 @@ All parameters except `url` are optional: | `verify_certs` | `bool` | Enable TLS cert validation (default `False`) | | `proxy` | `str` | HTTP/SOCKS proxy URL | | `no_proxy` | `list[str]` | Hosts that bypass the proxy | -| `cipher_string` | `str` | OpenSSL cipher string | -| `min_tls_version` | `str` | Minimum TLS version (`"1.0"`–`"1.3"`) | -| `max_tls_version` | `str` | Maximum TLS version (`"1.0"`–`"1.3"`) | +| `profile` | `str` | Connection profile: `modern` (default), `compatibility`, `chrome` | +| `cipher_string` | `str` | OpenSSL cipher string. Overrides the profile's, and pins it so the ladder will not move. With no `profile` named it selects `compatibility`, so no TLS version floor applies. `:!aNULL` is appended whenever `verify_certs` is on, since an anonymous suite would skip the check | +| `min_tls_version` | `str` | Minimum TLS version (`"3.0"` for SSLv3, or `"1.0"`–`"1.3"`) | +| `max_tls_version` | `str` | Maximum TLS version (`"3.0"` for SSLv3, or `"1.0"`–`"1.3"`) | | `retries` | `int` | Number of retries on failure | | `retry_wait_min_ms` | `int` | Minimum backoff between retries (ms) | | `retry_wait_max_ms` | `int` | Maximum backoff between retries (ms) | @@ -299,6 +302,11 @@ r = await client.request( | `no_proxy` | `list[str]` | Hosts that bypass the proxy | | `headers` | `list[tuple[str, str]]` | Request headers | | `retries` | `int` | Number of retries on failure | +| `redirect_cookies` | `bool` | Carry `Set-Cookie` across redirect hops | +| `profile` | `str` | Connection profile, as on `request()` | +| `cipher_string` | `str` | OpenSSL cipher string | +| `min_tls_version` | `str` | Minimum TLS version | +| `max_tls_version` | `str` | Maximum TLS version | ### DNS Pinning & Request-Line Control @@ -442,7 +450,9 @@ async def main(): asyncio.run(main()) ``` -`raw_connect()` takes all the same TLS knobs as `request()` — `verify_certs`, `cipher_string`, `min_tls_version`, `max_tls_version`, `resolve_ip`, `proxy`, `no_proxy`. `alpn_protocols` is a list of byte-strings (commonly `["h2", "http/1.1"]`) used in the TLS ALPN extension. After the handshake, `conn.negotiated_alpn` reports which one the server picked (or `None` if no ALPN was negotiated, including all plain-HTTP connections). +`raw_connect()` takes all the same TLS knobs as `request()` — `verify_certs`, `profile`, `cipher_string`, `min_tls_version`, `max_tls_version`, `resolve_ip`, `proxy`, `no_proxy`. `alpn_protocols` is a list of byte-strings (commonly `["h2", "http/1.1"]`) used in the TLS ALPN extension. After the handshake, `conn.negotiated_alpn` reports which one the server picked (or `None` if no ALPN was negotiated, including all plain-HTTP connections). + +`raw_connect`, and requests using `resolve_ip` or `request_target`, widen their cipher offer and retry when a handshake fails in a way that says the peer could not negotiate, the same as an ordinary request. They stop there: the other half of the ladder reacts to a *refusal* by changing what the client claims to be, and these are the callers who asked for exact control over one request, so re-sending it dressed differently would answer a question they did not ask. Naming a `profile`, `cipher_string` or TLS version pins the configuration and nothing widens at all. `conn.peer_ip` returns the actual IP address the OS connected to, same as `Response.peer_ip`. `None` when the connection went through a proxy. @@ -582,7 +592,9 @@ def http_mock(): ### 1. Build custom OpenSSL -blasthttp ships a script that downloads OpenSSL 3.3.2, compiles it with weak cipher support (`RC4`, `DES`, `3DES`, export ciphers, `SSLv3`), and installs it to `vendor/openssl/install/`. This only needs to be run once — the result is cached and reused. +blasthttp ships a script that downloads OpenSSL 3.3.2, compiles it with weak cipher support (`RC4`, `DES`, `3DES`, `SEED`) and `SSLv3`, and installs it to `vendor/openssl/install/`. The result is cached, keyed on the OpenSSL version, the target, and the feature flags, so changing any of those triggers a rebuild on the next run. + +Export ciphers are the one thing the build cannot provide: OpenSSL removed them outright in 1.1.0, so no build flag brings them back. ```bash ./scripts/build-openssl.sh diff --git a/antibot_rules.json b/antibot_rules.json new file mode 100644 index 0000000..2939f2f --- /dev/null +++ b/antibot_rules.json @@ -0,0 +1,135 @@ +{ + "refusal_statuses": [401, 403, 405, 406, 429, 503], + "rules": [ + { + "vendor": "cloudflare", + "outcome": "challenge", + "when": {"header_contains": {"name": "cf-mitigated", "value": "challenge"}}, + "why": "The unambiguous one, and the only header Cloudflare sets specifically to say it intervened." + }, + { + "vendor": "cloudflare", + "outcome": "challenge", + "when": {"all": [ + {"status": "refusal"}, + {"any": [ + {"body_contains": "/cdn-cgi/challenge-platform/h/"}, + {"body_contains": "window._cf_chl_opt"} + ]} + ]}, + "why": "The challenge page's own markers, for when cf-mitigated is missing. Only on a refusal: on a 200 they're the vendor's script loading normally, as in Turnstile's api.js. The /h/ path is the challenge orchestrator; the bare /cdn-cgi/challenge-platform/ prefix also matches the JavaScript detections script Cloudflare injects into every HTML page, origin error pages included." + }, + { + "vendor": "cloudflare", + "outcome": "blocked", + "when": {"all": [ + {"status": "refusal"}, + {"body_contains": "cf-error-details"} + ]}, + "why": "Cloudflare's own error page, as served for a firewall rule block (1020). It carries no cf-mitigated header." + }, + { + "vendor": "akamai", + "outcome": "blocked", + "when": {"all": [ + {"status": "refusal"}, + {"any": [ + {"body_contains": "edgesuite"}, + {"all": [ + {"any": [ + {"header_contains": {"name": "server", "value": "akamaighost"}}, + {"header_present": "akamai-grn"}, + {"header_present": "x-akam-sw-version"}, + {"header_present": "x-akamai-transformed"}, + {"header_contains": {"name": "server-timing", "value": "ak_p"}} + ]}, + {"body_contains": "Access Denied"} + ]} + ]} + ]}, + "why": "The edge error page. It HTML-entity-encodes its own punctuation, so the body carries errors.edgesuite.net and a search for errors.edgesuite.net never fires; the bare label edgesuite survives that. server: AkamaiGHost is not reliably present either: www.akamai.com's own 403 sends no server header and identifies itself through akamai-grn, x-akam-sw-version and an ak_p server-timing entry instead, so any one of the edge headers will do. Reference # on its own stays insufficient, because plenty of unrelated error pages carry one. Only on a refusal: edgesuite.net also serves customer assets, so a page that loaded fine can name it." + }, + { + "vendor": "akamai", + "outcome": "challenge", + "when": {"all": [ + {"status": [428]}, + {"any": [{"cookie_set": "sec_cpt"}, {"body_contains": "sec-cpt"}]} + ]}, + "why": "sec-cpt is an adaptive challenge that fires after _abck already passed, and it answers 428 rather than 403. Its own outcome, not a block." + }, + { + "vendor": "datadome", + "outcome": "challenge", + "when": {"body_contains": "geo.captcha-delivery.com"} + }, + { + "vendor": "perimeterx", + "outcome": "challenge", + "when": {"any": [{"body_contains": "/px/captcha"}, {"body_contains": "px-captcha"}]} + }, + { + "vendor": "imperva", + "outcome": "challenge", + "when": {"any": [ + {"body_contains": "_Incapsula_Resource?CWUDNSAI="}, + {"all": [ + {"body_contains": "_Incapsula_Resource"}, + {"not": {"body_contains": "&ns=1&cb="}} + ]} + ]}, + "why": "The &ns=1&cb= script is the loader Imperva adds to pages it let through, the origin's own error pages included, so on its own it's not an interception. Imperva's incident page frames itself as CWUDNSAI." + }, + { + "vendor": "awswaf", + "outcome": "challenge", + "when": {"header_present": "x-amzn-waf-action"}, + "why": "The header carries the action, and there is more than one: challenge for the silent JavaScript kind, captcha for the kind wanting a person. Matching only challenge missed wizzair.com, which answers 405 with captcha, so the ladder read a genuine interception as an ordinary refusal and declined to try anything else. Match the header's presence and let the value be detail." + }, + { + "vendor": "kasada", + "outcome": "blocked", + "when": {"all": [ + {"any": [{"header_present": "x-kpsdk-ct"}, {"body_contains": "KPSDK"}]}, + {"status": [429]} + ]}, + "why": "Kasada answers 429 rather than 403 for a refusal." + }, + { + "vendor": "kasada", + "outcome": "present", + "when": {"any": [{"header_present": "x-kpsdk-ct"}, {"body_contains": "KPSDK"}]} + }, + { + "vendor": "cloudflare", + "outcome": "present", + "when": {"any": [{"cookie_set": "__cf_bm"}, {"cookie_set": "cf_clearance"}]}, + "why": "From here on, sightings of a product on a response nothing above matched. Only a sighting, even on a refusal: these products set their cookies on every response, the origin's own 401s and 403s included, so a cookie is no evidence the product is what refused." + }, + { + "vendor": "akamai", + "outcome": "present", + "when": {"any": [{"cookie_set": "_abck"}, {"cookie_set": "ak_bmsc"}, {"cookie_set": "bm_sz"}]} + }, + { + "vendor": "datadome", + "outcome": "present", + "when": {"cookie_set": "datadome"} + }, + { + "vendor": "perimeterx", + "outcome": "present", + "when": {"cookie_prefix": "_px"} + }, + { + "vendor": "imperva", + "outcome": "present", + "when": {"any": [{"cookie_prefix": "visid_incap_"}, {"cookie_prefix": "incap_ses_"}]} + }, + { + "vendor": "f5", + "outcome": "present", + "when": {"header_contains": {"name": "server", "value": "volt-adc"}} + } + ] +} diff --git a/pyproject.toml b/pyproject.toml index 3b67538..b1ded7c 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -1,6 +1,8 @@ [build-system] requires = ["maturin>=1.0,<2.0"] -build-backend = "maturin" +# TEMPORARY: restore `build-backend = "maturin"` and drop backend-path before merging into dev. +build-backend = "openssl_build_backend" +backend-path = ["scripts"] [project] name = "blasthttp" diff --git a/scripts/build-openssl.sh b/scripts/build-openssl.sh index 74a0805..eb53aee 100755 --- a/scripts/build-openssl.sh +++ b/scripts/build-openssl.sh @@ -24,6 +24,27 @@ SOURCE_DIR="${VENDOR_DIR}/openssl-${OPENSSL_VERSION}" INSTALL_DIR="${VENDOR_DIR}/install" TARBALL="${VENDOR_DIR}/openssl-${OPENSSL_VERSION}.tar.gz" MARKER="${INSTALL_DIR}/.blasthttp-built" +PATCH_DIR="${SCRIPT_DIR}/openssl-patches" + +# The features that make this build different from a stock OpenSSL. +# +# `enable-ssl3` and `enable-ssl3-method` both have to be here. OpenSSL's +# Configure has a disable cascade that reads "if ssl3-method is off, turn ssl3 +# off too", and ssl3-method is off by default. So enable-ssl3 on its own gets +# cascaded straight back to no-ssl3, which is exactly what happened here for +# a long time: the flag was passed, configdata.pm recorded both `enable-ssl3` +# and `no-ssl3`, and the build came out with OPENSSL_NO_SSL3 defined and no +# SSLv3_method symbols. +FEATURE_FLAGS=( + enable-weak-ssl-ciphers + enable-ssl3 + enable-ssl3-method + no-shared + no-module + no-tests + -fPIC +) + # --- Cross-compilation support --- TARGET="${CARGO_BUILD_TARGET:-}" @@ -48,15 +69,30 @@ case "$TARGET" in ;; esac -# Skip if already built for the same target +# Identifies what the cached build actually is. Both the target and the +# feature flags belong in here: an existing checkout whose flags changed needs +# a rebuild just as much as one whose target changed, and keying on the target +# alone means a flag change is silently ignored until someone deletes the +# install directory by hand. +# Patches are part of the recipe too: editing one has to invalidate the cached +# build exactly as changing a flag does, or the change silently does nothing. +PATCH_HASH="none" +if [ -d "$PATCH_DIR" ] && compgen -G "$PATCH_DIR/*.patch" > /dev/null; then + PATCH_HASH=$(cat "$PATCH_DIR"/*.patch | sha256sum | cut -c1-16) +fi +BUILD_RECIPE="${TARGET:-native}|${OPENSSL_VERSION}|${FEATURE_FLAGS[*]}|patches=${PATCH_HASH}" + +# Skip only if the cached build was made the same way if [ -f "$MARKER" ]; then - BUILT_TARGET=$(cat "$MARKER" 2>/dev/null || true) - if [ "$BUILT_TARGET" = "$TARGET" ]; then + BUILT_RECIPE=$(cat "$MARKER" 2>/dev/null || true) + if [ "$BUILT_RECIPE" = "$BUILD_RECIPE" ]; then echo "=== OpenSSL ${OPENSSL_VERSION} already built for '${TARGET:-native}' at ${INSTALL_DIR} ===" echo "=== Delete ${INSTALL_DIR} to force rebuild ===" exit 0 fi - echo "=== Rebuilding OpenSSL: target changed from '${BUILT_TARGET:-native}' to '${TARGET:-native}' ===" + echo "=== Rebuilding OpenSSL: build recipe changed ===" + echo "=== was: ${BUILT_RECIPE:-(unknown)}" + echo "=== now: ${BUILD_RECIPE}" rm -rf "$INSTALL_DIR" fi @@ -91,6 +127,25 @@ fi echo "Extracting..." tar xzf "$TARBALL" -C "$VENDOR_DIR" +# --- Apply our patches --- +# +# Applied to a freshly extracted tree every time, so the patches never stack +# and a failure is always a real conflict rather than a second application. +# Every patch is required: a silent skip would produce a build that looks fine +# and behaves differently, which is the failure mode that hid the SSLv3 +# problem for months. +if [ -d "$PATCH_DIR" ]; then + for patch in "$PATCH_DIR"/*.patch; do + [ -e "$patch" ] || continue + echo "Applying $(basename "$patch")" + if ! patch -p1 -d "$SOURCE_DIR" -s < "$patch"; then + echo "ERROR: failed to apply $(basename "$patch")" >&2 + echo "The patch probably needs rebasing onto OpenSSL ${OPENSSL_VERSION}." >&2 + exit 1 + fi + done +fi + # --- Find cross-compiler if needed --- find_cross_cc() { local target="$1" @@ -147,12 +202,7 @@ cd "$SOURCE_DIR" COMMON_ARGS=( --prefix="$INSTALL_DIR" - enable-weak-ssl-ciphers - enable-ssl3 - no-shared - no-module - no-tests - -fPIC + "${FEATURE_FLAGS[@]}" ) if [ -n "$openssl_target" ]; then @@ -186,9 +236,9 @@ make -j"$NUM_JOBS" echo "Installing..." make install_sw -# Mark as complete (store target for cache invalidation) +# Mark as complete (store the full recipe so a flag or target change invalidates) mkdir -p "$INSTALL_DIR" -echo "$TARGET" > "$MARKER" +echo "$BUILD_RECIPE" > "$MARKER" echo "" echo "=== OpenSSL ${OPENSSL_VERSION} built successfully ===" diff --git a/scripts/openssl-patches/0001-renegotiation-info-instead-of-scsv.patch b/scripts/openssl-patches/0001-renegotiation-info-instead-of-scsv.patch new file mode 100644 index 0000000..8754e8c --- /dev/null +++ b/scripts/openssl-patches/0001-renegotiation-info-instead-of-scsv.patch @@ -0,0 +1,67 @@ +Signal renegotiation-info support with the extension, not the SCSV + +RFC 5746 section 3.4 lets a client advertise renegotiation-info support on an +initial ClientHello in either of two ways: the +TLS_EMPTY_RENEGOTIATION_INFO_SCSV pseudo-cipher (0x00ff), or an empty +renegotiation_info extension (0xff01). + +OpenSSL picks the SCSV. Every browser picks the extension. That one choice +costs a fingerprint match twice over: it adds a cipher suite nothing else +sends and omits an extension everything else sends, so both the cipher count +and the extension set differ from any browser. + +The switch is conditioned on the minimum protocol version rather than made +unconditional, because the SCSV exists for a reason. Pre-RFC5746 servers, the +very ones that motivated it, can mishandle an unfamiliar extension. So a +client whose floor is TLS 1.2 is treated as modern and sends the extension, +while one willing to speak TLS 1.1 or older keeps the SCSV and stock +behaviour. blasthttp's compatibility path stays byte-identical to unpatched +OpenSSL; only the impersonation path changes, and it always pins its floor at +TLS 1.2. + +On an initial handshake previous_client_finished_len is zero, so the existing +WPACKET_sub_memcpy_u8 call already emits the empty renegotiated_connection +field RFC 5746 requires. No new code is needed to build the body. + +--- a/ssl/statem/extensions_clnt.c ++++ b/ssl/statem/extensions_clnt.c +@@ -16,8 +16,19 @@ + unsigned int context, X509 *x, + size_t chainidx) + { +- /* Add RI if renegotiating */ +- if (!s->renegotiate) ++ /* ++ * Add RI if renegotiating. ++ * ++ * On an initial handshake RFC 5746 3.4 allows either this extension or ++ * the SCSV. Browsers always send the extension, so a client whose floor ++ * is TLS 1.2 does too. One willing to speak older protocols keeps the ++ * SCSV, which is safer against pre-RFC5746 servers that mishandle ++ * unfamiliar extensions. ssl_cipher_list_to_bytes() is gated on the same ++ * condition so exactly one of the two is ever sent. ++ */ ++ if (!s->renegotiate ++ && (SSL_CONNECTION_IS_DTLS(s) ++ || s->min_proto_version < TLS1_2_VERSION)) + return EXT_RETURN_NOT_SENT; + + if (!WPACKET_put_bytes_u16(pkt, TLSEXT_TYPE_renegotiate) +--- a/ssl/statem/statem_clnt.c ++++ b/ssl/statem/statem_clnt.c +@@ -4063,9 +4063,14 @@ + { + int i; + size_t totlen = 0, len, maxlen, maxverok = 0; ++ /* ++ * Send the SCSV only when the renegotiation_info extension is not being ++ * sent. tls_construct_ctos_renegotiate() is gated on the same version ++ * test, so support is signalled exactly once, one way or the other. ++ */ + int empty_reneg_info_scsv = !s->renegotiate + && (SSL_CONNECTION_IS_DTLS(s) +- || s->min_proto_version < TLS1_3_VERSION); ++ || s->min_proto_version < TLS1_2_VERSION); + SSL *ssl = SSL_CONNECTION_GET_SSL(s); + + /* Set disabled masks for this session */ diff --git a/scripts/openssl_build_backend.py b/scripts/openssl_build_backend.py new file mode 100644 index 0000000..ece864f --- /dev/null +++ b/scripts/openssl_build_backend.py @@ -0,0 +1,25 @@ +# TEMPORARY: remove before merging into dev, and restore `build-backend = "maturin"` in pyproject.toml. +# Builds the vendored OpenSSL before maturin runs, so unreleased 1.0 installs from a git source with just `uv sync`. +import subprocess +import sys +from pathlib import Path + +import maturin +from maturin import * # re-exports maturin's other PEP 517 hooks + +BUILD_OPENSSL = Path(__file__).resolve().parent / "build-openssl.sh" + + +def _build_openssl(): + # the script returns immediately once OpenSSL is built + subprocess.run(["bash", str(BUILD_OPENSSL)], check=True, stdout=sys.stderr) + + +def build_wheel(wheel_directory, config_settings=None, metadata_directory=None): + _build_openssl() + return maturin.build_wheel(wheel_directory, config_settings, metadata_directory) + + +def build_editable(wheel_directory, config_settings=None, metadata_directory=None): + _build_openssl() + return maturin.build_editable(wheel_directory, config_settings, metadata_directory) diff --git a/src/antibot.rs b/src/antibot.rs new file mode 100644 index 0000000..f3fcceb --- /dev/null +++ b/src/antibot.rs @@ -0,0 +1,317 @@ +//! Working out what a bot-management product did to a request. +//! +//! A scan has to tell three things apart that all arrive looking like a +//! response: the origin answered, a protection product answered instead, or +//! the product answered and handed us a session anyway. Collapsing those is +//! what turns a protected host into an apparently dead one, which is the +//! false negative this exists to prevent. +//! +//! Two rules, both learned by getting them wrong: +//! +//! 1. Never match a vendor's name in the body. `datadome.co`, `imperva.com` +//! and `kasada.io` all scored as challenges purely from their own marketing +//! copy. Everything here keys on headers, cookies, status, or a body string +//! that cannot occur in ordinary prose. +//! +//! 2. A response that merely *references* a vendor's script is not a challenge +//! page. Every page on a protected site carries the loader; only a page +//! shaped like an interception is a block. +//! +//! The rules themselves are data, in `antibot_rules.json`: the copy built in +//! is the default, and [`set_rules`] swaps in another at runtime, so a rule +//! can change without a rebuild. + +use std::fmt; +use std::sync::{Arc, LazyLock, RwLock}; + +use serde::Deserialize; + +/// Which product answered, named by the rule that matched. Deliberately +/// conservative: guessing a vendor wrong is worse than reporting unknown. +#[derive(Debug, Clone, PartialEq, Eq, Hash)] +pub struct Vendor(String); + +impl Vendor { + pub fn new(name: impl Into) -> Self { + Vendor(name.into()) + } + + /// A refusal no rule could put a name to. + pub fn unknown() -> Self { + Vendor::new("unknown") + } + + pub fn is_unknown(&self) -> bool { + self.0 == "unknown" + } +} + +impl fmt::Display for Vendor { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + f.write_str(&self.0) + } +} + +#[derive(Debug, Clone, PartialEq, Eq)] +pub enum Outcome { + /// Ordinary response, no protection product detected. + Ok, + /// A product is in front of this host and let us through, usually having + /// set a session cookie. The most useful signal there is: we passed + /// something that was actively looking. + Present(Vendor), + /// An interactive challenge. Out of reach for any HTTP client, since + /// answering means executing the page's JavaScript. + Challenge(Vendor), + /// A hard refusal, no challenge offered. + Blocked(Vendor), + /// Never got a usable response. + Error, +} + +impl Outcome { + /// Did we get real content? The bar the benchmark scores against. + pub fn got_through(&self) -> bool { + matches!(self, Outcome::Ok | Outcome::Present(_)) + } + + /// Is there positive evidence that a protection product intervened? + /// + /// The distinction between this and "we did not get through" matters, and + /// getting it wrong costs real requests. Most 403s are ordinary + /// authorization failures: an endpoint needing a token, a path that is not + /// public, a method that is not allowed. Those classify as + /// `Blocked(Unknown)` because they are refusals with no product signature, + /// and retrying them with different TLS settings is pure waste against a + /// server that will say 403 however it is asked. + /// + /// So a named vendor is the bar. It means something recognisable answered + /// instead of the origin, which is the only case where trying a different + /// profile is worth a second request. + /// + /// The cost of that conservatism is missing a product we cannot + /// fingerprint. The cost of the alternative is doubling the request count + /// of every scan that touches a 403, which is most of them. + pub fn indicates_protection(&self) -> bool { + self.vendor().is_some_and(|v| !v.is_unknown()) + } + + pub fn vendor(&self) -> Option<&Vendor> { + match self { + Outcome::Present(v) | Outcome::Challenge(v) | Outcome::Blocked(v) => Some(v), + _ => None, + } + } +} + +/// Everything a classification looks at. Headers and cookies are expected +/// lowercase-insensitive; `headers` holds `(name, value)` pairs as received, +/// duplicates and all. +pub struct ResponseFacts<'a> { + pub status: u16, + pub headers: &'a [(String, String)], + pub body: &'a str, +} + +impl ResponseFacts<'_> { + fn header_matches(&self, name: &str, needle: &str) -> bool { + self.headers + .iter() + .any(|(k, v)| k.eq_ignore_ascii_case(name) && v.to_ascii_lowercase().contains(needle)) + } + + fn has_header(&self, name: &str) -> bool { + self.headers + .iter() + .any(|(k, _)| k.eq_ignore_ascii_case(name)) + } + + /// Cookie names set by this response. Only the name is examined; values + /// are session data and carry nothing we want to match on. + fn set_cookie_names(&self) -> Vec { + self.headers + .iter() + .filter(|(k, _)| k.eq_ignore_ascii_case("set-cookie")) + .filter_map(|(_, v)| v.split('=').next()) + .map(|n| n.trim().to_ascii_lowercase()) + .collect() + } + + fn sets_cookie(&self, name: &str) -> bool { + let want = name.to_ascii_lowercase(); + self.set_cookie_names().contains(&want) + } + + fn sets_cookie_prefixed(&self, prefix: &str) -> bool { + let want = prefix.to_ascii_lowercase(); + self.set_cookie_names().iter().any(|n| n.starts_with(&want)) + } + + fn body_contains(&self, needle: &str) -> bool { + // Bodies can be large; a challenge marker is always near the top. + const WINDOW: usize = 64 * 1024; + let mut end = self.body.len().min(WINDOW); + // The cut can land in the middle of a multi-byte character, and + // slicing a str there panics rather than erring. A body truncated by + // max_body_size puts arbitrary bytes at the cap, so this is reachable + // from the wire. Walk back to the nearest boundary. + while end > 0 && !self.body.is_char_boundary(end) { + end -= 1; + } + self.body[..end].contains(needle) + } +} + +/// The rules a classification runs, as loaded from `antibot_rules.json`. +/// +/// Order matters, and the first rule to match decides: a challenge outranks a +/// mere sighting of the vendor, and a block outranks both, because a host that +/// challenges us has still told us more than one that refuses outright. +#[derive(Debug, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct Rules { + /// Statuses a protection product uses to refuse. 429 belongs because a rate + /// limit and a bot block are the same outcome for a scan, whoever sent it. + refusal_statuses: Vec, + rules: Vec, +} + +#[derive(Debug, Deserialize)] +#[serde(deny_unknown_fields)] +struct Rule { + vendor: String, + outcome: RuleOutcome, + /// What the outcome becomes when the status is a refusal. + on_refusal: Option, + when: Condition, + /// Why the rule is shaped the way it is. Not used, but worth keeping. + #[allow(dead_code)] + why: Option, +} + +#[derive(Debug, Clone, Copy, Deserialize)] +#[serde(rename_all = "snake_case")] +enum RuleOutcome { + Present, + Challenge, + Blocked, +} + +#[derive(Debug, Deserialize)] +#[serde(rename_all = "snake_case", deny_unknown_fields)] +enum Condition { + All(Vec), + Any(Vec), + Not(Box), + Status(Statuses), + HeaderPresent(String), + /// Case-insensitive, in both the name and the value. + HeaderContains { + name: String, + value: String, + }, + CookieSet(String), + CookiePrefix(String), + BodyContains(String), +} + +#[derive(Debug, Deserialize)] +#[serde(untagged)] +enum Statuses { + Codes(Vec), + Named(NamedStatuses), +} + +#[derive(Debug, Deserialize)] +#[serde(rename_all = "snake_case")] +enum NamedStatuses { + Refusal, +} + +impl Rules { + pub fn from_json(json: &str) -> Result { + let rules: Rules = serde_json::from_str(json)?; + if rules.rules.iter().any(|r| r.vendor.is_empty()) { + return Err(serde::de::Error::custom("every rule needs a vendor")); + } + Ok(rules) + } + + /// Classify one response. + pub fn classify(&self, facts: &ResponseFacts) -> Outcome { + let refusal = self.is_refusal(facts.status); + for rule in self.rules.iter().filter(|r| self.matches(&r.when, facts)) { + let outcome = match (refusal, rule.on_refusal) { + (true, Some(outcome)) => outcome, + _ => rule.outcome, + }; + // A sighting means the product let us through, which a refusal + // isn't, and it is no evidence the product is what refused. + if refusal && matches!(outcome, RuleOutcome::Present) { + continue; + } + return outcome.with(Vendor::new(&rule.vendor)); + } + // Nothing identifiable. A 404 or a 500 is still the server answering. + if refusal { + Outcome::Blocked(Vendor::unknown()) + } else { + Outcome::Ok + } + } + + fn is_refusal(&self, status: u16) -> bool { + self.refusal_statuses.contains(&status) + } + + fn matches(&self, condition: &Condition, facts: &ResponseFacts) -> bool { + match condition { + Condition::All(conditions) => conditions.iter().all(|c| self.matches(c, facts)), + Condition::Any(conditions) => conditions.iter().any(|c| self.matches(c, facts)), + Condition::Not(condition) => !self.matches(condition, facts), + Condition::Status(Statuses::Codes(codes)) => codes.contains(&facts.status), + Condition::Status(Statuses::Named(NamedStatuses::Refusal)) => { + self.is_refusal(facts.status) + } + Condition::HeaderPresent(name) => facts.has_header(name), + Condition::HeaderContains { name, value } => { + facts.header_matches(name, &value.to_ascii_lowercase()) + } + Condition::CookieSet(name) => facts.sets_cookie(name), + Condition::CookiePrefix(prefix) => facts.sets_cookie_prefixed(prefix), + Condition::BodyContains(needle) => facts.body_contains(needle), + } + } +} + +impl RuleOutcome { + fn with(self, vendor: Vendor) -> Outcome { + match self { + RuleOutcome::Present => Outcome::Present(vendor), + RuleOutcome::Challenge => Outcome::Challenge(vendor), + RuleOutcome::Blocked => Outcome::Blocked(vendor), + } + } +} + +impl Default for Rules { + /// The rules built in from `antibot_rules.json`. + fn default() -> Self { + Rules::from_json(include_str!("../antibot_rules.json")) + .expect("the built-in antibot_rules.json is valid") + } +} + +static RULES: LazyLock>> = + LazyLock::new(|| RwLock::new(Arc::new(Rules::default()))); + +/// Replace the rules every later classification runs, in this process. +pub fn set_rules(rules: Rules) { + *RULES.write().unwrap_or_else(|e| e.into_inner()) = Arc::new(rules); +} + +/// Classify one response with the current rules. +pub fn classify(facts: &ResponseFacts) -> Outcome { + let rules = RULES.read().unwrap_or_else(|e| e.into_inner()).clone(); + rules.classify(facts) +} diff --git a/src/batch.rs b/src/batch.rs index 25a0b55..244b371 100644 --- a/src/batch.rs +++ b/src/batch.rs @@ -136,6 +136,283 @@ fn merge_limiters( } } +// ── Host discovery scheduler ────────────────────────────────────── + +/// Stops a burst of requests to one host from each separately working out +/// which connection profile that host wants. +/// +/// The problem. The client remembers, per host, the profile that got through, +/// so a scan pays for the ladder once rather than once per request. At the +/// start of a batch that has not been paid yet: if fifty requests to one host +/// go out together, all fifty find the default refused and all fifty shift, +/// a hundred handshakes where fifty one would have done. +/// +/// What does not work, and this was built and measured before being thrown +/// away: a per-host lock, so the first request discovers and the rest wait on +/// it. It does cut the handshakes, and it is slower anyway, because a waiter +/// cannot be released until the leader has classified its response, +/// classification needs the body, and so the wait is a whole request long. A +/// protection product does not refuse the handshake; it takes the connection +/// and then answers 403, which puts the one fact worth knowing in the last +/// byte to arrive. Waiting also broke the rule that a slow request never +/// holds up faster ones behind it, which is the entire promise of +/// `send_batch_stream`. +/// +/// What works instead is to not wait. A request whose host is already being +/// worked out steps aside and lets a request for a *different* host have its +/// slot, then goes once there is an answer. Nothing idles, so the saving +/// costs no time. +/// +/// The condition is the whole design, and it comes straight from the test +/// that killed the lock: six requests, one host, concurrency six. Step aside +/// there and you are standing in the street, because every other request +/// wants the same host. So: **only step aside when there is undispatched work +/// for a different host.** When there is not, go and discover it yourself and +/// accept the duplicate. That makes this strictly better or equal to having +/// no scheduler at all, never worse, which is what the lock failed to be. +struct HostGate { + /// Someone is making the first request to this host, so an answer is + /// coming. + claimed: bool, + /// Flips once a request to this host finishes. A `watch` rather than a + /// `Notify` because the answer can land between a waiter reading the + /// state and awaiting on it, and a watch carries the value so that + /// waiter sees it instead of sleeping until its timeout. + answered: tokio::sync::watch::Sender, +} + +struct SchedulerState { + gates: std::collections::HashMap, + /// Requests not yet dispatched, in total and per host. Together these + /// answer the only question the scheduler asks: is there work for a + /// different host that could use the slot I would otherwise sit in? + remaining_total: usize, + remaining_by_host: std::collections::HashMap, + /// How many requests are standing aside right now, and the ceiling on + /// that. + /// + /// The ceiling is about *spawning*, so the two callers want different + /// numbers and passing one for both was a mistake worth recording. + /// + /// The stream driver takes a permit before it spawns, which is how it + /// keeps a million-URL batch from being resident all at once. A request + /// that stands aside hands that permit straight back, so the driver + /// spawns another, which may also stand aside, and so on: a batch + /// alternating between two slow hosts would spawn the lot. There the + /// ceiling is the concurrency limit and it is load-bearing. + /// + /// `send_batch` spawns every task up front regardless. Standing aside + /// creates nothing, so a ceiling protects nothing, and setting one only + /// throws the benefit away: measured over a grid of batch shapes, the + /// duplicate handshakes went to zero exactly when the ceiling was not + /// binding and were left completely untouched when it was. There the + /// limit is the batch size, which is to say no limit at all. + aside: usize, + aside_limit: usize, +} + +pub(crate) struct Scheduler { + state: std::sync::Mutex, +} + +/// What a request should do about its host. +pub(crate) enum Dispatch { + /// Go, and you are the one discovering this host. Hold the guard for the + /// request. + Lead(LeaderGuard), + /// Go. Either the host is already answered, or there was nothing else to + /// do with the slot so discovering it twice is the cheaper mistake. + Go, + /// Stand aside until this says true, then ask again. + Aside(tokio::sync::watch::Receiver, AsideGuard), +} + +/// Held for the duration of the first request to a host. Marks the host +/// answered when dropped, so a request that fails, times out or panics +/// releases whoever stepped aside for it instead of leaving them on the +/// timeout. +pub(crate) struct LeaderGuard { + sched: Arc, + host: String, +} + +impl Drop for LeaderGuard { + fn drop(&mut self) { + if let Ok(mut st) = self.sched.state.lock() + && let Some(gate) = st.gates.get_mut(&self.host) + { + // `send` refuses when nothing is subscribed and leaves the value + // alone, which is the common case here: usually the leader + // finishes with nobody yet standing aside. The answer would then + // be lost, and the next request to this host would stand aside + // waiting for news that had already been and gone. + gate.answered.send_replace(true); + } + } +} + +/// Held while a request stands aside, so the ceiling is released however the +/// wait ends. +pub(crate) struct AsideGuard { + sched: Arc, +} + +impl Drop for AsideGuard { + fn drop(&mut self) { + if let Ok(mut st) = self.sched.state.lock() { + st.aside = st.aside.saturating_sub(1); + } + } +} + +/// Host and port, matching how the client keys its per-host profile memory. +/// A URL that will not parse gets a bucket under its own text, which just +/// means it coordinates with nothing. +fn host_key(url: &str) -> String { + let Ok(uri) = url.parse::() else { + return url.to_string(); + }; + match uri.host() { + Some(h) => { + let default_port = if uri.scheme_str() == Some("https") { + 443 + } else { + 80 + }; + format!("{}:{}", h, uri.port_u16().unwrap_or(default_port)) + } + None => url.to_string(), + } +} + +impl Scheduler { + /// `aside_limit` is how many requests may stand aside at once. See the + /// field: the stream driver needs the concurrency limit, `send_batch` + /// needs no limit and passes the batch size. + fn new(configs: &[RequestConfig], aside_limit: usize) -> Arc { + // Pinned requests never consult the per-host memory, so they take no + // part in discovering it and `await_turn` lets them straight through. + let mut remaining_by_host: std::collections::HashMap = + std::collections::HashMap::new(); + let mut remaining_total = 0; + for c in configs.iter().filter(|c| !c.tls_is_pinned()) { + *remaining_by_host.entry(host_key(&c.url)).or_insert(0) += 1; + remaining_total += 1; + } + Arc::new(Scheduler { + state: std::sync::Mutex::new(SchedulerState { + gates: std::collections::HashMap::new(), + remaining_total, + remaining_by_host, + aside: 0, + aside_limit: aside_limit.max(1), + }), + }) + } + + /// Decide and commit in one lock, because deciding and then committing + /// separately lets two requests both look, both see no one discovering, + /// and both go: the duplicate this exists to prevent. + /// + /// `may_stand_aside` is false on the second ask, after a wait, so a + /// request stands aside at most once and cannot be starved by a host + /// that keeps being re-claimed. + fn dispatch(self: &Arc, host: &str, may_stand_aside: bool) -> Dispatch { + let Ok(mut st) = self.state.lock() else { + return Dispatch::Go; + }; + + if may_stand_aside && st.aside < st.aside_limit { + let busy = st + .gates + .get(host) + .is_some_and(|g| g.claimed && !*g.answered.borrow()); + let mine = st.remaining_by_host.get(host).copied().unwrap_or(0); + let elsewhere = st.remaining_total.saturating_sub(mine) > 0; + if busy && elsewhere { + let rx = st.gates[host].answered.subscribe(); + st.aside += 1; + return Dispatch::Aside( + rx, + AsideGuard { + sched: self.clone(), + }, + ); + } + } + + // Going, so take the slot. + st.remaining_total = st.remaining_total.saturating_sub(1); + if let Some(n) = st.remaining_by_host.get_mut(host) { + *n = n.saturating_sub(1); + } + let gate = st + .gates + .entry(host.to_string()) + .or_insert_with(|| HostGate { + claimed: false, + answered: tokio::sync::watch::Sender::new(false), + }); + if gate.claimed || *gate.answered.borrow() { + return Dispatch::Go; + } + gate.claimed = true; + Dispatch::Lead(LeaderGuard { + sched: self.clone(), + host: host.to_string(), + }) + } +} + +/// How long a request stands aside before giving up and going anyway. +/// +/// Only a backstop. The leader's guard releases waiters the moment its +/// request ends, errors included, so this fires when a leader is wedged in a +/// way its own timeout did not catch. Sized off the request timeout, since a +/// leader cannot legitimately outlive that. +fn step_aside_limit(config: &RequestConfig) -> Duration { + Duration::from_secs(config.timeout().clamp(1, 60)) +} + +/// Stand aside if asked to, then report whether this request is the one +/// discovering its host. +/// +/// Takes the caller's concurrency permit, if it is already holding one, and +/// gives it back only if it never stood aside. The two callers differ on +/// exactly this: `send_batch` has not taken a permit yet and passes `None`, +/// while the stream driver hands one over with the request and needs it +/// released for the wait, since a request standing aside is not in flight +/// and holding a slot as though it were is the lock all over again. +async fn await_turn( + sched: &Arc, + host: &str, + config: &RequestConfig, + permit: Option, +) -> ( + Option, + Option, +) { + if config.tls_is_pinned() { + return (None, permit); + } + match sched.dispatch(host, true) { + Dispatch::Lead(g) => (Some(g), permit), + Dispatch::Go => (None, permit), + Dispatch::Aside(mut rx, aside) => { + drop(permit); + if !*rx.borrow() { + let _ = tokio::time::timeout(step_aside_limit(config), rx.changed()).await; + } + drop(aside); + let lead = match sched.dispatch(host, false) { + Dispatch::Lead(g) => Some(g), + _ => None, + }; + (lead, None) + } + } +} + pub async fn send_batch( client: Arc, configs: Vec, @@ -145,20 +422,31 @@ pub async fn send_batch( ) -> Vec { let semaphore = Arc::new(tokio::sync::Semaphore::new(concurrency)); let limiter = merge_limiters(shared_limiter, rate_limit); + // No ceiling here: the tasks all exist either way, so standing aside + // costs nothing that going would not have cost anyway. + let sched = Scheduler::new(&configs, configs.len()); let mut handles = Vec::new(); for config in configs { - // Rate limit: pace dispatch before spawning the task - if let Some(ref limiter) = limiter { - limiter.acquire().await; - } - let client = client.clone(); let permit = semaphore.clone(); let url = config.url.clone(); + let sched = sched.clone(); + let limiter = limiter.clone(); let handle = tokio::spawn(async move { + // Stand aside before taking a permit rather than after, so the + // slot is genuinely free while we are standing there. Waiting + // with a permit in hand would be the lock this replaces. + let host = host_key(&config.url); + let (_lead, _) = await_turn(&sched, &host, &config, None).await; + let _permit = permit.acquire().await.unwrap(); + // Paced at the send, not the spawn, so a request that stood aside + // pays when it actually goes instead of bursting with the rest. + if let Some(limiter) = limiter { + limiter.acquire().await; + } let result = client.send(&config).await; BatchResult { url, result } }); @@ -203,9 +491,8 @@ pub async fn send_batch( /// `__anext__` (a `future_into_py` block running on the tokio runtime). /// /// Concurrency is gated *before* spawn by a semaphore acquire on the driver, -/// so at most `concurrency` requests are in-flight at any time. Rate-limit -/// acquire happens before the semaphore so dispatch pacing matches -/// `send_batch`. In-flight tasks are NOT cancelled if the consumer drops +/// so at most `concurrency` requests are in-flight at any time. The rate +/// limit is taken just before each send, as in `send_batch`. In-flight tasks are NOT cancelled if the consumer drops /// the stream — they run to completion and their sends fail silently. This /// also matches `send_batch`. pub fn send_batch_stream( @@ -217,23 +504,43 @@ pub fn send_batch_stream( ) -> impl Stream + Send + 'static { let limiter = merge_limiters(shared_limiter, rate_limit); let semaphore = Arc::new(tokio::sync::Semaphore::new(concurrency)); + // Ceiling here, because this driver spawns lazily and a request that + // stands aside gives its permit back for the driver to spawn against. + let sched = Scheduler::new(&configs, concurrency); stream::once(async move { let (tx, rx) = futures::channel::mpsc::unbounded::(); tokio::spawn(async move { for config in configs { - if let Some(ref l) = limiter { - l.acquire().await; - } let permit = match semaphore.clone().acquire_owned().await { Ok(p) => p, Err(_) => break, }; let client = client.clone(); let tx = tx.clone(); + let sched = sched.clone(); + let semaphore = semaphore.clone(); + let limiter = limiter.clone(); tokio::spawn(async move { + let host = host_key(&config.url); + // The driver holds the permit here, unlike `send_batch`, + // so standing aside means handing it back. The driver is + // parked on the next acquire and takes it immediately, + // which is what sends the slot to another host. Dispatch + // order is untouched; only this one request is late. + let (_lead, permit) = await_turn(&sched, &host, &config, Some(permit)).await; + // Re-take a slot only if we gave ours up. + let permit = match permit { + Some(p) => Some(p), + None => semaphore.acquire_owned().await.ok(), + }; + let _permit = permit; + // Paced at the send, as in `send_batch`. + if let Some(limiter) = limiter { + limiter.acquire().await; + } let url = config.url.clone(); let result = client.send(&config).await; let _ = tx.unbounded_send(BatchResult { url, result }); @@ -441,6 +748,120 @@ mod tests { ); } + /// Records when each send happened. + struct Recording { + inner: MockClient, + sends: std::sync::Mutex>, + } + + impl Recording { + fn new(delay: Duration) -> Self { + Recording { + inner: MockClient::new(200, "ok".to_string()).with_delay(delay), + sends: std::sync::Mutex::new(Vec::new()), + } + } + + fn sent_at(&self, url: &str) -> Instant { + self.sends + .lock() + .unwrap() + .iter() + .find(|(u, _)| u == url) + .unwrap() + .1 + } + } + + impl HttpClient for Recording { + async fn send(&self, config: &RequestConfig) -> Result { + self.sends + .lock() + .unwrap() + .push((config.url.clone(), Instant::now())); + self.inner.send(config).await + } + } + + #[tokio::test] + async fn test_requests_that_stood_aside_still_keep_the_rate() { + // Followers of a slow host stand aside while its first request runs. + // When it finishes they must go at the configured rate, not at once. + let client = Arc::new(Recording::new(Duration::from_secs(2))); + let configs: Vec = (0..8) + .map(|i| format!("https://a.com/{i}")) + .chain((0..8).map(|i| format!("https://b.com/{i}"))) + .map(RequestConfig::new) + .collect(); + + send_batch(client.clone(), configs, 50, Some(5.0), None).await; + + let sends: Vec = client.sends.lock().unwrap().iter().map(|s| s.1).collect(); + assert_eq!(sends.len(), 16); + let busiest = sends + .iter() + .map(|t| { + sends + .iter() + .filter(|u| **u >= *t && **u < *t + Duration::from_secs(1)) + .count() + }) + .max() + .unwrap(); + // 5 a second, plus one for the bucket's first token. + assert!( + busiest <= 6, + "{busiest} sends inside one second at a rate of 5" + ); + } + + fn pinned(url: &str) -> RequestConfig { + let mut c = RequestConfig::new(url.to_string()); + c.profile = Some("chrome131".to_string()); + c + } + + #[tokio::test] + async fn test_pinned_requests_do_not_stand_aside() { + // Pinned requests learn nothing from a host's first request, so + // waiting for it is pure latency. + let client = Arc::new(Recording::new(Duration::from_millis(500))); + let configs: Vec = ["a", "a", "a", "b", "b", "b"] + .iter() + .enumerate() + .map(|(i, h)| pinned(&format!("https://{h}.com/{i}"))) + .collect(); + + send_batch(client.clone(), configs, 50, None, None).await; + + let sends: Vec = client.sends.lock().unwrap().iter().map(|s| s.1).collect(); + let spread = *sends.iter().max().unwrap() - *sends.iter().min().unwrap(); + assert!( + spread < Duration::from_millis(200), + "sends spread over {spread:?}" + ); + } + + #[tokio::test] + async fn test_a_pinned_request_does_not_lead_a_host() { + // A pinned first request never records what the host wants, so the + // unpinned requests behind it must not wait on it. + let client = Arc::new(Recording::new(Duration::from_millis(500))); + let configs = vec![ + pinned("https://a.com/pinned"), + RequestConfig::new("https://a.com/open".to_string()), + RequestConfig::new("https://b.com/other".to_string()), + ]; + + send_batch(client.clone(), configs, 50, None, None).await; + + let waited = client.sent_at("https://a.com/open") - client.sent_at("https://a.com/pinned"); + assert!( + waited < Duration::from_millis(200), + "waited {waited:?} on a pinned request" + ); + } + #[tokio::test] async fn test_rate_limit_with_concurrency() { // Rate limit AND concurrency together @@ -648,3 +1069,189 @@ mod tests { ); } } + +#[cfg(test)] +mod scheduler_tests { + use super::*; + + fn cfgs(urls: &[&str]) -> Vec { + urls.iter() + .map(|u| RequestConfig::new(u.to_string())) + .collect() + } + + fn is_aside(d: &Dispatch) -> bool { + matches!(d, Dispatch::Aside(..)) + } + + #[test] + fn test_one_host_never_stands_aside() { + // The case that killed the previous attempt, as a unit test. Six + // requests to one host: standing aside means standing in the street, + // because there is no other host to hand the slot to. Every one of + // them has to go, even though that means discovering the host six + // times over. + let sched = Scheduler::new(&cfgs(&["https://a.test/"; 6]), 6); + let host = host_key("https://a.test/"); + + let first = sched.dispatch(&host, true); + assert!(matches!(first, Dispatch::Lead(_)), "first one leads"); + + for i in 0..5 { + let d = sched.dispatch(&host, true); + assert!(!is_aside(&d), "request {i} should have gone, not waited"); + } + } + + #[test] + fn test_another_host_waiting_is_what_makes_it_worth_stepping_aside() { + // Same shape, one difference: there is work for a different host, so + // the slot this request would have sat in has somewhere to go. + let sched = Scheduler::new( + &cfgs(&["https://a.test/", "https://a.test/", "https://b.test/"]), + 4, + ); + let a = host_key("https://a.test/"); + + // Bound, not matched in place: the guard marks the host answered when + // it drops, so letting it die at the end of the statement would mean + // the discovering request had already finished. + let lead = sched.dispatch(&a, true); + assert!(matches!(lead, Dispatch::Lead(_))); + assert!( + is_aside(&sched.dispatch(&a, true)), + "b.test is still queued" + ); + } + + #[test] + fn test_the_second_ask_always_goes() { + // After standing aside once a request goes regardless, so a host that + // keeps being re-claimed cannot starve the requests behind it. + let sched = Scheduler::new( + &cfgs(&["https://a.test/", "https://a.test/", "https://b.test/"]), + 4, + ); + let a = host_key("https://a.test/"); + + let _lead = sched.dispatch(&a, true); + assert!(is_aside(&sched.dispatch(&a, true))); + assert!(!is_aside(&sched.dispatch(&a, false)), "the second ask goes"); + } + + #[test] + fn test_an_answered_host_is_not_worth_waiting_for() { + let configs = cfgs(&["https://a.test/", "https://a.test/", "https://b.test/"]); + let sched = Scheduler::new(&configs, 4); + let a = host_key("https://a.test/"); + + let lead = sched.dispatch(&a, true); + assert!(matches!(lead, Dispatch::Lead(_))); + drop(lead); // the discovering request finished + + assert!( + !is_aside(&sched.dispatch(&a, true)), + "there is an answer now, so nothing to wait for" + ); + } + + #[test] + fn test_a_leader_that_dies_releases_the_host() { + // The guard marks the host answered on drop rather than on success, + // so a request that errors or panics does not leave everyone who + // stepped aside for it sitting on the timeout. + let configs = cfgs(&["https://a.test/", "https://a.test/", "https://b.test/"]); + let sched = Scheduler::new(&configs, 4); + let a = host_key("https://a.test/"); + + let Dispatch::Lead(lead) = sched.dispatch(&a, true) else { + panic!("expected to lead"); + }; + let Dispatch::Aside(rx, _guard) = sched.dispatch(&a, true) else { + panic!("expected to stand aside"); + }; + assert!(!*rx.borrow()); + + drop(lead); + assert!( + *rx.borrow(), + "dropping the leader should release the waiter" + ); + } + + #[test] + fn test_the_ceiling_bounds_how_many_can_be_standing_aside() { + // The stream driver's limit, which is the concurrency one. Without + // it, a batch alternating between two slow hosts has the driver spawn + // a task, watch it hand the permit straight back, spawn another, and + // so on until the whole batch is resident. `send_batch` wants the + // opposite and the test above covers that. + let mut urls = vec!["https://a.test/"; 10]; + urls.push("https://b.test/"); + let sched = Scheduler::new(&cfgs(&urls), 2); + let a = host_key("https://a.test/"); + + let _lead = sched.dispatch(&a, true); + let _one = sched.dispatch(&a, true); + let _two = sched.dispatch(&a, true); + assert!( + is_aside(&_one) && is_aside(&_two), + "both fit under a ceiling of 2" + ); + assert!( + !is_aside(&sched.dispatch(&a, true)), + "the third should go rather than pile up" + ); + } + + #[test] + fn test_no_ceiling_lets_a_whole_batch_stand_aside() { + // What `send_batch` passes, and why it differs from the stream. Its + // tasks are all spawned up front, so standing aside creates nothing + // and a ceiling protects nothing. Setting one anyway only throws the + // saving away: measured across a grid of batch shapes, a ceiling of + // the concurrency limit left the duplicate handshakes completely + // untouched wherever a batch was bigger than that limit, which is + // every batch worth scheduling. + let mut urls = vec!["https://a.test/"; 40]; + urls.push("https://b.test/"); + let configs = cfgs(&urls); + let sched = Scheduler::new(&configs, configs.len()); + let a = host_key("https://a.test/"); + + let _lead = sched.dispatch(&a, true); + let aside: Vec<_> = (0..39).map(|_| sched.dispatch(&a, true)).collect(); + assert!( + aside.iter().all(is_aside), + "every one of them should wait for the answer the first is fetching" + ); + } + + #[test] + fn test_a_freed_slot_lets_another_stand_aside() { + let mut urls = vec!["https://a.test/"; 10]; + urls.push("https://b.test/"); + let sched = Scheduler::new(&cfgs(&urls), 1); + let a = host_key("https://a.test/"); + + let _lead = sched.dispatch(&a, true); + let waiting = sched.dispatch(&a, true); + assert!(is_aside(&waiting)); + assert!(!is_aside(&sched.dispatch(&a, true)), "ceiling of 1 is full"); + + drop(waiting); + assert!(is_aside(&sched.dispatch(&a, true)), "and free again after"); + } + + #[test] + fn test_host_key_ignores_the_path_and_fills_in_the_port() { + assert_eq!( + host_key("https://a.test/one"), + host_key("https://a.test/two") + ); + assert_eq!(host_key("https://a.test/"), "a.test:443"); + assert_eq!(host_key("http://a.test/"), "a.test:80"); + assert_eq!(host_key("https://a.test:8443/"), "a.test:8443"); + assert_ne!(host_key("https://a.test/"), host_key("https://b.test/")); + } +} diff --git a/src/client/hyper.rs b/src/client/hyper.rs index 8318ff8..55b1ed6 100644 --- a/src/client/hyper.rs +++ b/src/client/hyper.rs @@ -1,12 +1,14 @@ -use super::{ClientError, HttpClient}; +use super::{ClientError, HttpClient, TlsFailure}; use crate::config::RequestConfig; use crate::debug::{DebugLog, debug_record, new_debug_log}; +use crate::profile::RungFailure; use crate::response::{CertInfo, RedirectHop, Response}; use http_body_util::BodyExt; use hyper_util::client::legacy::Client; use hyper_util::client::legacy::connect::HttpConnector; use hyper_util::rt::TokioExecutor; +use std::borrow::Cow; use std::collections::HashMap; use std::future::Future; use std::io::Read; @@ -58,6 +60,273 @@ fn sanitize_uri(url: &str) -> String { out } +/// Add the extensions a browser sends that OpenSSL does not send on its own. +/// +/// JA4 hashes the list of extension *types*, not their contents, so presence +/// is what the fingerprint turns on. Payload fidelity still matters to +/// anything reading the raw hello, and where a payload is cheap to get right +/// it is written out properly rather than left empty. +/// +/// `SSL_CTX_add_custom_ext` refuses any extension OpenSSL handles itself, so +/// this only covers the ones it has never heard of. `status_request` is set +/// through its own API for that reason, and `renegotiation_info` cannot be +/// done from out here at all. +fn add_browser_extensions( + builder: &mut openssl::ssl::SslConnectorBuilder, +) -> Result<(), ClientError> { + use openssl::ssl::ExtensionContext; + + // Each extension's payload needs its OWN type, and this is not a style + // choice. The `openssl` crate stashes the bytes an add-callback returns in + // per-connection ex_data keyed on the payload type: + // + // let idx = Ssl::cached_ex_index::>(); + // + // so every custom extension returning `Vec` shares one slot and + // overwrites the others. Registering three that way sends exactly one. + // Distinct newtypes give each its own slot. + macro_rules! payload_type { + ($name:ident) => { + struct $name(Vec); + impl AsRef<[u8]> for $name { + fn as_ref(&self) -> &[u8] { + &self.0 + } + } + }; + } + payload_type!(SignedCertTimestamp); + + // The ClientHello for sending, plus the places a server may answer. + // Registering CLIENT_HELLO alone makes OpenSSL reject a reply that arrives + // in a context with no handler, which breaks the handshake outright rather + // than merely looking wrong. + let ctx = ExtensionContext::CLIENT_HELLO + | ExtensionContext::TLS1_3_SERVER_HELLO + | ExtensionContext::TLS1_3_ENCRYPTED_EXTENSIONS + | ExtensionContext::TLS1_2_SERVER_HELLO; + + // Which codepoints OpenSSL will cede, probed rather than assumed: + // + // 0x0005 status_request refused for custom use, OpenSSL owns it + // 0x0012 signed_cert_timestamp accepted + // 0x001b compress_certificate refused, OpenSSL owns it + // 0x4469 application_settings accepted + // 0xfe0d encrypted_client_hello accepted + // 0xff01 renegotiation_info refused, OpenSSL owns it + // + // status_request is set per-connection instead, in the one place OpenSSL's + // own API for it works. renegotiation_info and the SCSV are handled by a + // patch to our OpenSSL build. + + // status_request (0x0005) is handled per-connection; see the connector. + + // signed_certificate_timestamp (0x0012): empty, as a client sends it. + let _ = builder.add_custom_ext( + 0x0012, + ctx, + |_ssl, _ctx, _cert| Ok(Some(SignedCertTimestamp(Vec::new()))), + |_ssl, _ctx, _data, _cert| Ok(()), + ); + + // ── Three Chrome extensions deliberately NOT sent ───────────── + // + // Chrome sends application_settings (0x4469), encrypted_client_hello + // (0xfe0d) and compress_certificate (0x001b). We send none of them, which + // costs three extensions against Chrome's JA4. Each was tried and each + // broke real sites, because advertising a protocol feature is a promise to + // implement it and a server that takes you up on it gets a client that + // cannot follow through. + // + // ALPS (0x4469): google.com negotiates it and then expects the ALPS + // settings exchange inside HTTP/2. We do not implement that, so the + // handshake completes and the request then fails. Needs the HTTP/2 + // work before it can be sent honestly. + // + // ECH (0xfe0d): Chrome sends a GREASE ECH, a decoy shaped precisely + // enough to pass a real parser. A hand-built imitation is not, and + // reddit.com rejects it at the handshake. Registering the server-reply + // contexts fixed cloudflare.com but not reddit.com, so the payload + // itself is the problem. Needs real ECH, which arrived in OpenSSL 3.5. + // + // compress_certificate (0x001b): our OpenSSL is built with + // OPENSSL_NO_COMP_ALG, so it cannot decompress a certificate. Enabling + // it means a compression library in the build, and the wheels + // cross-compile to six targets. + // + // The measurement that matters: without all three, akamai.com went from a + // hard 403 to 200 with a real session. Exact JA4 parity turned out not to + // be the bar. Plausibility was enough there, and a client that connects is + // worth more than one that matches a hash and cannot. + + Ok(()) +} + +/// Layers of a profile that can be disabled independently, for working out +/// which one a particular detector is reacting to. +/// +/// Driven by `BLASTHTTP_BISECT`, a comma-separated list of layers to leave +/// off: `tls`, `headers`, `http2`. This exists because "we are blocked" does +/// not say which part of the imitation is failing, and the alternative is +/// rebuilding between every experiment. +/// +/// ```text +/// BLASTHTTP_BISECT=headers,http2 # profile TLS only +/// BLASTHTTP_BISECT=tls # everything but TLS +/// ``` +pub(crate) fn bisect_disabled(layer: &str) -> bool { + match std::env::var("BLASTHTTP_BISECT") { + Ok(v) => v.split(',').any(|p| p.trim().eq_ignore_ascii_case(layer)), + Err(_) => false, + } +} + +/// Apply the cipher, signature-algorithm, group and version settings to a +/// connector builder. +/// +/// Shared because there are two places that build an SSL context, the pooled +/// connector and `connect_stream`, and they have historically been +/// copy-pasted siblings. Anything applied in only one of them silently gives +/// `raw_connect`, `resolve_ip` and `request_target` requests a different +/// fingerprint from ordinary ones. +/// +/// Precedence is: an explicit setting beats the profile, and the profile beats +/// the default. A caller who names a cipher string means it, and several +/// existing tests depend on that still being true. +/// The cipher list to offer: `base` untouched when verification is off, with +/// `!aNULL` appended when it is on. +/// +/// An anonymous suite sends no certificate, and OpenSSL skips verify-peer (and +/// the hostname check with it) when none arrives, so offering one to a caller +/// who set `verify_certs` would report a verified handshake that checked +/// nothing. This applies to a caller's own `cipher_string` too. +fn verified_cipher_list(base: &str, verify: bool) -> Cow<'_, str> { + if verify { + Cow::Owned(format!("{base}:!aNULL")) + } else { + Cow::Borrowed(base) + } +} + +fn apply_tls_settings( + builder: &mut openssl::ssl::SslConnectorBuilder, + config: &RequestConfig, +) -> Result<(), ClientError> { + use openssl::ssl::SslOptions; + + // A bisect of the TLS layer means "what the default would have sent", + // which is now a named profile rather than an absence. + let profile = if bisect_disabled("tls") { + crate::profile::default_profile() + } else { + config.resolved_profile() + }; + + // Security level 0: allow all ciphers including RC4 and DES. This is an + // offensive-first tool and needs to connect to anything. + builder.set_security_level(0); + + // `SslConnector::builder` sets NO_SSLV3. Clear it, or a server that speaks + // nothing newer stays unreachable even when asked for by name. Clearing + // the option only permits the protocol; what is actually offered is + // decided by the min/max version below, and a browser profile pins that + // floor at 1.2 so this never widens a profile's offer. + builder.clear_options(SslOptions::NO_SSLV3); + + if profile.tls.browser_extensions { + // Removing two extensions a browser does not send. + // + // encrypt_then_mac (0x0016) goes away by disabling the feature. + // padding (0x0015) is added by SSL_OP_TLSEXT_PADDING, which arrives + // switched on inside SSL_OP_ALL, so it has to be cleared rather than + // set. JA4 counts extensions, so both of these move the fingerprint. + // + // The `openssl` crate exposes neither flag, so the bits are taken from + // ssl.h directly: SSL_OP_TLSEXT_PADDING is SSL_OP_BIT(4) and + // SSL_OP_NO_ENCRYPT_THEN_MAC is SSL_OP_BIT(19). Both are stable parts + // of the public ABI. + const SSL_OP_TLSEXT_PADDING: u64 = 1 << 4; + const SSL_OP_NO_ENCRYPT_THEN_MAC: u64 = 1 << 19; + builder.set_options(SslOptions::from_bits_retain(SSL_OP_NO_ENCRYPT_THEN_MAC)); + builder.clear_options(SslOptions::from_bits_retain(SSL_OP_TLSEXT_PADDING)); + + add_browser_extensions(builder)?; + } + + // Each of these is optional: leaving one unset keeps OpenSSL's own list + // and order, which is what a profile imitating an ordinary OpenSSL client + // wants. + if let Some(suites) = profile.tls.ciphersuites { + builder.set_ciphersuites(suites).map_err(|e| { + ClientError::tls(format!( + "profile '{}' has an invalid TLS 1.3 suite list: {}", + profile.name, e + )) + })?; + } + if let Some(sigalgs) = profile.tls.sigalgs { + builder.set_sigalgs_list(sigalgs).map_err(|e| { + ClientError::tls(format!( + "profile '{}' has an invalid signature algorithm list: {}", + profile.name, e + )) + })?; + } + if let Some(groups) = profile.tls.groups { + builder.set_groups_list(groups).map_err(|e| { + ClientError::tls(format!( + "profile '{}' has an invalid group list: {}", + profile.name, e + )) + })?; + } + + // Always set the cipher list, never leave it alone. + // + // `SslConnector::builder` installs its own first, + // `DEFAULT:!aNULL:!eNULL:!MD5:!3DES:!DES:!RC4:!IDEA:!SEED:...`, and + // `set_security_level(0)` does NOT undo it. The security level governs how + // weak a negotiated cipher may be; the cipher list governs which ones are + // offered at all. Leaving it alone means RC4, DES, 3DES and SEED never + // reach the wire whatever the security level says, so a server speaking + // only one of them is unreachable, which is the whole reason this project + // builds its own OpenSSL. + let ciphers = verified_cipher_list( + config + .cipher_string + .as_deref() + .unwrap_or(profile.tls.cipher_list), + config.should_verify_certs(), + ); + builder + .set_cipher_list(&ciphers) + .map_err(|e| ClientError::tls(format!("invalid cipher string '{}': {}", ciphers, e)))?; + + let min_version = config + .min_tls_version + .as_deref() + .or(profile.tls.min_version); + if let Some(v) = min_version { + let version = parse_tls_version(v)?; + builder + .set_min_proto_version(Some(version)) + .map_err(|e| ClientError::tls(format!("failed to set min TLS version: {}", e)))?; + } + + let max_version = config + .max_tls_version + .as_deref() + .or(profile.tls.max_version); + if let Some(v) = max_version { + let version = parse_tls_version(v)?; + builder + .set_max_proto_version(Some(version)) + .map_err(|e| ClientError::tls(format!("failed to set max TLS version: {}", e)))?; + } + + Ok(()) +} + // Load the OpenSSL legacy provider once (for RC4, DES, etc.). // The provider is statically compiled into libcrypto via `no-module` build flag. // `Once` ensures this runs exactly once even across threads. @@ -90,6 +359,30 @@ type CertSlot = Arc>>; /// will lag, but the value will always be an IP that *was* used). type PeerSlot = Arc>>; +/// A failed TLS handshake, as the connector hands it to hyper-util. +/// +/// hyper-util wraps it in an error whose `Display` is just "client error +/// (Connect)", but keeps it as that error's `source()`, so +/// `classify_dispatch_error` finds it again by walking the chain. The reason +/// travels with the error of the request it belongs to. +#[derive(Debug)] +struct TlsHandshakeError { + failure: TlsFailure, + inner: Box, +} + +impl std::fmt::Display for TlsHandshakeError { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + self.inner.fmt(f) + } +} + +impl std::error::Error for TlsHandshakeError { + fn source(&self) -> Option<&(dyn std::error::Error + 'static)> { + Some(&*self.inner) + } +} + /// Build the lookup key for `PeerSlot` from a URI's host and port. /// HTTPS defaults to 443, everything else to 80 — matches what /// `HttpConnector` uses when it dials the OS resolver. @@ -197,6 +490,9 @@ fn extract_cert_info(ssl: &openssl::ssl::SslRef) -> Option { // Wraps HttpConnector with TLS handshake via openssl + tokio-openssl. #[derive(Clone)] struct OpenSslConnector { + /// Whether a browser profile is active. Needed in `call()` because + /// status_request can only be set once an `Ssl` exists. + profile_active: bool, http: HttpConnector, ssl: openssl::ssl::SslConnector, // Shared slot for cert info — written during handshake, read after response @@ -275,12 +571,13 @@ fn encode_alpn_protocols(protos: &[String]) -> Result, ClientError> { fn parse_tls_version(s: &str) -> Result { match s.to_lowercase().as_str() { + "3.0" | "ssl3" | "sslv3" | "ssl3.0" => Ok(openssl::ssl::SslVersion::SSL3), "1.0" | "tls1.0" | "tlsv1.0" => Ok(openssl::ssl::SslVersion::TLS1), "1.1" | "tls1.1" | "tlsv1.1" => Ok(openssl::ssl::SslVersion::TLS1_1), "1.2" | "tls1.2" | "tlsv1.2" => Ok(openssl::ssl::SslVersion::TLS1_2), "1.3" | "tls1.3" | "tlsv1.3" => Ok(openssl::ssl::SslVersion::TLS1_3), _ => Err(ClientError::other(format!( - "unknown TLS version '{}' (use 1.0, 1.1, 1.2, 1.3)", + "unknown TLS version '{}' (use 3.0 for SSLv3, or 1.0, 1.1, 1.2, 1.3)", s ))), } @@ -374,35 +671,13 @@ impl OpenSslConnector { openssl::ssl::SslConnector::builder(openssl::ssl::SslMethod::tls_client()) .map_err(|e| ClientError::tls(format!("SSL setup failed: {}", e)))?; - // Security level 0: allow all ciphers including RC4, DES, export. - // This is an offensive-first tool — we need to connect to anything. - builder.set_security_level(0); - if !config.should_verify_certs() { builder.set_verify(openssl::ssl::SslVerifyMode::NONE); } else { load_system_ca_certs(&mut builder)?; } - if let Some(ref ciphers) = config.cipher_string { - builder.set_cipher_list(ciphers).map_err(|e| { - ClientError::tls(format!("invalid cipher string '{}': {}", ciphers, e)) - })?; - } - - if let Some(ref min_ver) = config.min_tls_version { - let version = parse_tls_version(min_ver)?; - builder - .set_min_proto_version(Some(version)) - .map_err(|e| ClientError::tls(format!("failed to set min TLS version: {}", e)))?; - } - - if let Some(ref max_ver) = config.max_tls_version { - let version = parse_tls_version(max_ver)?; - builder - .set_max_proto_version(Some(version)) - .map_err(|e| ClientError::tls(format!("failed to set max TLS version: {}", e)))?; - } + apply_tls_settings(&mut builder, config)?; // ALPN: advertise HTTP/2 and HTTP/1.1 support during TLS handshake. // The wire format is length-prefixed: [2, b'h', b'2', 8, b'h', b't', ...]. @@ -429,6 +704,7 @@ impl OpenSslConnector { http.set_connect_timeout(Some(connect_timeout)); Ok(OpenSslConnector { + profile_active: config.resolved_profile().tls.browser_extensions, http, ssl, cert_slot, @@ -559,6 +835,7 @@ impl tower_service::Service for OpenSslConnector { None => self.http.call(uri), }; let ssl_connector = self.ssl.clone(); + let profile_active = self.profile_active; let cert_slot = self.cert_slot.clone(); let peer_slot = self.peer_slot.clone(); let connect_timeout = self.connect_timeout; @@ -569,7 +846,7 @@ impl tower_service::Service for OpenSslConnector { // Record peer IP for this fresh connection. Best-effort — // failure to read peer_addr (vanishingly rare) is not fatal. - if let (Some(key), Ok(peer)) = (slot_key, tcp_stream.peer_addr()) + if let (Some(key), Ok(peer)) = (slot_key.clone(), tcp_stream.peer_addr()) && let Ok(mut map) = peer_slot.lock() { map.insert(key, peer.ip()); @@ -621,6 +898,15 @@ impl tower_service::Service for OpenSslConnector { let mut ssl_conf = openssl::ssl::Ssl::new(ssl_connector.context()) .map_err(|e| -> Box { Box::new(e) })?; + + // status_request (0x0005). OpenSSL keeps this codepoint for itself + // and refuses a custom extension for it, and its own API, + // SSL_set_tlsext_status_type, is per-connection rather than + // per-context. So it has to be set here, where an Ssl finally + // exists, rather than alongside the other profile extensions. + if profile_active { + let _ = ssl_conf.set_status_type(openssl::ssl::StatusType::OCSP); + } // Only set SNI for hostnames, not IP addresses (SNI with IPs is invalid per RFC) if host.parse::().is_err() { ssl_conf @@ -644,19 +930,28 @@ impl tower_service::Service for OpenSslConnector { let mut stream = tokio_openssl::SslStream::new(ssl_conf, tcp_stream) .map_err(|e| -> Box { Box::new(e) })?; + // Attach why, so it survives being boxed away as hyper-util requires. tokio::time::timeout(connect_timeout, Pin::new(&mut stream).connect()) .await .map_err(|_| -> Box { - Box::new(std::io::Error::new( - std::io::ErrorKind::TimedOut, - format!( - "TLS handshake with {} timed out after {}s", - host, - connect_timeout.as_secs() - ), - )) + Box::new(TlsHandshakeError { + failure: TlsFailure::Timeout, + inner: Box::new(std::io::Error::new( + std::io::ErrorKind::TimedOut, + format!( + "TLS handshake with {} timed out after {}s", + host, + connect_timeout.as_secs() + ), + )), + }) })? - .map_err(|e| -> Box { Box::new(e) })?; + .map_err(|e| -> Box { + Box::new(TlsHandshakeError { + failure: TlsFailure::from_openssl(&e), + inner: Box::new(e), + }) + })?; // Extract cert info after successful handshake let cert_info = extract_cert_info(stream.ssl()); @@ -703,6 +998,12 @@ struct CachedClient { #[derive(Clone, Hash, Eq, PartialEq)] struct TlsKey { verify_certs: bool, + /// Part of the key because a profile changes the cipher list, the + /// signature algorithms, the groups and the version floor. Leaving it out + /// means the first request to a host fixes the TLS configuration for every + /// later one, so an impersonated request and a plain one would quietly + /// share a connection. + profile: Option, cipher_string: Option, min_tls_version: Option, max_tls_version: Option, @@ -713,6 +1014,7 @@ impl TlsKey { fn from_config(config: &RequestConfig) -> Self { TlsKey { verify_certs: config.should_verify_certs(), + profile: config.profile.clone(), cipher_string: config.cipher_string.clone(), min_tls_version: config.min_tls_version.clone(), max_tls_version: config.max_tls_version.clone(), @@ -734,11 +1036,24 @@ enum ConnMode { Socks5 { proxy_url: String, tls: TlsKey }, } +/// How long a host that refused every rung keeps getting one request instead of +/// the whole ladder. Long enough to stop doubling a scan's requests to it, short +/// enough that a host which relents is noticed. See `ladder_exhausted`. +const EXHAUSTED_TTL: Duration = Duration::from_secs(600); + pub struct HyperClient { // Clients cached by connection mode. A scan using an HTTP proxy may need // both a ForwardProxy client (for HTTP targets) and a Tunnel client (for // HTTPS targets), so we cache per-mode rather than a single client. cached: Mutex>, + // Which profile last got through to each host, so a scan does not re-walk + // the ladder on every request. Written only on success; see + // `remember_profile`. + host_profiles: Mutex>, + // When each host last refused every rung of the ladder. For a while after, + // a refusal from it is taken as the answer instead of walking the ladder + // again; see `ladder_exhausted`. + host_exhausted: Mutex>, } impl Default for HyperClient { @@ -751,6 +1066,8 @@ impl HyperClient { pub fn new() -> Self { HyperClient { cached: Mutex::new(std::collections::HashMap::new()), + host_profiles: Mutex::new(HashMap::new()), + host_exhausted: Mutex::new(HashMap::new()), } } @@ -818,7 +1135,34 @@ impl HyperClient { let cert_slot: CertSlot = Arc::new(Mutex::new(None)); let peer_slot: PeerSlot = Arc::new(Mutex::new(HashMap::new())); let connector = OpenSslConnector::new(config, cert_slot.clone(), peer_slot.clone())?; - let builder = Client::builder(TokioExecutor::new()); + let mut builder = Client::builder(TokioExecutor::new()); + + // Shape the HTTP/2 SETTINGS and connection window to the profile. + // + // Nothing here was being set at all before, so the values on the wire + // were hyper-util's defaults, which match no browser. These four are + // everything hyper-util exposes of the Akamai HTTP/2 fingerprint; the + // remaining two fields, HEADER_TABLE_SIZE and the pseudo-header order, + // need forks of hyper-util and h2 respectively and are left alone. + let http2_profile = if bisect_disabled("http2") { + None + } else { + config.resolved_profile().http2 + }; + if let Some(p) = http2_profile { + builder + .http2_initial_stream_window_size(p.initial_stream_window) + .http2_initial_connection_window_size(p.initial_connection_window) + .http2_max_header_list_size(p.max_header_list_size) + // `None` omits MAX_FRAME_SIZE entirely, which is what Chrome + // does. Our default announces 16384 and Chrome announces + // nothing, so leaving this unset is itself part of the match. + .http2_max_frame_size(p.max_frame_size) + // Adaptive window resizes the connection window as traffic + // flows, which would emit WINDOW_UPDATE frames no browser + // sends and undo the fixed value set above. + .http2_adaptive_window(false); + } let connector = match mode { ConnMode::Direct(_) => connector, @@ -872,31 +1216,203 @@ async fn dispatch_request( } debug_record(log, v, 1, " Sending request..."); - let hyper_response = client.request(request).await.map_err(|e| { - let msg = format!("request failed: {}", e); - let err_str = e.to_string().to_lowercase(); - if err_str.contains("ssl") || err_str.contains("tls") || err_str.contains("certificate") { - ClientError::tls(msg) - } else { - ClientError::connection(msg) - } - })?; + // dev collapsed the three client variants into one pooled client when it + // moved the target handshake inside the proxy tunnel, so there is nothing + // left to match on. The classifier stays: it reads back why the handshake + // actually failed instead of guessing from the error text. + let hyper_response = client + .request(request) + .await + .map_err(|e| classify_dispatch_error(&e))?; parse_response(hyper_response, config, log).await } +/// Turn a hyper-util client error into something that says what went wrong. +/// +/// The error itself carries almost nothing: its `Display` is the literal +/// `"client error (Connect)"` for every connect-time failure, so the substring +/// test this replaces could never match and classified every TLS failure as a +/// connection error. That also meant TLS failures were being retried, which +/// `test_tls_error_is_not_retryable` says they should not be. +/// +/// The connector attaches a failed handshake's reason as a `TlsHandshakeError` +/// somewhere down the `source()` chain. Without one, the failure happened +/// before TLS, and `error_from_chain` reports the cause. +fn classify_dispatch_error(e: &hyper_util::client::legacy::Error) -> ClientError { + let mut current: Option<&(dyn std::error::Error + 'static)> = Some(e); + while let Some(err) = current { + if let Some(tls) = err.downcast_ref::() { + return ClientError::tls_detailed( + format!( + "TLS handshake failed: {}", + describe_tls_failure(&tls.failure) + ), + tls.failure.clone(), + ); + } + current = err.source(); + } + error_from_chain("request failed", e) +} + +/// Build a `ClientError` from every error in `e`'s `source()` chain. +/// +/// hyper and hyper-util keep the cause below the top-level error. hyper-util's +/// top level reads "client error (Connect)" whatever went wrong, the +/// connector's error under it says "tcp connect error", and only the +/// `io::Error` under that says "Connection refused". So the message is the +/// whole chain, and a timeout anywhere in it makes this a timeout rather than +/// a connection error, which is what the direct path reports for the same +/// failure. +fn error_from_chain(context: &str, e: &(dyn std::error::Error + 'static)) -> ClientError { + let mut parts = Vec::new(); + let mut timed_out = false; + let mut current = Some(e); + while let Some(err) = current { + if let Some(io) = err.downcast_ref::() + && io.kind() == std::io::ErrorKind::TimedOut + { + timed_out = true; + } + parts.push(err.to_string()); + current = err.source(); + } + + let message = format!("{}: {}", context, parts.join(": ")); + if timed_out { + ClientError::timeout(message) + } else { + ClientError::connection(message) + } +} + +/// A human-readable one-liner for a `TlsFailure`. +fn describe_tls_failure(failure: &TlsFailure) -> String { + match failure { + TlsFailure::NoSharedCipher => "no cipher suite in common".to_string(), + TlsFailure::UnsupportedProtocol => "no protocol version in common".to_string(), + TlsFailure::Alert { description } => format!("peer sent an alert: {}", description), + TlsFailure::CertificateVerify { detail } => { + format!("certificate verification failed: {}", detail) + } + TlsFailure::Reset => "connection closed during the handshake".to_string(), + TlsFailure::Timeout => "handshake timed out".to_string(), + TlsFailure::Other { detail } => detail.clone(), + } +} + +/// A connection that was established, and what it took to establish it. +pub(crate) struct Connected { + pub stream: Box, + pub cert_info: Option, + pub alpn: Option, + pub peer_ip: Option, + /// The profile the handshake finally succeeded under, which is not + /// necessarily the one asked for. `dispatch_direct` builds its request + /// from this so the headers cannot drift away from the TLS underneath + /// them, which is the failure mode a half-applied profile produces. + pub profile: &'static crate::profile::ConnectionProfile, + pub attempts: Vec, +} + /// Opens a fresh TCP connection (optionally to a resolved IP instead of DNS) /// and performs TLS if HTTPS (with SNI set to the original hostname). -/// Returns the connected stream and any certificate info collected during -/// the TLS handshake. /// /// Shared setup used by `dispatch_direct` (one-shot hyper requests over an /// un-pooled socket) and by callers that need a long-lived, unframed handle /// to a TCP or TLS stream. +/// +/// Widens the offer and tries again when a handshake fails in a way that says +/// the peer could not negotiate, which is the same breadth rung the pooled +/// path climbs. Reaching a server too old for the default is the reason this +/// library carries its own OpenSSL, and without this these callers lost it +/// when the default narrowed: a virtualhost sweep across an estate with one +/// legacy appliance in it would simply not see the appliance. +/// +/// It stops there. The pooled ladder has a second rung that reacts to a +/// *refusal* by changing what the client claims to be, and that one would be +/// wrong here. `resolve_ip` and `request_target` exist for probes whose point +/// is how one exact request is answered, so re-sending a probe dressed +/// differently answers a question nobody asked, doubles the requests, and +/// muddies the result. A raw connection has no response to classify at all. pub(crate) async fn connect_stream( target_uri: &http::Uri, config: &RequestConfig, log: &DebugLog, +) -> Result { + let mut rung = config.resolved_profile(); + let mut tried: Vec<&'static str> = Vec::new(); + let mut attempts: Vec = Vec::new(); + + loop { + tried.push(rung.name); + let rung_config = config.with_profile(rung.name); + + match connect_once(target_uri, &rung_config, log).await { + Ok((stream, cert_info, alpn, peer_ip)) => { + return Ok(Connected { + stream, + cert_info, + alpn, + peer_ip, + profile: rung, + attempts, + }); + } + Err(e) => { + attempts.push(crate::report::Attempt::new( + rung.name, + crate::report::AttemptOutcome::HandshakeFailed { + reason: e.message.clone(), + }, + )); + + // A pinned config ends where the caller put it. Naming a + // cipher string, a TLS version or a profile means they meant + // it, and on this path more than any other: these are the + // callers who asked for exact control. + let widen = !config.tls_is_pinned() + && tried.len() < crate::profile::MAX_RUNGS + && e.tls_failure + .as_ref() + .is_some_and(|f| f.suggests_wider_offer()); + + match widen + .then(|| { + crate::profile::next_rung(rung, RungFailure::CouldNotNegotiate, &tried) + }) + .flatten() + { + Some(next) => { + debug_record( + log, + config.verbosity, + 1, + &format!( + " handshake failed under '{}', widening to '{}'", + rung.name, next.name + ), + ); + rung = next; + } + None => { + let mut e = e; + e.attempts = attempts; + return Err(e); + } + } + } + } + } +} + +/// One connection attempt under one profile. The retrying lives in +/// `connect_stream`; this just does as it is told. +async fn connect_once( + target_uri: &http::Uri, + config: &RequestConfig, + log: &DebugLog, ) -> Result< ( Box, @@ -1047,18 +1563,13 @@ pub(crate) async fn connect_stream( openssl::ssl::SslConnector::builder(openssl::ssl::SslMethod::tls_client()) .map_err(|e| ClientError::tls(format!("SSL setup failed: {}", e)))?; - ssl_builder.set_security_level(0); - if !config.should_verify_certs() { ssl_builder.set_verify(openssl::ssl::SslVerifyMode::NONE); } else { load_system_ca_certs(&mut ssl_builder)?; } - if let Some(ref ciphers) = config.cipher_string { - ssl_builder - .set_cipher_list(ciphers) - .map_err(|e| ClientError::tls(format!("invalid cipher string '{}': {}", ciphers, e)))?; - } + + apply_tls_settings(&mut ssl_builder, config)?; if let Some(ref min_ver) = config.min_tls_version { let version = parse_tls_version(min_ver)?; ssl_builder @@ -1085,9 +1596,16 @@ pub(crate) async fn connect_stream( .map_err(|e| ClientError::tls(format!("failed to set ALPN: {}", e)))?; let ssl_connector = ssl_builder.build(); + let profile_active = config.resolved_profile().tls.browser_extensions; let mut ssl_conf = openssl::ssl::Ssl::new(ssl_connector.context()) .map_err(|e| ClientError::tls(format!("SSL conf failed: {}", e)))?; + // See the pooled path: OpenSSL keeps status_request for itself and its API + // for it is per-connection, so it can only be set once an Ssl exists. + if profile_active { + let _ = ssl_conf.set_status_type(openssl::ssl::StatusType::OCSP); + } + // SNI = original hostname, NOT the resolved IP if host.parse::().is_err() { ssl_conf @@ -1120,7 +1638,13 @@ pub(crate) async fn connect_stream( config.timeout() )) })? - .map_err(|e| ClientError::tls(format!("TLS handshake failed: {}", e)))?; + .map_err(|e| { + let failure = TlsFailure::from_openssl(&e); + ClientError::tls_detailed( + format!("TLS handshake failed: {}", describe_tls_failure(&failure)), + failure, + ) + })?; let cert_info = extract_cert_info(tls_stream.ssl()); let negotiated_alpn = tls_stream @@ -1145,11 +1669,27 @@ async fn dispatch_direct( target_uri: &http::Uri, config: &RequestConfig, log: &DebugLog, -) -> Result<(SingleResponse, Option, Option), ClientError> { +) -> Result< + ( + SingleResponse, + Option, + Option, + Vec, + ), + ClientError, +> { let v = config.verbosity; - let (stream, cert_info, alpn, peer_ip) = connect_stream(target_uri, config, log).await?; - let io = hyper_util::rt::TokioIo::new(stream); - let h2 = alpn.as_deref() == Some("h2"); + let conn = connect_stream(target_uri, config, log).await?; + let (cert_info, peer_ip) = (conn.cert_info, conn.peer_ip); + + // Build the request under whatever profile the handshake settled on, not + // whatever was asked for. If `connect_stream` widened to `compatibility` + // to reach an old server, the headers have to widen with it, or the + // request states something the connection underneath it contradicts. + let config = &config.with_profile(conn.profile.name); + + let io = hyper_util::rt::TokioIo::new(conn.stream); + let h2 = conn.alpn.as_deref() == Some("h2"); // HTTP/2 carries the target in `:path`, which hyper derives from the // request URI, so there is no request-line for `request_target` to @@ -1221,18 +1761,32 @@ async fn dispatch_direct( } debug_record(log, v, 1, " Sending request via dispatch_direct..."); - let hyper_response = sender.send_request(request).await.map_err(|e| { - let msg = format!("dispatch_direct request failed: {}", e); - let err_str = e.to_string().to_lowercase(); - if err_str.contains("ssl") || err_str.contains("tls") || err_str.contains("certificate") { - ClientError::tls(msg) - } else { - ClientError::connection(msg) - } - })?; + // TLS is already done by this point, and a failure there was reported + // with its reason by `connect_stream`, so whatever fails here is the + // connection. + let hyper_response = sender + .send_request(request) + .await + .map_err(|e| error_from_chain("dispatch_direct request failed", &e))?; let resp = parse_response(hyper_response, config, log).await?; - Ok((resp, cert_info, peer_ip)) + + // `conn.attempts` holds the handshakes that failed on the way here. The + // one that worked is this response, and it can only be classified now + // that there is a body, so it is recorded here rather than in + // `connect_stream`, which never sees one. + let mut attempts = conn.attempts; + attempts.push(crate::report::Attempt::from_outcome( + conn.profile.name, + resp.status, + &crate::antibot::classify(&crate::antibot::ResponseFacts { + status: resp.status, + headers: &resp.headers, + body: &String::from_utf8_lossy(&resp.body_bytes), + }), + )); + + Ok((resp, cert_info, peer_ip, attempts)) } /// The two one-shot senders `dispatch_direct` can end up holding, picked by @@ -1331,6 +1885,14 @@ async fn dispatch_forward_proxy( parse_response(hyper_response, config, log).await } +/// Headers that say which browser a request claims to be. +const IDENTITY_HEADERS: [&str; 4] = [ + "user-agent", + "sec-ch-ua", + "sec-ch-ua-mobile", + "sec-ch-ua-platform", +]; + fn build_request( uri: &http::Uri, config: &RequestConfig, @@ -1391,11 +1953,55 @@ fn build_request( builder = builder.header("Host", authority.as_str()); } - if !has_custom_ua { - builder = builder.header("User-Agent", "blasthttp/0.1.0"); - } - if !has_custom_ae { - builder = builder.header("Accept-Encoding", "gzip, deflate, br"); + // With a profile active, send its full header set in the browser's order + // rather than this client's two defaults. + // + // This has to travel with the TLS half or it makes matters worse. Browser + // headers over a non-browser handshake state something checkably false, + // and a contradiction is easier to act on than unfamiliarity: udemy.com + // answers an honest curl with 200 and the same client wearing only a + // Chrome User-Agent with 403. + // + // A caller's own header wins over the profile's, since naming one + // explicitly means it, except the browser identity once the ladder has + // switched profile (below). + // A profile with no headers of its own means the client's own minimal + // defaults, which is what `compatibility` and `modern` both want. + let header_profile = if bisect_disabled("headers") { + crate::profile::default_profile() + } else { + config.resolved_profile() + }; + // When the ladder chose the profile, its browser identity replaces the + // caller's; every other header the caller set still wins. + let profile_owns = |name: &str| { + config.profile_from_ladder + && IDENTITY_HEADERS + .iter() + .any(|h| h.eq_ignore_ascii_case(name)) + && header_profile + .headers + .iter() + .any(|(p, _)| p.eq_ignore_ascii_case(name)) + }; + if !header_profile.headers.is_empty() { + for (name, value) in header_profile.headers { + let already_set = + !profile_owns(name) && custom.iter().any(|(k, _)| k.eq_ignore_ascii_case(name)); + if !already_set { + builder = builder.header(*name, *value); + } + } + } else { + if !has_custom_ua { + builder = builder.header( + "User-Agent", + concat!("blasthttp/", env!("CARGO_PKG_VERSION")), + ); + } + if !has_custom_ae { + builder = builder.header("Accept-Encoding", "gzip, deflate, br"); + } } // Emit the caller's headers, folding any redirect-chain cookies into the @@ -1430,6 +2036,9 @@ fn build_request( builder = builder.header(name.as_str(), merged); continue; } + if profile_owns(name) { + continue; + } builder = builder.header(name.as_str(), value.as_str()); } } @@ -1687,6 +2296,12 @@ fn retry_backoff(attempt: u32, min_wait: Duration, max_wait: Duration) -> Durati impl HttpClient for HyperClient { async fn send(&self, config: &RequestConfig) -> Result { config.validate_proxy().map_err(ClientError::other)?; + // Checked once, at the edge. `resolved_profile` deliberately falls + // back rather than erroring, because it is called from deep inside the + // connector where there is nothing useful to do with a failure; this + // is the place that turns a typo into a message instead of silently + // giving the caller a profile they did not ask for. + config.validate_profile().map_err(ClientError::other)?; let timeout_duration = Duration::from_secs(config.timeout()); let max_retries = config.max_retries(); let min_wait = config.retry_wait_min(); @@ -1712,8 +2327,10 @@ impl HttpClient for HyperClient { tokio::time::sleep(backoff).await; } - match tokio::time::timeout(timeout_duration, self.send_inner(config, &log)).await { - Ok(Ok(response)) => { + // One deadline for the whole request, redirects and ladder included. + let deadline = tokio::time::Instant::now() + timeout_duration; + match self.send_inner(config, &log, deadline).await { + Ok(response) => { let status = response.status; if super::ErrorKind::Status(status).is_retryable() && attempt < max_retries { debug_record( @@ -1730,7 +2347,7 @@ impl HttpClient for HyperClient { } return Ok(response); } - Ok(Err(e)) => { + Err(e) => { if e.kind.is_retryable() && attempt < max_retries { debug_record( &log, @@ -1743,12 +2360,6 @@ impl HttpClient for HyperClient { } return Err(e); } - Err(_) => { - return Err(ClientError::timeout(format!( - "request timed out after {}s", - config.timeout() - ))); - } } } @@ -1757,13 +2368,187 @@ impl HttpClient for HyperClient { } impl HyperClient { + /// One attempt at one hop with one profile. + /// + /// Everything that used to sit inline in the redirect loop, extracted so + /// the ladder can call it more than once. The `?` operators that used to + /// short-circuit out of `send_inner` now surface here as an `Err` the + /// ladder can decide about, which is the substance of the change rather + /// than the extraction itself. + async fn attempt_hop( + &self, + config: &RequestConfig, + uri: &http::Uri, + log: &DebugLog, + hop_cookies: Option<&str>, + ) -> Result<(SingleResponse, Option), ClientError> { + // Decide the connection mode for the *current* target host on every + // hop, not just the first. A redirect can send the request to a + // different host, and the proxy / no_proxy decision has to follow it. + // Freezing the first hop's choice would otherwise let a request that + // started direct keep connecting directly after a redirect onto a + // proxied host (leaking traffic past the proxy), and let a request + // that started proxied keep using the proxy after a redirect onto a + // no_proxy host. Clients are cached by mode, so hops that share a mode + // reuse the same client. + let mode = Self::conn_mode(config, uri)?; + + // Forward proxy: dispatch directly via TCP + http1::SendRequest + // (bypasses hyper Client's URI normalization to preserve absolute-form) + let proxy_url_for_fwd = if let ConnMode::ForwardProxy(ref url) = mode { + Some(url.clone()) + } else { + None + }; + + let cached = if proxy_url_for_fwd.is_some() { + None + } else { + Some(self.get_or_build(config, &mode)?) + }; + + let resp = if let Some(ref proxy_url) = proxy_url_for_fwd { + dispatch_forward_proxy(proxy_url, uri, config, log, hop_cookies).await? + } else { + dispatch_request( + &cached.as_ref().unwrap().inner, + uri, + config, + log, + hop_cookies, + ) + .await? + }; + + Ok((resp, cached)) + } + + /// Which profile this hop starts on. + /// + /// A pinned config starts and ends on what the caller asked for. Otherwise + /// a host we have succeeded against before starts on whatever worked, + /// which is the whole point of remembering: the ladder costs a wasted + /// request every time it walks, and a scan hits the same host repeatedly. + fn starting_profile( + &self, + config: &RequestConfig, + uri: &http::Uri, + ) -> &'static crate::profile::ConnectionProfile { + if config.tls_is_pinned() { + return config.resolved_profile(); + } + if let Some(key) = peer_slot_key(uri) + && let Ok(map) = self.host_profiles.lock() + && let Some(p) = map.get(&key) + { + return p; + } + config.resolved_profile() + } + + /// The next rung, or `None` to stop. + /// + /// Returns `None` immediately for a pinned config: naming a cipher string, + /// a TLS version or a profile means the caller meant it, and quietly + /// substituting another profile would put something on the wire they did + /// not ask for. + fn next_rung_for( + &self, + config: &RequestConfig, + current: &crate::profile::ConnectionProfile, + failure: RungFailure, + tried: &[&'static str], + ) -> Option<&'static crate::profile::ConnectionProfile> { + if config.tls_is_pinned() || tried.len() >= crate::profile::MAX_RUNGS { + return None; + } + crate::profile::next_rung(current, failure, tried) + } + + /// Whether this host recently refused every rung, so a refusal from it now + /// should be taken as the answer rather than spending another request on + /// the next rung. + /// + /// Without this, only success was remembered, so a host that refuses + /// everything had the whole ladder walked again for every request to it: + /// two requests where one tells you the same thing, on exactly the hosts + /// most likely to rate-limit or ban. It expires, unlike the success memory, + /// because a refusal is a judgement the product makes from things that + /// change (rate, reputation, its own rules), and what it refused ten minutes + /// ago it may let through now. + fn ladder_exhausted(&self, config: &RequestConfig, uri: &http::Uri) -> bool { + if config.tls_is_pinned() { + return false; + } + peer_slot_key(uri).is_some_and(|key| self.exhausted_within_ttl(&key)) + } + + fn exhausted_within_ttl(&self, key: &str) -> bool { + self.host_exhausted + .lock() + .ok() + .and_then(|map| map.get(key).copied()) + .is_some_and(|at| at.elapsed() < EXHAUSTED_TTL) + } + + /// Record that a host refused every rung of the ladder. + fn remember_exhausted(&self, config: &RequestConfig, uri: &http::Uri) { + if config.tls_is_pinned() { + return; + } + if let Some(key) = peer_slot_key(uri) + && let Ok(mut map) = self.host_exhausted.lock() + { + map.insert(key, Instant::now()); + } + } + + /// Record that a profile worked against a host. + /// + /// Only ever called after an attempt got through. A failure alone is a + /// hypothesis and could be the host having a bad minute; a success is + /// evidence. Recording failures would let one flake pin a host to a worse + /// profile for the rest of a scan. + /// + /// A pinned request records nothing, which matches `starting_profile` + /// refusing to read the memory when pinned. The memory is a cache of what + /// the ladder found out, and a caller naming a profile is not the ladder + /// finding anything out. Recording it would mean one deliberate + /// `profile="chrome"` request quietly turned every later request to that + /// host into a browser claim, which is the change most likely to make + /// things worse: PerimeterX passes an honest client and refuses a + /// half-convincing browser. It would also make the result depend on the + /// order two independent requests happened to run in. + fn remember_profile( + &self, + config: &RequestConfig, + uri: &http::Uri, + profile: &'static crate::profile::ConnectionProfile, + ) { + if config.tls_is_pinned() { + return; + } + if let Some(key) = peer_slot_key(uri) { + if let Ok(mut map) = self.host_profiles.lock() { + map.insert(key.clone(), profile); + } + // whatever it refused before, it lets us through now + if let Ok(mut map) = self.host_exhausted.lock() { + map.remove(&key); + } + } + } + async fn send_inner( &self, config: &RequestConfig, log: &DebugLog, + deadline: tokio::time::Instant, ) -> Result { let v = config.verbosity; let start = Instant::now(); + let timed_out = + || ClientError::timeout(format!("request timed out after {}s", config.timeout())); let sanitized_url = sanitize_uri(&config.url); let mut uri: http::Uri = sanitized_url.parse().map(normalize_empty_path).map_err( @@ -1797,7 +2582,10 @@ impl HyperClient { // Bypasses the cached connection pool — opens a fresh TCP connection. if config.resolve_ip.is_some() || config.request_target.is_some() { let redirect_chain: Vec = Vec::new(); - let (resp, cert_info, peer_ip) = dispatch_direct(&uri, config, log).await?; + let (resp, cert_info, peer_ip, attempts) = + tokio::time::timeout_at(deadline, dispatch_direct(&uri, config, log)) + .await + .map_err(|_| timed_out())??; let hop_ms = start.elapsed().as_millis(); debug_record(log, v, 1, &format!("<- {} ({}ms)", resp.status, hop_ms)); @@ -1811,6 +2599,7 @@ impl HyperClient { status: resp.status, headers: resp.headers, body_bytes: resp.body_bytes, + attempts, elapsed_ms, redirect_chain, cert_info, @@ -1854,35 +2643,140 @@ impl HyperClient { // that started proxied keep using the proxy after a redirect onto a // no_proxy host. Clients are cached by mode, so hops that share a // mode reuse the same client. - let mode = Self::conn_mode(config, &uri)?; + // Walk the profile ladder for this hop. + // + // Per hop rather than per request, because a redirect can land on + // a differently protected host. Inside `send_inner` rather than in + // `send`, so it does not spend the caller's retry budget: those are + // different questions, one being "did this flake" and the other + // "is this the wrong way to talk to this host". + let mut rung = self.starting_profile(config, &uri); + let mut tried: Vec<&'static str> = Vec::new(); + let mut attempts: Vec = Vec::new(); + // The last refusal received, so an escalation that then fails + // outright cannot turn a real response into an error. + let mut refusal = None; + let (resp, cached) = loop { + tried.push(rung.name); + let rung_config = config.with_profile(rung.name); + + let hop = self.attempt_hop(&rung_config, &uri, log, hop_cookies.as_deref()); + match tokio::time::timeout_at(deadline, hop) + .await + .unwrap_or_else(|_| Err(timed_out())) + { + Ok((resp, cached)) => { + let outcome = crate::antibot::classify(&crate::antibot::ResponseFacts { + status: resp.status, + headers: &resp.headers, + body: &String::from_utf8_lossy(&resp.body_bytes), + }); + + // Only walk the ladder when something recognisable + // refused us. A bare 403 with no product signature is + // an ordinary refusal, and retrying it differently + // just spends another request to be told the same + // thing. + attempts.push(crate::report::Attempt::from_outcome( + rung.name, + resp.status, + &outcome, + )); - // Forward proxy: dispatch directly via TCP + http1::SendRequest - // (bypasses hyper Client's URI normalization to preserve absolute-form) - let is_forward_proxy = matches!(&mode, ConnMode::ForwardProxy(_)); - let proxy_url_for_fwd = if let ConnMode::ForwardProxy(ref url) = mode { - Some(url.clone()) - } else { - None - }; + if outcome.got_through() || !outcome.indicates_protection() { + // Remember what worked, but only on a real + // success. A failure is a hypothesis; this is + // evidence. + if outcome.got_through() { + self.remember_profile(config, &uri, rung); + } + break (resp, cached); + } - // For non-forward-proxy modes, get the cached hyper Client. - let cached = if is_forward_proxy { - None - } else { - Some(self.get_or_build(config, &mode)?) - }; + // A host that recently refused every rung gets one + // request, not the whole ladder again. + let next = if self.ladder_exhausted(config, &uri) { + None + } else { + self.next_rung_for(config, rung, RungFailure::Refused, &tried) + }; + match next { + Some(next) => { + debug_record( + log, + v, + 1, + &format!( + " {} refused under '{}', trying '{}'", + resp.status, rung.name, next.name + ), + ); + refusal = Some((resp, cached)); + rung = next; + } + // Out of rungs: the refusal is the answer. Hand it + // back as a response rather than an error, so the + // caller can read the status, the headers and + // whatever the challenge page said. + None => { + // Only a ladder that actually ran out counts; a + // single refusal while already exhausted must not + // keep extending it, or the host is never re-tried. + if tried.len() > 1 { + self.remember_exhausted(config, &uri); + } + break (resp, cached); + } + } + } + Err(mut e) => { + attempts.push(crate::report::Attempt::new( + rung.name, + crate::report::AttemptOutcome::HandshakeFailed { + reason: e.message.clone(), + }, + )); - let resp = if let Some(ref proxy_url) = proxy_url_for_fwd { - dispatch_forward_proxy(proxy_url, &uri, config, log, hop_cookies.as_deref()).await? - } else { - dispatch_request( - &cached.as_ref().unwrap().inner, - &uri, - config, - log, - hop_cookies.as_deref(), - ) - .await? + let widens = e + .tls_failure + .as_ref() + .is_some_and(|f| f.suggests_wider_offer()); + let next = if widens { + self.next_rung_for(config, rung, RungFailure::CouldNotNegotiate, &tried) + } else { + None + }; + match next { + Some(next) => { + debug_record( + log, + v, + 1, + &format!( + " handshake failed under '{}', widening to '{}'", + rung.name, next.name + ), + ); + rung = next; + } + // Nothing else to try. An earlier rung's refusal is + // still the answer; otherwise report the error, + // carrying what was tried. + None => { + if let Some(kept) = refusal.take() { + // Running out of time says nothing about + // the host, so it doesn't count as exhausted. + if e.kind != super::ErrorKind::Timeout { + self.remember_exhausted(config, &uri); + } + break kept; + } + e.attempts = attempts; + return Err(e); + } + } + } + } }; let hop_ms = start.elapsed().as_millis(); debug_record(log, v, 1, &format!("<- {} ({}ms)", resp.status, hop_ms)); @@ -1998,6 +2892,8 @@ impl HyperClient { status: resp.status, headers: resp.headers, body_bytes: resp.body_bytes, + // What the ladder tried on this, the final, hop. + attempts, elapsed_ms, redirect_chain, cert_info, @@ -2215,6 +3111,7 @@ enum Codec { Gzip, Deflate, Brotli, + Zstd, Identity, } @@ -2226,6 +3123,7 @@ impl Codec { "gzip" | "x-gzip" => Some(Codec::Gzip), "deflate" => Some(Codec::Deflate), "br" => Some(Codec::Brotli), + "zstd" => Some(Codec::Zstd), "identity" => Some(Codec::Identity), _ => None, } @@ -2236,6 +3134,7 @@ impl Codec { Codec::Gzip => "gzip", Codec::Deflate => "deflate", Codec::Brotli => "brotli", + Codec::Zstd => "zstd", Codec::Identity => "identity", } } @@ -2248,6 +3147,7 @@ fn decode_one(codec: Codec, data: &[u8], max_size: usize) -> Inflated { Codec::Gzip => read_bounded(flate2::read::GzDecoder::new(data), max_size, codec), Codec::Deflate => decode_deflate(data, max_size), Codec::Brotli => read_bounded(brotli::Decompressor::new(data, 4096), max_size, codec), + Codec::Zstd => decode_zstd(data, max_size), Codec::Identity => Inflated::Whole(data.to_vec()), } } @@ -2298,6 +3198,56 @@ fn decode_deflate(data: &[u8], max_size: usize) -> Inflated { } } +/// RFC 9659's window limit for `Content-Encoding: zstd`. A frame asking for +/// more is refused rather than allowed to make us allocate it. +const ZSTD_MAX_WINDOW: u64 = 8 * 1024 * 1024; + +/// Undo `Content-Encoding: zstd`, which may hold several frames back to back, +/// including skippable ones. The decoder reads one frame at a time, so this +/// walks them, stopping at `max_size` bytes of output. +fn decode_zstd(data: &[u8], max_size: usize) -> Inflated { + use ruzstd::decoding::StreamingDecoder; + use ruzstd::decoding::errors::{FrameDecoderError, ReadFrameHeaderError}; + + let fail = |out: Vec, e: String| { + if out.is_empty() { + Inflated::Failed(format!("zstd decompression failed: {}", e)) + } else { + Inflated::Partial(out, format!("zstd stream did not decode cleanly: {}", e)) + } + }; + + let mut src = data; + let mut out = Vec::new(); + loop { + let decoder = match StreamingDecoder::new_with_max_window_size(&mut src, ZSTD_MAX_WINDOW) { + Ok(d) => d, + Err(FrameDecoderError::ReadFrameHeaderError(ReadFrameHeaderError::SkipFrame { + length, + .. + })) => { + let Some(rest) = src.get(length as usize..) else { + return fail(out, "skippable frame runs past the end".to_string()); + }; + src = rest; + if src.is_empty() { + break; + } + continue; + } + Err(e) => return fail(out, e.to_string()), + }; + let room = (max_size - out.len()) as u64; + if let Err(e) = decoder.take(room).read_to_end(&mut out) { + return fail(out, e.to_string()); + } + if out.len() >= max_size || src.is_empty() { + break; + } + } + Inflated::Whole(out) +} + /// Whether `data` starts with something that reads as a zlib header (RFC 1950 /// 2.2): compression method 8 in the low nibble of CMF, and a CMF/FLG pair /// that is a multiple of 31. True for every zlib stream, and true for a raw @@ -2393,6 +3343,81 @@ where mod tests { use super::*; + /// Stands in for hyper-util's errors: says nothing itself, and keeps the + /// cause in `source()`. + #[derive(Debug)] + struct Wrapper(&'static str, std::io::Error); + + impl std::fmt::Display for Wrapper { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.write_str(self.0) + } + } + + impl std::error::Error for Wrapper { + fn source(&self) -> Option<&(dyn std::error::Error + 'static)> { + Some(&self.1) + } + } + + #[test] + fn test_error_from_chain_includes_the_cause() { + let e = Wrapper( + "tcp connect error", + std::io::Error::from(std::io::ErrorKind::ConnectionRefused), + ); + let err = error_from_chain("request failed", &e); + assert_eq!(err.kind, super::super::ErrorKind::Connection); + assert!( + err.message + .starts_with("request failed: tcp connect error: "), + "{}", + err.message + ); + assert!( + err.message.to_lowercase().contains("refused"), + "{}", + err.message + ); + } + + #[test] + fn test_error_from_chain_finds_a_timeout_below_the_top() { + let e = Wrapper( + "client error (Connect)", + std::io::Error::new(std::io::ErrorKind::TimedOut, "proxy tunnel timed out"), + ); + let err = error_from_chain("request failed", &e); + assert_eq!(err.kind, super::super::ErrorKind::Timeout); + assert!( + err.message.contains("proxy tunnel timed out"), + "{}", + err.message + ); + } + + #[test] + fn test_exhausted_host_expires() { + let client = HyperClient::new(); + assert!(!client.exhausted_within_ttl("example.com:443")); + + client + .host_exhausted + .lock() + .unwrap() + .insert("example.com:443".to_string(), Instant::now()); + assert!(client.exhausted_within_ttl("example.com:443")); + + // a refusal from longer ago than the TTL no longer counts, so the ladder is walked again + let long_ago = Instant::now() - EXHAUSTED_TTL - Duration::from_secs(1); + client + .host_exhausted + .lock() + .unwrap() + .insert("example.com:443".to_string(), long_ago); + assert!(!client.exhausted_within_ttl("example.com:443")); + } + #[test] fn test_backoff_attempt_0() { let min = Duration::from_secs(1); @@ -2588,11 +3613,16 @@ mod tests { out } + fn zstd_bytes(data: &[u8]) -> Vec { + ruzstd::encoding::compress_to_vec(data, ruzstd::encoding::CompressionLevel::Fastest) + } + fn each_codec() -> Vec<(&'static str, Vec)> { vec![ ("gzip", gzip_bytes(SAMPLE)), ("deflate", deflate_bytes(SAMPLE)), ("br", brotli_bytes(SAMPLE)), + ("zstd", zstd_bytes(SAMPLE)), ] } @@ -2959,6 +3989,68 @@ mod tests { assert_eq!(decoded("gzip,gzip", &twice, NO_LIMIT), SAMPLE); } + // ── zstd ────────────────────────────────────────────────────── + + #[test] + fn test_zstd_reads_every_frame() { + // A body may be several frames back to back; all of them are content. + let mut two = zstd_bytes(b"first "); + two.extend(zstd_bytes(b"second")); + assert_eq!(decoded("zstd", &two, NO_LIMIT), b"first second"); + } + + #[test] + fn test_zstd_skips_skippable_frames() { + // Magic 0x184D2A50, a little-endian length, then that many bytes. + let mut body = vec![0x50, 0x2A, 0x4D, 0x18, 3, 0, 0, 0, b'x', b'y', b'z']; + body.extend(zstd_bytes(SAMPLE)); + assert_eq!(decoded("zstd", &body, NO_LIMIT), SAMPLE); + } + + #[test] + fn test_zstd_output_is_bounded() { + let zeros = vec![0u8; 200_000]; + assert_eq!(decoded("zstd", &zstd_bytes(&zeros), 4096).len(), 4096); + // The cap spans frames too. + let mut two = zstd_bytes(&zeros); + two.extend(zstd_bytes(&zeros)); + assert_eq!(decoded("zstd", &two, 250_000).len(), 250_000); + } + + #[test] + fn test_zstd_refuses_an_oversized_window() { + // A frame header asking for a 16 MiB window, over RFC 9659's 8 MiB. + // Refused before anything is allocated, and the body comes back raw. + let body = [0x28, 0xB5, 0x2F, 0xFD, 0x00, 14 << 3]; + let (raw, reason) = undecoded("zstd", &body, NO_LIMIT); + assert_eq!(raw, body); + assert!( + reason.contains("zstd"), + "reason should name the codec: {}", + reason + ); + } + + #[test] + fn test_zstd_flags_a_corrupt_stream() { + let big = SAMPLE.repeat(400); + let mut z = zstd_bytes(&big); + let mid = z.len() / 2; + z[mid] ^= 0xff; + let out = decompress("zstd", &z, NO_LIMIT); + assert!( + out.decode_error.is_some(), + "a corrupted zstd body must be flagged" + ); + assert_ne!(out.body, big); + } + + #[test] + fn test_zstd_stacks_with_other_codings() { + let stacked = zstd_bytes(&gzip_bytes(SAMPLE)); + assert_eq!(decoded("gzip, zstd", &stacked, NO_LIMIT), SAMPLE); + } + #[test] fn test_decompress_ignores_identity_entries_in_a_list() { let gz = gzip_bytes(SAMPLE); @@ -3150,4 +4242,37 @@ mod tests { assert!(TunnelProxy::from_url("ftp://proxy:21").is_err()); assert!(TunnelProxy::from_url("http://:8080").is_err()); } + + // ── verified_cipher_list ────────────────────────────────────── + + #[test] + fn test_unverified_list_is_untouched() { + assert_eq!(verified_cipher_list("ALL", false), "ALL"); + assert_eq!(verified_cipher_list("RC4-SHA", false), "RC4-SHA"); + assert_eq!( + verified_cipher_list("ADH-AES128-SHA", false), + "ADH-AES128-SHA" + ); + } + + #[test] + fn test_unverified_list_is_borrowed() { + assert!(matches!( + verified_cipher_list("ALL", false), + Cow::Borrowed("ALL") + )); + } + + #[test] + fn test_verifying_drops_anonymous_suites() { + assert_eq!(verified_cipher_list("ALL", true), "ALL:!aNULL"); + } + + #[test] + fn test_verifying_drops_them_from_a_caller_list_too() { + assert_eq!( + verified_cipher_list("ADH-AES128-SHA", true), + "ADH-AES128-SHA:!aNULL" + ); + } } diff --git a/src/client/mock.rs b/src/client/mock.rs index 0dbd625..8ea18e9 100644 --- a/src/client/mock.rs +++ b/src/client/mock.rs @@ -75,6 +75,8 @@ impl HttpClient for ErrorKindMockClient { Err(ClientError { message: self.message.clone(), kind: self.kind.clone(), + tls_failure: None, + attempts: Vec::new(), }) } } @@ -117,6 +119,8 @@ impl HttpClient for SequenceMockClient { return Err(ClientError { message: format!("mock error on call {}", call), kind: self.fail_kind.clone(), + tls_failure: None, + attempts: Vec::new(), }); } @@ -126,6 +130,7 @@ impl HttpClient for SequenceMockClient { status: self.success_status, headers: Vec::new(), body_bytes, + attempts: Vec::new(), elapsed_ms: 0, redirect_chain: Vec::new(), cert_info: None, @@ -178,6 +183,7 @@ impl HttpClient for MockClient { status: self.status, headers: self.headers.clone(), body_bytes, + attempts: Vec::new(), elapsed_ms: 0, redirect_chain: Vec::new(), cert_info: None, diff --git a/src/client/mod.rs b/src/client/mod.rs index c16ced2..b09e93e 100644 --- a/src/client/mod.rs +++ b/src/client/mod.rs @@ -42,10 +42,156 @@ impl ErrorKind { } } +/// Why a TLS handshake failed, in enough detail to decide what to do about it. +/// +/// The distinction that matters is whether a *broader* offer would plausibly +/// help. A peer that shares no cipher with us might accept an older one; a peer +/// whose certificate we rejected will reject it again just as hard, and +/// retrying with weaker crypto there would be both futile and alarming. +/// +/// Derived from OpenSSL's reason codes rather than its message text. The text +/// is stable enough to log and far too loose to branch on. +#[derive(Debug, Clone, PartialEq, Eq)] +pub enum TlsFailure { + /// No cipher suite in common. The clearest signal that a wider offer is + /// worth trying. + NoSharedCipher, + /// No protocol version in common, or the peer rejected the one we picked. + UnsupportedProtocol, + /// The peer sent a fatal alert. `description` is OpenSSL's reason text. + /// handshake_failure most often means the same thing as NoSharedCipher, + /// but the peer declined to say which, so it is kept separate. + Alert { + description: String, + }, + /// Certificate verification failed. Never a reason to widen the offer. + CertificateVerify { + detail: String, + }, + /// The connection died mid-handshake with no alert. Typical of old servers + /// and middleboxes choking on a large or unfamiliar ClientHello, and also + /// of something actively refusing us, which is why it is not on its own + /// treated as a cipher problem. + Reset, + /// The handshake did not complete in time. + Timeout, + Other { + detail: String, + }, +} + +impl TlsFailure { + /// Would offering more ciphers and older protocol versions plausibly help? + /// + /// `Alert` is in here, and that is the case that matters in practice. A + /// server with no cipher in common with us does not send a neatly labelled + /// "no shared cipher"; that reason code is what OpenSSL raises when *we* + /// work it out locally. What actually comes back from the wire is a fatal + /// handshake_failure alert with no explanation. Measured against an + /// RC4-only server offered AES: the error is + /// `ssl/tls alert handshake failure`, not `no shared cipher`. Excluding + /// alerts here would mean never widening the offer in the single most + /// common case. + /// + /// Still conservative about the rest. A certificate rejection, a timeout + /// and a reset all say nothing about our cipher list, and widening in + /// response would send deprecated crypto to a peer that never asked for + /// it. + pub fn suggests_wider_offer(&self) -> bool { + matches!( + self, + TlsFailure::NoSharedCipher | TlsFailure::UnsupportedProtocol | TlsFailure::Alert { .. } + ) + } + + /// Classify an OpenSSL handshake error. + pub fn from_openssl(err: &openssl::ssl::Error) -> Self { + // Reason codes from the vendored sslerr.h. + const NO_SHARED_CIPHER: i32 = 193; + const NO_PROTOCOLS_AVAILABLE: i32 = 191; + const UNSUPPORTED_PROTOCOL: i32 = 258; + const UNSUPPORTED_SSL_VERSION: i32 = 259; + const WRONG_VERSION_NUMBER: i32 = 267; + const VERSION_TOO_HIGH: i32 = 166; + const VERSION_TOO_LOW: i32 = 396; + const CERTIFICATE_VERIFY_FAILED: i32 = 134; + const TLSV1_ALERT_PROTOCOL_VERSION: i32 = 1070; + const TLSV1_ALERT_INSUFFICIENT_SECURITY: i32 = 1071; + const SSLV3_ALERT_HANDSHAKE_FAILURE: i32 = 1040; + const SSLV3_ALERT_ILLEGAL_PARAMETER: i32 = 1047; + + // Scan the whole stack for a code we recognise rather than reading + // only the first. OpenSSL pushes several errors for one failure and + // the most specific is not reliably at the front: a generic + // "ssl handshake failure" often sits on top of the reason that + // actually explains it. + if let Some(stack) = err.ssl_error() { + let mut fallback: Option = None; + + for e in stack.errors() { + let reason = e.reason().unwrap_or("").to_string(); + let classified = match e.reason_code() { + NO_SHARED_CIPHER => Some(TlsFailure::NoSharedCipher), + // The peer telling us our ciphers are too weak is the same + // actionable fact as having none in common, from the other + // direction. + TLSV1_ALERT_INSUFFICIENT_SECURITY => Some(TlsFailure::NoSharedCipher), + NO_PROTOCOLS_AVAILABLE + | UNSUPPORTED_PROTOCOL + | UNSUPPORTED_SSL_VERSION + | WRONG_VERSION_NUMBER + | VERSION_TOO_HIGH + | VERSION_TOO_LOW + | TLSV1_ALERT_PROTOCOL_VERSION => Some(TlsFailure::UnsupportedProtocol), + CERTIFICATE_VERIFY_FAILED => { + Some(TlsFailure::CertificateVerify { detail: reason }) + } + SSLV3_ALERT_HANDSHAKE_FAILURE | SSLV3_ALERT_ILLEGAL_PARAMETER => { + Some(TlsFailure::Alert { + description: reason, + }) + } + _ => { + // Unrecognised: keep the first one as a description of + // last resort, but carry on looking for a code we know. + fallback.get_or_insert(TlsFailure::Other { detail: reason }); + None + } + }; + if let Some(failure) = classified { + return failure; + } + } + + if let Some(failure) = fallback { + return failure; + } + } + + // No reason stack: the failure was below TLS. SYSCALL with an io cause + // is a reset; without one it is an unexpected EOF, which OpenSSL + // reports the same way. + match err.io_error() { + Some(io) if io.kind() == std::io::ErrorKind::TimedOut => TlsFailure::Timeout, + Some(_) => TlsFailure::Reset, + None => TlsFailure::Reset, + } + } +} + #[derive(Debug, Clone)] pub struct ClientError { pub message: String, pub kind: ErrorKind, + /// Structured detail when the failure was a TLS handshake, so callers can + /// branch on why rather than substring-matching `message`. + pub tls_failure: Option, + /// What the profile ladder tried before giving up. + /// + /// Empty when nothing laddered, which is every error raised before a hop + /// begins. More than one entry means the client already tried the obvious + /// alternatives, so the caller need not. + pub attempts: Vec, } impl ClientError { @@ -53,6 +199,8 @@ impl ClientError { ClientError { message, kind: ErrorKind::Connection, + tls_failure: None, + attempts: Vec::new(), } } @@ -60,6 +208,8 @@ impl ClientError { ClientError { message, kind: ErrorKind::Timeout, + tls_failure: None, + attempts: Vec::new(), } } @@ -67,6 +217,8 @@ impl ClientError { ClientError { message, kind: ErrorKind::Tls, + tls_failure: None, + attempts: Vec::new(), } } @@ -74,6 +226,8 @@ impl ClientError { ClientError { message, kind: ErrorKind::InvalidUrl, + tls_failure: None, + attempts: Vec::new(), } } @@ -81,6 +235,8 @@ impl ClientError { ClientError { message, kind: ErrorKind::TooManyRedirects, + tls_failure: None, + attempts: Vec::new(), } } @@ -88,6 +244,52 @@ impl ClientError { ClientError { message, kind: ErrorKind::Status(status), + tls_failure: None, + attempts: Vec::new(), + } + } + + /// A TLS failure that knows why. Prefer this over `tls` wherever an + /// `openssl::ssl::Error` is in hand, since the reason is what the profile + /// ladder branches on and it cannot be recovered from the message text. + /// + /// A reset is a connection failure and a timeout is a timeout, whichever + /// stage they hit; `tls_failure` still says it was during the handshake. + pub fn tls_detailed(message: String, failure: TlsFailure) -> Self { + let kind = match failure { + TlsFailure::Reset => ErrorKind::Connection, + TlsFailure::Timeout => ErrorKind::Timeout, + _ => ErrorKind::Tls, + }; + ClientError { + message, + kind, + tls_failure: Some(failure), + attempts: Vec::new(), + } + } + + /// What this failure amounts to for a caller deciding what to do next. + /// + /// A peer that could not negotiate with us is reported separately from one + /// we never reached: the first is alive and wants cryptography we declined + /// or cannot do, which is actionable, and the second is not. Once + /// `compatibility`, the widest offer, has failed too, legacy TLS is not + /// what is missing. + pub fn conclusion(&self) -> crate::report::Conclusion { + let widest_tried = self + .attempts + .iter() + .any(|a| a.profile == crate::profile::COMPATIBILITY.name); + match &self.tls_failure { + Some(f) if f.suggests_wider_offer() && !widest_tried => { + crate::report::Conclusion::NeedsLegacyTls { + reason: self.message.clone(), + } + } + _ => crate::report::Conclusion::Unreachable { + reason: self.message.clone(), + }, } } @@ -95,6 +297,8 @@ impl ClientError { ClientError { message, kind: ErrorKind::Other, + tls_failure: None, + attempts: Vec::new(), } } } @@ -329,3 +533,69 @@ mod tests { assert_eq!(config.max_retries(), 0); } } + +#[cfg(test)] +mod tls_failure_tests { + use super::*; + + #[test] + fn test_negotiation_failures_suggest_a_wider_offer() { + // The three shapes a "we could not agree" answer arrives in. + assert!(TlsFailure::NoSharedCipher.suggests_wider_offer()); + assert!(TlsFailure::UnsupportedProtocol.suggests_wider_offer()); + assert!( + TlsFailure::Alert { + description: "ssl/tls alert handshake failure".to_string(), + } + .suggests_wider_offer(), + "a handshake_failure alert is what a real cipher mismatch looks \ + like on the wire; excluding it would defeat the ladder in its \ + most common case" + ); + } + + #[test] + fn test_other_failures_do_not_suggest_a_wider_offer() { + // Widening the offer in response to any of these would send weaker + // crypto to a peer whose objection had nothing to do with our ciphers. + assert!( + !TlsFailure::CertificateVerify { + detail: "self signed certificate".to_string(), + } + .suggests_wider_offer() + ); + assert!(!TlsFailure::Reset.suggests_wider_offer()); + assert!(!TlsFailure::Timeout.suggests_wider_offer()); + assert!( + !TlsFailure::Other { + detail: "something else".to_string(), + } + .suggests_wider_offer() + ); + } + + #[test] + fn test_constructors_leave_tls_failure_unset() { + // Only the TLS-aware constructor carries detail; everything else must + // not claim to know why a handshake failed. + assert!(ClientError::connection("x".into()).tls_failure.is_none()); + assert!(ClientError::timeout("x".into()).tls_failure.is_none()); + assert!(ClientError::tls("x".into()).tls_failure.is_none()); + + let detailed = ClientError::tls_detailed("x".into(), TlsFailure::NoSharedCipher); + assert_eq!(detailed.kind, ErrorKind::Tls); + assert_eq!(detailed.tls_failure, Some(TlsFailure::NoSharedCipher)); + } + + #[test] + fn test_a_handshake_reset_or_timeout_keeps_its_own_kind() { + let reset = ClientError::tls_detailed("x".into(), TlsFailure::Reset); + assert_eq!(reset.kind, ErrorKind::Connection); + assert!(reset.kind.is_retryable()); + assert_eq!(reset.tls_failure, Some(TlsFailure::Reset)); + + let timeout = ClientError::tls_detailed("x".into(), TlsFailure::Timeout); + assert_eq!(timeout.kind, ErrorKind::Timeout); + assert_eq!(timeout.tls_failure, Some(TlsFailure::Timeout)); + } +} diff --git a/src/client/raw.rs b/src/client/raw.rs index fad3a85..05f7177 100644 --- a/src/client/raw.rs +++ b/src/client/raw.rs @@ -36,13 +36,12 @@ impl RawConnection { ClientError::invalid_url(format!("invalid URL '{}': {}", uri_str, e)) })?; let log = new_debug_log(); - let (stream, cert_info, negotiated_alpn, peer_ip) = - connect_stream(&uri, config, &log).await?; + let conn = connect_stream(&uri, config, &log).await?; Ok(Self { - stream: Arc::new(Mutex::new(Some(stream))), - cert_info, - negotiated_alpn, - peer_ip, + stream: Arc::new(Mutex::new(Some(conn.stream))), + cert_info: conn.cert_info, + negotiated_alpn: conn.alpn, + peer_ip: conn.peer_ip, }) } diff --git a/src/config.rs b/src/config.rs index be31eaa..6ddf987 100644 --- a/src/config.rs +++ b/src/config.rs @@ -28,6 +28,17 @@ pub struct RequestConfig { /// `corp`), single IPs, CIDR ranges (`10.0.0.0/8`), or `*` for all. #[serde(default)] pub no_proxy: Vec, + /// Browser profile to impersonate, e.g. `"chrome"`. Applies a matching + /// TLS configuration, HTTP/2 settings and default headers together. + /// + /// Applied as a unit on purpose. A profile whose layers disagree is worse + /// than none: a client claiming to be Chrome whose handshake says + /// otherwise has stated something checkably false. + /// + /// An explicit `cipher_string`, `min_tls_version` or `max_tls_version` + /// overrides the profile's value for that field, on the grounds that a + /// caller naming something specific means it. + pub profile: Option, pub cipher_string: Option, pub min_tls_version: Option, pub max_tls_version: Option, @@ -53,6 +64,11 @@ pub struct RequestConfig { pub alpn_protocols: Option>, #[serde(default)] pub verbosity: u8, + /// Set when the ladder, not the caller, chose `profile`. The profile's + /// browser identity headers then replace the caller's, since a switch + /// made because of a denial has to claim one browser consistently. + #[serde(skip)] + pub profile_from_ladder: bool, } impl RequestConfig { @@ -70,6 +86,7 @@ impl RequestConfig { verify_certs: None, proxy: None, no_proxy: Vec::new(), + profile: None, cipher_string: None, min_tls_version: None, max_tls_version: None, @@ -81,6 +98,96 @@ impl RequestConfig { resolve_ip: None, alpn_protocols: None, verbosity: 0, + profile_from_ladder: false, + } + } + + /// The connection profile this request should use. + /// + /// Infallible, because every request has a profile now: naming none means + /// the default rather than "no profile", and the code that applies one no + /// longer has an absent case to branch on. An unrecognised name falls back + /// to the default here; `validate_profile` is what turns it into an error, + /// so a typo is caught once at the edge rather than silently changing + /// behaviour deep in the connector. + pub fn resolved_profile(&self) -> &'static crate::profile::ConnectionProfile { + if let Some(name) = self.profile.as_deref() { + return crate::profile::by_name(name) + .unwrap_or_else(|_| crate::profile::default_profile()); + } + // Asking for a pre-TLS1.2 floor is asking for the legacy profile, + // whether or not the caller said so. The default offers only modern + // ciphers, none of which exist in SSLv3 or TLS 1.0, so honouring the + // version alone produces "no ciphers available": a request that names + // a protocol, and is told that protocol is impossible. + // + // Naming a cipher string is the same request from the other side: the + // caller wants exactly those ciphers, and the default's TLS 1.2 floor + // would make RC4 or 3DES against a TLS 1.0 server impossible. Under + // `compatibility` the string replaces its list and no floor applies. + // + // The ladder cannot rescue either, because both pin the configuration + // and pinning is what stops the ladder moving. + if self.wants_legacy_protocol() || self.cipher_string.is_some() { + return crate::profile::by_name("compatibility") + .unwrap_or_else(|_| crate::profile::default_profile()); + } + crate::profile::default_profile() + } + + /// Did the caller ask to speak something older than TLS 1.2? + fn wants_legacy_protocol(&self) -> bool { + let is_legacy = |v: &str| { + matches!( + v.to_ascii_lowercase().as_str(), + "3.0" + | "ssl3" + | "sslv3" + | "ssl3.0" + | "1.0" + | "tls1.0" + | "tlsv1.0" + | "1.1" + | "tls1.1" + | "tlsv1.1" + ) + }; + self.min_tls_version.as_deref().is_some_and(is_legacy) + || self.max_tls_version.as_deref().is_some_and(is_legacy) + } + + /// This config with a different profile selected. + /// + /// Used by the ladder to retry a hop differently. A clone rather than a + /// parameter threaded through the connector because `TlsKey`, and so the + /// cached-client lookup, is derived from the config: changing the profile + /// anywhere else would key the cache on one profile while building the + /// connector from another. + pub fn with_profile(&self, name: &str) -> RequestConfig { + let mut c = self.clone(); + c.profile_from_ladder = !self.tls_is_pinned(); + c.profile = Some(name.to_string()); + c + } + + /// Whether the caller pinned TLS settings themselves. + /// + /// When they did, the ladder must not move: naming a cipher string, a TLS + /// version or a profile means they meant it, and silently substituting + /// another profile would send something they did not ask for. + /// `tests/tls_integration.rs` depends on this. + pub fn tls_is_pinned(&self) -> bool { + self.profile.is_some() + || self.cipher_string.is_some() + || self.min_tls_version.is_some() + || self.max_tls_version.is_some() + } + + /// Reject an unknown profile name. + pub fn validate_profile(&self) -> Result<(), String> { + match self.profile.as_deref() { + Some(name) => crate::profile::by_name(name).map(|_| ()), + None => Ok(()), } } @@ -271,6 +378,27 @@ fn ip_in_cidr(ip: IpAddr, net: IpAddr, prefix_len: u8) -> bool { mod tests { use super::*; + fn profile_of(c: &RequestConfig) -> &'static str { + c.resolved_profile().name + } + + #[test] + fn a_cipher_string_alone_selects_compatibility() { + let mut c = RequestConfig::new("https://example.com".to_string()); + assert_eq!(profile_of(&c), "modern"); + c.cipher_string = Some("RC4-MD5".to_string()); + assert_eq!(profile_of(&c), "compatibility"); + } + + #[test] + fn a_named_profile_wins_over_a_cipher_string() { + // Asking for both means the profile's version floor applies too. + let mut c = RequestConfig::new("https://example.com".to_string()); + c.cipher_string = Some("RC4-MD5".to_string()); + c.profile = Some("modern".to_string()); + assert_eq!(profile_of(&c), "modern"); + } + fn pats(v: &[&str]) -> Vec { v.iter().map(|s| s.to_string()).collect() } diff --git a/src/lib.rs b/src/lib.rs index a2f112b..733c6c8 100644 --- a/src/lib.rs +++ b/src/lib.rs @@ -1,9 +1,12 @@ +pub mod antibot; pub mod batch; pub mod client; pub mod config; pub mod cookies; pub mod debug; pub mod h2; +pub mod profile; +pub mod report; pub mod response; #[cfg(feature = "python")] diff --git a/src/main.rs b/src/main.rs index 691c41d..0b82dfa 100644 --- a/src/main.rs +++ b/src/main.rs @@ -69,6 +69,10 @@ struct Cli { #[arg(long = "no-proxy", num_args = 1)] no_proxy: Vec, + /// Browser profile to imitate (e.g. "chrome") + #[arg(long)] + profile: Option, + /// OpenSSL cipher string (e.g. "ALL", "HIGH", "RC4-SHA") #[arg(long)] ciphers: Option, @@ -120,6 +124,7 @@ fn build_config(cli: &Cli, url: String) -> RequestConfig { config.max_body_size = cli.max_body_size; config.proxy = cli.proxy.clone(); config.no_proxy = cli.no_proxy.clone(); + config.profile = cli.profile.clone(); config.cipher_string = cli.ciphers.clone(); config.min_tls_version = cli.min_tls.clone(); config.max_tls_version = cli.max_tls.clone(); diff --git a/src/mock.rs b/src/mock.rs index 1552c27..6e5ff36 100644 --- a/src/mock.rs +++ b/src/mock.rs @@ -15,7 +15,8 @@ // (auth tuple → header, cookies dict → header, raise_error, etc.) // is the consumer's responsibility, not ours. -use crate::python::PyResponse; +use crate::client::ClientError; +use crate::python::{PyResponse, TransportError, client_error_to_py}; use crate::response::Response; use pyo3::exceptions::{PyRuntimeError, PyStopAsyncIteration, PyTypeError}; use pyo3::prelude::*; @@ -276,6 +277,7 @@ fn make_response( status: status_code, headers, body_bytes: body, + attempts: Vec::new(), elapsed_ms: 0, redirect_chain: Vec::new(), cert_info: None, @@ -438,10 +440,12 @@ fn plan_dispatch( } let Some(handler) = pick_handler(py, state, url, method, &req_headers_dict, body)? else { - return Err(PyRuntimeError::new_err(format!( + // A TransportError like the real client's for an unreachable host, + // so `except TransportError` handles both the same way. + return Err(client_error_to_py(ClientError::connection(format!( "No mock response registered for {} {}", method, url - ))); + )))); }; match handler.kind { @@ -668,9 +672,12 @@ impl PyBlasthttpMock { } /// Dispatch a single request through the mock. Mirrors the - /// `BlastHTTP.request` shape — most kwargs (timeout, verify_certs, - /// retries, etc.) are accepted but ignored, since they don't apply - /// to a mock. + /// `BlastHTTP.request` shape. Kwargs the mock has no use for (timeout, + /// verify_certs, profile, retries and the rest) are accepted and ignored + /// when the URL is intercepted, since nothing is dialled. When the URL is + /// excluded and the call goes to the real client they are forwarded + /// verbatim, because then they describe a connection that actually + /// happens. #[pyo3(signature = ( url, method=None, @@ -704,6 +711,7 @@ impl PyBlasthttpMock { files, follow_redirects, max_redirects, + _kwargs, ); } @@ -796,9 +804,11 @@ impl PyBlasthttpMock { Some(loc) => { hops += 1; if hops > max_hops { - return Err(PyRuntimeError::new_err(format!( - "exceeded {} redirects starting from mock dispatch", - max_hops + return Err(client_error_to_py(ClientError::too_many_redirects( + format!( + "exceeded {} redirects starting from mock dispatch", + max_hops + ), ))); } current_url = loc; @@ -950,6 +960,7 @@ impl PyBlasthttpMock { files: Option>, follow_redirects: Option, max_redirects: Option, + rest: Option>, ) -> PyResult> { let Some(ref client) = self.real_client else { return Err(PyRuntimeError::new_err(format!( @@ -957,6 +968,18 @@ impl PyBlasthttpMock { ))); }; let kwargs = PyDict::new(py); + // Everything the mock does not name itself goes through untouched. + // A mocked URL can ignore `timeout`, `profile` or `verify_certs` + // because nothing is dialled, but a passed-through one is a real + // request and dropping them here would quietly give it different + // TLS, a different profile and no timeout from what the caller + // asked for. Set first, so the named arguments below win on the + // (impossible from Python, but cheap to guarantee) chance of overlap. + if let Some(ref extra) = rest { + for (k, val) in extra.iter() { + kwargs.set_item(k, val)?; + } + } if let Some(m) = method { kwargs.set_item("method", m)?; } @@ -1046,7 +1069,7 @@ impl PyMockBatchIterator { return Python::attach(|py| -> PyResult> { match resp_or_err { Ok(resp) => batch_result_obj(py, &entry.url, Some(resp), None), - Err(e) => batch_result_obj(py, &entry.url, None, Some(e.to_string())), + Err(e) => batch_error_obj(py, &entry.url, e), } }); } @@ -1138,6 +1161,23 @@ fn batch_result_obj( Ok(cls.call((url,), Some(&kwargs))?.unbind()) } +/// Build a failed `blasthttp.BatchResult` from a mock dispatch error. +/// +/// A `TransportError` goes on `transport_error`, with its bare message as +/// `error` the way a real batch reports it. Anything else, like an exception +/// from a user's callback, keeps the old `"Type: message"` string. +fn batch_error_obj(py: Python<'_>, url: &str, e: PyErr) -> PyResult> { + let value = e.value(py); + if !value.is_instance_of::() { + return batch_result_obj(py, url, None, Some(e.to_string())); + } + let cls = py.import("blasthttp")?.getattr("BatchResult")?; + let kwargs = PyDict::new(py); + kwargs.set_item("error", value.str()?)?; + kwargs.set_item("transport_error", value)?; + Ok(cls.call((url,), Some(&kwargs))?.unbind()) +} + // ── Submodule registration ──────────────────────────────────────── pub fn register_mock_submodule<'py>(parent: &Bound<'py, PyModule>) -> PyResult<()> { diff --git a/src/profile.rs b/src/profile.rs new file mode 100644 index 0000000..8254071 --- /dev/null +++ b/src/profile.rs @@ -0,0 +1,604 @@ +//! Connection profiles: what we put on the wire, and what we claim to be. +//! +//! A profile is data, not code, because browsers ship every few weeks and one +//! that has to be rewritten each time will not be kept current. +//! +//! Profiles differ on two independent axes, which is why a single ladder from +//! "safe" to "aggressive" does not describe them: +//! +//! breadth narrow current ciphers <-> everything, SSLv3 upward +//! claim no identity claim <-> claiming to be a browser +//! +//! | profile | breadth | claim | for | +//! |---|---|---|---| +//! | [`COMPATIBILITY`] | everything | honest | ancient servers, TLS enumeration | +//! | [`MODERN`] | narrow | honest | ordinary traffic | +//! | [`CHROME_131`] | Chrome's list | Chrome | hosts that reward imitation | +//! +//! Two measured findings shape this. +//! +//! A profile must be applied whole or not at all. Browser headers over a +//! non-browser handshake are worse than honest ones: `udemy.com` returns 200 +//! to an honest curl and 403 with `cf-mitigated: challenge` to the same client +//! wearing only a Chrome User-Agent. +//! +//! And claiming to be a browser is not free. The User-Agent is not what gets +//! scored, it selects which scoring applies: claim a browser and the client is +//! checked against the claim. On `priceline.com`, plain blasthttp gets through +//! 3/3 and our Chrome profile 0/3, while curl_cffi, whose imitation survives +//! the check, gets 3/3. So `MODERN` exists to be unremarkable without +//! inviting that inspection. + +/// The TLS half of a browser profile. +/// +/// Fields map onto the OpenSSL calls that set them. TLS 1.3 suites and TLS 1.2 +/// suites are separate because OpenSSL configures them through different +/// functions: `SSL_CTX_set_ciphersuites` and `SSL_CTX_set_cipher_list`. Setting +/// only the latter, which is all blasthttp did before this, leaves the 1.3 +/// suites at OpenSSL's defaults in OpenSSL's order. +#[derive(Debug, Clone, Copy)] +pub struct TlsProfile { + /// TLS 1.3 suites, in order, colon separated + /// (`SSL_CTX_set_ciphersuites`). `None` leaves OpenSSL's own list and + /// order, which is what a profile wanting to look like an ordinary + /// OpenSSL client should do. + pub ciphersuites: Option<&'static str>, + /// TLS 1.2 and below, in order, colon separated (`SSL_CTX_set_cipher_list`). + pub cipher_list: &'static str, + /// Signature algorithms, in order. JA4 hashes this list as sent rather + /// than sorted, so the order is part of the fingerprint. `None` leaves + /// OpenSSL's. + pub sigalgs: Option<&'static str>, + /// Supported groups, in order. These do not affect JA4, which counts only + /// the presence of the extension and not its contents, but they are + /// visible to anything reading the raw hello. `None` leaves OpenSSL's. + pub groups: Option<&'static str>, + /// `None` leaves the floor and ceiling entirely to OpenSSL, which at + /// security level 0 with NO_SSLV3 cleared means SSLv3 through TLS 1.3. + pub min_version: Option<&'static str>, + pub max_version: Option<&'static str>, + pub alpn: &'static [&'static str], + /// Send the extensions a browser sends and suppress the two it does not. + /// + /// An explicit field rather than "is a profile set", which is how this + /// worked before there was more than one profile. Left implicit, naming + /// today's behaviour as a profile would have switched certificate-status + /// requests and SCT on, and padding and encrypt_then_mac off, for the very + /// configuration that is supposed to be byte-identical to what shipped. + pub browser_extensions: bool, +} + +/// The HTTP/2 half of a profile. +/// +/// Values are what the browser puts in its SETTINGS frame and its +/// connection-level WINDOW_UPDATE, which together make up most of what is +/// commonly called the Akamai HTTP/2 fingerprint. +/// +/// Two fields of that fingerprint are deliberately absent because our stack +/// cannot reach them without forking dependencies: +/// +/// HEADER_TABLE_SIZE (setting 1): hyper carries it internally but +/// hyper-util does not expose it, so there is no way to set it from here. +/// +/// Pseudo-header order: `h2` hardcodes method, scheme, authority, path in +/// `Iter::next()`, and Chrome sends method, authority, scheme, path. +/// Changing it means forking h2, hyper and hyper-util, since each layer +/// translates a fixed set of options rather than passing them through. +#[derive(Debug, Clone, Copy)] +pub struct Http2Profile { + /// SETTINGS 4, INITIAL_WINDOW_SIZE. + pub initial_stream_window: u32, + /// The connection window. The WINDOW_UPDATE a browser sends is this minus + /// the protocol default of 65535, so Chrome's 15663105 increment comes + /// from asking for 15 MiB. + pub initial_connection_window: u32, + /// SETTINGS 6, MAX_HEADER_LIST_SIZE. + pub max_header_list_size: u32, + /// SETTINGS 5, MAX_FRAME_SIZE. `None` leaves it out, which is what Chrome + /// does; our default sends 16384 and Chrome sends nothing. + pub max_frame_size: Option, +} + +/// A complete profile: TLS, plus the HTTP layers that have to agree with it. +#[derive(Debug, Clone, Copy)] +pub struct ConnectionProfile { + pub name: &'static str, + pub tls: TlsProfile, + /// `None` leaves hyper-util's HTTP/2 defaults, which match no browser but + /// are the right answer for a profile not pretending to be one. + pub http2: Option, + /// Default request headers, in the order they go on the wire. HTTP/1.1 + /// header order is preserved by this client, so this ordering is real. + /// Empty means the client's own minimal defaults. + pub headers: &'static [(&'static str, &'static str)], + /// Whether this profile claims to be a browser. + /// + /// The axis the ladder moves along on a policy block, and the reason it is + /// a field rather than something inferred from the headers: what matters + /// is whether we invited a browser-shaped inspection, not which browser we + /// named. + pub claims_browser: bool, +} + +/// Chrome 131 on Windows. +/// +/// Chosen over a newer Chrome deliberately. Chrome 133 and later sign with +/// ML-DSA (signature algorithms 0x0904 through 0x0906), which OpenSSL 3.3.2 +/// does not know, so its signature_algorithms list cannot be reproduced and +/// its JA4 is out of reach. Chrome 131's list is entirely standard. +/// +/// The cipher and extension sets are identical between Chrome 124 and 150, so +/// this is not a stale profile in the way the version number suggests; only +/// the signature algorithms moved. Both Chrome 131 and 124 produce +/// `t13d1516h2_8daaf6152771_02713d6af862`. +pub const CHROME_131: ConnectionProfile = ConnectionProfile { + name: "chrome131", + tls: TlsProfile { + // 1301, 1302, 1303. + ciphersuites: Some( + "TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256", + ), + // c02b, c02f, c02c, c030, cca9, cca8, c013, c014, 009c, 009d, 002f, 0035. + cipher_list: "ECDHE-ECDSA-AES128-GCM-SHA256:\ + ECDHE-RSA-AES128-GCM-SHA256:\ + ECDHE-ECDSA-AES256-GCM-SHA384:\ + ECDHE-RSA-AES256-GCM-SHA384:\ + ECDHE-ECDSA-CHACHA20-POLY1305:\ + ECDHE-RSA-CHACHA20-POLY1305:\ + ECDHE-RSA-AES128-SHA:\ + ECDHE-RSA-AES256-SHA:\ + AES128-GCM-SHA256:\ + AES256-GCM-SHA384:\ + AES128-SHA:\ + AES256-SHA", + // 0403, 0804, 0401, 0503, 0805, 0501, 0806, 0601, in that order. + sigalgs: Some( + "ECDSA+SHA256:RSA-PSS+SHA256:RSA+SHA256:\ + ECDSA+SHA384:RSA-PSS+SHA384:RSA+SHA384:\ + RSA-PSS+SHA512:RSA+SHA512", + ), + // Chrome also offers X25519MLKEM768 (4588) ahead of these, which + // OpenSSL gained in 3.5 and ours cannot send. It does not affect JA4, + // but it is visible in the raw hello, and a client claiming to be a + // current Chrome while offering no post-quantum share is itself worth + // noticing. + groups: Some("X25519:P-256:P-384"), + min_version: Some("1.2"), + max_version: Some("1.3"), + alpn: &["h2", "http/1.1"], + browser_extensions: true, + }, + // Chrome's Akamai fingerprint is + // 1:65536;2:0;4:6291456;6:262144|15663105|0|m,a,s,p. ENABLE_PUSH is + // already 0 because hyper always disables push. + http2: Some(Http2Profile { + initial_stream_window: 6_291_456, + // 15 MiB. The WINDOW_UPDATE that results is 15728640 - 65535 = + // 15663105, which is what Chrome sends. + initial_connection_window: 15_728_640, + max_header_list_size: 262_144, + max_frame_size: None, + }), + headers: &[ + ( + "sec-ch-ua", + "\"Google Chrome\";v=\"131\", \"Chromium\";v=\"131\", \"Not_A Brand\";v=\"24\"", + ), + ("sec-ch-ua-mobile", "?0"), + ("sec-ch-ua-platform", "\"Windows\""), + ("upgrade-insecure-requests", "1"), + ( + "user-agent", + "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 \ + (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36", + ), + ( + "accept", + "text/html,application/xhtml+xml,application/xml;q=0.9,\ + image/avif,image/webp,image/apng,*/*;q=0.8,\ + application/signed-exchange;v=b3;q=0.7", + ), + ("sec-fetch-site", "none"), + ("sec-fetch-mode", "navigate"), + ("sec-fetch-user", "?1"), + ("sec-fetch-dest", "document"), + ("accept-encoding", "gzip, deflate, br, zstd"), + ("accept-language", "en-US,en;q=0.9"), + ("priority", "u=0, i"), + ], + claims_browser: true, +}; + +/// Everything, SSLv3 upward, with an honest User-Agent. +/// +/// A faithful transcription of what blasthttp sent before profiles existed, +/// and it has to stay that way: `tests/legacy_default.rs` and +/// `test_compat_path_keeps_the_scsv` are its specification. Every `None` here +/// is deliberate, meaning "leave OpenSSL alone", because that is what the +/// unnamed default did. +/// +/// Honest but conspicuous. The cipher list resolves to 105 suites, and the +/// resulting hello has never been observed in roughly 17 billion connections +/// recorded by tlsfingerprint.io. That is the cost of reaching a server from +/// 2003, and it is why it is no longer the thing you get by accident. +pub const COMPATIBILITY: ConnectionProfile = ConnectionProfile { + name: "compatibility", + tls: TlsProfile { + ciphersuites: None, + // Every suite the build provides except the eNULL ones. Includes the + // aNULL suites, which skip authentication: worth reaching for a + // scanner while verification is off, and dropped whenever it is on + // (see `verified_cipher_list`). + cipher_list: "ALL", + sigalgs: None, + groups: None, + min_version: None, + max_version: None, + alpn: &["h2", "http/1.1"], + browser_extensions: false, + }, + http2: None, + headers: &[], + claims_browser: false, +}; + +/// Narrow current ciphers, no identity claim. +/// +/// Sits between the other two: `COMPATIBILITY` is honest but offers 105 suites +/// and every protocol version back to SSLv3; `CHROME_131` looks ordinary but +/// makes a claim it cannot fully back. This offers a current, boring cipher +/// list and claims nothing. +/// +/// What it buys, measured: +/// +/// JA4 goes from `t13d10512h2` to `t13d1113h2`. Eleven suites instead of a +/// hundred and five, which is a shape plenty of software has, where 105 is +/// a shape essentially nothing has. +/// +/// SSLv3, TLS 1.0 and TLS 1.1 stop being offered. A client advertising +/// SSLv3 in 2026 is conspicuous on its own. +/// +/// What it does NOT buy, also measured, and worth recording because it was the +/// original argument for this profile: it is not "unremarkable in real +/// traffic". Checked against tlsfingerprint.io, which indexes roughly 17 +/// billion observed connections, all three of our profiles come back never +/// seen, `MODERN` included. Narrowing the cipher list fixes the JA4 and does +/// nothing for the rest of the hello, which stays distinctive for reasons +/// OpenSSL gives us no way to change: no GREASE anywhere, and extensions in +/// OpenSSL's fixed order rather than a browser's shuffled one. +/// +/// So the case for it rests on the JA4 and the version floor, not on blending +/// in. Blending in is not available on this TLS stack. +/// +/// It cannot reach a server that speaks only deprecated ciphers. That is what +/// falling back to `COMPATIBILITY` is for. +pub const MODERN: ConnectionProfile = ConnectionProfile { + name: "modern", + tls: TlsProfile { + ciphersuites: Some( + "TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:TLS_AES_128_GCM_SHA256", + ), + // ECDHE with AEAD, which is what current guidance and every modern + // library converge on. Twelve suites rather than a hundred and five. + cipher_list: "ECDHE-ECDSA-AES128-GCM-SHA256:\ + ECDHE-RSA-AES128-GCM-SHA256:\ + ECDHE-ECDSA-AES256-GCM-SHA384:\ + ECDHE-RSA-AES256-GCM-SHA384:\ + ECDHE-ECDSA-CHACHA20-POLY1305:\ + ECDHE-RSA-CHACHA20-POLY1305:\ + DHE-RSA-AES128-GCM-SHA256:\ + DHE-RSA-AES256-GCM-SHA384", + sigalgs: None, + groups: Some("X25519:P-256:P-384:P-521"), + // No SSLv3, TLS 1.0 or 1.1. Offering them is one of the louder things + // the old default did. + min_version: Some("1.2"), + max_version: Some("1.3"), + alpn: &["h2", "http/1.1"], + browser_extensions: false, + }, + http2: None, + headers: &[], + claims_browser: false, +}; + +/// Look a profile up by name. +/// +/// Unknown names are an error rather than silently resolving to the default. +/// They used to: a typo bought you whatever the default was, with nothing said +/// about it, which is a poor trade once the choice of profile decides whether +/// a request is inspected. +pub fn by_name(name: &str) -> Result<&'static ConnectionProfile, String> { + match name.to_ascii_lowercase().as_str() { + "compatibility" | "compat" => Ok(&COMPATIBILITY), + "modern" => Ok(&MODERN), + // "chrome" tracks the newest Chrome we can actually reproduce, which + // is what a caller asking for "chrome" wants. + "chrome" | "chrome131" | "browser" => Ok(&CHROME_131), + other => Err(format!( + "unknown profile '{}' (known: compatibility, modern, chrome)", + other + )), + } +} + +/// The profile a request gets when it does not name one. +/// +/// `MODERN` rather than `COMPATIBILITY`, which is the breaking change in 1.0. +/// A request that meets a server too old for it does not fail: the ladder +/// widens to `COMPATIBILITY` on a negotiation failure, at the cost of one +/// extra handshake against such a server and nothing at all against the rest. +/// +/// The old default offered 105 cipher suites and every protocol version back +/// to SSLv3 to every host on the internet, to stay reachable for the small +/// number that need it. This inverts that: be ordinary by default, and pay +/// for breadth only where breadth is actually wanted. +pub fn default_profile() -> &'static ConnectionProfile { + &MODERN +} + +#[cfg(test)] +// These assertions are deliberately about constants: the profiles are data, +// and the point of the tests is that editing that data fails loudly. Clippy +// is right that they fold at compile time and wrong that this makes them +// pointless. +#[allow(clippy::assertions_on_constants)] +mod tests { + use super::*; + + #[test] + fn test_lookup_resolves_and_rejects() { + assert_eq!(by_name("chrome").unwrap().name, "chrome131"); + assert_eq!(by_name("CHROME131").unwrap().name, "chrome131"); + assert_eq!(by_name("modern").unwrap().name, "modern"); + assert_eq!(by_name("compat").unwrap().name, "compatibility"); + + // An unknown name is an error, not a silent fallback. It used to be a + // fallback, which meant a typo bought you the default with nothing + // said about it. + let err = by_name("netscape").unwrap_err(); + assert!( + err.contains("netscape"), + "error should name the input: {err}" + ); + assert!( + err.contains("compatibility"), + "error should list the options" + ); + } + + #[test] + fn test_compatibility_leaves_openssl_alone() { + // This profile has to stay byte-identical to what shipped before + // profiles existed, and every `None` is how that is expressed. + // `tests/legacy_default.rs` is the end-to-end proof; this is the + // cheap guard that catches an accidental edit. + let c = COMPATIBILITY.tls; + assert_eq!(c.cipher_list, "ALL"); + assert!(c.ciphersuites.is_none(), "would reorder the TLS 1.3 suites"); + assert!(c.sigalgs.is_none()); + assert!(c.groups.is_none()); + assert!( + c.min_version.is_none() && c.max_version.is_none(), + "a version floor would put SSLv3 and TLS 1.0 out of reach" + ); + assert!( + !c.browser_extensions, + "browser extensions would add SCT and status_request and drop \ + padding and encrypt_then_mac, none of which the old default did" + ); + assert!( + COMPATIBILITY.http2.is_none(), + "would change the SETTINGS frame" + ); + assert!(COMPATIBILITY.headers.is_empty()); + assert!(!COMPATIBILITY.claims_browser); + } + + #[test] + fn test_modern_is_narrow_and_honest() { + let m = MODERN.tls; + // The point of this profile: a short list instead of 105 suites. + assert_eq!(m.cipher_list.split(':').count(), 8); + assert_eq!(m.ciphersuites.unwrap().split(':').count(), 3); + assert_eq!(m.min_version, Some("1.2"), "no SSLv3, TLS 1.0 or 1.1"); + assert!( + !m.browser_extensions && !MODERN.claims_browser && MODERN.headers.is_empty(), + "modern must not claim to be a browser; that is the whole \ + difference between it and chrome131" + ); + } + + #[test] + fn test_chrome_matches_the_target_shape() { + // 12 TLS 1.2 suites plus 3 TLS 1.3 suites is the 15 JA4 counts for + // Chrome. Getting this wrong moves the JA4 header. + let c = CHROME_131.tls; + assert_eq!(c.cipher_list.split(':').count(), 12); + assert_eq!(c.ciphersuites.unwrap().split(':').count(), 3); + assert_eq!(c.sigalgs.unwrap().split(':').count(), 8); + assert!(c.browser_extensions); + assert!(CHROME_131.claims_browser); + assert!(CHROME_131.http2.is_some()); + } + + #[test] + fn test_chrome_headers_are_in_browser_order() { + // Order is part of the fingerprint, so guard both ends of it. + let names: Vec<&str> = CHROME_131.headers.iter().map(|(n, _)| *n).collect(); + assert_eq!(names.first(), Some(&"sec-ch-ua")); + assert_eq!(names.last(), Some(&"priority")); + assert!(names.contains(&"sec-fetch-dest")); + } + + #[test] + fn test_only_chrome_claims_to_be_a_browser() { + // The axis the ladder moves along on a policy block. + assert!(!COMPATIBILITY.claims_browser); + assert!(!MODERN.claims_browser); + assert!(CHROME_131.claims_browser); + } +} + +// ── The ladder ──────────────────────────────────────────────────── + +/// Why an attempt did not produce a usable response. +/// +/// The two variants move along different axes and must not be conflated. A +/// transport failure says the peer could not negotiate with us, which is about +/// breadth. A policy block says it negotiated fine and then refused, which is +/// about what we claimed to be. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum RungFailure { + /// The handshake failed in a way suggesting a wider offer might work. + /// Retrying is free: no application data was sent. + CouldNotNegotiate, + /// A clean handshake, then a refusal or a challenge. + Refused, +} + +/// Which profile to try next, if any. +/// +/// Deliberately a decision rather than a fixed list, because the two axes have +/// different triggers and a linear chain cannot express that. +/// +/// On a negotiation failure, widen: `modern` and `chrome131` both offer narrow +/// modern ciphers, so the answer is `compatibility`, which offers everything +/// back to SSLv3. From `compatibility` there is nowhere wider to go. +/// +/// On a refusal, change what we claim. Empirically rather than by rule: the +/// evidence points both ways depending on the host, so try the other side once +/// and let the result decide. Akamai, Kasada and Cloudflare deployments have +/// rewarded the browser claim; PerimeterX has punished it, on every target +/// tested. Nothing here inspects the vendor, because two deployments of one +/// product routinely disagree. +/// `tried` is every profile already attempted for this hop, and it is not +/// optional bookkeeping. Without it the honest and browser profiles oscillate: +/// a refusal moves `modern` to `chrome131`, and a refusal there moves it back +/// to `modern`, forever. Termination is a property of this function rather +/// than of the cap at the call site. +pub fn next_rung( + current: &ConnectionProfile, + failure: RungFailure, + tried: &[&str], +) -> Option<&'static ConnectionProfile> { + let candidate = match failure { + RungFailure::CouldNotNegotiate => { + if current.name == COMPATIBILITY.name { + // Already offering everything; the same bytes would go out + // again and waste a handshake on a server that just refused + // them. + None + } else { + Some(&COMPATIBILITY) + } + } + RungFailure::Refused => { + if current.claims_browser { + // Drop the claim. A client that says it is Chrome gets checked + // against that; one that says nothing does not. + Some(&MODERN) + } else if current.name == COMPATIBILITY.name { + // Both honest and conspicuous, so there are two things to + // change. Narrowing is the smaller step and leaves the browser + // claim available afterwards. + Some(&MODERN) + } else { + Some(&CHROME_131) + } + } + }?; + + if tried.contains(&candidate.name) { + None + } else { + Some(candidate) + } +} + +/// How many attempts one hop may make. +/// +/// Three is the longest sensible walk: compatibility narrows to modern, modern +/// claims a browser, and that is the end of it. `next_rung` already terminates +/// on its own by refusing to revisit; this is the belt to that braces, because +/// the cost of being wrong is an unbounded request loop against someone else's +/// server. +pub const MAX_RUNGS: usize = 3; + +#[cfg(test)] +mod ladder_tests { + use super::*; + + #[test] + fn test_negotiation_failure_widens_to_compatibility() { + assert_eq!( + next_rung(&MODERN, RungFailure::CouldNotNegotiate, &["modern"]).map(|p| p.name), + Some("compatibility") + ); + assert_eq!( + next_rung(&CHROME_131, RungFailure::CouldNotNegotiate, &["chrome131"]).map(|p| p.name), + Some("compatibility") + ); + } + + #[test] + fn test_nowhere_wider_than_compatibility() { + // Offering everything already; a second attempt would send the same + // bytes and waste a handshake on a server that just refused them. + assert!( + next_rung( + &COMPATIBILITY, + RungFailure::CouldNotNegotiate, + &["compatibility"] + ) + .is_none() + ); + } + + #[test] + fn test_refusal_drops_a_browser_claim() { + // The PerimeterX case: claiming Chrome invited an inspection we + // failed, so stop claiming it. + let next = next_rung(&CHROME_131, RungFailure::Refused, &["chrome131"]).unwrap(); + assert_eq!(next.name, "modern"); + assert!(!next.claims_browser); + } + + #[test] + fn test_refusal_from_an_honest_profile_tries_the_claim() { + // The Akamai case: honest gets refused, a browser claim gets through. + assert_eq!( + next_rung(&MODERN, RungFailure::Refused, &["modern"]).map(|p| p.name), + Some("chrome131") + ); + } + + #[test] + fn test_the_walk_terminates_within_the_cap() { + // Guards the cap against a future edit to next_rung that cycles. + for start in [&COMPATIBILITY, &MODERN, &CHROME_131] { + for failure in [RungFailure::CouldNotNegotiate, RungFailure::Refused] { + let mut seen = vec![start.name]; + let mut current = start; + while let Some(next) = next_rung(current, failure, &seen) { + assert!( + !seen.contains(&next.name), + "ladder revisited {} starting from {} on {:?}", + next.name, + start.name, + failure + ); + seen.push(next.name); + current = next; + assert!( + seen.len() <= MAX_RUNGS, + "walk from {} on {:?} exceeded MAX_RUNGS: {:?}", + start.name, + failure, + seen + ); + } + } + } + } +} diff --git a/src/python.rs b/src/python.rs index 4f217e1..6a5b01c 100644 --- a/src/python.rs +++ b/src/python.rs @@ -15,10 +15,12 @@ use std::time::Duration; use tokio::sync::Mutex as TokioMutex; use tokio::time::Instant; +use crate::antibot as blasthttp_antibot; use crate::batch::{self, BatchResult, RateLimiter}; use crate::client::HttpClient; use crate::client::hyper::HyperClient; use crate::client::raw; +use crate::client::{ClientError, ErrorKind, TlsFailure}; use crate::config::RequestConfig; use crate::multipart; use crate::response::{CertInfo, RedirectHop, Response, ResponseHash}; @@ -380,6 +382,84 @@ pyo3::create_exception!( "Raised by Response.raise_for_status() when the response status is 4xx or 5xx." ); +// ── Turning a ClientError into a Python exception ───────────────── + +pyo3::create_exception!( + blasthttp, + TransportError, + pyo3::exceptions::PyRuntimeError, + "Raised when a request never produced a response: DNS, TCP, TLS or timeout." +); + +/// Convert a `ClientError` into a Python exception that keeps what we know. +/// +/// Every call site used to do `PyRuntimeError::new_err(e.message)`, which threw +/// away `kind` and left Python to substring-match a sentence to find out +/// whether a host was unreachable, timed out, or refused our ciphers. The +/// message stays the same so existing `except RuntimeError` keeps working +/// (TransportError subclasses it), but the detail is now attached as +/// attributes: +/// +/// `kind` one of connection, timeout, tls, invalid_url, +/// too_many_redirects, status, other +/// `tls_failure` when the handshake failed, why: no_shared_cipher, +/// unsupported_protocol, alert, certificate_verify, reset, +/// timeout, other. `None` otherwise. +/// `retryable` whether trying the same request again might help. +/// Render a `Conclusion` as the `(name, vendor)` pair Python sees. +fn conclusion_to_py(c: &crate::report::Conclusion) -> (String, String) { + use crate::report::Conclusion; + let (name, vendor) = match c { + Conclusion::Reached => ("reached", None), + Conclusion::Blocked { vendor } => ("blocked", vendor.clone()), + Conclusion::NeedsBrowser { vendor } => ("needs_browser", vendor.clone()), + Conclusion::NeedsLegacyTls { .. } => ("needs_legacy_tls", None), + Conclusion::Unreachable { .. } => ("unreachable", None), + }; + (name.to_string(), vendor.unwrap_or_else(|| "-".to_string())) +} + +pub(crate) fn client_error_to_py(e: ClientError) -> PyErr { + let kind = match &e.kind { + ErrorKind::Connection => "connection", + ErrorKind::Timeout => "timeout", + ErrorKind::Tls => "tls", + ErrorKind::InvalidUrl => "invalid_url", + ErrorKind::TooManyRedirects => "too_many_redirects", + ErrorKind::Status(_) => "status", + ErrorKind::Other => "other", + }; + let tls_failure = e.tls_failure.as_ref().map(|f| match f { + TlsFailure::NoSharedCipher => "no_shared_cipher", + TlsFailure::UnsupportedProtocol => "unsupported_protocol", + TlsFailure::Alert { .. } => "alert", + TlsFailure::CertificateVerify { .. } => "certificate_verify", + TlsFailure::Reset => "reset", + TlsFailure::Timeout => "timeout", + TlsFailure::Other { .. } => "other", + }); + let retryable = e.kind.is_retryable(); + + let (conclusion, conclusion_vendor) = conclusion_to_py(&e.conclusion()); + let attempts: Vec<(String, String)> = e + .attempts + .iter() + .map(|a| (a.profile.clone(), format!("{:?}", a.outcome))) + .collect(); + + let err = TransportError::new_err(e.message); + Python::attach(|py| { + let v = err.value(py); + let _ = v.setattr("kind", kind); + let _ = v.setattr("tls_failure", tls_failure); + let _ = v.setattr("retryable", retryable); + let _ = v.setattr("conclusion", conclusion); + let _ = v.setattr("conclusion_vendor", conclusion_vendor); + let _ = v.setattr("attempts", attempts); + }); + err +} + // ── Response wrapper ────────────────────────────────────────────── #[pyclass(name = "Response")] @@ -468,6 +548,9 @@ impl PyResponse { status, headers: headers.unwrap_or_default(), body_bytes, + // A response built from Python was not produced by the ladder, so + // it has nothing to report about what was tried. + attempts: Vec::new(), elapsed_ms, redirect_chain: redirect_chain .map(|hops| hops.into_iter().map(|h| h.inner.clone()).collect()) @@ -649,6 +732,83 @@ impl PyResponse { /// Debug messages collected during the request. /// Always populated — Python side can log/display as needed. + /// What a bot-management product did to this request, if anything. + /// + /// Returns a tuple of `(outcome, vendor)`, both lowercase strings: + /// + /// outcome ok | present | challenge | blocked + /// vendor cloudflare | akamai | datadome | perimeterx | imperva | + /// f5 | kasada | unknown | "-" when nothing was detected + /// + /// `present` means a product is in front of the host and let us through, + /// usually having issued a session cookie. That is a more useful answer + /// than a bare 200: it says we passed something that was actively looking. + /// + /// `challenge` is the one to branch on. A blocked request might succeed + /// with different settings; a challenge will not, because answering it + /// means running the page's JavaScript. It is the signal to hand the URL + /// to a real browser rather than retry. + /// What this request amounts to, as a single verdict a caller can branch + /// on. Returns `(conclusion, vendor)`: + /// + /// reached got the page, or the server's own answer such as + /// a 404 or a 500 + /// blocked refused. vendor names the product when one was + /// recognised; "-" is an ordinary refusal (auth, + /// forbidden path, method not allowed) that a + /// different connection shape won't change + /// needs_browser a JavaScript challenge; hand this URL to a real + /// browser, because no HTTP client passes this tier + /// needs_legacy_tls the peer wants cryptography we declined or cannot do + /// unreachable never got a response + /// + /// `needs_browser` is the one worth acting on. It means retrying with + /// different settings cannot help. A `blocked` with a named vendor is + /// the only refusal the ladder tries other settings against, so by the + /// time it is reported they have been tried. + #[getter] + fn conclusion(&self) -> (String, String) { + conclusion_to_py(&self.inner.conclusion()) + } + + /// What the profile ladder tried, in order, as `(profile, outcome, status)` + /// tuples. Usually one entry; more than one means the first approach did + /// not work and the client moved. + #[getter] + fn attempts(&self) -> Vec<(String, String, Option)> { + use crate::report::AttemptOutcome; + self.inner + .attempts + .iter() + .map(|a| { + let (kind, status) = match &a.outcome { + AttemptOutcome::Reached { status } => ("reached", Some(*status)), + AttemptOutcome::Refused { status, .. } => ("refused", Some(*status)), + AttemptOutcome::Challenged { status, .. } => ("challenged", Some(*status)), + AttemptOutcome::HandshakeFailed { .. } => ("handshake_failed", None), + }; + (a.profile.clone(), kind.to_string(), status) + }) + .collect() + } + + #[getter] + fn protection(&self) -> (String, String) { + let outcome = self.inner.protection(); + let name = match &outcome { + blasthttp_antibot::Outcome::Ok => "ok", + blasthttp_antibot::Outcome::Present(_) => "present", + blasthttp_antibot::Outcome::Challenge(_) => "challenge", + blasthttp_antibot::Outcome::Blocked(_) => "blocked", + blasthttp_antibot::Outcome::Error => "error", + }; + let vendor = outcome + .vendor() + .map(|v| v.to_string()) + .unwrap_or_else(|| "-".to_string()); + (name.to_string(), vendor) + } + #[getter] fn debug_log(&self) -> Vec { self.inner.debug_log.clone() @@ -703,6 +863,13 @@ struct PyBatchResult { url: String, response: Option, error: Option, + /// The failure behind `error`, kept so `transport_error` can say more + /// than the message. Turned into a Python object only when asked for, so + /// a batch doesn't take the GIL for every failed request. + client_error: Option, + /// A `TransportError` handed to the constructor, which wins over + /// `client_error`. + transport_error: Option>, } #[pymethods] @@ -711,15 +878,30 @@ impl PyBatchResult { /// fixture mocks. Provide either `response` (a Response instance) /// for success, or `error` (a string) for failure. Both default to /// None; passing neither yields a degenerate "no response, no error" - /// result that `success` reports as False. + /// result that `success` reports as False. `transport_error`, if + /// given, must be a `TransportError`. #[new] - #[pyo3(signature = (url, response=None, error=None))] - fn new(url: String, response: Option>, error: Option) -> Self { - PyBatchResult { + #[pyo3(signature = (url, response=None, error=None, transport_error=None))] + fn new( + url: String, + response: Option>, + error: Option, + transport_error: Option>, + ) -> PyResult { + if let Some(ref t) = transport_error + && !t.is_instance_of::() + { + return Err(pyo3::exceptions::PyTypeError::new_err( + "transport_error must be a TransportError", + )); + } + Ok(PyBatchResult { url, response: response.map(|r| r.inner.clone()), error, - } + client_error: None, + transport_error: transport_error.map(Bound::unbind), + }) } #[getter] @@ -743,6 +925,19 @@ impl PyBatchResult { self.error.clone() } + /// The `TransportError` that `request()` would have raised for this + /// failure, with `kind`, `tls_failure`, `retryable`, `conclusion` and + /// `attempts`. None if the request succeeded. + #[getter] + fn transport_error(&self, py: Python<'_>) -> Option> { + if let Some(ref t) = self.transport_error { + return Some(t.clone_ref(py)); + } + self.client_error + .clone() + .map(|e| client_error_to_py(e).into_value(py).into_any()) + } + /// True if the request succeeded #[getter] fn success(&self) -> bool { @@ -759,14 +954,16 @@ impl PyBatchResult { } fn to_py_batch_result(r: BatchResult) -> PyBatchResult { - let (response, error) = match r.result { - Ok(resp) => (Some(resp), None), - Err(e) => (None, Some(e.message)), + let (response, error, client_error) = match r.result { + Ok(resp) => (Some(resp), None, None), + Err(e) => (None, Some(e.message.clone()), Some(e)), }; PyBatchResult { url: r.url, response, error, + client_error, + transport_error: None, } } @@ -919,6 +1116,7 @@ impl BlastHTTP { verify_certs=None, proxy=None, no_proxy=None, + profile=None, cipher_string=None, min_tls_version=None, max_tls_version=None, @@ -947,6 +1145,7 @@ impl BlastHTTP { verify_certs: Option, proxy: Option, no_proxy: Option>, + profile: Option, cipher_string: Option, min_tls_version: Option, max_tls_version: Option, @@ -973,6 +1172,7 @@ impl BlastHTTP { verify_certs, proxy, no_proxy: no_proxy.unwrap_or_default(), + profile, cipher_string, min_tls_version, max_tls_version, @@ -984,6 +1184,7 @@ impl BlastHTTP { resolve_ip, alpn_protocols, verbosity: 0, + profile_from_ladder: false, }; let limiter = self.rate_limiter.clone(); @@ -993,10 +1194,7 @@ impl BlastHTTP { if let Some(ref limiter) = limiter { limiter.acquire().await; } - let response = client - .send(&config) - .await - .map_err(|e| PyRuntimeError::new_err(e.message))?; + let response = client.send(&config).await.map_err(client_error_to_py)?; Ok(PyResponse { inner: response, headers_cache: OnceLock::new(), @@ -1126,6 +1324,10 @@ impl BlastHTTP { /// Download a URL directly to a local file. /// Returns the file path on success. /// max_size: maximum bytes to download (None = no limit, uses default 10MB) + /// + /// Takes the same connection knobs as `request`. Without them the profile + /// ladder runs as usual, so a download from a host that needs legacy TLS + /// or a browser profile reaches it the same way an ordinary request does. #[pyo3(signature = ( url, path, @@ -1137,6 +1339,10 @@ impl BlastHTTP { headers=None, retries=None, redirect_cookies=None, + profile=None, + cipher_string=None, + min_tls_version=None, + max_tls_version=None, ))] #[allow(clippy::too_many_arguments)] fn download<'py>( @@ -1152,6 +1358,10 @@ impl BlastHTTP { headers: Option>, retries: Option, redirect_cookies: Option, + profile: Option, + cipher_string: Option, + min_tls_version: Option, + max_tls_version: Option, ) -> PyResult> { let config = RequestConfig { url, @@ -1166,9 +1376,10 @@ impl BlastHTTP { verify_certs, proxy, no_proxy: no_proxy.unwrap_or_default(), - cipher_string: None, - min_tls_version: None, - max_tls_version: None, + profile, + cipher_string, + min_tls_version, + max_tls_version, retries, retry_wait_min_ms: None, retry_wait_max_ms: None, @@ -1177,6 +1388,7 @@ impl BlastHTTP { resolve_ip: None, alpn_protocols: None, verbosity: 0, + profile_from_ladder: false, }; let limiter = self.rate_limiter.clone(); @@ -1186,10 +1398,7 @@ impl BlastHTTP { if let Some(ref limiter) = limiter { limiter.acquire().await; } - let response = client - .send(&config) - .await - .map_err(|e| PyRuntimeError::new_err(e.message))?; + let response = client.send(&config).await.map_err(client_error_to_py)?; // Write body bytes to file let mut file = std::fs::File::create(&path).map_err(|e| { @@ -1218,6 +1427,7 @@ impl BlastHTTP { #[pyo3(signature = ( url, verify_certs=None, + profile=None, cipher_string=None, min_tls_version=None, max_tls_version=None, @@ -1233,6 +1443,7 @@ impl BlastHTTP { py: Python<'py>, url: String, verify_certs: Option, + profile: Option, cipher_string: Option, min_tls_version: Option, max_tls_version: Option, @@ -1244,6 +1455,7 @@ impl BlastHTTP { ) -> PyResult> { let mut config = RequestConfig::new(url.clone()); config.verify_certs = verify_certs; + config.profile = profile; config.cipher_string = cipher_string; config.min_tls_version = min_tls_version; config.max_tls_version = max_tls_version; @@ -1264,7 +1476,7 @@ impl BlastHTTP { } let conn = raw::RawConnection::connect(&url, &config) .await - .map_err(|e| PyRuntimeError::new_err(e.message))?; + .map_err(client_error_to_py)?; Ok(PyRawConnection { inner: Arc::new(conn), rate_limiter: limiter_for_conn, @@ -1299,10 +1511,7 @@ impl PyRawConnection { if let Some(ref limiter) = limiter { limiter.acquire().await; } - inner - .send_bytes(&data) - .await - .map_err(|e| PyRuntimeError::new_err(e.message))?; + inner.send_bytes(&data).await.map_err(client_error_to_py)?; Ok(()) }) } @@ -1330,7 +1539,7 @@ impl PyRawConnection { let data = inner .read_raw(max_bytes, timeout_ms) .await - .map_err(|e| PyRuntimeError::new_err(e.message))?; + .map_err(client_error_to_py)?; Ok(data) }) } @@ -1339,10 +1548,7 @@ impl PyRawConnection { fn close<'py>(&self, py: Python<'py>) -> PyResult> { let inner = self.inner.clone(); future_into_py(py, async move { - inner - .close() - .await - .map_err(|e| PyRuntimeError::new_err(e.message))?; + inner.close().await.map_err(client_error_to_py)?; Ok(()) }) } @@ -1404,6 +1610,8 @@ struct PyBatchConfig { #[pyo3(get, set)] no_proxy: Option>, #[pyo3(get, set)] + profile: Option, + #[pyo3(get, set)] cipher_string: Option, #[pyo3(get, set)] min_tls_version: Option, @@ -1439,6 +1647,7 @@ impl PyBatchConfig { verify_certs=None, proxy=None, no_proxy=None, + profile=None, cipher_string=None, min_tls_version=None, max_tls_version=None, @@ -1465,6 +1674,7 @@ impl PyBatchConfig { verify_certs: Option, proxy: Option, no_proxy: Option>, + profile: Option, cipher_string: Option, min_tls_version: Option, max_tls_version: Option, @@ -1490,6 +1700,7 @@ impl PyBatchConfig { verify_certs, proxy, no_proxy, + profile, cipher_string, min_tls_version, max_tls_version, @@ -1520,6 +1731,7 @@ impl Clone for PyBatchConfig { verify_certs: self.verify_certs, proxy: self.proxy.clone(), no_proxy: self.no_proxy.clone(), + profile: self.profile.clone(), cipher_string: self.cipher_string.clone(), min_tls_version: self.min_tls_version.clone(), max_tls_version: self.max_tls_version.clone(), @@ -1553,6 +1765,7 @@ impl PyBatchConfig { verify_certs: self.verify_certs, proxy: self.proxy, no_proxy: self.no_proxy.unwrap_or_default(), + profile: self.profile, cipher_string: self.cipher_string, min_tls_version: self.min_tls_version, max_tls_version: self.max_tls_version, @@ -1564,6 +1777,7 @@ impl PyBatchConfig { resolve_ip: self.resolve_ip, alpn_protocols: self.alpn_protocols, verbosity: 0, + profile_from_ladder: false, }) } } @@ -2025,6 +2239,27 @@ fn register_h2_submodule<'py>(parent: &Bound<'py, PyModule>) -> PyResult<()> { Ok(()) } +/// Replace the anti-bot rules every response in this process is classified +/// with, from a JSON file shaped like the built-in `antibot_rules.json`. +/// `None` restores the built-in rules. Raises ValueError on a bad file. +#[pyfunction] +#[pyo3(signature = (path=None))] +fn set_antibot_rules(path: Option) -> PyResult<()> { + let rules = match path { + None => crate::antibot::Rules::default(), + Some(path) => { + let json = std::fs::read_to_string(&path).map_err(|e| { + pyo3::exceptions::PyValueError::new_err(format!("{}: {e}", path.display())) + })?; + crate::antibot::Rules::from_json(&json).map_err(|e| { + pyo3::exceptions::PyValueError::new_err(format!("{}: {e}", path.display())) + })? + } + }; + crate::antibot::set_rules(rules); + Ok(()) +} + // ── Module registration ─────────────────────────────────────────── #[pymodule] @@ -2043,7 +2278,9 @@ fn blasthttp(m: &Bound<'_, PyModule>) -> PyResult<()> { m.add_class::()?; m.add_class::()?; m.add_class::()?; + m.add_function(wrap_pyfunction!(set_antibot_rules, m)?)?; m.add("HTTPStatusError", m.py().get_type::())?; + m.add("TransportError", m.py().get_type::())?; register_h2_submodule(m)?; crate::mock::register_mock_submodule(m)?; register_headers_as_mapping(m)?; diff --git a/src/report.rs b/src/report.rs new file mode 100644 index 0000000..4ea3b6c --- /dev/null +++ b/src/report.rs @@ -0,0 +1,192 @@ +//! What was tried, and what it means. +//! +//! A request that walks the profile ladder makes several attempts, and the +//! caller sees only the last one. That is fine when it worked and useless when +//! it did not: "403" says nothing about whether a different approach exists, +//! whether one was already tried, or whether anything would have helped. +//! +//! The consumer this is for is BBOT, which has to tell four outcomes apart +//! that all look like "no page": +//! +//! - the host is dead +//! - the host needs legacy TLS we declined to offer +//! - a product blocked us, and here is which +//! - a product served a JavaScript challenge, so no HTTP client will ever +//! get through and the URL should go to a real browser +//! +//! Only the last of those justifies launching a browser, and it must be a +//! field rather than something inferred by grepping our response body for +//! `cf-mitigated`. + +use serde::{Deserialize, Serialize}; + +use crate::antibot::Outcome; + +/// What one rung of the ladder produced. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +pub enum AttemptOutcome { + /// Got a response the classifier was happy with. + Reached { status: u16 }, + /// A clean handshake, then a refusal. + Refused { + status: u16, + #[serde(skip_serializing_if = "Option::is_none")] + vendor: Option, + }, + /// A challenge: answering it needs JavaScript. + Challenged { + status: u16, + #[serde(skip_serializing_if = "Option::is_none")] + vendor: Option, + }, + /// Never got as far as a response. + HandshakeFailed { reason: String }, +} + +/// One rung: which profile, and what happened under it. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +pub struct Attempt { + pub profile: String, + pub outcome: AttemptOutcome, +} + +impl Attempt { + pub fn new(profile: &str, outcome: AttemptOutcome) -> Self { + Attempt { + profile: profile.to_string(), + outcome, + } + } + + /// Build from a classifier verdict. + pub fn from_outcome(profile: &str, status: u16, outcome: &Outcome) -> Self { + let vendor = outcome + .vendor() + .filter(|v| !v.is_unknown()) + .map(|v| v.to_string()); + let o = match outcome { + Outcome::Ok | Outcome::Present(_) => AttemptOutcome::Reached { status }, + Outcome::Challenge(_) => AttemptOutcome::Challenged { status, vendor }, + Outcome::Blocked(_) | Outcome::Error => AttemptOutcome::Refused { status, vendor }, + }; + Attempt::new(profile, o) + } +} + +/// What the whole request amounts to, once every rung has been tried. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +pub enum Conclusion { + /// Got the page, or the server's own answer such as a 404 or a 500. + Reached, + /// Refused. `vendor` names the product when one was recognised, and the + /// ladder has already tried what it could; `None` is an ordinary refusal + /// from the origin (auth, forbidden path) that no connection shape changes. + Blocked { + #[serde(skip_serializing_if = "Option::is_none")] + vendor: Option, + }, + /// A JavaScript challenge. **The signal to hand this URL to a real + /// browser.** No HTTP client passes this tier, curl_cffi included, and + /// retrying with different connection settings will not change it. + NeedsBrowser { + #[serde(skip_serializing_if = "Option::is_none")] + vendor: Option, + }, + /// The peer could not negotiate with us and we had nothing wider to offer. + /// Distinct from `Unreachable`: the host is alive and talking, it just + /// wants cryptography we declined or cannot do. + NeedsLegacyTls { reason: String }, + /// Never got a response at all. + Unreachable { reason: String }, +} + +impl Conclusion { + /// Should the caller hand this to a browser? + /// + /// The single question BBOT needs answered, exposed as a predicate so it + /// does not have to match on the variant. + pub fn needs_browser(&self) -> bool { + matches!(self, Conclusion::NeedsBrowser { .. }) + } + + /// Work out the conclusion from the last outcome and what was tried. + pub fn from_attempts(last: &Outcome, attempts: &[Attempt]) -> Self { + let vendor = last + .vendor() + .filter(|v| !v.is_unknown()) + .map(|v| v.to_string()); + match last { + Outcome::Ok | Outcome::Present(_) => Conclusion::Reached, + Outcome::Challenge(_) => Conclusion::NeedsBrowser { vendor }, + Outcome::Blocked(_) => Conclusion::Blocked { vendor }, + // A classifier error with attempts behind it means something went + // wrong that was not a refusal. + Outcome::Error => Conclusion::Unreachable { + reason: attempts + .last() + .map(|a| format!("{:?}", a.outcome)) + .unwrap_or_else(|| "no response".to_string()), + }, + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::antibot::Vendor; + + #[test] + fn test_a_challenge_asks_for_a_browser_and_a_block_does_not() { + // The distinction the whole escalation path rests on. A block might + // yield to different settings; a challenge needs JavaScript and never + // will, so only one of these justifies starting a browser. + let challenge = + Conclusion::from_attempts(&Outcome::Challenge(Vendor::new("cloudflare")), &[]); + assert!(challenge.needs_browser()); + assert_eq!( + challenge, + Conclusion::NeedsBrowser { + vendor: Some("cloudflare".to_string()) + } + ); + + let blocked = Conclusion::from_attempts(&Outcome::Blocked(Vendor::new("akamai")), &[]); + assert!(!blocked.needs_browser()); + } + + #[test] + fn test_an_unnamed_vendor_is_left_unnamed() { + // Reporting "unknown" as though it were a product would invite a + // consumer to branch on it. An absent vendor is the honest answer. + let blocked = Conclusion::from_attempts(&Outcome::Blocked(Vendor::unknown()), &[]); + assert_eq!(blocked, Conclusion::Blocked { vendor: None }); + } + + #[test] + fn test_reaching_it_is_not_an_escalation() { + for outcome in [Outcome::Ok, Outcome::Present(Vendor::new("akamai"))] { + let c = Conclusion::from_attempts(&outcome, &[]); + assert_eq!(c, Conclusion::Reached); + assert!(!c.needs_browser()); + } + } + + #[test] + fn test_attempts_record_the_profile_that_produced_them() { + // The log is only useful if it says which rung did what. + let a = Attempt::from_outcome( + "modern", + 403, + &Outcome::Challenge(Vendor::new("perimeterx")), + ); + assert_eq!(a.profile, "modern"); + assert_eq!( + a.outcome, + AttemptOutcome::Challenged { + status: 403, + vendor: Some("perimeterx".to_string()), + } + ); + } +} diff --git a/src/response.rs b/src/response.rs index b9651f4..c4b7820 100644 --- a/src/response.rs +++ b/src/response.rs @@ -32,6 +32,13 @@ pub struct Response { /// Debug messages collected during the request (for Python-side inspection) #[serde(skip_serializing)] pub debug_log: Vec, + /// What the profile ladder tried on the final hop, in order. + /// + /// Usually one entry. More than one means the first approach did not work + /// and the client moved, which is worth knowing even on success: it says + /// this host needs something other than the default. + #[serde(default)] + pub attempts: Vec, /// Why `body_bytes` is not decoded content, when it isn't. /// /// `None` on any ordinary response, including one with no @@ -79,7 +86,7 @@ impl Serialize for Response { /// callers that just want lazy memory behavior should access /// fields directly instead of serializing. fn serialize(&self, s: S) -> Result { - let mut out = s.serialize_struct("Response", 13)?; + let mut out = s.serialize_struct("Response", 14)?; out.serialize_field("url", &self.url)?; out.serialize_field("status", &self.status)?; out.serialize_field("headers", &self.headers)?; @@ -89,6 +96,7 @@ impl Serialize for Response { out.serialize_field("hash", self.hash())?; out.serialize_field("elapsed_ms", &self.elapsed_ms)?; out.serialize_field("redirect_chain", &self.redirect_chain)?; + out.serialize_field("attempts", &self.attempts)?; if let Some(ref c) = self.cert_info { out.serialize_field("cert_info", c)?; } else { @@ -121,6 +129,7 @@ impl Clone for Response { status: self.status, headers: self.headers.clone(), body_bytes: self.body_bytes.clone(), + attempts: self.attempts.clone(), elapsed_ms: self.elapsed_ms, redirect_chain: self.redirect_chain.clone(), cert_info: self.cert_info.clone(), @@ -147,6 +156,33 @@ impl Response { /// Body decoded as UTF-8 (lossy — invalid sequences become U+FFFD). /// Computed on first access and cached. + /// What, if anything, a bot-management product did to this request. + /// + /// Computed from the response rather than stored, so it stays correct if a + /// caller builds a `Response` by hand. Cheap: the classifier reads headers, + /// cookies, status and at most the first 64 KB of the body. + /// + /// The distinction worth acting on is `Challenge`. A blocked request might + /// succeed with different connection settings; a challenge will not, + /// because answering it means executing the page's JavaScript. That is the + /// signal to hand the URL to a real browser instead of retrying. + /// What this request amounts to, as a single verdict. + /// + /// `Conclusion::NeedsBrowser` is the one to branch on: it means a + /// JavaScript challenge, which no HTTP client passes, so the URL should go + /// to a real browser rather than be retried. + pub fn conclusion(&self) -> crate::report::Conclusion { + crate::report::Conclusion::from_attempts(&self.protection(), &self.attempts) + } + + pub fn protection(&self) -> crate::antibot::Outcome { + crate::antibot::classify(&crate::antibot::ResponseFacts { + status: self.status, + headers: &self.headers, + body: self.body(), + }) + } + pub fn body(&self) -> &str { self.body_cache .get_or_init(|| match std::str::from_utf8(&self.body_bytes) { @@ -452,3 +488,74 @@ mod tests { assert!(cookies.is_empty()); } } + +#[cfg(test)] +mod protection_tests { + use super::*; + use crate::antibot::{Outcome, Vendor}; + + fn response_with(status: u16, headers: &[(&str, &str)], body: &str) -> Response { + Response { + url: "https://example.com/".to_string(), + status, + headers: headers + .iter() + .map(|(k, v)| (k.to_string(), v.to_string())) + .collect(), + body_bytes: body.as_bytes().to_vec(), + attempts: Vec::new(), + elapsed_ms: 1, + redirect_chain: Vec::new(), + cert_info: None, + peer_ip: None, + request_url: "https://example.com/".to_string(), + request_method: "GET".to_string(), + debug_log: Vec::new(), + decode_error: None, + body_cache: OnceLock::new(), + raw_headers_cache: OnceLock::new(), + cookies_cache: OnceLock::new(), + hash_cache: OnceLock::new(), + } + } + + #[test] + fn test_response_classifies_itself() { + // The classifier is only useful if a caller can reach it from the + // response it already has. This is the path BBOT takes. + // + // Headers are the ones www.akamai.com actually returns, which notably + // include no `server` header at all. + let blocked = response_with( + 403, + &[ + ("akamai-grn", "0.6d551702.1790644996.315f62a4"), + ("content-type", "text/html"), + ], + "\nAccess Denied\n", + ); + assert_eq!( + blocked.protection(), + Outcome::Blocked(Vendor::new("akamai")) + ); + } + + #[test] + fn test_challenge_is_distinguishable_from_a_block() { + // The distinction the whole escalation path rests on: a block might + // succeed with different settings, a challenge never will. + let challenge = response_with( + 403, + &[("cf-mitigated", "challenge"), ("server", "cloudflare")], + "Just a moment...", + ); + assert_eq!( + challenge.protection(), + Outcome::Challenge(Vendor::new("cloudflare")) + ); + + let ordinary = response_with(200, &[("server", "nginx")], "hello"); + assert_eq!(ordinary.protection(), Outcome::Ok); + assert!(ordinary.protection().got_through()); + } +} diff --git a/tests/antibot_classify.rs b/tests/antibot_classify.rs new file mode 100644 index 0000000..69f97c4 --- /dev/null +++ b/tests/antibot_classify.rs @@ -0,0 +1,473 @@ +// The outcome classifier, against real captured responses. +// +// Fixtures are trimmed from responses actually observed on 2026-09-28, not +// invented. The Akamai one in particular is the exact page `www.akamai.com` +// returned to blasthttp while returning 200 with real content to a +// Chrome-impersonating client from the same address in the same minute. That +// pair is the clearest single measurement of the problem this effort exists to +// fix, so it is worth having as a regression fixture. + +use blasthttp::antibot::{Outcome, ResponseFacts, Vendor, classify}; + +fn hdrs(pairs: &[(&str, &str)]) -> Vec<(String, String)> { + pairs + .iter() + .map(|(k, v)| (k.to_string(), v.to_string())) + .collect() +} + +fn facts<'a>(status: u16, headers: &'a [(String, String)], body: &'a str) -> ResponseFacts<'a> { + ResponseFacts { + status, + headers, + body, + } +} + +// ── Real captures ───────────────────────────────────────────────── + +/// Exactly what www.akamai.com served blasthttp, 366 bytes, reproduced 3/3. +const AKAMAI_DENIED: &str = concat!( + "\nAccess Denied\n\n

Access Denied

\n \n", + "You don't have permission to access \"http://www.akamai.com/\" ", + "on this server.

\nReference #18.d3952a17.1790609813.f486746\n", + "

https://errors.edgesuite.net/18.d3952a17.1790609813.f486746\n", + "\n\n" +); + +#[test] +fn test_akamai_access_denied_is_a_block() { + // These are the headers www.akamai.com actually returned, not a plausible + // set. Two things about them broke the first version of the classifier and + // are the reason this fixture is verbatim: + // + // - there is no `server` header at all, so a rule requiring + // `server: AkamaiGHost` never fired + // - the body entity-encodes its own punctuation, so a search for + // `errors.edgesuite.net` never matched `errors.edgesuite.net` + // + // Both were caught by running the live benchmark, which classified this + // exact response as Blocked(Unknown). + let h = hdrs(&[ + ("mime-version", "1.0"), + ("content-type", "text/html"), + ("content-length", "368"), + ("server-timing", "cdn-cache; desc=HIT"), + ( + "server-timing", + "ak_p; desc=\"1790644996525_35083629_828334756_15_11554_9_12_-\";dur=1", + ), + ("x-akam-sw-version", "0.5.0"), + ("akamai-grn", "0.6d551702.1790644996.315f62a4"), + ( + "set-cookie", + "akaas_production=2147483647~rv=32~id=e124a124de5a16eb7688b22c80f8e95d;", + ), + ]); + assert_eq!( + classify(&facts(403, &h, AKAMAI_DENIED)), + Outcome::Blocked(Vendor::new("akamai")) + ); +} + +#[test] +fn test_akamai_block_detected_without_a_server_header() { + // Narrower version of the above: the edge markers alone have to carry it. + let h = hdrs(&[("akamai-grn", "0.6d551702.1790644996.315f62a4")]); + assert_eq!( + classify(&facts( + 403, + &h, + "\nAccess Denied\n" + )), + Outcome::Blocked(Vendor::new("akamai")) + ); +} + +#[test] +fn test_entity_encoded_edgesuite_still_matches() { + // No Akamai headers whatsoever, just the encoded error page body. + let h = hdrs(&[("content-type", "text/html")]); + assert_eq!( + classify(&facts(403, &h, AKAMAI_DENIED)), + Outcome::Blocked(Vendor::new("akamai")) + ); +} + +#[test] +fn test_akamai_markers_on_a_page_that_loaded_are_not_a_block() { + // edgesuite.net also hosts customer assets, and a page can say "Access + // Denied" without being refused. Neither is a block on a 200. + let assets = r#""#; + let article = "

Fixing Access Denied errors

"; + for (h, body) in [ + (hdrs(&[("content-type", "text/html")]), assets), + ( + hdrs(&[("akamai-grn", "0.6d551702.1790644996.315f62a4")]), + article, + ), + ] { + let outcome = classify(&facts(200, &h, body)); + assert!(outcome.got_through(), "{body} classified as {outcome:?}"); + } +} + +#[test] +fn test_akamai_session_cookie_on_success_is_present() { + // The other half of the same comparison: through, and handed a session. + let h = hdrs(&[ + ("server", "AkamaiNetStorage"), + ( + "set-cookie", + "_abck=A7F1...~0~YAAQ...; Path=/; Domain=.akamai.com", + ), + ("set-cookie", "bm_sz=8C2...; Path=/"), + ]); + let outcome = classify(&facts(200, &h, "real content")); + assert_eq!(outcome, Outcome::Present(Vendor::new("akamai"))); + assert!(outcome.got_through()); +} + +#[test] +fn test_cloudflare_managed_challenge() { + // nowsecure.nl and the scrapingcourse challenge pages both answer this + // way. curl_cffi does not pass it either: this tier wants JavaScript, and + // no amount of fingerprint work reaches it. + let h = hdrs(&[ + ("server", "cloudflare"), + ("cf-mitigated", "challenge"), + ("cf-ray", "a423d7b45e21b0e8-ATL"), + ]); + assert_eq!( + classify(&facts(403, &h, "Just a moment...")), + Outcome::Challenge(Vendor::new("cloudflare")) + ); +} + +#[test] +fn test_cloudflare_script_is_not_a_challenge() { + // Turnstile's api.js names the challenge platform, but it's a script that + // loaded fine, not an interception. + let h = hdrs(&[ + ("server", "cloudflare"), + ("content-type", "application/javascript; charset=UTF-8"), + ]); + let body = r#"var u="/cdn-cgi/challenge-platform/";function render(){}"#; + assert_ne!( + classify(&facts(200, &h, body)), + Outcome::Challenge(Vendor::new("cloudflare")) + ); +} + +#[test] +fn test_cloudflare_javascript_detections_page_is_not_a_challenge() { + // Cloudflare injects this script tag into ordinary pages on sites with + // JavaScript detections on. + let h = hdrs(&[("server", "cloudflare"), ("content-type", "text/html")]); + let body = r#"welcome"#; + assert_ne!( + classify(&facts(200, &h, body)), + Outcome::Challenge(Vendor::new("cloudflare")) + ); +} + +#[test] +fn test_cloudflare_challenge_body_on_a_refusal_is_a_challenge() { + // Without cf-mitigated, the markers still identify the challenge page. + let h = hdrs(&[("server", "cloudflare")]); + let body = ""; + for status in [403, 503] { + assert_eq!( + classify(&facts(status, &h, body)), + Outcome::Challenge(Vendor::new("cloudflare")) + ); + } +} + +#[test] +fn test_cloudflare_waf_block_is_not_a_challenge() { + // A plain WAF refusal carries no cf-mitigated header. Calling it a + // challenge would suggest a JS tier that is not actually there. + let h = hdrs(&[("server", "cloudflare"), ("cf-ray", "a423d7b45e21b0e8-ATL")]); + let outcome = classify(&facts(403, &h, "error 1020")); + assert!( + matches!(outcome, Outcome::Blocked(_)), + "expected a block, got {:?}", + outcome + ); + assert_ne!(outcome, Outcome::Challenge(Vendor::new("cloudflare"))); +} + +#[test] +fn test_cloudflare_challenge_orchestrator_on_a_refusal_is_a_challenge() { + let h = hdrs(&[("server", "cloudflare")]); + let body = r#""#; + assert_eq!( + classify(&facts(403, &h, body)), + Outcome::Challenge(Vendor::new("cloudflare")) + ); +} + +#[test] +fn test_cloudflare_error_page_is_a_cloudflare_block() { + let h = hdrs(&[("server", "cloudflare")]); + let body = r#"

Error 1020

"#; + assert_eq!( + classify(&facts(403, &h, body)), + Outcome::Blocked(Vendor::new("cloudflare")) + ); +} + +#[test] +fn test_origin_refusal_behind_a_bot_product_is_not_vendor_evidence() { + // Bot products set their cookies and inject their scripts on every + // response, the origin's own refusals included. Those alone must not name + // the product as what refused, or every ordinary 401/403 walks the ladder. + let jsd = r#"

401 Unauthorized

"#; + for (cookie, body) in [ + ("__cf_bm=xyz; Path=/", jsd), + ("_abck=A7F1~0~YAAQ; Path=/", "forbidden"), + ("datadome=abc; Path=/", "forbidden"), + ("_px3=abc; Path=/", "forbidden"), + ("incap_ses_123=abc; Path=/", "forbidden"), + ] { + let h = hdrs(&[("set-cookie", cookie)]); + for status in [401, 403, 405] { + let outcome = classify(&facts(status, &h, body)); + assert_eq!( + outcome, + Outcome::Blocked(Vendor::unknown()), + "{cookie} at {status}" + ); + assert!(!outcome.indicates_protection()); + } + } +} + +#[test] +fn test_cloudflare_bm_cookie_on_success_is_present() { + // www.cloudflare.com sets __cf_bm on an ordinary 200. + let h = hdrs(&[ + ("server", "cloudflare"), + ("set-cookie", "__cf_bm=hNs8cwmP7rgTn3Us...; Path=/; Secure"), + ]); + assert_eq!( + classify(&facts(200, &h, "content")), + Outcome::Present(Vendor::new("cloudflare")) + ); +} + +#[test] +fn test_akamai_sec_cpt_428_is_its_own_outcome() { + // Fires after _abck has already passed, and answers 428 rather than 403. + // Folding it into "blocked" would hide that we got further than usual. + let h = hdrs(&[ + ("server", "AkamaiGHost"), + ("set-cookie", "sec_cpt=abc123; Path=/"), + ]); + assert_eq!( + classify(&facts(428, &h, r#"{"chlg_duration":30,"difficulty":5}"#)), + Outcome::Challenge(Vendor::new("akamai")) + ); +} + +/// The shape of Imperva's incident page, with its identifiers replaced. +const IMPERVA_INCIDENT: &str = concat!( + r#""#, + r#""#, + r#""# +); + +/// The loader Imperva adds to the end of pages it let through. +const IMPERVA_LOADER: &str = r#""#; + +#[test] +fn test_imperva_incident_page_is_a_challenge() { + let h = hdrs(&[("x-cdn", "Imperva")]); + for status in [200, 403, 503] { + assert_eq!( + classify(&facts(status, &h, IMPERVA_INCIDENT)), + Outcome::Challenge(Vendor::new("imperva")) + ); + } +} + +#[test] +fn test_imperva_loader_on_a_page_is_not_a_challenge() { + // The origin's own page, a 200 or its own 403, with Imperva's loader + // appended. + let h = hdrs(&[("x-cdn", "Imperva")]); + let page = format!( + "IIS Windows Serverwelcome{IMPERVA_LOADER}" + ); + let refusal = format!("

403 Forbidden

{IMPERVA_LOADER}"); + for (status, body) in [(200, &page), (403, &refusal)] { + assert_ne!( + classify(&facts(status, &h, body)), + Outcome::Challenge(Vendor::new("imperva")) + ); + } +} + +#[test] +fn test_imperva_script_without_the_loader_form_is_a_challenge() { + // Anything else naming the resource still counts, as before. + let h = hdrs(&[]); + let body = r#""#; + assert_eq!( + classify(&facts(200, &h, body)), + Outcome::Challenge(Vendor::new("imperva")) + ); +} + +// ── The false positives that caught the first pass ──────────────── + +#[test] +fn test_vendor_homepages_are_not_challenges() { + // Every one of these scored CHALLENGE on the first attempt, purely + // because a body grep found the vendor's own name in its marketing copy. + // A vendor homepage is a terrible probe and the classifier must not be + // fooled by one. + for (host, body) in [ + ( + "datadome.co", + "

DataDome

DataDome protects against bots. \ + Learn why DataDome is trusted.

", + ), + ( + "imperva.com", + "

Imperva

Imperva Incapsula delivers \ + application security.

", + ), + ( + "kasada.io", + "

Kasada

Kasada stops bots cold.

", + ), + ( + "humansecurity.com", + "

HUMAN, formerly PerimeterX, defends against bots.

", + ), + ] { + let h = hdrs(&[("content-type", "text/html")]); + assert_eq!( + classify(&facts(200, &h, body)), + Outcome::Ok, + "{} marketing copy was misread as a protection response", + host + ); + } +} + +#[test] +fn test_reference_number_alone_is_not_akamai() { + // Plenty of error pages carry a reference number. Without the edgesuite + // host or the AkamaiGHost server header it means nothing. + let h = hdrs(&[("server", "nginx")]); + let outcome = classify(&facts( + 403, + &h, + "

Access Denied

Reference #12345

", + )); + assert_eq!(outcome, Outcome::Blocked(Vendor::unknown())); +} + +// ── Ordinary responses ──────────────────────────────────────────── + +#[test] +fn test_plain_success_is_ok() { + let h = hdrs(&[("server", "nginx"), ("content-type", "text/html")]); + let outcome = classify(&facts(200, &h, "hello")); + assert_eq!(outcome, Outcome::Ok); + assert!(outcome.got_through()); +} + +#[test] +fn test_ordinary_404_is_not_a_block() { + // A missing page is a real answer from the origin, and counting it as a + // block would make an ordinary scan look like it was being refused. + let h = hdrs(&[("server", "nginx")]); + assert_eq!(classify(&facts(404, &h, "not found")), Outcome::Ok); +} + +#[test] +fn test_server_errors_are_answers() { + // A 500 or a 502 is the server answering, not a failure to reach it. + let h = hdrs(&[("server", "nginx")]); + for status in [400, 410, 500, 502, 504] { + let outcome = classify(&facts(status, &h, "error")); + assert_eq!(outcome, Outcome::Ok, "status {status}"); + } +} + +#[test] +fn test_unidentified_refusal_reports_unknown_vendor() { + // Better to say a block happened and admit we cannot name who did it than + // to guess a vendor. + let h = hdrs(&[("server", "nginx")]); + let outcome = classify(&facts(403, &h, "forbidden")); + assert_eq!(outcome, Outcome::Blocked(Vendor::unknown())); + assert!(!outcome.got_through()); +} + +#[test] +fn test_aws_waf_captcha_is_a_challenge() { + // wizzair.com answers 405 with `captcha` rather than `challenge`. A rule + // matching only the latter classified this as an ordinary refusal, which + // meant the profile ladder saw no reason to try anything else against a + // host that a browser profile does in fact reach. + let h = hdrs(&[ + ("server", "CloudFront"), + ("x-amzn-waf-action", "captcha"), + ("x-cache", "Error from cloudfront"), + ]); + let outcome = classify(&facts(405, &h, "")); + assert_eq!(outcome, Outcome::Challenge(Vendor::new("awswaf"))); + assert!( + outcome.indicates_protection(), + "the ladder only moves on positive evidence of a product" + ); +} + +#[test] +fn test_a_bare_403_is_not_evidence_of_protection() { + // The counterpart, and the reason `indicates_protection` exists at all. + // Most 403s are ordinary authorization failures, and retrying each one + // with different TLS settings would double the request count of any scan + // that touches one. + let h = hdrs(&[("server", "nginx")]); + let outcome = classify(&facts(403, &h, "forbidden")); + assert_eq!(outcome, Outcome::Blocked(Vendor::unknown())); + assert!(!outcome.indicates_protection()); +} + +#[test] +fn test_a_body_cut_mid_character_does_not_panic() { + // `max_body_size` cuts the body at a byte count, so the last character can + // be left half-written. The classifier scans a 64 KiB window from the top, + // and slicing a str at a byte that is not a character boundary panics + // rather than erring, which took the whole request down. + let h = hdrs(&[("server", "nginx")]); + + // One single-byte character, then two-byte ones, puts every boundary on an + // odd offset. The window ends at 65536, which is therefore inside a + // character rather than between two. + let body = format!("a{}", "\u{5ae}".repeat(40_000)); + assert!(body.len() > 65_536); + assert!(!body.is_char_boundary(65_536)); + + let outcome = classify(&facts(200, &h, &body)); + assert_eq!(outcome, Outcome::Ok); +} + +#[test] +fn test_a_marker_in_a_multibyte_body_is_still_found() { + // Walking back to a boundary must not cost us the match itself. + let h = hdrs(&[("server", "cloudflare")]); + let body = format!("{}window._cf_chl_opt", "\u{5ae}".repeat(10)); + assert_eq!( + classify(&facts(403, &h, &body)), + Outcome::Challenge(Vendor::new("cloudflare")) + ); +} diff --git a/tests/antibot_rules.rs b/tests/antibot_rules.rs new file mode 100644 index 0000000..33185b5 --- /dev/null +++ b/tests/antibot_rules.rs @@ -0,0 +1,115 @@ +// Loading anti-bot rules from JSON: what a rules file can say, and how a bad +// one fails. + +use blasthttp::antibot::{Outcome, ResponseFacts, Rules, Vendor}; + +fn facts<'a>(status: u16, headers: &'a [(String, String)], body: &'a str) -> ResponseFacts<'a> { + ResponseFacts { + status, + headers, + body, + } +} + +fn header(name: &str, value: &str) -> Vec<(String, String)> { + vec![(name.to_string(), value.to_string())] +} + +const CUSTOM: &str = r#"{ + "refusal_statuses": [403], + "rules": [ + { + "vendor": "examplewaf", + "outcome": "challenge", + "when": {"all": [{"status": "refusal"}, {"body_contains": "example-challenge"}]} + }, + { + "vendor": "examplewaf", + "outcome": "present", + "on_refusal": "blocked", + "when": {"cookie_prefix": "exwaf_"} + }, + { + "vendor": "otherwaf", + "outcome": "blocked", + "when": {"all": [ + {"header_contains": {"name": "X-Edge", "value": "OtherWAF"}}, + {"not": {"status": [200]}} + ]} + } + ] +}"#; + +#[test] +fn test_a_rules_file_can_name_a_new_vendor() { + let rules = Rules::from_json(CUSTOM).unwrap(); + let none = vec![]; + assert_eq!( + rules.classify(&facts(403, &none, "example-challenge")), + Outcome::Challenge(Vendor::new("examplewaf")) + ); + // the same body on a status the file doesn't call a refusal + assert_eq!( + rules.classify(&facts(503, &none, "example-challenge")), + Outcome::Ok + ); +} + +#[test] +fn test_on_refusal_changes_the_outcome_on_a_refusal_only() { + let rules = Rules::from_json(CUSTOM).unwrap(); + let cookie = header("set-cookie", "exwaf_session=abc; Path=/"); + assert_eq!( + rules.classify(&facts(200, &cookie, "welcome")), + Outcome::Present(Vendor::new("examplewaf")) + ); + assert_eq!( + rules.classify(&facts(403, &cookie, "go away")), + Outcome::Blocked(Vendor::new("examplewaf")) + ); +} + +#[test] +fn test_header_contains_ignores_case() { + let rules = Rules::from_json(CUSTOM).unwrap(); + let edge = header("x-edge", "otherwaf-7"); + assert_eq!( + rules.classify(&facts(404, &edge, "")), + Outcome::Blocked(Vendor::new("otherwaf")) + ); + assert_eq!(rules.classify(&facts(200, &edge, "")), Outcome::Ok); +} + +#[test] +fn test_a_rules_file_replaces_the_built_in_rules() { + let rules = Rules::from_json(CUSTOM).unwrap(); + let mitigated = header("cf-mitigated", "challenge"); + assert_eq!( + Rules::default().classify(&facts(403, &mitigated, "")), + Outcome::Challenge(Vendor::new("cloudflare")) + ); + assert_eq!( + rules.classify(&facts(403, &mitigated, "")), + Outcome::Blocked(Vendor::unknown()) + ); +} + +#[test] +fn test_a_bad_rules_file_is_refused() { + for bad in [ + // not JSON + "{", + // a condition that doesn't exist + r#"{"refusal_statuses": [], "rules": [{"vendor": "x", "outcome": "challenge", "when": {"body_matches": "x"}}]}"#, + // an outcome that doesn't exist + r#"{"refusal_statuses": [], "rules": [{"vendor": "x", "outcome": "maybe", "when": {"body_contains": "x"}}]}"#, + // a misspelled field + r#"{"refusal_statuses": [], "rules": [{"vendor": "x", "outcome": "challenge", "wen": {"body_contains": "x"}}]}"#, + // no vendor + r#"{"refusal_statuses": [], "rules": [{"vendor": "", "outcome": "challenge", "when": {"body_contains": "x"}}]}"#, + // no rules at all + r#"{"refusal_statuses": []}"#, + ] { + assert!(Rules::from_json(bad).is_err(), "accepted: {bad}"); + } +} diff --git a/tests/antibot_set_rules.rs b/tests/antibot_set_rules.rs new file mode 100644 index 0000000..9f6d2b5 --- /dev/null +++ b/tests/antibot_set_rules.rs @@ -0,0 +1,31 @@ +// `set_rules` swaps the rules every later classification in the process +// runs. Its own test binary, since the rules are process-wide. + +use blasthttp::antibot::{Outcome, ResponseFacts, Rules, Vendor, classify, set_rules}; + +#[test] +fn test_set_rules_replaces_the_rules_classify_runs() { + let none = vec![]; + let page = ResponseFacts { + status: 200, + headers: &none, + body: "example-challenge", + }; + assert_eq!(classify(&page), Outcome::Ok); + + set_rules( + Rules::from_json( + r#"{"refusal_statuses": [403], "rules": [ + {"vendor": "examplewaf", "outcome": "challenge", "when": {"body_contains": "example-challenge"}} + ]}"#, + ) + .unwrap(), + ); + assert_eq!( + classify(&page), + Outcome::Challenge(Vendor::new("examplewaf")) + ); + + set_rules(Rules::default()); + assert_eq!(classify(&page), Outcome::Ok); +} diff --git a/tests/burst_scheduler.rs b/tests/burst_scheduler.rs new file mode 100644 index 0000000..1f51676 --- /dev/null +++ b/tests/burst_scheduler.rs @@ -0,0 +1,156 @@ +// Does a burst against one host stop re-discovering that host once there is +// other work to get on with? +// +// The ladder trigger here is an RC4-only server rather than a protection +// product: a stock client offers `modern`, cannot negotiate, and widens to +// `compatibility`. Two attempts, visible in `Response::attempts`, and no +// vendor spoofing needed. What the scheduler does with them is the same +// either way. + +mod support; + +use blasthttp::batch::send_batch; +use blasthttp::client::hyper::HyperClient; +use blasthttp::config::RequestConfig; +use openssl::ssl::SslVersion; +use std::sync::Arc; +use support::tls_server::{TlsServerConfig, TlsTestServer}; + +fn config(url: &str) -> RequestConfig { + let mut c = RequestConfig::new(url.to_string()); + c.verify_certs = Some(false); + c.timeout_seconds = Some(5); + c +} + +async fn legacy_server() -> TlsTestServer { + TlsTestServer::start(TlsServerConfig { + cipher_list: Some("RC4-SHA".to_string()), + max_tls_version: Some(SslVersion::TLS1_2), + ..Default::default() + }) + .await +} + +async fn modern_server() -> TlsTestServer { + TlsTestServer::start(TlsServerConfig { + min_tls_version: Some(SslVersion::TLS1_2), + ..Default::default() + }) + .await +} + +fn attempts(results: &[blasthttp::batch::BatchResult], host: &str) -> usize { + results + .iter() + .filter(|r| r.url.contains(host)) + .filter_map(|r| r.result.as_ref().ok()) + .map(|r| r.attempts.len()) + .sum() +} + +fn reached(results: &[blasthttp::batch::BatchResult]) -> usize { + results + .iter() + .filter(|r| r.result.as_ref().is_ok_and(|resp| resp.status == 200)) + .count() +} + +#[tokio::test] +async fn test_a_burst_discovers_a_host_once_when_there_is_other_work() { + let legacy = legacy_server().await; + let modern = modern_server().await; + + let legacy_host = legacy.addr.port().to_string(); + let mut configs = Vec::new(); + for _ in 0..4 { + configs.push(config(&legacy.url())); + } + for _ in 0..4 { + configs.push(config(&modern.url())); + } + + let client = Arc::new(HyperClient::new()); + let results = send_batch(client, configs, 8, None, None).await; + + assert_eq!( + reached(&results), + 8, + "every request should still get through" + ); + + // Four requests to a host needing a shift. Without the scheduler each + // discovers it alone: 4 x 2 = 8 handshakes. With it, one discovers and + // the other three are told: 2 + 3 = 5. Asserting on "fewer than eight" + // rather than "exactly five" because the modern server's requests can + // finish and free the other host before all four legacy ones have even + // started, and how many get the news is a scheduling detail. + let a = attempts(&results, &legacy_host); + assert!( + a < 8, + "expected the burst to share what it learned, got {a} attempts for 4 requests" + ); + assert!( + a >= 5, + "cannot be fewer than one discovery plus three, got {a}" + ); + + legacy.shutdown().await; + modern.shutdown().await; +} + +#[tokio::test] +async fn test_a_burst_with_nothing_else_to_do_does_not_wait() { + // The case that killed the previous attempt. Every request wants the same + // host, so standing aside would be standing idle, and the right answer is + // to pay for the duplicate discovery instead. Four requests, two attempts + // each, nobody waiting on anybody. + let legacy = legacy_server().await; + let legacy_host = legacy.addr.port().to_string(); + + let configs: Vec<_> = (0..4).map(|_| config(&legacy.url())).collect(); + let client = Arc::new(HyperClient::new()); + + let started = std::time::Instant::now(); + let results = send_batch(client, configs, 4, None, None).await; + let elapsed = started.elapsed(); + + assert_eq!(reached(&results), 4); + assert_eq!( + attempts(&results, &legacy_host), + 8, + "with no other host to hand the slot to, every request discovers" + ); + // Four concurrent local requests. If any of them had stood aside waiting + // for another, this would be serial rather than parallel. + assert!( + elapsed < std::time::Duration::from_secs(2), + "took {elapsed:?}, which means something waited" + ); + + legacy.shutdown().await; +} + +#[tokio::test] +async fn test_a_host_that_cannot_be_reached_does_not_strand_the_rest() { + // The leader guard marks the host answered when it drops, not when it + // succeeds. Without that, a burst against a dead host would have one + // request fail and the others sit on the stand-aside timeout. + let modern = modern_server().await; + let mut configs: Vec<_> = (0..4).map(|_| config("https://127.0.0.1:1/")).collect(); + configs.push(config(&modern.url())); + + let client = Arc::new(HyperClient::new()); + let started = std::time::Instant::now(); + let results = send_batch(client, configs, 5, None, None).await; + let elapsed = started.elapsed(); + + assert_eq!(results.len(), 5); + assert_eq!(reached(&results), 1, "only the live server answers"); + assert!( + elapsed < std::time::Duration::from_secs(3), + "took {elapsed:?}: a failed leader should release the others at once" + ); + + modern.shutdown().await; +} diff --git a/tests/connect_errors.rs b/tests/connect_errors.rs new file mode 100644 index 0000000..0759d68 --- /dev/null +++ b/tests/connect_errors.rs @@ -0,0 +1,163 @@ +//! A connect failure should say why it failed, on both the pooled path and the +//! direct one. hyper-util's own error reads "client error (Connect)" whatever +//! the cause, so these check that the cause underneath makes it out. + +mod support; + +use blasthttp::client::hyper::HyperClient; +use blasthttp::client::{ErrorKind, HttpClient, TlsFailure}; +use blasthttp::config::RequestConfig; +use openssl::ssl::SslVersion; +use support::tls_server::{TlsServerConfig, TlsTestServer}; +use tokio::net::TcpListener; + +/// A local port with nothing listening on it. +async fn closed_port() -> u16 { + let listener = TcpListener::bind("127.0.0.1:0").await.unwrap(); + listener.local_addr().unwrap().port() +} + +fn config_for(url: String) -> RequestConfig { + let mut config = RequestConfig::new(url); + config.timeout_seconds = Some(5); + config.retries = Some(0); + config +} + +#[tokio::test] +async fn test_refused_connection_says_so_on_the_pooled_path() { + let port = closed_port().await; + let config = config_for(format!("http://127.0.0.1:{}/", port)); + + let err = HyperClient::new().send(&config).await.unwrap_err(); + + assert_eq!(err.kind, ErrorKind::Connection); + assert!( + err.message.to_lowercase().contains("refused"), + "message should give the cause: {}", + err.message + ); +} + +#[tokio::test] +async fn test_refused_connection_says_so_on_the_direct_path() { + let port = closed_port().await; + let mut config = config_for(format!("http://localhost:{}/", port)); + // resolve_ip sends the request down the direct path + config.resolve_ip = Some("127.0.0.1".to_string()); + + let err = HyperClient::new().send(&config).await.unwrap_err(); + + assert_eq!(err.kind, ErrorKind::Connection); + assert!( + err.message.to_lowercase().contains("refused"), + "message should give the cause: {}", + err.message + ); +} + +#[tokio::test] +async fn test_an_earlier_tls_failure_does_not_relabel_a_later_error() { + // A TLS failure belongs to the request that hit it. Once the server is + // gone, the next request to the same host:port is refused, and that has + // to read as a connection error (retryable), not as the old TLS failure. + let server = TlsTestServer::start(TlsServerConfig { + max_tls_version: Some(SslVersion::TLS1), + ..Default::default() + }) + .await; + let mut config = config_for(server.url()); + config.verify_certs = Some(false); + // Pinned at 1.2, so the handshake fails and nothing widens past it. + config.min_tls_version = Some("1.2".to_string()); + + let client = HyperClient::new(); + let first = client.send(&config).await.unwrap_err(); + assert_eq!(first.kind, ErrorKind::Tls, "setup: {}", first.message); + assert!(first.tls_failure.is_some()); + + server.shutdown().await; + + let second = client.send(&config).await.unwrap_err(); + assert_eq!( + second.kind, + ErrorKind::Connection, + "a refused connection was reported as: {}", + second.message + ); + assert!(second.tls_failure.is_none()); +} + +/// Accepts connections, reads the ClientHello and hangs up. Counts how many +/// connections it saw. +async fn hangs_up_on_hello() -> (u16, std::sync::Arc) { + use tokio::io::AsyncReadExt; + let listener = TcpListener::bind("127.0.0.1:0").await.unwrap(); + let port = listener.local_addr().unwrap().port(); + let seen = std::sync::Arc::new(std::sync::atomic::AtomicUsize::new(0)); + let counter = seen.clone(); + tokio::spawn(async move { + loop { + let (mut sock, _) = listener.accept().await.unwrap(); + counter.fetch_add(1, std::sync::atomic::Ordering::SeqCst); + let mut buf = [0u8; 4096]; + let _ = sock.read(&mut buf).await; + drop(sock); + } + }); + (port, seen) +} + +#[tokio::test] +async fn test_a_hangup_during_the_handshake_is_a_retryable_connection_error() { + let (port, seen) = hangs_up_on_hello().await; + let mut config = config_for(format!("https://127.0.0.1:{}/", port)); + config.retries = Some(1); + config.retry_wait_min_ms = Some(10); + config.retry_wait_max_ms = Some(10); + + let err = HyperClient::new().send(&config).await.unwrap_err(); + + assert_eq!(err.kind, ErrorKind::Connection, "{}", err.message); + assert_eq!(err.tls_failure, Some(TlsFailure::Reset)); + assert_eq!(seen.load(std::sync::atomic::Ordering::SeqCst), 2); +} + +/// Accepts connections and never answers them. +async fn silent_server() -> u16 { + let listener = TcpListener::bind("127.0.0.1:0").await.unwrap(); + let port = listener.local_addr().unwrap().port(); + tokio::spawn(async move { + let mut held = Vec::new(); + loop { + held.push(listener.accept().await.unwrap().0); + } + }); + port +} + +#[tokio::test] +async fn test_a_handshake_timeout_reads_the_same_on_both_paths() { + // A peer that accepts and never answers the ClientHello is a timeout on + // the pooled path and the direct one alike. + let port = silent_server().await; + let mut seen = Vec::new(); + for direct in [false, true] { + let mut config = config_for(format!("https://127.0.0.1:{}/", port)); + config.timeout_seconds = Some(1); + if direct { + config.resolve_ip = Some("127.0.0.1".to_string()); + } + + let err = HyperClient::new().send(&config).await.unwrap_err(); + + assert_eq!( + err.kind, + ErrorKind::Timeout, + "direct={direct}: {}", + err.message + ); + seen.push(err.tls_failure); + } + assert_eq!(seen[0], seen[1]); +} diff --git a/tests/fingerprint_baseline.rs b/tests/fingerprint_baseline.rs new file mode 100644 index 0000000..de7f3f2 --- /dev/null +++ b/tests/fingerprint_baseline.rs @@ -0,0 +1,326 @@ +// What our ClientHello actually looks like, measured offline. +// +// These tests stand up a local TLS server that records the hello the client +// sends, so the fingerprint can be checked in CI with no network and no +// third-party service. That matters for two reasons: the public fingerprint +// endpoints rate-limit, and several of the ones the ecosystem cites are dead +// or, in one case, now serving a gambling site. +// +// The pinned values below are a regression gate, not an aspiration. Changing +// the TLS configuration is supposed to change them; the point is that it +// cannot happen silently. When one of these fails, read the `ja4_raw` output +// in the failure message, confirm the move was intended, and update the pin. + +mod support; + +use blasthttp::client::HttpClient; +use blasthttp::client::hyper::HyperClient; +use blasthttp::config::RequestConfig; +use support::tls_server::{CapturedHello, TlsServerConfig, TlsTestServer}; + +/// Make one ordinary request at a capturing server and return what it saw. +async fn capture_default_hello() -> CapturedHello { + let server = TlsTestServer::start(TlsServerConfig::default()).await; + + let mut config = RequestConfig::new(server.url()); + config.verify_certs = Some(false); + config.timeout_seconds = Some(5); + + let client = HyperClient::new(); + let _ = client.send(&config).await; + + let hello = server + .hello + .lock() + .expect("hello slot poisoned") + .clone() + .expect("server recorded no ClientHello"); + + server.shutdown().await; + hello +} + +#[tokio::test] +async fn test_capture_sees_a_plausible_hello() { + // Guards the harness itself. If capture breaks, every other test in this + // file would go green against an empty hello and prove nothing. + let hello = capture_default_hello().await; + + assert!( + !hello.ciphers.is_empty(), + "captured no cipher suites, so capture is broken" + ); + assert!( + !hello.extensions.is_empty(), + "captured no extensions, so capture is broken" + ); + assert!( + hello.extensions.contains(&0x000d), + "expected signature_algorithms in the hello" + ); + assert!( + !hello.sig_algs.is_empty(), + "signature_algorithms extension present but parsed empty" + ); +} + +#[tokio::test] +async fn test_no_sni_for_an_ip_literal() { + // The server is addressed by IP, and SNI is not sent for IP literals. This + // pins the `i` in the JA4 header and is the reason these tests read `t13i` + // rather than the `t13d` a hostname would give. + let hello = capture_default_hello().await; + assert!( + !hello.has_sni, + "SNI should be absent when connecting to an IP literal" + ); +} + +#[tokio::test] +async fn test_default_ja4_is_pinned() { + let hello = capture_default_hello().await; + let actual = support::ja4::ja4(&hello); + + // The default is now `modern`: 11 cipher suites, TLS 1.2 and 1.3 only. + // + // 12 extensions rather than 11, because a TLS 1.2 floor trips our OpenSSL + // patch into sending renegotiation_info (0xff01) instead of the SCSV. That + // is the intended behaviour of the patch and not incidental: a client + // whose floor is 1.2 signals the modern way, and one willing to speak + // older protocols keeps the SCSV. + // + // Reads `t13i` rather than `t13d` because this server is addressed by IP, + // so no SNI is sent. Chrome 131 for contrast is t13d1516h2. + const EXPECTED: &str = "t13i1112h2"; + + let header = actual.split('_').next().unwrap_or_default(); + assert_eq!( + header, + EXPECTED, + "default JA4 header moved.\n was: {}\n now: {}\n raw: {}", + EXPECTED, + header, + support::ja4::ja4_raw(&hello), + ); +} + +#[tokio::test] +async fn test_compatibility_ja4_is_pinned() { + // What the default used to be, and still is when asked for by name. This + // pin is `compatibility`'s specification along with legacy_default.rs: if + // it moves, the profile has stopped being a faithful transcription of what + // shipped before profiles existed. + // + // 105 suites renders as 99 because the JA4 count field is two digits and + // the spec caps it. Worth knowing on its own: past 99, JA4 cannot tell 105 + // from 200. + // + // Reads `t13d` and 12 extensions, not `t13i` and 11, because this capture + // addresses the server by hostname so SNI is sent. The IP-addressed + // equivalent is t13i9911h2. + let hello = capture_profile_hello("compatibility").await; + let header = support::ja4::ja4(&hello); + let header = header.split('_').next().unwrap_or_default(); + + assert_eq!( + header, + "t13d9912h2", + "compatibility JA4 moved.\n raw: {}", + support::ja4::ja4_raw(&hello), + ); +} + +#[tokio::test] +async fn test_grease_is_absent() { + // We send no GREASE anywhere, which JA4 ignores but a detector checking + // the raw hello does not: every modern browser emits GREASE in several + // places and almost nothing scripted does, so its absence is a cheap and + // reliable tell. OpenSSL has no API for it at any version. + // + // When stealth mode arrives this expectation inverts. Until then it + // records where we stand. + let hello = capture_default_hello().await; + + let greased: Vec = hello + .ciphers + .iter() + .chain(hello.extensions.iter()) + .chain(hello.groups.iter()) + .copied() + .filter(|v| support::ja4::is_grease(*v)) + .collect(); + + assert!( + greased.is_empty(), + "expected no GREASE from the OpenSSL stack, found {:04x?}", + greased + ); +} + +#[tokio::test] +async fn test_compatibility_offers_legacy_protocol_versions() { + // SSLv3 through TLS 1.3 all offered, which is what makes an ancient server + // reachable. No longer what an unconfigured request sends: the default is + // `modern`, and a legacy server is reached by the ladder falling back to + // this profile instead. + let hello = capture_profile_hello("compatibility").await; + + for (version, label) in [ + (0x0300u16, "SSLv3"), + (0x0301, "TLS 1.0"), + (0x0302, "TLS 1.1"), + (0x0303, "TLS 1.2"), + (0x0304, "TLS 1.3"), + ] { + assert!( + hello.supported_versions.contains(&version), + "{} missing from supported_versions: {:04x?}", + label, + hello.supported_versions + ); + } +} + +#[tokio::test] +async fn test_no_browser_only_extensions_yet() { + // Extensions the browser profile sends and the default does not. Named + // here so the gap stays legible. + // + // renegotiation_info (0xff01) is deliberately absent from this list: the + // default now sends it, because `modern` sets a TLS 1.2 floor and the + // OpenSSL patch keys on exactly that. It is not a browser-only extension, + // it is what any client with a modern floor should send. + let hello = capture_default_hello().await; + + for (ext, name) in [ + (0x0005u16, "status_request"), + (0x0012, "signed_certificate_timestamp"), + (0x001b, "compress_certificate"), + (0x44cd, "application_settings (ALPS)"), + (0xfe0d, "encrypted_client_hello"), + ] { + assert!( + !hello.extensions.contains(&ext), + "{} ({:#06x}) is now sent; stealth work has started, update this test", + name, + ext + ); + } +} + +// ── Chrome profile ──────────────────────────────────────────────── + +/// Capture a hello made with a named profile, addressed by hostname so SNI is +/// sent and the result is comparable with a browser's. +async fn capture_profile_hello(profile: &str) -> CapturedHello { + let server = TlsTestServer::start(TlsServerConfig::default()).await; + let url = format!("https://localhost:{}/", server.addr.port()); + + let mut config = RequestConfig::new(url); + config.verify_certs = Some(false); + config.timeout_seconds = Some(5); + config.profile = Some(profile.to_string()); + + let client = HyperClient::new(); + let _ = client.send(&config).await; + + let hello = server + .hello + .lock() + .expect("hello slot poisoned") + .clone() + .expect("server recorded no ClientHello"); + server.shutdown().await; + hello +} + +#[tokio::test] +async fn test_chrome_profile_ja4_is_pinned() { + let hello = capture_profile_hello("chrome").await; + let got = support::ja4::ja4(&hello); + + // Chrome 131 and 124, confirmed against tls.peet.ws, is + // t13d1516h2_8daaf6152771_02713d6af862. + // + // The cipher segment matches it exactly. The extension count does not, and + // is not meant to: three of Chrome's extensions are deliberately not sent + // because advertising them broke real sites. See add_browser_extensions() + // for which and why. + // + // Worth recording next to the pin, because it is the finding that shaped + // the whole approach: this non-matching JA4 is enough to turn akamai.com + // from a hard 403 into a 200 with a real session. Exact parity was not the + // bar. A client that connects beats one that matches a hash and cannot. + const EXPECTED: &str = "t13d1513h2_8daaf6152771_1eb89897b454"; + + assert_eq!( + got, + EXPECTED, + "Chrome profile JA4 moved.\n raw: {}", + support::ja4::ja4_raw(&hello), + ); +} + +#[tokio::test] +async fn test_chrome_profile_cipher_list_matches_chrome_exactly() { + // The cipher segment is the one part that does match Chrome, and it only + // does because of the OpenSSL patch: stock OpenSSL appends + // TLS_EMPTY_RENEGOTIATION_INFO_SCSV to the list, which no browser sends, + // and that one extra suite changed the hash. Guarded separately so a + // regression there is unmistakable. + let hello = capture_profile_hello("chrome").await; + let got = support::ja4::ja4(&hello); + let cipher_hash = got.split('_').nth(1).unwrap_or_default(); + + assert_eq!( + cipher_hash, "8daaf6152771", + "cipher list no longer matches Chrome 131" + ); + assert!( + !hello.ciphers.contains(&0x00ff), + "the renegotiation SCSV is back in the cipher list: {:04x?}", + hello.ciphers + ); +} + +#[tokio::test] +async fn test_chrome_profile_sends_renegotiation_info() { + // The other half of the same patch. OpenSSL sends the SCSV precisely when + // it is not sending this extension, so these two tests fail together. + let hello = capture_profile_hello("chrome").await; + assert!( + hello.extensions.contains(&0xff01), + "renegotiation_info missing, so the OpenSSL patch is not in this build" + ); +} + +#[tokio::test] +async fn test_chrome_profile_drops_non_browser_extensions() { + // padding and encrypt_then_mac are OpenSSL habits no browser has. + let hello = capture_profile_hello("chrome").await; + for (ext, name) in [(0x0015u16, "padding"), (0x0016, "encrypt_then_mac")] { + assert!( + !hello.extensions.contains(&ext), + "{} ({:#06x}) should not be sent under a browser profile", + name, + ext + ); + } +} + +#[tokio::test] +async fn test_compat_path_keeps_the_scsv() { + // The OpenSSL patch is conditioned on the TLS floor, so `compatibility`, + // which sets no floor, stays byte-identical to stock OpenSSL: the SCSV + // goes out and renegotiation_info does not. `modern` is the other side of + // that same condition, and test_default_ja4_is_pinned covers it. + let hello = capture_profile_hello("compatibility").await; + assert!( + hello.ciphers.contains(&0x00ff), + "compatibility path lost the SCSV, so the patch is too broad" + ); + assert!( + !hello.extensions.contains(&0xff01), + "compatibility path gained renegotiation_info, so the patch is too broad" + ); +} diff --git a/tests/h2_path.rs b/tests/h2_path.rs index 895c762..1f26309 100644 --- a/tests/h2_path.rs +++ b/tests/h2_path.rs @@ -4,8 +4,7 @@ //! a URL with no path (`https://host?q=1`) has to be given one before it gets //! there. Servers reject `:path: ?q=1` with a 400. -#[allow(dead_code)] -mod tls_server; +mod support; use blasthttp::client::HttpClient; use blasthttp::client::hyper::HyperClient; @@ -15,7 +14,7 @@ use hyper::body::Bytes; use hyper_util::rt::{TokioExecutor, TokioIo}; use std::net::SocketAddr; use std::sync::{Arc, Mutex}; -use tls_server::{TlsServerConfig, build_h2_acceptor}; +use support::tls_server::{TlsServerConfig, build_h2_acceptor}; use tokio::net::TcpListener; /// Serve HTTP/2 over TLS until the test ends, recording every `:path` that diff --git a/tests/ja4_conformance.rs b/tests/ja4_conformance.rs new file mode 100644 index 0000000..4959953 --- /dev/null +++ b/tests/ja4_conformance.rs @@ -0,0 +1,167 @@ +// Does our JA4 agree with everyone else's? +// +// The baseline tests pin what we send, but a JA4 implementation that is wrong +// in a self-consistent way would pass all of them and still be useless for +// comparing against a browser. This checks the algorithm itself against a +// captured real-browser ClientHello whose JA4 is published. +// +// Reference data is the Chrome 150 signature from lexiforest/curl-impersonate, +// `tests/signatures/chrome_150.0.7871.127_macOS.yaml` (MIT licensed, copyright +// the curl_cffi developers). Its JA4 is corroborated by four independent +// projects across three different fingerprinting services, which is about as +// much agreement as this area offers. +// +// Worth knowing when adding vectors: fingerprint values are not portable +// between calculators. One handshake observed by seven services produced three +// different JA3 hashes and two different JA4s. Only add a vector whose expected +// value has independent corroboration, or this file starts asserting one +// service's quirks. + +mod support; + +use support::tls_server::CapturedHello; + +/// GREASE placeholder. Which of the sixteen values appears does not matter, +/// since every one of them is stripped before hashing. +const GREASE: u16 = 0x0a0a; + +/// Chrome 150 on macOS, as captured by curl-impersonate. +fn chrome_150() -> CapturedHello { + CapturedHello { + // Modern hellos pin this at TLS 1.2 and put the real list in + // supported_versions. + legacy_version: 0x0303, + ciphers: vec![ + GREASE, 4865, 4866, 4867, 49195, 49199, 49196, 49200, 52393, 52392, 49171, 49172, 156, + 157, 47, 53, + ], + // In the order Chrome sent them. Chrome permutes this per connection, + // which is exactly why JA4 sorts before hashing and why the order here + // does not affect the result. + extensions: vec![ + GREASE, 0x44cd, 0x002d, 0x001b, 0x0033, 0x002b, 0xff01, 0xfe0d, 0x0017, 0x000b, 0x0005, + 0x0023, 0x000a, 0x0012, 0x0010, 0x000d, 0x0000, 0x1a1a, + ], + groups: vec![GREASE, 4588, 29, 23, 24], + // The first three are ML-DSA, new in recent Chrome. Order is + // significant: JA4 does not sort this list. + sig_algs: vec![ + 2308, 2309, 2310, 1027, 2052, 1025, 1283, 2053, 1281, 2054, 1537, + ], + alpn: vec!["h2".to_string(), "http/1.1".to_string()], + supported_versions: vec![GREASE, 0x0304, 0x0303], + has_sni: true, + } +} + +#[test] +fn test_ja4_matches_published_chrome_150() { + const PUBLISHED: &str = "t13d1516h2_8daaf6152771_806a8c22fdea"; + + let computed = support::ja4::ja4(&chrome_150()); + + assert_eq!( + computed, + PUBLISHED, + "JA4 disagrees with the published value for Chrome 150.\n raw: {}", + support::ja4::ja4_raw(&chrome_150()), + ); +} + +#[test] +fn test_ja4_header_decodes_as_expected() { + // Spelled out so a failure says which field moved rather than just that a + // string differs. 16 ciphers and 18 extensions were sent; one and two of + // those are GREASE, giving the 15 and 16 in the header. + let hello = chrome_150(); + let computed = support::ja4::ja4(&hello); + let header = computed.split('_').next().unwrap(); + + assert_eq!(&header[0..1], "t", "protocol should be TCP"); + assert_eq!( + &header[1..3], + "13", + "supported_versions tops out at TLS 1.3" + ); + assert_eq!(&header[3..4], "d", "SNI was present"); + assert_eq!(&header[4..6], "15", "16 ciphers less 1 GREASE"); + assert_eq!(&header[6..8], "16", "18 extensions less 2 GREASE"); + assert_eq!(&header[8..10], "h2", "first and last char of first ALPN"); +} + +#[test] +fn test_extension_order_does_not_change_ja4() { + // The property the whole stealth plan rests on. Chrome shuffles its + // extension order every connection, so if ordering moved the hash, no + // stable Chrome JA4 could exist. It also means our inability to control + // extension order under OpenSSL costs us nothing on this metric. + let mut shuffled = chrome_150(); + shuffled.extensions.reverse(); + + assert_eq!( + support::ja4::ja4(&chrome_150()), + support::ja4::ja4(&shuffled), + "reversing the extension order changed the JA4" + ); +} + +#[test] +fn test_grease_does_not_change_ja4() { + // The other half of the same argument: GREASE is stripped, so not emitting + // any costs nothing here either. It is still visible to anything reading + // the raw hello, which is tracked separately in fingerprint_baseline.rs. + let mut without = chrome_150(); + without.ciphers.retain(|c| !support::ja4::is_grease(*c)); + without.extensions.retain(|e| !support::ja4::is_grease(*e)); + without.groups.retain(|g| !support::ja4::is_grease(*g)); + without + .supported_versions + .retain(|v| !support::ja4::is_grease(*v)); + + assert_eq!( + support::ja4::ja4(&chrome_150()), + support::ja4::ja4(&without), + "removing GREASE changed the JA4" + ); +} + +#[test] +fn test_counts_cap_at_99() { + // The JA4 header is fixed width: one character of protocol, two of + // version, one of SNI, two of cipher count, two of extension count, two of + // ALPN. A three-digit count would make it unparseable, so the spec caps + // both counts at 99. + // + // This is not academic for us. Our legacy cipher list is 105 suites, so we + // sit past the cap, and `tls.peet.ws` renders that as `105` where we + // render `99`. Their hashes and ours agree exactly, so the underlying + // reading is identical and only the count field diverges. We follow the + // spec, because the whole purpose here is comparing against browser JA4s + // and every browser is far below the cap. Once a stealth profile ships at + // 15 suites the divergence disappears on its own. + let mut absurd = chrome_150(); + absurd.ciphers = (1u16..=150).collect(); + absurd.extensions = (1u16..=150).collect(); + + let header = support::ja4::ja4(&absurd); + let header = header.split('_').next().unwrap(); + + assert_eq!(&header[4..6], "99", "cipher count should cap at 99"); + assert_eq!(&header[6..8], "99", "extension count should cap at 99"); + assert_eq!(header.len(), 10, "header must stay 10 characters wide"); +} + +#[test] +fn test_sig_alg_order_does_change_ja4() { + // Signature algorithms are the one list JA4 leaves in the order sent, so + // a profile that gets the values right and the order wrong still fails to + // match. Guards against anyone "tidying up" by sorting them. + let mut reordered = chrome_150(); + reordered.sig_algs.reverse(); + + assert_ne!( + support::ja4::ja4(&chrome_150()), + support::ja4::ja4(&reordered), + "signature algorithm order should be significant" + ); +} diff --git a/tests/ladder.rs b/tests/ladder.rs new file mode 100644 index 0000000..261c06c --- /dev/null +++ b/tests/ladder.rs @@ -0,0 +1,254 @@ +//! The profile ladder, seen from the response a caller gets back. + +use blasthttp::client::hyper::HyperClient; +use blasthttp::client::{ErrorKind, HttpClient}; +use blasthttp::config::RequestConfig; +use blasthttp::report::{AttemptOutcome, Conclusion}; +use tokio::io::{AsyncReadExt, AsyncWriteExt}; +use tokio::net::TcpListener; + +/// Answers one connection with `status`, `headers` and `body`, then stops +/// listening, so every later connection to the port is refused. +async fn serve_once(status: &str, headers: &str, body: &'static str) -> u16 { + let listener = TcpListener::bind("127.0.0.1:0").await.unwrap(); + let port = listener.local_addr().unwrap().port(); + let reply = format!( + "HTTP/1.1 {status}\r\n{headers}content-length: {}\r\nconnection: close\r\n\r\n{body}", + body.len() + ); + tokio::spawn(async move { + let (mut sock, _) = listener.accept().await.unwrap(); + drop(listener); + let mut buf = [0u8; 4096]; + let _ = sock.read(&mut buf).await; + let _ = sock.write_all(reply.as_bytes()).await; + let _ = sock.shutdown().await; + }); + port +} + +fn config(port: u16) -> RequestConfig { + let mut config = RequestConfig::new(format!("http://127.0.0.1:{port}/")); + config.timeout_seconds = Some(5); + config.retries = Some(0); + config +} + +#[tokio::test] +async fn test_a_failed_escalation_keeps_the_refusal_it_had() { + // A refusal moves the ladder on; if the next rung never gets a response, + // the caller still gets the refusal, not the later error. + let port = serve_once( + "403 Forbidden", + "cf-mitigated: challenge\r\nserver: cloudflare\r\n", + "Just a moment...", + ) + .await; + + let resp = HyperClient::new() + .send(&config(port)) + .await + .unwrap_or_else(|e| panic!("the 403 was lost to: {}", e.message)); + + assert_eq!(resp.status, 403); + assert!(matches!(resp.conclusion(), Conclusion::NeedsBrowser { .. })); + assert!( + resp.attempts.len() >= 2, + "the escalation should be recorded: {:?}", + resp.attempts + ); + assert!(matches!( + resp.attempts[0].outcome, + AttemptOutcome::Challenged { status: 403, .. } + )); + assert!(matches!( + resp.attempts.last().unwrap().outcome, + AttemptOutcome::HandshakeFailed { .. } + )); +} + +#[tokio::test] +async fn test_a_not_found_page_is_reached() { + // A 404 is the server answering. It must not read as an unreachable host. + let port = serve_once("404 Not Found", "server: nginx\r\n", "not found").await; + + let resp = HyperClient::new().send(&config(port)).await.unwrap(); + + assert_eq!(resp.status, 404); + assert_eq!(resp.conclusion(), Conclusion::Reached); + assert_eq!( + resp.attempts[0].outcome, + AttemptOutcome::Reached { status: 404 } + ); +} + +/// Answers the first connection with a Cloudflare challenge after `delay`, +/// then accepts every later connection and never answers it. +async fn slow_challenge_then_silence(delay: std::time::Duration) -> u16 { + let listener = TcpListener::bind("127.0.0.1:0").await.unwrap(); + let port = listener.local_addr().unwrap().port(); + tokio::spawn(async move { + let mut first = true; + let mut held = Vec::new(); + loop { + let (mut sock, _) = listener.accept().await.unwrap(); + if !first { + held.push(sock); + continue; + } + first = false; + tokio::spawn(async move { + let mut buf = [0u8; 4096]; + let _ = sock.read(&mut buf).await; + tokio::time::sleep(delay).await; + let body = "Just a moment..."; + let reply = format!( + "HTTP/1.1 403 Forbidden\r\ncf-mitigated: challenge\r\n\ + content-length: {}\r\nconnection: close\r\n\r\n{body}", + body.len() + ); + let _ = sock.write_all(reply.as_bytes()).await; + let _ = sock.shutdown().await; + }); + } + }); + port +} + +#[tokio::test] +async fn test_an_escalation_that_runs_out_of_time_keeps_the_refusal() { + // The refusal arrives most of the way through the timeout and the next + // rung never answers. The caller gets the refusal, not a timeout. + let port = slow_challenge_then_silence(std::time::Duration::from_millis(1500)).await; + let mut config = config(port); + config.timeout_seconds = Some(2); + + let resp = HyperClient::new() + .send(&config) + .await + .unwrap_or_else(|e| panic!("the 403 was lost to: {}", e.message)); + + assert_eq!(resp.status, 403); + match &resp.attempts.last().unwrap().outcome { + AttemptOutcome::HandshakeFailed { reason } => { + assert!(reason.contains("timed out"), "{reason}") + } + other => panic!("the last rung should have timed out: {other:?}"), + } +} + +#[tokio::test] +async fn test_a_server_that_never_answers_still_times_out() { + let listener = TcpListener::bind("127.0.0.1:0").await.unwrap(); + let port = listener.local_addr().unwrap().port(); + tokio::spawn(async move { + let mut held = Vec::new(); + loop { + held.push(listener.accept().await.unwrap().0); + } + }); + let mut config = config(port); + config.timeout_seconds = Some(1); + + let err = HyperClient::new().send(&config).await.unwrap_err(); + + assert_eq!(err.kind, ErrorKind::Timeout); + assert_eq!(err.message, "request timed out after 1s"); +} + +type Seen = std::sync::Arc>>>; + +/// Refuses the first request with a Cloudflare challenge and answers every +/// later one with a 200, recording each request's headers. +async fn challenge_first_then_ok() -> (u16, Seen) { + let listener = TcpListener::bind("127.0.0.1:0").await.unwrap(); + let port = listener.local_addr().unwrap().port(); + let seen: Seen = Default::default(); + let record = seen.clone(); + tokio::spawn(async move { + loop { + let (mut sock, _) = listener.accept().await.unwrap(); + let mut buf = vec![0u8; 16384]; + let n = sock.read(&mut buf).await.unwrap_or(0); + let head = String::from_utf8_lossy(&buf[..n]).to_string(); + let headers: Vec<(String, String)> = head + .lines() + .skip(1) + .filter_map(|l| l.split_once(':')) + .map(|(k, v)| (k.trim().to_ascii_lowercase(), v.trim().to_string())) + .collect(); + let first = { + let mut s = record.lock().unwrap(); + s.push(headers); + s.len() == 1 + }; + let reply = if first { + "HTTP/1.1 403 Forbidden\r\ncf-mitigated: challenge\r\ncontent-length: 0\r\nconnection: close\r\n\r\n" + } else { + "HTTP/1.1 200 OK\r\ncontent-length: 2\r\nconnection: close\r\n\r\nok" + }; + let _ = sock.write_all(reply.as_bytes()).await; + let _ = sock.shutdown().await; + } + }); + (port, seen) +} + +fn values<'a>(headers: &'a [(String, String)], name: &str) -> Vec<&'a str> { + headers + .iter() + .filter(|(k, _)| k == name) + .map(|(_, v)| v.as_str()) + .collect() +} + +fn caller_headers() -> Vec<(String, String)> { + [ + ("User-Agent", "custom/1.0"), + ("Accept", "application/json"), + ("X-Added", "1"), + ("Cookie", "session=abc"), + ] + .into_iter() + .map(|(k, v)| (k.to_string(), v.to_string())) + .collect() +} + +#[tokio::test] +async fn test_a_switch_after_a_denial_claims_one_browser() { + // First contact keeps the caller's User-Agent. Once a denial moves the + // ladder to chrome131, its browser identity replaces the caller's, and + // everything else the caller added stays. + let (port, seen) = challenge_first_then_ok().await; + let mut config = config(port); + config.headers = Some(caller_headers()); + + let resp = HyperClient::new().send(&config).await.unwrap(); + assert_eq!(resp.status, 200); + + let seen = seen.lock().unwrap(); + assert_eq!(seen.len(), 2); + assert_eq!(values(&seen[0], "user-agent"), ["custom/1.0"]); + + let switched = &seen[1]; + let ua = values(switched, "user-agent"); + assert_eq!(ua.len(), 1, "one User-Agent, got {ua:?}"); + assert!(ua[0].contains("Chrome/131"), "{}", ua[0]); + assert!(values(switched, "sec-ch-ua")[0].contains("131")); + assert_eq!(values(switched, "accept"), ["application/json"]); + assert_eq!(values(switched, "x-added"), ["1"]); + assert_eq!(values(switched, "cookie"), ["session=abc"]); +} + +#[tokio::test] +async fn test_a_profile_the_caller_named_keeps_their_user_agent() { + let (port, seen) = challenge_first_then_ok().await; + let mut config = config(port); + config.headers = Some(caller_headers()); + config.profile = Some("chrome131".to_string()); + + HyperClient::new().send(&config).await.unwrap(); + + let seen = seen.lock().unwrap(); + assert_eq!(values(&seen[0], "user-agent"), ["custom/1.0"]); +} diff --git a/tests/legacy_default.rs b/tests/legacy_default.rs new file mode 100644 index 0000000..e5d59f9 --- /dev/null +++ b/tests/legacy_default.rs @@ -0,0 +1,734 @@ +// Full-stack tests: legacy ciphers and protocols must work WITHOUT the caller +// naming them. +// +// The custom OpenSSL build (scripts/build-openssl.sh, enable-weak-ssl-ciphers) +// exists so blasthttp can reach ancient servers. README.md advertises "All TLS +// ciphers available by default". These tests hold that promise to its word: +// every client config below sets `verify_certs` (the cert is self-signed) and +// a timeout, and nothing else. No `cipher_string`, no `min_tls_version`, no +// `max_tls_version`. +// +// Each test stands up a real TLS server pinned to one legacy cipher or +// protocol version, then makes a real request through the ordinary client +// path. + +mod support; + +use blasthttp::client::HttpClient; +use blasthttp::client::hyper::HyperClient; +use blasthttp::config::RequestConfig; +use blasthttp::report::Conclusion; +use openssl::ssl::SslVersion; +use support::tls_server::{TlsServerConfig, TlsTestServer}; + +/// A client config with NO TLS tuning of any kind. +/// +/// `verify_certs` is off because the test server's cert is self-signed, and +/// that is orthogonal to which ciphers we offer. Everything else is left at +/// the default, which is the entire point of these tests. +fn default_config(url: &str) -> RequestConfig { + let mut config = RequestConfig::new(url.to_string()); + config.verify_certs = Some(false); + config.timeout_seconds = Some(5); + config +} + +/// Stand up a server that speaks exactly one legacy cipher, then connect with +/// a stock client and assert we got the page. +async fn assert_reachable_with_cipher(cipher: &str) { + let server = TlsTestServer::start(TlsServerConfig { + cipher_list: Some(cipher.to_string()), + // None of these ciphers exist in TLS 1.3, so the server has to sit at + // 1.2 or below for the pin to mean anything. + max_tls_version: Some(SslVersion::TLS1_2), + ..Default::default() + }) + .await; + + let config = default_config(&server.url()); + let client = HyperClient::new(); + let result = client.send(&config).await; + + assert!( + result.is_ok(), + "a stock client should reach a {cipher}-only server, but got: {:?}", + result.as_ref().err() + ); + assert_eq!(result.unwrap().status, 200); + + server.shutdown().await; +} + +/// Same, for a server pinned to one legacy protocol version. +async fn assert_reachable_with_version(label: &str, version: SslVersion) { + let server = TlsTestServer::start(TlsServerConfig { + min_tls_version: Some(version), + max_tls_version: Some(version), + ..Default::default() + }) + .await; + + let config = default_config(&server.url()); + let client = HyperClient::new(); + let result = client.send(&config).await; + + assert!( + result.is_ok(), + "a stock client should reach a {label}-only server, but got: {:?}", + result.as_ref().err() + ); + assert_eq!(result.unwrap().status, 200); + + server.shutdown().await; +} + +// ── Legacy ciphers, no client configuration ─────────────────────── + +#[tokio::test] +async fn test_rc4_reachable_by_default() { + assert_reachable_with_cipher("RC4-SHA").await; +} + +#[tokio::test] +async fn test_rc4_md5_reachable_by_default() { + assert_reachable_with_cipher("RC4-MD5").await; +} + +#[tokio::test] +async fn test_3des_reachable_by_default() { + assert_reachable_with_cipher("DES-CBC3-SHA").await; +} + +#[tokio::test] +async fn test_seed_reachable_by_default() { + assert_reachable_with_cipher("SEED-SHA").await; +} + +#[tokio::test] +async fn test_camellia_reachable_by_default() { + assert_reachable_with_cipher("CAMELLIA128-SHA").await; +} + +#[tokio::test] +async fn test_anon_dh_reachable_by_default() { + // Anonymous DH: no certificate at all. Worth reaching, since a server + // that allows it is itself the finding. + assert_reachable_with_cipher("ADH-AES128-SHA").await; +} + +// ── Null encryption is deliberately NOT in the default ──────────── + +#[tokio::test] +async fn test_null_cipher_not_offered_by_default() { + // A server offering only eNULL should be out of reach for a stock client. + // Connecting would look like TLS while encrypting nothing, so it has to be + // an explicit decision rather than something we fall into. + let server = TlsTestServer::start(TlsServerConfig { + cipher_list: Some("NULL-SHA".to_string()), + max_tls_version: Some(SslVersion::TLS1_2), + ..Default::default() + }) + .await; + + let config = default_config(&server.url()); + let client = HyperClient::new(); + let result = client.send(&config).await; + + assert!( + result.is_err(), + "null encryption must not be reachable without asking for it" + ); + + server.shutdown().await; +} + +#[tokio::test] +async fn test_null_cipher_reachable_when_asked_for() { + // ...but naming it explicitly still works, so the capability is available + // to anyone who actually wants it. + let server = TlsTestServer::start(TlsServerConfig { + cipher_list: Some("NULL-SHA".to_string()), + max_tls_version: Some(SslVersion::TLS1_2), + ..Default::default() + }) + .await; + + let mut config = default_config(&server.url()); + config.cipher_string = Some("NULL-SHA".to_string()); + + let client = HyperClient::new(); + let result = client.send(&config).await; + + assert!( + result.is_ok(), + "explicit NULL-SHA should still connect: {:?}", + result.as_ref().err() + ); + + server.shutdown().await; +} + +// ── Legacy protocol versions, no client configuration ───────────── + +#[tokio::test] +async fn test_sslv3_reachable_by_default() { + assert_reachable_with_version("SSLv3", SslVersion::SSL3).await; +} + +#[tokio::test] +async fn test_sslv3_reachable_when_pinned_explicitly() { + // The version can also be named, which had no spelling at all before: + // `parse_tls_version` only knew 1.0 through 1.3, so SSLv3 was + // unrequestable regardless of what the build supported. + let server = TlsTestServer::start(TlsServerConfig { + min_tls_version: Some(SslVersion::SSL3), + max_tls_version: Some(SslVersion::SSL3), + ..Default::default() + }) + .await; + + let mut config = default_config(&server.url()); + config.min_tls_version = Some("3.0".to_string()); + config.max_tls_version = Some("3.0".to_string()); + + let client = HyperClient::new(); + let result = client.send(&config).await; + + assert!( + result.is_ok(), + "explicitly pinned SSLv3 should connect: {:?}", + result.as_ref().err() + ); + assert_eq!(result.unwrap().status, 200); + + server.shutdown().await; +} + +#[tokio::test] +async fn test_tls10_reachable_by_default() { + assert_reachable_with_version("TLS 1.0", SslVersion::TLS1).await; +} + +#[tokio::test] +async fn test_tls11_reachable_by_default() { + assert_reachable_with_version("TLS 1.1", SslVersion::TLS1_1).await; +} + +// ── Control: this one already works, so a failure here means the +// harness is broken rather than the client ────────────────────── + +#[tokio::test] +async fn test_modern_still_reachable_by_default() { + let server = TlsTestServer::start(TlsServerConfig { + min_tls_version: Some(SslVersion::TLS1_2), + ..Default::default() + }) + .await; + + let config = default_config(&server.url()); + let client = HyperClient::new(); + let result = client.send(&config).await; + + assert!( + result.is_ok(), + "control case failed, so the harness is suspect: {:?}", + result.as_ref().err() + ); + + server.shutdown().await; +} + +// ── What a cold burst costs when there is nothing else to do ────── + +#[tokio::test] +async fn test_a_cold_burst_rediscovers_when_it_has_nothing_else_to_do() { + // Concurrent requests straight at `client.send`, all to one host, with no + // batch around them. Each walks the ladder alone, because the per-host + // memory is only written once something has succeeded and nothing has + // yet. Two handshakes apiece where one would have done. + // + // That is the right answer here and the test says so rather than treating + // it as a defect. The only way to share the discovery is for these + // requests to wait on each other, and a waiter cannot be released until + // the leader has classified its response, which needs the body, which + // means the wait is a whole request long. Paying for fewer handshakes + // with dead time is a bad trade, and it breaks the rule that a slow + // request never holds up faster ones. + // + // A batch does better, because a batch has somewhere else to send the + // slot: see `tests/burst_scheduler.rs`. The saving is only available when + // there is other work, and this test is the case where there is not. + const N: usize = 6; + + let server = TlsTestServer::start(TlsServerConfig { + cipher_list: Some("RC4-SHA".to_string()), + max_tls_version: Some(SslVersion::TLS1_2), + ..Default::default() + }) + .await; + + let client = std::sync::Arc::new(HyperClient::new()); + let url = server.url(); + + let mut tasks = Vec::new(); + for _ in 0..N { + let client = client.clone(); + let config = default_config(&url); + tasks.push(tokio::spawn(async move { client.send(&config).await })); + } + + let (mut attempts, mut reached) = (0usize, 0usize); + for t in tasks { + if let Ok(Ok(resp)) = t.await { + attempts += resp.attempts.len(); + if resp.status == 200 { + reached += 1; + } + } + } + + // Correctness is what matters and it holds: every request gets through, + // the ladder just runs more often than it strictly needs to. + assert_eq!(reached, N, "every request should get through"); + assert!( + attempts >= N, + "expected at least one attempt each, got {attempts}" + ); + + server.shutdown().await; +} + +// ── The same promise on the direct and raw paths ────────────────── +// +// `resolve_ip`, `request_target` and `raw_connect` do not go through the +// pooled client, so they did not go through the ladder either. That was fine +// while the default offered every cipher OpenSSL has; once it narrowed to +// eleven, these callers quietly lost the legacy reach the custom build exists +// to give them. A virtualhost sweep is exactly where an old appliance turns +// up, and exactly where it would have gone unseen. + +#[tokio::test] +async fn test_resolve_ip_reaches_a_legacy_server() { + let server = TlsTestServer::start(TlsServerConfig { + cipher_list: Some("RC4-SHA".to_string()), + max_tls_version: Some(SslVersion::TLS1_2), + ..Default::default() + }) + .await; + + let mut config = default_config(&server.url()); + config.resolve_ip = Some("127.0.0.1".to_string()); + + let client = HyperClient::new(); + let result = client.send(&config).await; + + assert!( + result.is_ok(), + "resolve_ip should reach an RC4-only server the same way the pooled \ + path does: {:?}", + result.as_ref().err() + ); + let resp = result.unwrap(); + assert_eq!(resp.status, 200); + + // And it should say how it got there. Two rungs: the default could not + // negotiate, the wider one could. + let profiles: Vec<&str> = resp.attempts.iter().map(|a| a.profile.as_str()).collect(); + assert_eq!( + profiles, + vec!["modern", "compatibility"], + "expected the report to show the widening, got {profiles:?}" + ); + + server.shutdown().await; +} + +#[tokio::test] +async fn test_request_target_reaches_a_legacy_server() { + let server = TlsTestServer::start(TlsServerConfig { + cipher_list: Some("RC4-SHA".to_string()), + max_tls_version: Some(SslVersion::TLS1_2), + ..Default::default() + }) + .await; + + let mut config = default_config(&server.url()); + config.request_target = Some("/".to_string()); + + let client = HyperClient::new(); + let result = client.send(&config).await; + + assert!( + result.is_ok(), + "request_target takes the same un-pooled path and needs the same \ + reach: {:?}", + result.as_ref().err() + ); + assert_eq!(result.unwrap().status, 200); + + server.shutdown().await; +} + +#[tokio::test] +async fn test_raw_connect_reaches_a_legacy_server() { + // A raw connection has no response to classify, so only the cipher half + // of the ladder can apply to it. That half is the half it needs: byte- + // level work against an old appliance starts with reaching the appliance. + let server = TlsTestServer::start(TlsServerConfig { + cipher_list: Some("RC4-SHA".to_string()), + max_tls_version: Some(SslVersion::TLS1_2), + ..Default::default() + }) + .await; + + let url = server.url(); + let config = default_config(&url); + let result = blasthttp::client::raw::RawConnection::connect(&url, &config).await; + + assert!( + result.is_ok(), + "raw_connect should reach an RC4-only server with no cipher_string: \ + {:?}", + result.as_ref().err() + ); + + // Write a request and read the answer, rather than stopping at "the + // handshake returned Ok". A connection that cannot carry bytes is not a + // connection, and the harness serves each connection on its own accept + // loop iteration, so leaving one open with nothing written parks the + // server in a read that never returns and the shutdown below deadlocks. + let conn = result.unwrap(); + conn.send_bytes(b"GET / HTTP/1.1\r\nHost: 127.0.0.1\r\nConnection: close\r\n\r\n") + .await + .expect("write over the RC4 connection"); + let data = conn + .read_raw(4096, Some(5000)) + .await + .expect("read back over the RC4 connection"); + assert!( + data.starts_with(b"HTTP/1.1 200"), + "expected a response over the raw connection, got {:?}", + String::from_utf8_lossy(&data[..data.len().min(80)]) + ); + let _ = conn.close().await; + + server.shutdown().await; +} + +#[tokio::test] +async fn test_the_direct_path_does_not_widen_past_a_pin() { + // The counterpart, and the more important one on this path. These are the + // callers who asked for exact control, so a named cipher string has to be + // the last word: widening past it would put suites on the wire the caller + // deliberately left off, and a TLS enumeration sweep would report every + // server as speaking everything. + let server = TlsTestServer::start(TlsServerConfig { + cipher_list: Some("RC4-SHA".to_string()), + max_tls_version: Some(SslVersion::TLS1_2), + ..Default::default() + }) + .await; + + let mut config = default_config(&server.url()); + config.resolve_ip = Some("127.0.0.1".to_string()); + config.cipher_string = Some("AES128-SHA".to_string()); + + let client = HyperClient::new(); + let result = client.send(&config).await; + + assert!( + result.is_err(), + "a pinned cipher string that the server does not speak must fail \ + rather than be widened behind the caller's back" + ); + + server.shutdown().await; +} + +#[tokio::test] +async fn test_a_pinned_request_does_not_teach_the_host_memory() { + // The memory is a cache of what the ladder worked out, so a request that + // never laddered has nothing to contribute to it. + // + // Found by a mock test rather than reasoning: a passthrough request with + // profile="chrome" was followed by one with no profile, and the second + // sent Chrome's headers. One deliberate browser request had silently + // converted the host for the rest of the client's life. That is the worst + // direction for it to drift in, because claiming to be a browser is what + // invites a detector to check the claim, and it made the outcome depend + // on the order two unrelated requests ran in. + let server = TlsTestServer::start(TlsServerConfig { + min_tls_version: Some(SslVersion::TLS1_2), + ..Default::default() + }) + .await; + + let client = HyperClient::new(); + + let mut pinned = default_config(&server.url()); + pinned.profile = Some("chrome".to_string()); + let first = client.send(&pinned).await.expect("pinned request"); + assert_eq!( + first.attempts.first().map(|a| a.profile.as_str()), + Some("chrome131"), + "the pinned request should have used what it named" + ); + + let second = client + .send(&default_config(&server.url())) + .await + .expect("unpinned request"); + assert_eq!( + second.attempts.first().map(|a| a.profile.as_str()), + Some("modern"), + "an unpinned request should start at the default, not inherit the \ + profile a previous pinned request named" + ); + + server.shutdown().await; +} + +// ── Anonymous suites must not survive turning verification on ───── +// +// The other half of `test_anon_dh_reachable_by_default`: with verification off +// anonymous suites stay reachable, and with it on they are never offered, since +// a suite with no certificate would make the check pass while checking nothing. + +/// An anonymous-DH-only server. These suites only exist at TLS 1.2 and below. +async fn anon_dh_server() -> TlsTestServer { + TlsTestServer::start(TlsServerConfig { + cipher_list: Some("ADH-AES128-SHA".to_string()), + max_tls_version: Some(SslVersion::TLS1_2), + ..Default::default() + }) + .await +} + +/// Ask for verification against an anonymous-DH server and assert we do not +/// reach it. `cipher_string` is whatever the caller would have set. +async fn assert_unreachable_with_verification(cipher_string: Option<&str>) { + let server = anon_dh_server().await; + + let mut config = default_config(&server.url()); + config.verify_certs = Some(true); + config.cipher_string = cipher_string.map(str::to_string); + + let result = HyperClient::new().send(&config).await; + assert!( + result.is_err(), + "verify_certs was on, so a server with no certificate must be out of \ + reach, but the request succeeded with status {:?}", + result.as_ref().ok().map(|r| r.status) + ); + + server.shutdown().await; +} + +#[tokio::test] +async fn test_anon_dh_unreachable_when_verifying() { + // No pin, so this walks the ladder: `compatibility` must not bring the + // anonymous suites back either. + assert_unreachable_with_verification(None).await; +} + +#[tokio::test] +async fn test_anon_dh_unreachable_when_verifying_with_explicit_ciphers() { + // `ALL` admits the anonymous suites. The caller also asked for + // verification, and that is the half we honor. + assert_unreachable_with_verification(Some("ALL")).await; +} + +#[tokio::test] +async fn test_anon_dh_unreachable_when_verifying_on_direct_path() { + // `resolve_ip` builds its TLS context through `connect_stream`, not the + // pooled connector, so cover that path too. + let server = anon_dh_server().await; + + let mut config = default_config(&server.url()); + config.verify_certs = Some(true); + config.resolve_ip = Some("127.0.0.1".to_string()); + + let result = HyperClient::new().send(&config).await; + assert!( + result.is_err(), + "the direct path must exclude anonymous suites too, but got status {:?}", + result.as_ref().ok().map(|r| r.status) + ); + + server.shutdown().await; +} + +#[tokio::test] +async fn test_anon_dh_reachable_on_direct_path_without_verification() { + // With verification off the same path still reaches it. + let server = anon_dh_server().await; + + let mut config = default_config(&server.url()); + config.resolve_ip = Some("127.0.0.1".to_string()); + + let result = HyperClient::new().send(&config).await; + assert!( + result.is_ok(), + "verification is off, so the direct path should still reach an \ + anonymous-DH server: {:?}", + result.as_ref().err() + ); + assert_eq!(result.unwrap().status, 200); + + server.shutdown().await; +} + +/// The cipher suites `compatibility` offers, with verification on or off. +async fn compatibility_hello_ciphers(verify: bool) -> Vec { + let server = TlsTestServer::start(TlsServerConfig::default()).await; + + let mut config = default_config(&server.url()); + config.profile = Some("compatibility".to_string()); + config.verify_certs = Some(verify); + // The self-signed cert fails verification; only the hello matters here. + let _ = HyperClient::new().send(&config).await; + + let hello = server + .hello + .lock() + .expect("hello slot poisoned") + .clone() + .expect("server recorded no ClientHello"); + server.shutdown().await; + hello.ciphers +} + +#[tokio::test] +async fn test_verification_removes_only_anonymous_suites() { + // TLS_DH_anon_WITH_AES_128_CBC_SHA (ADH-AES128-SHA). + const ADH_AES128_SHA: u16 = 0x0034; + + let unverified = compatibility_hello_ciphers(false).await; + let verified = compatibility_hello_ciphers(true).await; + + assert!( + unverified.contains(&ADH_AES128_SHA), + "with verification off, compatibility must still offer anonymous suites" + ); + assert!( + !verified.contains(&ADH_AES128_SHA), + "with verification on, anonymous suites must not be offered" + ); + // Removing them must cost nothing else: what remains is the unverified + // offer, in the same order, minus the anonymous suites. + let mut remaining = unverified.iter().filter(|c| verified.contains(c)); + assert!( + verified.iter().all(|c| remaining.next() == Some(c)), + "verification should only remove suites, not add or reorder them" + ); + assert!( + verified.len() < unverified.len(), + "expected fewer suites with verification on" + ); +} + +// ── A cipher string alone carries no version floor ──────────────── +// +// Naming ciphers is asking for exactly those, and it pins the configuration, +// so the ladder will not widen it. If the default's TLS 1.2 floor came along, +// RC4 against a TLS 1.0 server would be impossible to ask for. + +/// A server that speaks only TLS 1.0 with RC4-MD5. +async fn rc4_md5_tls10_server() -> TlsTestServer { + TlsTestServer::start(TlsServerConfig { + cipher_list: Some("RC4-MD5".to_string()), + min_tls_version: Some(SslVersion::TLS1), + max_tls_version: Some(SslVersion::TLS1), + ..Default::default() + }) + .await +} + +#[tokio::test] +async fn test_cipher_string_alone_reaches_a_tls10_server() { + let server = rc4_md5_tls10_server().await; + + let mut config = default_config(&server.url()); + config.cipher_string = Some("RC4-MD5".to_string()); + + let result = HyperClient::new().send(&config).await; + assert!( + result.is_ok(), + "cipher_string with no version set should reach a TLS 1.0 RC4-MD5 \ + server: {:?}", + result.as_ref().err() + ); + assert_eq!(result.unwrap().status, 200); + + server.shutdown().await; +} + +#[tokio::test] +async fn test_cipher_string_alone_reaches_a_tls10_server_on_direct_path() { + let server = rc4_md5_tls10_server().await; + + let mut config = default_config(&server.url()); + config.cipher_string = Some("RC4-MD5".to_string()); + config.resolve_ip = Some("127.0.0.1".to_string()); + + let result = HyperClient::new().send(&config).await; + assert!( + result.is_ok(), + "the direct path should honour a bare cipher_string the same way: {:?}", + result.as_ref().err() + ); + assert_eq!(result.unwrap().status, 200); + + server.shutdown().await; +} + +// ── What a failure concludes ────────────────────────────────────── + +#[tokio::test] +async fn test_a_failure_after_the_widest_offer_does_not_ask_for_legacy_tls() { + // No profile offers a NULL cipher, so modern and compatibility both fail. + // Legacy TLS was offered and refused; asking for it again helps nothing. + let server = TlsTestServer::start(TlsServerConfig { + cipher_list: Some("NULL-SHA".to_string()), + max_tls_version: Some(SslVersion::TLS1_2), + ..Default::default() + }) + .await; + + let err = HyperClient::new() + .send(&default_config(&server.url())) + .await + .unwrap_err(); + + let profiles: Vec<&str> = err.attempts.iter().map(|a| a.profile.as_str()).collect(); + assert_eq!(profiles, ["modern", "compatibility"]); + assert!( + matches!(err.conclusion(), Conclusion::Unreachable { .. }), + "got {:?}", + err.conclusion() + ); + + server.shutdown().await; +} + +#[tokio::test] +async fn test_a_pin_that_stops_the_ladder_still_asks_for_legacy_tls() { + // The pin is why nothing wider was tried, so unpinning is the answer. + let server = TlsTestServer::start(TlsServerConfig { + max_tls_version: Some(SslVersion::TLS1), + ..Default::default() + }) + .await; + let mut config = default_config(&server.url()); + config.min_tls_version = Some("1.2".to_string()); + + let err = HyperClient::new().send(&config).await.unwrap_err(); + + assert!( + matches!(err.conclusion(), Conclusion::NeedsLegacyTls { .. }), + "got {:?}", + err.conclusion() + ); + + server.shutdown().await; +} diff --git a/tests/live_benchmark.rs b/tests/live_benchmark.rs new file mode 100644 index 0000000..cc9c8ec --- /dev/null +++ b/tests/live_benchmark.rs @@ -0,0 +1,289 @@ +// Live measurement against real services. Opt-in, never part of CI. +// +// cargo test --test live_benchmark -- --ignored --nocapture +// +// Everything here needs the network and some of it needs third-party services +// that rate-limit, go away, or change behaviour between runs. None of that +// belongs in a gate. These print a scorecard and assert only that the +// measurement itself worked, because an assertion like "we are not blocked" +// would fail for reasons entirely outside this repository. +// +// The deterministic gates live in fingerprint_baseline.rs (what we send), +// ja4_conformance.rs (whether our JA4 agrees with everyone else's) and +// antibot_classify.rs (how outcomes are read). Those run offline in CI. + +mod support; + +use blasthttp::antibot::{Outcome, ResponseFacts, classify}; +use blasthttp::client::HttpClient; +use blasthttp::client::hyper::HyperClient; +use blasthttp::config::RequestConfig; +use support::tls_server::{TlsServerConfig, TlsTestServer}; + +fn config(url: &str) -> RequestConfig { + config_with(url, None) +} + +fn config_with(url: &str, profile: Option<&str>) -> RequestConfig { + let mut c = RequestConfig::new(url.to_string()); + c.verify_certs = Some(false); + c.timeout_seconds = Some(30); + c.follow_redirects = Some(true); + c.profile = profile.map(|p| p.to_string()); + c +} + +async fn fetch(url: &str) -> Option<(u16, Vec<(String, String)>, String)> { + fetch_with(url, None).await +} + +async fn fetch_with( + url: &str, + profile: Option<&str>, +) -> Option<(u16, Vec<(String, String)>, String)> { + let client = HyperClient::new(); + match client.send(&config_with(url, profile)).await { + Ok(r) => { + let body = r.body().to_string(); + Some((r.status, r.headers.clone(), body)) + } + Err(e) => { + println!(" request failed: {}", e.message); + None + } + } +} + +/// Our own ClientHello, captured locally, addressed by hostname so SNI is sent +/// and the result is directly comparable with what a remote service computes. +async fn local_ja4() -> String { + let server = TlsTestServer::start(TlsServerConfig::default()).await; + let url = format!("https://localhost:{}/", server.addr.port()); + + let client = HyperClient::new(); + let _ = client.send(&config(&url)).await; + + let hello = server + .hello + .lock() + .expect("hello slot poisoned") + .clone() + .expect("no ClientHello captured"); + server.shutdown().await; + + support::ja4::ja4(&hello) +} + +#[tokio::test] +#[ignore = "needs network; run with --ignored"] +async fn live_ja4_agrees_with_a_remote_observer() { + // Cross-checks the whole capture path. Our local reading and a third + // party's should describe the same client, since it is the same client. + // A mismatch means either the capture is wrong or something between us and + // them is rewriting the handshake, and both are worth knowing. + let local = local_ja4().await; + println!("\n local JA4: {}", local); + + let Some((status, _, body)) = fetch("https://tls.peet.ws/api/all").await else { + println!(" remote observer unreachable, skipping comparison"); + return; + }; + assert_eq!(status, 200, "fingerprint service did not answer"); + + let parsed: serde_json::Value = serde_json::from_str(&body).expect("expected JSON"); + let remote = parsed["tls"]["ja4"].as_str().unwrap_or("(absent)"); + println!(" remote JA4: {}", remote); + + // Not an assertion either way. JA4 implementations genuinely disagree: + // one handshake shown to seven services produced three different JA3s and + // two different JA4s. + // + // One divergence here is known and expected. The JA4 header is fixed + // width and the spec caps both counts at 99; our legacy list is 105 + // suites, so we print `99` where peet.ws prints `105`. The two hash + // segments match exactly, which says the underlying reading is the same + // and only the rendering differs. A stealth profile at 15 suites makes it + // go away. + if local == remote { + println!(" agree"); + } else { + let (l, r) = (local.split('_').next(), remote.split('_').next()); + if local.split('_').skip(1).eq(remote.split('_').skip(1)) { + println!(" headers differ ({:?} vs {:?}), hashes agree", l, r); + println!(" (expected while our cipher count is over the 99 cap)"); + } else { + println!(" hashes differ, worth investigating which of us is wrong"); + } + } +} + +#[tokio::test] +#[ignore = "needs network; run with --ignored"] +async fn live_fingerprint_prevalence() { + // The check nothing else provides: does our ClientHello occur in real + // traffic at all? A profile can match a published JA4 exactly and still be + // a shape no real client emits, and no signature comparison catches that. + // + // Measured 2026-09-28: before the legacy cipher work our hello had been + // seen 903 times across roughly 17B observations; after, it does not + // appear at all. A real Chrome sits above 450M. + let Some((_, _, body)) = fetch("https://tls.tlsfingerprint.io/api/client-fingerprint").await + else { + println!(" fingerprint database unreachable, skipping"); + return; + }; + + let parsed: serde_json::Value = match serde_json::from_str(&body) { + Ok(v) => v, + Err(_) => { + println!(" unexpected response shape, skipping"); + return; + } + }; + let Some(id) = parsed["norm_hex_id"].as_str() else { + println!(" no norm_hex_id in response, skipping"); + return; + }; + + println!("\n our ClientHello id: {}", id); + + let url = format!( + "https://tlsfingerprint.io/api/tls/fingerprints/{}/exists", + id + ); + let Some((_, _, seen)) = fetch(&url).await else { + return; + }; + + let parsed: serde_json::Value = serde_json::from_str(&seen).unwrap_or_default(); + match parsed["total"].as_u64() { + Some(n) => println!(" observed {} times in real traffic", n), + None => println!(" never observed in real traffic (the loudest signal we emit)"), + } +} + +#[tokio::test] +#[ignore = "needs network; run with --ignored"] +async fn live_block_scorecard() { + // One GET per target, classified structurally, scored per tier. + // + // Scoring across tiers as one number is worse than useless, which an + // earlier version of this proved by reporting "1/4" when the true bypass + // count was zero. Two ways that number lied: + // + // - the one success was the control, which protects nothing, so passing + // it demonstrates nothing + // - it counted the JS tier in the denominator, which no HTTP client + // reaches, so the score could never approach 4/4 however good the + // work got + // + // So: controls are verified and not scored, and only the passive tier is + // scored, because that is the only tier this work can move. + // + // Do not grow this into a list of ordinary commercial sites, and do not + // put it on a schedule. + + /// What a target is for, which decides whether and how it is scored. + enum Tier { + /// Protects nothing. Confirms the harness reaches the internet and + /// reads an ordinary response correctly. + Control, + /// Passive fingerprinting. The tier stealth mode exists to beat, and + /// the only one worth a score. + Passive, + /// Interactive JavaScript challenge. Out of reach for any HTTP client, + /// curl_cffi included. Reported so a change would be visible, never + /// counted against us. + Javascript, + } + + let targets = [ + ( + Tier::Control, + "cloudflare.com", + "https://www.cloudflare.com/", + ), + // The vendor's own property, and the clearest discriminator we have: + // it refused blasthttp and served a Chrome-impersonating client 219KB + // from the same address in the same minute. + (Tier::Passive, "akamai.com", "https://www.akamai.com/"), + ( + Tier::Javascript, + "scrapingcourse cf-challenge", + "https://www.scrapingcourse.com/cloudflare-challenge", + ), + ( + Tier::Javascript, + "scrapingcourse antibot", + "https://www.scrapingcourse.com/antibot-challenge", + ), + ]; + + // Each target twice: once plain, once impersonating. The delta between + // the two columns is the only number that says whether this work does + // anything, and running them in one pass keeps the comparison honest + // about time of day, network path and whatever the site is doing today. + async fn outcome_of(url: &str, profile: Option<&str>) -> Outcome { + match fetch_with(url, profile).await { + Some((status, headers, body)) => classify(&ResponseFacts { + status, + headers: &headers, + body: &body, + }), + None => Outcome::Error, + } + } + + println!( + " {:9} {:30} {:24} profile=chrome", + "tier", "target", "plain" + ); + let (mut passive_plain, mut passive_profiled, mut passive_total) = (0, 0, 0); + let mut control_failed = false; + + for (tier, label, url) in targets { + let plain = outcome_of(url, None).await; + let profiled = outcome_of(url, Some("chrome")).await; + + let tag = match tier { + Tier::Control => { + if !plain.got_through() { + control_failed = true; + } + "control" + } + Tier::Passive => { + passive_total += 1; + if plain.got_through() { + passive_plain += 1; + } + if profiled.got_through() { + passive_profiled += 1; + } + "passive" + } + Tier::Javascript => "js", + }; + println!( + " [{:7}] {:30} {:24} {:?}", + tag, + label, + format!("{:?}", plain), + profiled + ); + } + + println!(); + println!( + " passive-fingerprint tier: {}/{} plain, {}/{} with profile", + passive_plain, passive_total, passive_profiled, passive_total + ); + println!(" javascript tier: not scored, unreachable without a browser"); + println!(" controls: not scored"); + + assert!( + !control_failed, + "a control target was blocked, so this run says nothing about us: \ + either the network path is interfering or the site changed" + ); +} diff --git a/tests/proxy_tls.rs b/tests/proxy_tls.rs index db37397..9bd4b91 100644 --- a/tests/proxy_tls.rs +++ b/tests/proxy_tls.rs @@ -4,14 +4,13 @@ //! that goes through the tunnel: a TLS handshake starts with 0x16, and a //! plaintext request starts with the method. -#[allow(dead_code)] -mod tls_server; +mod support; use blasthttp::client::HttpClient; use blasthttp::client::hyper::HyperClient; use blasthttp::config::RequestConfig; use std::sync::{Arc, Mutex}; -use tls_server::{TlsServerConfig, TlsTestServer}; +use support::tls_server::{TlsServerConfig, TlsTestServer}; use tokio::io::{AsyncReadExt, AsyncWriteExt}; use tokio::net::{TcpListener, TcpStream}; diff --git a/tests/python/test_antibot_rules.py b/tests/python/test_antibot_rules.py new file mode 100644 index 0000000..5d046a8 --- /dev/null +++ b/tests/python/test_antibot_rules.py @@ -0,0 +1,51 @@ +"""Replacing the anti-bot rules from Python, with a JSON file shaped like the built-in one.""" + +import json + +import pytest + +import blasthttp + +CHALLENGE_PAGE = b"example-challenge" + + +def protection(body): + return blasthttp.Response("https://example.test/", 200, headers=[], body=body).protection + + +@pytest.fixture +def restore_rules(): + yield + blasthttp.set_antibot_rules() + + +def test_a_rules_file_replaces_the_rules(tmp_path, restore_rules): + rules = tmp_path / "rules.json" + rules.write_text( + json.dumps( + { + "refusal_statuses": [403], + "rules": [ + {"vendor": "examplewaf", "outcome": "challenge", "when": {"body_contains": "example-challenge"}} + ], + } + ) + ) + assert protection(CHALLENGE_PAGE) == ("ok", "-") + blasthttp.set_antibot_rules(str(rules)) + assert protection(CHALLENGE_PAGE) == ("challenge", "examplewaf") + blasthttp.set_antibot_rules() + assert protection(CHALLENGE_PAGE) == ("ok", "-") + + +def test_a_bad_rules_file_raises_and_keeps_the_rules(tmp_path, restore_rules): + rules = tmp_path / "rules.json" + rules.write_text('{"refusal_statuses": [], "rules": [{"vendor": "x", "outcome": "maybe", "when": {}}]}') + with pytest.raises(ValueError, match="rules.json"): + blasthttp.set_antibot_rules(str(rules)) + with pytest.raises(ValueError, match="missing.json"): + blasthttp.set_antibot_rules(str(tmp_path / "missing.json")) + assert protection(b'') == ( + "challenge", + "imperva", + ) diff --git a/tests/python/test_mock.py b/tests/python/test_mock.py index 6e40b49..452094f 100644 --- a/tests/python/test_mock.py +++ b/tests/python/test_mock.py @@ -333,6 +333,64 @@ async def test_batch_unmatched_url_yields_error_result(): assert "No mock response" in results[0].error +# ── errors match the real client ───────────────────────────────── + + +@pytest.mark.asyncio +async def test_unmatched_url_raises_transport_error(): + """The real client raises TransportError for a host it can't reach, so the + mock does too. Code written as `except TransportError` that branches on + `kind` would otherwise crash under the mock on any URL a test didn't mock.""" + mock = BlasthttpMock() + with pytest.raises(blasthttp.TransportError) as exc: + await mock.request("http://nothing/") + assert "No mock response" in str(exc.value) + assert exc.value.kind == "connection" + assert exc.value.conclusion == "unreachable" + + +@pytest.mark.asyncio +async def test_unmatched_url_is_still_a_runtime_error(): + mock = BlasthttpMock() + with pytest.raises(RuntimeError): + await mock.request("http://nothing/") + + +@pytest.mark.asyncio +async def test_redirect_loop_raises_transport_error(): + mock = BlasthttpMock() + mock.add_response(url="http://loop/", status_code=302, headers={"Location": "http://loop/"}) + with pytest.raises(blasthttp.TransportError) as exc: + await mock.request("http://loop/", follow_redirects=True, max_redirects=2) + assert exc.value.kind == "too_many_redirects" + assert exc.value.retryable is False + + +@pytest.mark.asyncio +async def test_batch_unmatched_url_carries_transport_error(): + mock = BlasthttpMock() + results = await mock.request_batch([blasthttp.BatchConfig("http://nothing/")]) + t = results[0].transport_error + assert isinstance(t, blasthttp.TransportError) + assert t.kind == "connection" + assert results[0].error == str(t) + + +@pytest.mark.asyncio +async def test_batch_callback_error_has_no_transport_error(): + """An exception from a test's own callback is not a transport failure, so + it keeps the plain error string and nothing more.""" + mock = BlasthttpMock() + + def boom(request): + raise ValueError("callback broke") + + mock.add_callback(boom, url="http://cb/") + results = await mock.request_batch([blasthttp.BatchConfig("http://cb/")]) + assert "callback broke" in results[0].error + assert results[0].transport_error is None + + # ── should_intercept + pass-through ────────────────────────────── @@ -388,3 +446,52 @@ async def handler(reader, writer): finally: server.close() await server.wait_closed() + + +@pytest.mark.asyncio +async def test_passthrough_forwards_the_kwargs_it_ignores(): + """A mocked URL can ignore `profile` because nothing is dialled. A + passed-through one cannot: it is a real connection, and dropping the + kwarg silently gives it different TLS and different headers from what + the caller asked for. BBOT runs its whole local-target suite through + this path, so a dropped kwarg there is a test that passes while + exercising the wrong thing.""" + + async def handler(reader, writer): + received = [] + await reader.readuntil(b"\r\n") + while (line := await reader.readuntil(b"\r\n")) != b"\r\n": + received.append(line.decode("latin1").rstrip("\r\n")) + body = "\n".join(received).encode() + writer.write(b"HTTP/1.1 200 OK\r\nContent-Length: " + str(len(body)).encode() + b"\r\n\r\n" + body) + await writer.drain() + writer.close() + + server = await asyncio.start_server(handler, "127.0.0.1", 0) + addr = server.sockets[0].getsockname() + base = f"http://{addr[0]}:{addr[1]}/" + try: + real = blasthttp.BlastHTTP() + mock = BlasthttpMock(real_client=real, should_mock_fn=lambda host: host != "127.0.0.1") + + r = await mock.request(base, profile="chrome") + assert "Mozilla/5.0" in r.text, f"profile was dropped on the way through: {r.text}" + + # And the default still arrives as itself, so the test above is + # measuring the kwarg rather than a client that always sends Chrome. + r2 = await mock.request(base) + assert "blasthttp/" in r2.text + finally: + server.close() + await server.wait_closed() + + +@pytest.mark.asyncio +async def test_passthrough_rejects_a_kwarg_the_real_client_does_not_take(): + """The flip side of forwarding. An intercepted URL swallows anything, + which is right when nothing is dialled, but on the way through the real + client's signature is the one that counts and a typo should be audible.""" + real = blasthttp.BlastHTTP() + mock = BlasthttpMock(real_client=real, should_mock_fn=lambda host: host != "127.0.0.1") + with pytest.raises(TypeError): + await mock.request("http://127.0.0.1:1/", not_a_real_kwarg=1) diff --git a/tests/python/test_profiles_and_report.py b/tests/python/test_profiles_and_report.py new file mode 100644 index 0000000..220a684 --- /dev/null +++ b/tests/python/test_profiles_and_report.py @@ -0,0 +1,303 @@ +"""The 1.0 Python surface: connection profiles, and the report a caller reads +to decide what to do next. + +This is the contract BBOT consumes. It had no test coverage at all until this +file, which mattered more than usual because the interesting part is not +whether a request succeeds but whether a *failure* is legible enough to act +on, and that is exactly the path nothing exercised. + +Everything here runs against a local server, so it is deterministic and needs +no network. What it cannot cover locally is the ladder actually shifting +profiles against a real protection product; the Rust integration tests do that +against a TLS server that can refuse a handshake, and `bench/` does it against +real sites. +""" + +import asyncio + +import pytest +import pytest_asyncio + +import blasthttp + + +# ── Local server ───────────────────────────────────────────────── + + +async def _handler(reader, writer): + """Serves paths that stand in for the things a caller has to tell apart. + + / ordinary 200 + /echo-headers 200 whose body is the request headers it received + /cf-challenge 403 shaped like a Cloudflare managed challenge, unless the + request carries the cookie pass=1 + /akamai-block 403 shaped like an Akamai edge refusal + /plain-403 403 with no product signature at all + """ + try: + request_line = await reader.readuntil(b"\r\n") + received = [] + while True: + line = await reader.readuntil(b"\r\n") + if line == b"\r\n": + break + received.append(line.decode("latin1").rstrip("\r\n")) + + try: + _method, path, _ver = request_line.decode().split(" ", 2) + except ValueError: + path = "/" + + def respond(status, headers, body): + raw = f"HTTP/1.1 {status} X\r\n".encode() + for k, v in headers: + raw += f"{k}: {v}\r\n".encode() + raw += b"Content-Length: " + str(len(body)).encode() + b"\r\n" + raw += b"Connection: close\r\n\r\n" + body + writer.write(raw) + + if path == "/echo-headers": + body = "\n".join(received).encode() + respond(200, [("Content-Type", "text/plain")], body) + elif path == "/cf-challenge" and not any(h.lower().startswith("cookie:") and "pass=1" in h for h in received): + # cf-mitigated is the header Cloudflare sets specifically to say it + # intervened, and is what separates a challenge from a WAF block. + respond( + 403, + [("server", "cloudflare"), ("cf-mitigated", "challenge")], + b"Just a moment...", + ) + elif path == "/akamai-block": + # No `server` header at all, which is what www.akamai.com's own 403 + # actually sends, and the reason a rule keyed on it matched nothing. + respond( + 403, + [("akamai-grn", "0.6d551702.1790644996.315f62a4")], + b"\nAccess Denied\n", + ) + elif path == "/plain-403": + respond(403, [("server", "nginx")], b"forbidden") + else: + respond(200, [("Content-Type", "text/plain")], b"hello") + + await writer.drain() + except Exception: + pass + finally: + writer.close() + + +@pytest_asyncio.fixture +async def server(): + srv = await asyncio.start_server(_handler, "127.0.0.1", 0) + port = srv.sockets[0].getsockname()[1] + async with srv: + yield f"http://127.0.0.1:{port}" + + +@pytest.fixture +def client(): + return blasthttp.BlastHTTP() + + +# ── Profiles ───────────────────────────────────────────────────── + + +async def test_default_profile_sends_an_honest_user_agent(server, client): + """The default claims nothing. That is the whole point of it: claiming to + be a browser selects which scrutiny applies, and a client that cannot back + the claim is worse off than one that never made it.""" + r = await client.request(f"{server}/echo-headers") + assert "blasthttp/" in r.text + assert "Mozilla/5.0" not in r.text + + +async def test_chrome_profile_sends_browser_headers_in_order(server, client): + r = await client.request(f"{server}/echo-headers", profile="chrome") + sent = [line.split(":", 1)[0].lower() for line in r.text.splitlines() if ":" in line] + + assert "Mozilla/5.0" in r.text + for expected in ("sec-ch-ua", "sec-fetch-dest", "accept-language", "priority"): + assert expected in sent, f"{expected} missing: {sent}" + + # Order is part of the fingerprint, not decoration, so a profile whose + # headers arrive shuffled has not been applied properly. + assert sent.index("sec-ch-ua") < sent.index("user-agent") < sent.index("accept") + + +async def test_compatibility_profile_is_also_honest(server, client): + r = await client.request(f"{server}/echo-headers", profile="compatibility") + assert "blasthttp/" in r.text + assert "Mozilla/5.0" not in r.text + + +async def test_unknown_profile_is_an_error(server, client): + """It used to resolve silently to the default, so a typo bought you + whatever that was with nothing said about it. Once the choice of profile + decides whether a request gets inspected, that is not a good trade.""" + with pytest.raises(Exception) as exc: + await client.request(f"{server}/", profile="chrmoe") + assert "chrmoe" in str(exc.value).lower() + + +async def test_an_explicit_header_beats_the_profile(server, client): + r = await client.request( + f"{server}/echo-headers", + profile="chrome", + headers=[("user-agent", "mine/1.0")], + ) + assert "mine/1.0" in r.text + assert "Chrome/131" not in r.text + + +# ── The report ─────────────────────────────────────────────────── + + +async def test_an_ordinary_response_reports_reached(server, client): + r = await client.request(f"{server}/") + assert r.conclusion == ("reached", "-") + assert r.protection == ("ok", "-") + assert [a[0] for a in r.attempts] == ["modern"] + assert r.attempts[0][1] == "reached" + assert r.attempts[0][2] == 200 + + +async def test_a_challenge_asks_for_a_browser(server, client): + """The signal BBOT keys on. A challenge needs JavaScript, so no amount of + retrying with different connection settings will help and the URL should + go to a real browser instead.""" + r = await client.request(f"{server}/cf-challenge") + assert r.status_code == 403 + assert r.protection == ("challenge", "cloudflare") + assert r.conclusion == ("needs_browser", "cloudflare") + + +async def test_a_host_that_refuses_every_rung_gets_one_request_after(server, client): + """Only success used to be remembered, so a host that refused everything had + the whole ladder walked again for every request: two requests where one + tells you the same thing, on the hosts most likely to rate-limit or ban.""" + url = f"{server}/cf-challenge" + + r = await client.request(url) + assert [a[0] for a in r.attempts] == ["modern", "chrome131"] + assert r.conclusion == ("needs_browser", "cloudflare") + + # recently refused on every rung: one request, and the refusal is the answer + r = await client.request(url) + assert [a[0] for a in r.attempts] == ["modern"] + assert r.conclusion == ("needs_browser", "cloudflare") + + # getting through clears it, so the next refusal walks the ladder again + r = await client.request(url, headers=[("cookie", "pass=1")]) + assert r.conclusion == ("reached", "-") + r = await client.request(url) + assert [a[0] for a in r.attempts] == ["modern", "chrome131"] + + +async def test_a_block_is_not_a_challenge(server, client): + """The distinction that decides whether to launch a browser. A block might + yield to a different approach; a challenge will not.""" + r = await client.request(f"{server}/akamai-block") + assert r.protection == ("blocked", "akamai") + conclusion, vendor = r.conclusion + assert conclusion == "blocked" + assert vendor == "akamai" + assert conclusion != "needs_browser" + + +async def test_a_bare_403_names_no_vendor(server, client): + """Most 403s are ordinary authorization failures. Reporting one as though a + product refused us would invite a caller to act on nothing, and internally + it is what stops the ladder spending a second request on every one.""" + r = await client.request(f"{server}/plain-403") + assert r.protection == ("blocked", "unknown") + assert r.conclusion == ("blocked", "-") + + +# ── Failures ───────────────────────────────────────────────────── + + +async def test_a_refused_connection_is_legible(client): + """Before 1.0 every transport failure arrived as a RuntimeError whose only + content was a sentence, so a caller could not tell a dead host from a + refused cipher without parsing English.""" + with pytest.raises(blasthttp.TransportError) as exc: + await client.request("http://127.0.0.1:1/", timeout=5) + + e = exc.value + assert e.kind == "connection" + assert e.tls_failure is None + assert e.retryable is True + assert e.conclusion == "unreachable" + + +async def test_transport_error_is_still_a_runtime_error(client): + """Subclassing RuntimeError on purpose, so callers written against the old + behaviour keep working.""" + with pytest.raises(RuntimeError): + await client.request("http://127.0.0.1:1/", timeout=5) + + +async def test_a_failed_batch_request_carries_the_transport_error(client): + """A batch used to report only the message, so a caller had to parse it to + tell a timeout from a refused connection from a TLS failure.""" + results = await client.request_batch([blasthttp.BatchConfig("http://127.0.0.1:1/", timeout=5)]) + r = results[0] + assert isinstance(r.transport_error, blasthttp.TransportError) + assert r.transport_error.kind == "connection" + assert r.transport_error.retryable is True + assert r.transport_error.conclusion == "unreachable" + assert r.error == str(r.transport_error) + + +async def test_a_successful_batch_request_has_no_transport_error(server, client): + results = await client.request_batch([blasthttp.BatchConfig(f"{server}/")]) + assert results[0].transport_error is None + + +def test_batch_result_takes_only_a_transport_error(): + with pytest.raises(TypeError): + blasthttp.BatchResult("http://x/", error="e", transport_error=ValueError("e")) + + +async def test_batch_results_carry_the_report(server, client): + cfgs = [ + blasthttp.BatchConfig(f"{server}/"), + blasthttp.BatchConfig(f"{server}/cf-challenge"), + ] + results = await client.request_batch(cfgs, concurrency=2) + by_url = {r.url: r for r in results} + + assert by_url[f"{server}/"].response.conclusion == ("reached", "-") + assert by_url[f"{server}/cf-challenge"].response.conclusion == ( + "needs_browser", + "cloudflare", + ) + + +# ── download ───────────────────────────────────────────────────── + + +async def test_download_takes_a_profile(server, client, tmp_path): + """`download` used to hardcode no profile with no way to pass one, so a + file behind a host that only answers a browser was unreachable through it + while the identical URL through `request` was fine.""" + dest = tmp_path / "headers.txt" + await client.download(f"{server}/echo-headers", str(dest), profile="chrome") + assert "Mozilla/5.0" in dest.read_text() + + +async def test_download_defaults_to_the_honest_profile(server, client, tmp_path): + """Which makes the test above about the argument rather than about a + client that always claims to be Chrome.""" + dest = tmp_path / "headers.txt" + await client.download(f"{server}/echo-headers", str(dest)) + text = dest.read_text() + assert "blasthttp/" in text + assert "Mozilla/5.0" not in text + + +async def test_download_rejects_an_unknown_profile(server, client, tmp_path): + with pytest.raises(Exception) as exc: + await client.download(f"{server}/", str(tmp_path / "x"), profile="chrmoe") + assert "chrmoe" in str(exc.value).lower() diff --git a/tests/support/ja4.rs b/tests/support/ja4.rs new file mode 100644 index 0000000..6e06060 --- /dev/null +++ b/tests/support/ja4.rs @@ -0,0 +1,182 @@ +// JA4 over a captured ClientHello. +// +// JA4 is FoxIO's TLS client fingerprint (BSD-3; the JA4+ family that includes +// JA4H is under a separate non-commercial licence and is deliberately not +// implemented here). It replaced JA3 because JA3 preserves ordering and counts +// GREASE, and Chrome shuffles its extension order and emits fresh GREASE on +// every connection, so one Chrome build produces a different JA3 per +// handshake. JA4 sorts and strips, so it holds still. +// +// That property is why this is the metric worth gating on: extension order and +// GREASE are the two things OpenSSL cannot control, and JA4 ignores both. +// +// JA4 = __ +// a protocol, version, SNI, cipher count, extension count, ALPN +// b first 12 hex of sha256 over the sorted cipher list +// c first 12 hex of sha256 over the sorted extension list, then the +// signature algorithms in the order sent +// +// Operates on a `CapturedHello` from the sibling `tls_server` module. + +use super::tls_server::CapturedHello; + +/// RFC 8701 GREASE values: both bytes equal, low nibble `a`. +/// 0x0a0a, 0x1a1a, ... 0xfafa. +pub fn is_grease(v: u16) -> bool { + (v >> 8) == (v & 0x00ff) && (v & 0x000f) == 0x000a +} + +fn strip_grease(values: &[u16]) -> Vec { + values.iter().copied().filter(|v| !is_grease(*v)).collect() +} + +fn sha256_prefix12(input: &str) -> String { + let digest = openssl::hash::hash(openssl::hash::MessageDigest::sha256(), input.as_bytes()) + .expect("sha256 of an in-memory string cannot fail"); + digest + .iter() + .map(|b| format!("{:02x}", b)) + .collect::()[..12] + .to_string() +} + +/// A list that contributes nothing hashes to zeros rather than to the hash of +/// an empty string, which is what the spec calls for. +fn hash_list(values: &[u16]) -> String { + if values.is_empty() { + return "000000000000".to_string(); + } + let joined = values + .iter() + .map(|v| format!("{:04x}", v)) + .collect::>() + .join(","); + sha256_prefix12(&joined) +} + +/// Two characters naming the negotiated TLS version. +fn version_code(hello: &CapturedHello) -> &'static str { + // `supported_versions` is authoritative when present. The legacy field is + // pinned at TLS 1.2 on any modern hello regardless of what is really + // offered, so reading it alone would call every TLS 1.3 client 1.2. + let versions = strip_grease(&hello.supported_versions); + let highest = versions + .iter() + .copied() + .max() + .unwrap_or(hello.legacy_version); + match highest { + 0x0304 => "13", + 0x0303 => "12", + 0x0302 => "11", + 0x0301 => "10", + 0x0300 => "s3", + _ => "00", + } +} + +/// First and last character of the first ALPN value: `h2` stays `h2`, +/// `http/1.1` becomes `h1`. `00` when no ALPN was offered. +fn alpn_code(hello: &CapturedHello) -> String { + match hello.alpn.first() { + None => "00".to_string(), + Some(first) => { + let chars: Vec = first.chars().collect(); + match (chars.first(), chars.last()) { + (Some(a), Some(z)) if a.is_ascii() && z.is_ascii() => format!("{}{}", a, z), + _ => "99".to_string(), + } + } + } +} + +/// Compute the JA4 string for a captured hello. +pub fn ja4(hello: &CapturedHello) -> String { + let ciphers = strip_grease(&hello.ciphers); + let extensions = strip_grease(&hello.extensions); + + // Counts are capped at 99 because the field is two digits wide. + let cipher_count = ciphers.len().min(99); + let ext_count = extensions.len().min(99); + + let a = format!( + "t{}{}{:02}{:02}{}", + version_code(hello), + if hello.has_sni { "d" } else { "i" }, + cipher_count, + ext_count, + alpn_code(hello), + ); + + let mut sorted_ciphers = ciphers.clone(); + sorted_ciphers.sort_unstable(); + let b = hash_list(&sorted_ciphers); + + // server_name and ALPN are counted in the header but left out of the hash, + // since both vary with the request rather than with the client. + let mut sorted_exts: Vec = extensions + .iter() + .copied() + .filter(|e| *e != 0x0000 && *e != 0x0010) + .collect(); + sorted_exts.sort_unstable(); + + // Signature algorithms keep the order they were sent in, and GREASE has to + // come out of them too. Chrome 152 puts a fresh greased value at the head + // of this list every handshake, so an implementation that keeps it reports + // a different third segment on every connection. + let sig_algs = strip_grease(&hello.sig_algs); + + let c = if sorted_exts.is_empty() { + "000000000000".to_string() + } else { + let ext_part = sorted_exts + .iter() + .map(|v| format!("{:04x}", v)) + .collect::>() + .join(","); + let sig_part = sig_algs + .iter() + .map(|v| format!("{:04x}", v)) + .collect::>() + .join(","); + sha256_prefix12(&format!("{}_{}", ext_part, sig_part)) + }; + + format!("{}_{}_{}", a, b, c) +} + +/// The readable form: same fields, nothing hashed. What you actually want when +/// a fingerprint moved and you need to know which byte did it. +pub fn ja4_raw(hello: &CapturedHello) -> String { + let ciphers = strip_grease(&hello.ciphers); + let extensions = strip_grease(&hello.extensions); + + let mut sorted_ciphers = ciphers.clone(); + sorted_ciphers.sort_unstable(); + let mut sorted_exts: Vec = extensions + .iter() + .copied() + .filter(|e| *e != 0x0000 && *e != 0x0010) + .collect(); + sorted_exts.sort_unstable(); + + let hex = |vs: &[u16]| { + vs.iter() + .map(|v| format!("{:04x}", v)) + .collect::>() + .join(",") + }; + + format!( + "t{}{}{:02}{:02}{}_{}_{}_{}", + version_code(hello), + if hello.has_sni { "d" } else { "i" }, + ciphers.len().min(99), + extensions.len().min(99), + alpn_code(hello), + hex(&sorted_ciphers), + hex(&sorted_exts), + hex(&strip_grease(&hello.sig_algs)), + ) +} diff --git a/tests/support/mod.rs b/tests/support/mod.rs new file mode 100644 index 0000000..1163676 --- /dev/null +++ b/tests/support/mod.rs @@ -0,0 +1,14 @@ +// Shared helpers for the integration tests. +// +// These live under `tests/support/` rather than directly in `tests/` because +// Cargo compiles every top-level file there as its own test binary. A helper +// module sitting at that level becomes a test target with no tests in it, and +// one that fails to build on its own the moment it references a sibling. +// +// Not every test uses every helper, so the modules allow dead code rather than +// making each consumer carry an allow attribute. + +#![allow(dead_code)] + +pub mod ja4; +pub mod tls_server; diff --git a/tests/support/tls_server.rs b/tests/support/tls_server.rs new file mode 100644 index 0000000..f22d2a0 --- /dev/null +++ b/tests/support/tls_server.rs @@ -0,0 +1,445 @@ +// Shared test helper: a minimal HTTPS server using OpenSSL directly. +// Accepts one connection, does TLS handshake, sends a hardcoded HTTP response, +// then shuts down. Configurable cipher list and TLS version range. + +use openssl::asn1::Asn1Time; +use openssl::bn::BigNum; +use openssl::hash::MessageDigest; +use openssl::pkey::PKey; +use openssl::rsa::Rsa; +use openssl::ssl::{AlpnError, SslAcceptor, SslAcceptorBuilder, SslMethod, SslOptions, SslVersion}; +use openssl::x509::X509; +use openssl::x509::extension::SubjectAlternativeName; +use std::net::SocketAddr; +use tokio::io::{AsyncReadExt, AsyncWriteExt}; +use tokio::net::TcpListener; + +/// The parts of a ClientHello that make up a TLS fingerprint. +/// +/// Read back server-side from the handshake the client under test actually +/// sent, so it reflects what went on the wire rather than what the config +/// asked for. Those differ more often than you would like: `cipher_string` +/// only governs TLS 1.2 and below, for instance, so the 1.3 suite list comes +/// from somewhere else entirely. +#[derive(Debug, Clone, Default)] +pub struct CapturedHello { + /// The legacy `client_version` field, which is pinned at TLS 1.2 on any + /// modern hello. The real version lives in `supported_versions`. + pub legacy_version: u16, + /// Cipher suites in the order offered, GREASE included. + pub ciphers: Vec, + /// Extension types in the order sent, GREASE included. + pub extensions: Vec, + /// `supported_groups` (0x000a) contents, GREASE included. + pub groups: Vec, + /// `signature_algorithms` (0x000d) contents, in the order sent. + pub sig_algs: Vec, + /// ALPN protocol names offered, in order. + pub alpn: Vec, + /// `supported_versions` (0x002b) contents, GREASE included. + pub supported_versions: Vec, + /// Whether a `server_name` (0x0000) extension was present. + pub has_sni: bool, +} + +/// Where a capturing server leaves what it saw. `None` until a client +/// completes enough of a handshake to send its hello. +pub type HelloSlot = std::sync::Arc>>; + +pub struct TlsTestServer { + pub addr: SocketAddr, + /// What the client sent, once it has connected. + pub hello: HelloSlot, + shutdown_tx: tokio::sync::oneshot::Sender<()>, + handle: tokio::task::JoinHandle<()>, +} + +pub struct TlsServerConfig { + pub cipher_list: Option, + pub min_tls_version: Option, + pub max_tls_version: Option, + pub response_body: String, + pub response_status: u16, + /// DNS SANs for the self-signed cert. Default: ["localhost"]. + pub san_dns: Vec, + /// IP SANs for the self-signed cert. Default: ["127.0.0.1"]. + pub san_ip: Vec, +} + +impl Default for TlsServerConfig { + fn default() -> Self { + TlsServerConfig { + cipher_list: None, + min_tls_version: None, + max_tls_version: None, + response_body: "OK".to_string(), + response_status: 200, + san_dns: vec!["localhost".to_string()], + san_ip: vec!["127.0.0.1".to_string()], + } + } +} + +// Generate a self-signed cert + key pair in memory (no files needed) +fn generate_self_signed(config: &TlsServerConfig) -> (PKey, X509) { + let rsa = Rsa::generate(2048).unwrap(); + let pkey = PKey::from_rsa(rsa).unwrap(); + + let mut builder = X509::builder().unwrap(); + builder.set_version(2).unwrap(); + + let serial = BigNum::from_u32(1).unwrap(); + builder + .set_serial_number(&serial.to_asn1_integer().unwrap()) + .unwrap(); + + let mut name = openssl::x509::X509NameBuilder::new().unwrap(); + name.append_entry_by_text("CN", "localhost").unwrap(); + let name = name.build(); + builder.set_issuer_name(&name).unwrap(); + builder.set_subject_name(&name).unwrap(); + + let not_before = Asn1Time::days_from_now(0).unwrap(); + let not_after = Asn1Time::days_from_now(1).unwrap(); + builder.set_not_before(¬_before).unwrap(); + builder.set_not_after(¬_after).unwrap(); + + builder.set_pubkey(&pkey).unwrap(); + + let mut san = SubjectAlternativeName::new(); + for dns in &config.san_dns { + san.dns(dns); + } + for ip in &config.san_ip { + san.ip(ip); + } + let san = san.build(&builder.x509v3_context(None, None)).unwrap(); + builder.append_extension(san).unwrap(); + + builder.sign(&pkey, MessageDigest::sha256()).unwrap(); + let cert = builder.build(); + + (pkey, cert) +} + +// ── ClientHello capture ─────────────────────────────────────────── +// +// Parsed straight off the wire rather than through OpenSSL's +// `SSL_CTX_set_client_hello_cb`. That callback looks like the obvious tool and +// quietly under-reports: `SSL_client_hello_get1_extensions_present` only +// returns extensions the *server's* OpenSSL has a definition for, so anything +// it has not heard of is invisible. Since the extensions worth testing are +// precisely the ones OpenSSL does not implement, ALPS and ECH among them, that +// made the harness blind to the thing it exists to measure. +// +// Reading the bytes has two further benefits we need later: extension order +// survives, and so does GREASE. Both are invisible through the callback. + +/// Split a big-endian `u16` array, as TLS encodes cipher and extension lists. +fn be_u16s(bytes: &[u8]) -> Vec { + bytes + .chunks_exact(2) + .map(|c| u16::from_be_bytes([c[0], c[1]])) + .collect() +} + +/// A cursor that refuses to run off the end, so a malformed or truncated hello +/// returns what was understood instead of panicking inside a test server. +struct Reader<'a> { + buf: &'a [u8], + pos: usize, +} + +impl<'a> Reader<'a> { + fn new(buf: &'a [u8]) -> Self { + Reader { buf, pos: 0 } + } + fn take(&mut self, n: usize) -> Option<&'a [u8]> { + let end = self.pos.checked_add(n)?; + if end > self.buf.len() { + return None; + } + let out = &self.buf[self.pos..end]; + self.pos = end; + Some(out) + } + fn u8(&mut self) -> Option { + self.take(1).map(|b| b[0]) + } + fn u16(&mut self) -> Option { + self.take(2).map(|b| u16::from_be_bytes([b[0], b[1]])) + } + /// A TLS vector with a one- or two-byte length prefix. + fn vec8(&mut self) -> Option<&'a [u8]> { + let n = self.u8()? as usize; + self.take(n) + } + fn vec16(&mut self) -> Option<&'a [u8]> { + let n = self.u16()? as usize; + self.take(n) + } +} + +/// Parse a ClientHello out of one or more TLS records. +/// +/// Returns `None` for anything that is not a well-formed handshake, which in a +/// test means the client failed before saying anything useful. +pub fn parse_client_hello(raw: &[u8]) -> Option { + // Reassemble the handshake across records. A hello carrying a real + // key_share and an ECH extension runs past 1400 bytes and is routinely + // split, so reading only the first record would truncate it. + let mut handshake = Vec::new(); + let mut r = Reader::new(raw); + while r.pos < raw.len() { + let content_type = r.u8()?; + let _record_version = r.u16()?; + let fragment = r.vec16()?; + if content_type != 0x16 { + return None; // not a handshake record + } + handshake.extend_from_slice(fragment); + } + + let mut h = Reader::new(&handshake); + if h.u8()? != 0x01 { + return None; // not a ClientHello + } + let len = { + let b = h.take(3)?; + ((b[0] as usize) << 16) | ((b[1] as usize) << 8) | b[2] as usize + }; + let body = h.take(len)?; + + let mut b = Reader::new(body); + let legacy_version = b.u16()?; + b.take(32)?; // random + b.vec8()?; // session id + let ciphers = be_u16s(b.vec16()?); + b.vec8()?; // compression methods + + let mut captured = CapturedHello { + legacy_version, + ciphers, + ..Default::default() + }; + + // Extensions are optional in the grammar, though nothing modern omits them. + let Some(ext_block) = b.vec16() else { + return Some(captured); + }; + + let mut e = Reader::new(ext_block); + while e.pos < ext_block.len() { + let Some(ext_type) = e.u16() else { break }; + let Some(data) = e.vec16() else { break }; + captured.extensions.push(ext_type); + + let mut d = Reader::new(data); + match ext_type { + 0x0000 => captured.has_sni = true, + 0x000a => { + if let Some(v) = d.vec16() { + captured.groups = be_u16s(v); + } + } + 0x000d => { + if let Some(v) = d.vec16() { + captured.sig_algs = be_u16s(v); + } + } + 0x0010 => { + if let Some(list) = d.vec16() { + let mut l = Reader::new(list); + while l.pos < list.len() { + match l.vec8() { + Some(name) => captured + .alpn + .push(String::from_utf8_lossy(name).into_owned()), + None => break, + } + } + } + } + // supported_versions is the odd one out: a one-byte length prefix + // where the others use two. + 0x002b => { + if let Some(v) = d.vec8() { + captured.supported_versions = be_u16s(v); + } + } + _ => {} + } + } + + Some(captured) +} + +fn build_acceptor(config: &TlsServerConfig) -> SslAcceptor { + acceptor_builder(config).build() +} + +/// An acceptor that only speaks HTTP/2, for tests that serve with hyper's +/// h2 server instead of the hardcoded HTTP/1.1 response above. +pub fn build_h2_acceptor(config: &TlsServerConfig) -> SslAcceptor { + let mut builder = acceptor_builder(config); + builder.set_alpn_select_callback(|_, client| { + openssl::ssl::select_next_proto(b"\x02h2", client).ok_or(AlpnError::NOACK) + }); + builder.build() +} + +fn acceptor_builder(config: &TlsServerConfig) -> SslAcceptorBuilder { + // Load the legacy provider so the server can use weak ciphers too + load_legacy_provider(); + + // Build from scratch — no mozilla presets that restrict ciphers/versions. + // mozilla_intermediate sets SSL options that can disable TLS 1.3. + let mut builder = SslAcceptor::mozilla_intermediate(SslMethod::tls()).unwrap(); + builder.set_security_level(0); + // Clear ALL version restrictions and options + builder.set_min_proto_version(None).unwrap(); + builder.set_max_proto_version(None).unwrap(); + // Remove any SSL options that might disable specific protocol versions, + // SSLv3 included, so a test can pin the server to it. + builder.clear_options( + SslOptions::NO_TLSV1_3 + | SslOptions::NO_TLSV1_2 + | SslOptions::NO_TLSV1_1 + | SslOptions::NO_TLSV1 + | SslOptions::NO_SSLV3, + ); + // Accept all ciphers by default + builder + .set_cipher_list("ALL:COMPLEMENTOFALL:eNULL") + .unwrap(); + // TLS 1.3 ciphersuites are configured separately in OpenSSL 3.x + builder + .set_ciphersuites( + "TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:TLS_AES_128_GCM_SHA256", + ) + .unwrap(); + + let (pkey, cert) = generate_self_signed(config); + builder.set_private_key(&pkey).unwrap(); + builder.set_certificate(&cert).unwrap(); + + if let Some(ref ciphers) = config.cipher_list { + builder.set_cipher_list(ciphers).unwrap(); + } + + if let Some(min_ver) = config.min_tls_version { + builder.set_min_proto_version(Some(min_ver)).unwrap(); + } + + if let Some(max_ver) = config.max_tls_version { + builder.set_max_proto_version(Some(max_ver)).unwrap(); + } + + builder +} + +// Load legacy provider for the test server (same as our client does) +static INIT_LEGACY: std::sync::Once = std::sync::Once::new(); + +fn load_legacy_provider() { + INIT_LEGACY.call_once(|| { + let _default = openssl::provider::Provider::try_load(None, "default", true) + .expect("failed to load default provider"); + let _legacy = openssl::provider::Provider::try_load(None, "legacy", true) + .expect("failed to load legacy provider"); + std::mem::forget(_default); + std::mem::forget(_legacy); + }); +} + +impl TlsTestServer { + pub async fn start(config: TlsServerConfig) -> Self { + let hello: HelloSlot = std::sync::Arc::new(std::sync::Mutex::new(None)); + let acceptor = build_acceptor(&config); + + // Bind to random port on localhost + let listener = TcpListener::bind("127.0.0.1:0").await.unwrap(); + let addr = listener.local_addr().unwrap(); + + let (shutdown_tx, shutdown_rx) = tokio::sync::oneshot::channel::<()>(); + + let hello_slot = hello.clone(); + let handle = tokio::spawn(async move { + let response = format!( + "HTTP/1.1 {} OK\r\nContent-Length: {}\r\nConnection: close\r\n\r\n{}", + config.response_status, + config.response_body.len(), + config.response_body, + ); + + // Serve connections until shutdown, rather than exactly one. + // + // One was enough while a failed handshake was the end of the + // story. It is not any more: the profile ladder retries a refused + // handshake with a wider offer, so every legacy-cipher test needs + // at least two connections. With a single-shot server the second + // attempt gets ECONNREFUSED and the test fails for a reason that + // has nothing to do with what it is testing. + let mut shutdown_rx = shutdown_rx; + loop { + tokio::select! { + result = listener.accept() => { + let Ok((tcp_stream, _)) = result else { continue }; + + // Read the ClientHello off the wire and parse it + // before TLS touches it. Peeking rather than consuming + // keeps the stream intact, so no replay buffer is + // needed and the handshake proceeds as it would have. + // + // Overwritten per connection, so after a ladder walk + // the slot holds the LAST hello, which is the one that + // succeeded. + let mut peek = vec![0u8; 8192]; + if let Ok(n) = tcp_stream.peek(&mut peek).await + && let Some(parsed) = parse_client_hello(&peek[..n]) + && let Ok(mut slot) = hello_slot.lock() + { + *slot = Some(parsed); + } + + let ssl = openssl::ssl::Ssl::new(acceptor.context()).unwrap(); + let mut tls_stream = match tokio_openssl::SslStream::new(ssl, tcp_stream) { + Ok(s) => s, + Err(_) => continue, + }; + + // A failed handshake is an ordinary event here, not + // the end: it is what the cipher and version mismatch + // tests are producing on purpose, and what the ladder + // then retries. + if std::pin::Pin::new(&mut tls_stream).accept().await.is_err() { + continue; + } + + let mut buf = [0u8; 4096]; + let _ = tls_stream.read(&mut buf).await; + let _ = tls_stream.write_all(response.as_bytes()).await; + let _ = tls_stream.shutdown().await; + } + _ = &mut shutdown_rx => break, + } + } + }); + + TlsTestServer { + addr, + hello, + shutdown_tx, + handle, + } + } + + pub fn url(&self) -> String { + format!("https://127.0.0.1:{}", self.addr.port()) + } + + pub async fn shutdown(self) { + let _ = self.shutdown_tx.send(()); + let _ = self.handle.await; + } +} diff --git a/tests/tls_integration.rs b/tests/tls_integration.rs index e4b39a7..a4ffa9c 100644 --- a/tests/tls_integration.rs +++ b/tests/tls_integration.rs @@ -2,13 +2,13 @@ // These spin up a real TLS server using our custom OpenSSL build, // then connect with blasthttp's HyperClient to verify end-to-end behavior. -mod tls_server; +mod support; use blasthttp::client::HttpClient; use blasthttp::client::hyper::HyperClient; use blasthttp::config::RequestConfig; use openssl::ssl::SslVersion; -use tls_server::{TlsServerConfig, TlsTestServer}; +use support::tls_server::{TlsServerConfig, TlsTestServer}; fn make_config(url: &str) -> RequestConfig { let mut config = RequestConfig::new(url.to_string()); diff --git a/tests/tls_server.rs b/tests/tls_server.rs deleted file mode 100644 index ecf5f5c..0000000 --- a/tests/tls_server.rs +++ /dev/null @@ -1,232 +0,0 @@ -// Shared test helper: a minimal HTTPS server using OpenSSL directly. -// Accepts one connection, does TLS handshake, sends a hardcoded HTTP response, -// then shuts down. Configurable cipher list and TLS version range. - -use openssl::asn1::Asn1Time; -use openssl::bn::BigNum; -use openssl::hash::MessageDigest; -use openssl::pkey::PKey; -use openssl::rsa::Rsa; -use openssl::ssl::{AlpnError, SslAcceptor, SslAcceptorBuilder, SslMethod, SslOptions, SslVersion}; -use openssl::x509::X509; -use openssl::x509::extension::SubjectAlternativeName; -use std::net::SocketAddr; -use tokio::io::{AsyncReadExt, AsyncWriteExt}; -use tokio::net::TcpListener; - -pub struct TlsTestServer { - pub addr: SocketAddr, - shutdown_tx: tokio::sync::oneshot::Sender<()>, - handle: tokio::task::JoinHandle<()>, -} - -pub struct TlsServerConfig { - pub cipher_list: Option, - pub min_tls_version: Option, - pub max_tls_version: Option, - pub response_body: String, - pub response_status: u16, - /// DNS SANs for the self-signed cert. Default: ["localhost"]. - pub san_dns: Vec, - /// IP SANs for the self-signed cert. Default: ["127.0.0.1"]. - pub san_ip: Vec, -} - -impl Default for TlsServerConfig { - fn default() -> Self { - TlsServerConfig { - cipher_list: None, - min_tls_version: None, - max_tls_version: None, - response_body: "OK".to_string(), - response_status: 200, - san_dns: vec!["localhost".to_string()], - san_ip: vec!["127.0.0.1".to_string()], - } - } -} - -// Generate a self-signed cert + key pair in memory (no files needed) -fn generate_self_signed(config: &TlsServerConfig) -> (PKey, X509) { - let rsa = Rsa::generate(2048).unwrap(); - let pkey = PKey::from_rsa(rsa).unwrap(); - - let mut builder = X509::builder().unwrap(); - builder.set_version(2).unwrap(); - - let serial = BigNum::from_u32(1).unwrap(); - builder - .set_serial_number(&serial.to_asn1_integer().unwrap()) - .unwrap(); - - let mut name = openssl::x509::X509NameBuilder::new().unwrap(); - name.append_entry_by_text("CN", "localhost").unwrap(); - let name = name.build(); - builder.set_issuer_name(&name).unwrap(); - builder.set_subject_name(&name).unwrap(); - - let not_before = Asn1Time::days_from_now(0).unwrap(); - let not_after = Asn1Time::days_from_now(1).unwrap(); - builder.set_not_before(¬_before).unwrap(); - builder.set_not_after(¬_after).unwrap(); - - builder.set_pubkey(&pkey).unwrap(); - - let mut san = SubjectAlternativeName::new(); - for dns in &config.san_dns { - san.dns(dns); - } - for ip in &config.san_ip { - san.ip(ip); - } - let san = san.build(&builder.x509v3_context(None, None)).unwrap(); - builder.append_extension(san).unwrap(); - - builder.sign(&pkey, MessageDigest::sha256()).unwrap(); - let cert = builder.build(); - - (pkey, cert) -} - -fn build_acceptor(config: &TlsServerConfig) -> SslAcceptor { - acceptor_builder(config).build() -} - -/// An acceptor that only speaks HTTP/2, for tests that serve with hyper's -/// h2 server instead of the hardcoded HTTP/1.1 response above. -#[allow(dead_code)] -pub fn build_h2_acceptor(config: &TlsServerConfig) -> SslAcceptor { - let mut builder = acceptor_builder(config); - builder.set_alpn_select_callback(|_, client| { - openssl::ssl::select_next_proto(b"\x02h2", client).ok_or(AlpnError::NOACK) - }); - builder.build() -} - -fn acceptor_builder(config: &TlsServerConfig) -> SslAcceptorBuilder { - // Load the legacy provider so the server can use weak ciphers too - load_legacy_provider(); - - // Build from scratch — no mozilla presets that restrict ciphers/versions. - // mozilla_intermediate sets SSL options that can disable TLS 1.3. - let mut builder = SslAcceptor::mozilla_intermediate(SslMethod::tls()).unwrap(); - builder.set_security_level(0); - // Clear ALL version restrictions and options - builder.set_min_proto_version(None).unwrap(); - builder.set_max_proto_version(None).unwrap(); - // Remove any SSL options that might disable specific TLS versions - builder.clear_options( - SslOptions::NO_TLSV1_3 - | SslOptions::NO_TLSV1_2 - | SslOptions::NO_TLSV1_1 - | SslOptions::NO_TLSV1, - ); - // Accept all ciphers by default - builder - .set_cipher_list("ALL:COMPLEMENTOFALL:eNULL") - .unwrap(); - // TLS 1.3 ciphersuites are configured separately in OpenSSL 3.x - builder - .set_ciphersuites( - "TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:TLS_AES_128_GCM_SHA256", - ) - .unwrap(); - - let (pkey, cert) = generate_self_signed(config); - builder.set_private_key(&pkey).unwrap(); - builder.set_certificate(&cert).unwrap(); - - if let Some(ref ciphers) = config.cipher_list { - builder.set_cipher_list(ciphers).unwrap(); - } - - if let Some(min_ver) = config.min_tls_version { - builder.set_min_proto_version(Some(min_ver)).unwrap(); - } - - if let Some(max_ver) = config.max_tls_version { - builder.set_max_proto_version(Some(max_ver)).unwrap(); - } - - builder -} - -// Load legacy provider for the test server (same as our client does) -static INIT_LEGACY: std::sync::Once = std::sync::Once::new(); - -fn load_legacy_provider() { - INIT_LEGACY.call_once(|| { - let _default = openssl::provider::Provider::try_load(None, "default", true) - .expect("failed to load default provider"); - let _legacy = openssl::provider::Provider::try_load(None, "legacy", true) - .expect("failed to load legacy provider"); - std::mem::forget(_default); - std::mem::forget(_legacy); - }); -} - -impl TlsTestServer { - pub async fn start(config: TlsServerConfig) -> Self { - let acceptor = build_acceptor(&config); - - // Bind to random port on localhost - let listener = TcpListener::bind("127.0.0.1:0").await.unwrap(); - let addr = listener.local_addr().unwrap(); - - let (shutdown_tx, shutdown_rx) = tokio::sync::oneshot::channel::<()>(); - - let handle = tokio::spawn(async move { - let response = format!( - "HTTP/1.1 {} OK\r\nContent-Length: {}\r\nConnection: close\r\n\r\n{}", - config.response_status, - config.response_body.len(), - config.response_body, - ); - - tokio::select! { - result = listener.accept() => { - if let Ok((tcp_stream, _)) = result { - // Async TLS handshake via tokio-openssl - let ssl = openssl::ssl::Ssl::new(acceptor.context()).unwrap(); - let mut tls_stream = match tokio_openssl::SslStream::new(ssl, tcp_stream) { - Ok(s) => s, - Err(_) => return, - }; - - // Perform the TLS accept handshake - if std::pin::Pin::new(&mut tls_stream).accept().await.is_err() { - // Handshake failed — expected for cipher/version mismatch tests - return; - } - - // Read the HTTP request (we don't care about contents) - let mut buf = [0u8; 4096]; - let _ = tls_stream.read(&mut buf).await; - - // Send response - let _ = tls_stream.write_all(response.as_bytes()).await; - let _ = tls_stream.shutdown().await; - } - } - _ = shutdown_rx => { - // Shutdown requested - } - } - }); - - TlsTestServer { - addr, - shutdown_tx, - handle, - } - } - - pub fn url(&self) -> String { - format!("https://127.0.0.1:{}", self.addr.port()) - } - - pub async fn shutdown(self) { - let _ = self.shutdown_tx.send(()); - let _ = self.handle.await; - } -}