BlastHTTP 1.0 - Connection Profile

CHANGELOG.md

@@ -1,5 +1,130 @@
 # Changelog
 
+## 1.0.0
+
+Breaking: a request that names no profile now sends `modern`, which is 11
+suites and TLS 1.2 or 1.3. A server too old for that is still reached, by
+the ladder falling back to `compatibility` on the second handshake.
+
+- **Connection profiles.** Three of them. `compatibility` offers every suite
+  the build provides and every protocol version back to SSLv3. `modern` offers
+  11 current suites and claims nothing. `chrome131` imitates Chrome's TLS,
+  HTTP/2 settings and headers together. Select one with `profile=`; an unknown
+  name is now an error rather than silently falling back
+- **Automatic shifting between them.** A handshake failure that suggests the
+  peer could not negotiate widens the offer; a refusal from a recognised
+  protection product changes what the client claims to be. Per redirect hop,
+  since a redirect can land on a differently protected host, and without
+  spending the caller's retry budget, which is a different question. A profile
+  that gets through is remembered per host so a scan does not re-walk the
+  ladder on every request. Naming a `profile`, `cipher_string`,
+  `min_tls_version` or `max_tls_version` pins the configuration and the ladder
+  does not move
+- The ladder only moves on positive evidence that a product intervened, not on
+  any refusal. Most 403s are ordinary authorization failures, and retrying each
+  one with different TLS settings would double the request count of any scan
+  that touches one
+- A host that refuses every rung is remembered too, for ten minutes: during
+  that time a refusal from it is taken as the answer after one request, rather
+  than walking the ladder again. Only success used to be remembered, so a host
+  that refused everything cost two requests for every one a scan sent it, on
+  exactly the hosts most likely to rate-limit or ban. Getting through clears
+  it, and it expires because a refusal is a judgement that changes
+- **`Response.protection`** reports what a bot-management product did:
+  `(outcome, vendor)` where outcome is ok, present, challenge or blocked. A
+  404 or a 500 is ok: the server answered and no product intervened
+- **`Response.conclusion`** reduces that to one verdict: reached, blocked,
+  needs_browser, needs_legacy_tls or unreachable. `needs_browser` means a
+  JavaScript challenge, which no HTTP client passes, so the URL should go to a
+  real browser rather than be retried. A `blocked` with vendor `-` is an
+  ordinary refusal from the origin, not a product
+- **`Response.attempts`** lists what the ladder tried, in order
+- Imperva's `_Incapsula_Resource` loader script no longer counts as a
+  challenge. Imperva appends it to pages it let through, so ordinary pages and
+  the origin's own error pages were judged `needs_browser`. Imperva's incident
+  page still counts
+- **The anti-bot rules are data.** What `protection` recognises now lives in
+  `antibot_rules.json` instead of code, and `blasthttp.set_antibot_rules(path)`
+  (`antibot::set_rules` in Rust) replaces them at runtime, so a rule can change
+  without a rebuild. `Vendor` is a name rather than a fixed list, so a rules
+  file can add a product
+- **TLS failures now say why.** The reason used to be destroyed in transit:
+  hyper-util's error displays as the literal "client error (Connect)" for every
+  connect-time failure, so a cipher mismatch, a rejected certificate, a DNS
+  failure and a refused connection were indistinguishable. Failures now raise
+  `TransportError` carrying `kind`, `tls_failure`, `retryable`, `conclusion`
+  and `attempts`. `TransportError` subclasses `RuntimeError`, so existing
+  handlers keep working
+- Fixes a consequence of that: TLS failures were being retried, because they
+  were misclassified as connection errors, contradicting the rule that says
+  they should not be
+- Connect failures before TLS say why too. A refused connection, a DNS failure
+  or a proxy that refuses the tunnel now includes the cause, e.g. "tcp connect
+  error: Connection refused (os error 111)", not only "client error
+  (Connect)", and a connect timeout is reported as a timeout instead of a
+  retryable connection error
+- Batch failures carry the same detail. `BatchResult.transport_error` is the
+  `TransportError` that `request()` would have raised, with `kind`,
+  `retryable`, `conclusion` and the rest. `error` is still the message string
+- `blasthttp.mock` raises `TransportError` where the real client would: for a
+  URL with no mock response (`kind="connection"`) and for too many redirects
+  (`kind="too_many_redirects"`). A mocked batch request for an unmocked URL
+  gets a `transport_error` too
+- Asking for a pre-TLS1.2 version without naming a profile now selects
+  `compatibility`, rather than failing with "no ciphers available" because the
+  default offers nothing that exists in SSLv3
+- `BLASTHTTP_BISECT=tls,headers,http2` disables profile layers independently,
+  for finding out which one a detector is reacting to
+- Batches no longer have every request in an opening burst work out the same
+  host's profile separately. A request whose host is already being discovered
+  stands aside and lets a request for a *different* host take its slot, then
+  goes once there is an answer. Six requests to a host needing a shift went
+  from twelve handshakes to seven, with nothing idling: standing aside only
+  happens when there is undispatched work elsewhere, so when a batch is all
+  one host every request goes immediately and discovers it again, which is the
+  cheaper mistake. A slow request still never holds up faster ones behind it.
+  Measured over a grid of batch shapes, the duplicate handshakes go to zero in
+  every one: a scan of 2000 URLs across 20 hosts drops 62 handshakes, 500 URLs
+  across 10 hosts drops 139, and wall time is unchanged. A batch no deeper than
+  its own concurrency limit pays one extra round trip for it, since there the
+  discovery cannot be overlapped with anything
+- `raw_connect`, and requests using `resolve_ip` or `request_target`, widen
+  their cipher offer and retry on a handshake the peer could not negotiate,
+  the same as an ordinary request. They stop at that rung: the other one
+  reacts to a refusal by changing what the client claims to be, and these are the callers who asked for exact
+  control over one request
+- `download()` takes `profile`, `cipher_string`, `min_tls_version`,
+  `max_tls_version` and `redirect_cookies`. It hardcoded no profile with no
+  way to pass one, so a file behind a host that only answers a browser was
+  unreachable through `download` while the same URL through `request` was fine
+- `blasthttp.mock` forwards every kwarg to the real client on a passthrough
+  request. It named six and dropped the rest, so a URL excluded from mocking
+  was dialled with different TLS, a different profile and no timeout from what
+  the caller asked for
+- A request that names a profile no longer writes it into the per-host memory.
+  Reads already skipped the memory when the caller pinned something; writes
+  did not, so one deliberate `profile="chrome"` request turned every later
+  request to that host into a browser claim, and made the outcome depend on
+  the order two unrelated requests ran in
+- Legacy ciphers are reachable, which is what the custom OpenSSL build has always been for. `compatibility` offers every suite the build provides, so a server speaking only RC4, RC4-MD5, 3DES, SEED, Camellia or anonymous DH is reachable without passing `cipher_string` yourself
+- Null-encryption suites are the deliberate exception. They stay reachable through an explicit `cipher_string` but are never offered by any profile, since negotiating one by accident returns a connection that looks like TLS and encrypts nothing
+- The profile's cipher list applies on both TLS paths, the pooled client and the `resolve_ip` / `request_target` / `raw_connect` path, which build their SSL contexts separately
+- SSLv3 works. `min_tls_version` / `max_tls_version` / `--min-tls` / `--max-tls` accept `3.0` (also `ssl3`, `sslv3`), and an SSLv3-only server is reachable without asking for it
+- The OpenSSL build cache is keyed on the feature flags and version as well as the target, so changing what the build asks for triggers a rebuild. The first run after upgrading rebuilds once
+- Export ciphers are not supported. OpenSSL removed them outright in 1.1.0, so no build flag brings them back
+- A TLS failure is reported only for the request that hit it. A later error to the same host, such as a refused connection, is classified on its own and retried as usual
+- A `cipher_string` with no `profile` named selects `compatibility`, the same as asking for a pre-1.2 TLS version, so no TLS 1.2 floor applies and `cipher_string="RC4-MD5"` reaches a TLS 1.0 server. Naming a profile as well keeps that profile's floor
+- `Content-Encoding: zstd` is decoded, including bodies of several frames. `chrome131` advertises it, as Chrome does, and Cloudflare answers with it. A frame asking for a window over 8 MiB (RFC 9659's limit) is refused and the body returned undecoded, so a response can't make the client allocate an arbitrary window
+- With `verify_certs` on, anonymous (aNULL) suites are never offered, including from an explicit `cipher_string`. They send no certificate, so verification would report success while checking nothing. With verification off, nothing is excluded
+- A refusal that sends the ladder to another profile is kept. If that next attempt gets no response at all (a failed handshake, a reset, a refused connection), the caller gets the refusal, with the failed attempt listed in `attempts`, rather than the error. The same holds when the timeout runs out during that next attempt: `timeout` still bounds the whole request, redirects and ladder included, but a refusal already received is returned instead of a timeout error
+- Akamai's block rule only fires on a refusal status. edgesuite.net also serves customer assets, so a 200 page loading one was classified as blocked and re-sent
+- A product's cookie or injected script on a refusal no longer names it as what refused. Bot-management products set their cookies and Cloudflare injects its JavaScript detections script on every response, the origin's own 401s and 403s included, so an ordinary auth failure behind a CDN walked the ladder and could report `needs_browser`. Cloudflare's own error page (a firewall-rule block such as 1020) is recognised as a Cloudflare block instead
+- A handshake failure concludes `needs_legacy_tls` only when `compatibility` wasn't tried, such as when a pin stopped the ladder. Once the widest offer has failed too, as with a server wanting a client certificate, the conclusion is `unreachable`
+- A connection reset during the TLS handshake is a `connection` error and retried, as before 1.0, rather than a non-retryable `tls` one. `tls_failure` still reads `reset`
+- A batch's rate limit is taken when a request is sent rather than when it's spawned, so requests that stood aside for a host's first request go at the configured rate when it finishes instead of all at once
+- A batch request with its TLS pinned (`profile`, `cipher_string` or a TLS version) skips host discovery. It never reads or writes the per-host memory, so it no longer stands aside for a host's first request, and never makes unpinned requests wait on it
+- When the ladder switches to `chrome131` after a denial, Chrome's User-Agent and client hints replace the caller's, so the request doesn't claim Chrome 131 in its handshake and hints and another browser in its User-Agent. First contact keeps the caller's User-Agent, every other header and cookie the caller added stays, and naming the profile yourself keeps your headers as before
+
 ## 0.10.1
 
 - Python 3.15 support: PyO3 0.30, and 3.15 wheels are published

Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "blasthttp"
-version = "0.10.2"
+version = "1.0.0"
 edition = "2024"
 description = "Offensive-first HTTP library with Python bindings"
 license = "AGPL-3.0"
@@ -45,6 +45,7 @@ http = "1"
 # Decompression
 flate2 = "1"
 brotli = "9"
+ruzstd = "0.9.1"
 
 # Public Suffix List — decides where a cookie's `Domain` stops being allowed
 # to widen. Without it `Domain=com` or `Domain=github.io` is accepted and the
@@ -70,5 +71,15 @@ pyo3-async-runtimes = { version = "0.30", features = ["attributes", "tokio-runti
 [dev-dependencies]
 # TLS test server — uses our custom OpenSSL to serve with weak ciphers
 tokio-test = "0.4"
+
+# Raw SSL_client_hello_get* bindings, for reading back the ClientHello we sent.
+# The safe `openssl` wrapper exposes ciphers and versions but not the extension
+# list, and the extension list is most of a fingerprint. Both of these are
+# already in the tree as transitive dependencies of `openssl`, so they pull
+# nothing new in; `foreign-types-shared` is only here for the `as_ptr()` that
+# hands those bindings an `SSL *`.
+openssl-sys = "0.9"
+foreign-types-shared = "0.1"
+
 # h2 test server, for checking what the client puts on the wire over HTTP/2
 hyper = { version = "1", features = ["server"] }