Repository navigation
3.1 documentation update - #3613
Merged
Merged
Conversation
liquidsec
changed the title
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## dev #3613 +/- ##
======================================
+ Coverage 91% 91% +1%
======================================
Files 465 465
Lines 51797 51927 +130
======================================
+ Hits 46905 47039 +134
+ Misses 4892 4888 -4 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Contributor
📊 Performance Benchmark Report
📈 Detailed Results (All Benchmarks)
🎯 Performance Summary+ 2 improvements 🚀
! 1 regression ⚠️
28 unchanged ✅🔍 Significant Changes (>10%)
🐍 Python Version 3.11.18 |
en0f
requested changes
Nov 6, 2026
en0f
left a comment
en0f
left a comment
There was a problem hiding this comment.
Reviewed at 88fef73.
🔴 The User-Agent note and --user-agent help overstate the change
Only requests made through api_request() send BBOT/<version> (bbot/modules/base.py:1383). urlscan, dockerhub, docker_pull, azure_tenant, the postman modules and wayback's CDX lookup still call self.helpers.request(), which sends web.user_agent (web.py:155). Someone who allowlists on BBOT/<version> after reading the migration guide will miss those requests. Describe the mechanism rather than listing modules, for example "Modules that query third-party APIs through BBOT's API request helper now send User-Agent: BBOT/<version> ... Not every module uses the helper yet, so some third-party requests still carry web.user_agent; don't rely on BBOT/<version> for allowlisting." Make the help text "most API calls send BBOT/", then rerun bbot/scripts/docs.py so advanced.md picks it up.
🔴 WAF Handling says GET parameters are always skipped
With try_get_as_post on, lightfuzz retests GET parameters as POST bodies even in bypass mode (lightfuzz.py:392-398). lightfuzz-xss turns that on for exactly this purpose, and so does lightfuzz-heavy. Suggested wording: "Cookie and header parameters are skipped, since they have no body to pad. GET parameters are skipped too, unless try_get_as_post is on (as in lightfuzz-heavy and lightfuzz-xss), in which case they are retested as POST bodies and handled like POST parameters above."
🔴 lightfuzz-xss table says POST fuzzing is Disabled
The PR adds a row to this table and the body says the tables now match their presets, but lightfuzz-xss doesn't set disable_post (default False) and does set try_get_as_post: True. Its own description says "POST fuzzing enabled". Make the row "Enabled, and try_get_as_post is on (GET params retested as POST)".
🔴 No validation output in the PR body
git.md § Pull requests asks for the commands run and their real output. Dropping dns.threads: 25 from subdomain-enum changes what every subdomain-enum scan runs with. Please paste the dns section of bbot -p subdomain-enum --current-preset-full from before and after, showing threads go from 25 to 50. Plain --current-preset won't show it, because the preset no longer sets threads at all.
🟢 The numbers were checked against the code
The DNS tip matches defaults.yml (50 per resolver, 256 total) and tells readers to raise dns.max_concurrency when they add resolvers, instead of promising a linear speedup. The ~5K nameserver count matches the live list (4,877). Removing threads: 25 catches a real regression from #3606, where the preset had started halving the new default.
🟢 Tag examples now match cloudcheck
cloudcheck emits the type and the provider as separate tags (cdn, github), and output.md, events.md and custom_yara_rules.md now show exactly that. configuration.md no longer names options that don't exist.
Reproduce this review
# save the script below as verify-3613.sh and the patch as control-3613.patch, then:
bash verify-3613.sh blacklanternsecurity/bbot 88fef73a49b0530e1212036f50a0b99dafb163f2
# expected: SUMMARY 9 PASS, 5 FAIL, 0 SKIP, 0 BROKEN (V01 V02 V03 V13 V14 fail)
# control: bash verify-3613.sh blacklanternsecurity/bbot 88fef73a49b0530e1212036f50a0b99dafb163f2 "$PWD/control-3613.patch"
# expected: SUMMARY 14 PASS, 0 FAIL, 0 SKIP, 0 BROKEN (the suggested wording makes V01 V02 V03 V13 V14 pass)#!/usr/bin/env bash
# verify-3613.sh: reproducible checks for blacklanternsecurity/bbot#3613
# Usage: bash verify-3613.sh <owner/repo> <sha> [<control-patch>]
# Changes nothing outside a mktemp directory and nothing on GitHub (PR-VF-02).
#
# CONTROL: patch control-3613.patch
# CONTROL-EXPECT: V01=PASS V02=PASS V03=PASS V13=PASS V14=PASS
set -uo pipefail
REPO="${1:?owner/repo}"; SHA="${2:?sha}"; PATCH="${3:-}"
PASS=0; FAIL=0; SKIP=0; BROKEN=0
pass() { echo "PASS $1 $2"; PASS=$((PASS+1)); }
fail() { echo "FAIL $1 $2 -- $3"; FAIL=$((FAIL+1)); }
skip() { echo "SKIP $1 $2 -- $3"; SKIP=$((SKIP+1)); }
broken() { echo "BROKEN $1 $2 -- $3"; BROKEN=$((BROKEN+1)); }
summary() { echo "SUMMARY $PASS PASS, $FAIL FAIL, $SKIP SKIP, $BROKEN BROKEN"; [ "$FAIL" -eq 0 ] && [ "$BROKEN" -eq 0 ]; }
echo "script_sha256 $(shasum -a 256 "$0" | cut -d' ' -f1)"
echo "git $(git --version | cut -d' ' -f3)"
echo "gh $(gh --version | head -1 | cut -d' ' -f3)"
WORK="$(mktemp -d)"; trap 'rm -rf "$WORK"' EXIT
export GIT_CONFIG_GLOBAL=/dev/null GIT_CONFIG_NOSYSTEM=1 GIT_TERMINAL_PROMPT=0
GIT=(git -c credential.helper= -c 'credential.helper=!gh auth git-credential')
"${GIT[@]}" clone --quiet --filter=blob:none --no-checkout "https://github.com/$REPO.git" "$WORK/src" || { broken V00 "clone" "clone failed"; summary; exit 1; }
cd "$WORK/src"
"${GIT[@]}" cat-file -e "$SHA^{commit}" 2>/dev/null || "${GIT[@]}" fetch --quiet origin "+refs/pull/*/head:refs/remotes/pr/*" || true
"${GIT[@]}" checkout --quiet --detach "$SHA" || { broken V00 "checkout $SHA" "checkout failed"; summary; exit 1; }
[ "$(git rev-parse HEAD)" = "$SHA" ] || { broken V00 "HEAD is $SHA" "HEAD mismatch"; summary; exit 1; }
if [ -n "$PATCH" ]; then git apply "$PATCH" || { broken V00 "apply control patch" "git apply failed"; summary; exit 1; }; fi
# ---- checks: each line states the desired property --------------------------
# Modules that call a third-party API with self.helpers.request() rather than api_request(),
# so they send the scan's web.user_agent, not BBOT/<version>.
OFFENDERS=""
for f in bbot/modules/urlscan.py bbot/modules/dockerhub.py bbot/modules/templates/postman.py bbot/modules/wayback.py; do
[ -f "$f" ] || continue
if grep -qE '(base_url|api_url|site_url)\s*=\s*f?"https?://' "$f" && grep -qE 'self\.helpers\.request\(' "$f"; then
OFFENDERS="$OFFENDERS $(basename "$f" .py)"
fi
done
if grep -q 'kwargs\["headers"\]\["User-Agent"\] = f"BBOT/{__version__}"' bbot/modules/base.py; then
pass V11 "api_request() sends User-Agent BBOT/<version>"
else
fail V11 "api_request() sends User-Agent BBOT/<version>" "base.py pattern not found"
fi
MIG=docs/migration/3.1_breaking_changes.md
UA_SECTION=$(awk '/^### API requests identify as BBOT/{s=1;next} /^(---|### )/{s=0} s' "$MIG")
QUAL='not every module|some third-party|api_request|API request helper|not all'
ua_unqualified() { [ -n "$UA_SECTION" ] && echo "$UA_SECTION" | grep -q 'BBOT/<version>' && ! echo "$UA_SECTION" | grep -qiE "$QUAL"; }
if ua_unqualified && [ -n "$OFFENDERS" ]; then
fail V01 "3.1 migration note's User-Agent claim matches the modules" "claims all third-party API requests send BBOT/<version>; still on web.user_agent:$OFFENDERS"
else
pass V01 "3.1 migration note's User-Agent claim matches the modules"
fi
help_unqualified() { grep -E 'BBOT/<version>' "$1" | grep -i 'user-agent' | grep -viqE '\((most|some)[^)]*BBOT/<version>'; }
if [ -n "$OFFENDERS" ] && { help_unqualified bbot/scanner/preset/args.py || help_unqualified docs/scanning/advanced.md; }; then
fail V02 "--user-agent help text matches the modules" "args.py or advanced.md says API calls send BBOT/<version> without qualification; still on web.user_agent:$OFFENDERS"
else
pass V02 "--user-agent help text matches the modules"
fi
LF=bbot/modules/lightfuzz/lightfuzz.py
WAF_ROW=$(grep -F '`try_bypasses` (default)' docs/modules/lightfuzz.md)
if echo "$WAF_ROW" | grep -q 'GET' && ! echo "$WAF_ROW" | grep -q 'try_get_as_post' \
&& grep -q 'if self.try_get_as_post and original_type == "GETPARAM":' "$LF" \
&& grep -q 'try_get_as_post: True' bbot/presets/web/lightfuzz-xss.yml; then
fail V03 "lightfuzz WAF Handling describes GET params under try_get_as_post" "doc says GET params are always skipped; lightfuzz.py retests them as POST in bypass mode and lightfuzz-xss turns that on"
else
pass V03 "lightfuzz WAF Handling describes GET params under try_get_as_post"
fi
if grep -q 'requires-python = ">=3.11' pyproject.toml && grep -q 'badge/python-3.11+' README.md && grep -q 'BBOT requires Python 3.11+' docs/troubleshooting.md; then
pass V04 "README badge and troubleshooting say 3.11+, matching requires-python"
else
fail V04 "README badge and troubleshooting say 3.11+, matching requires-python" "mismatch"
fi
if grep -qE '^ threads: 50$' bbot/defaults.yml && grep -qE '^ max_concurrency: 256$' bbot/defaults.yml \
&& grep -q 'default 50 queries per resolver' docs/scanning/tips_and_tricks.md && grep -q '256 total' docs/scanning/tips_and_tricks.md; then
pass V05 "DNS speed tip matches dns.threads=50 and dns.max_concurrency=256"
else
fail V05 "DNS speed tip matches dns.threads=50 and dns.max_concurrency=256" "mismatch"
fi
if ! grep -qE '^\s+threads:' bbot/presets/subdomain-enum.yml && ! grep -q 'threads: 25' README.md && ! grep -q 'threads: 25' docs/scanning/presets_list.md; then
pass V06 "subdomain-enum no longer lowers dns.threads, and README/presets_list match"
else
fail V06 "subdomain-enum no longer lowers dns.threads, and README/presets_list match" "threads still set somewhere"
fi
CC=bbot/modules/internal/cloudcheck.py
if grep -q 'event.add_tag(provider_type)' "$CC" && grep -q 'event.add_tag(provider_name)' "$CC" \
&& ! grep -rqE 'cdn-github|cloud-azure|cdn-akamai|cdn-ip' docs/scanning/output.md docs/scanning/events.md docs/modules/custom_yara_rules.md; then
pass V07 "tag examples use cloudcheck's separate type and provider tags"
else
fail V07 "tag examples use cloudcheck's separate type and provider tags" "old cdn-x tags remain or code differs"
fi
ok=1
grep -q 'avoid_wafs: always' bbot/presets/web/lightfuzz-light.yml || ok=0
grep -q 'avoid_wafs: try_bypasses' bbot/presets/web/lightfuzz.yml || ok=0
grep -q 'avoid_wafs: never' bbot/presets/web/lightfuzz-max.yml || ok=0
grep -q 'avoid_wafs: try_bypasses' bbot/presets/web/lightfuzz-xss.yml || ok=0
grep -q 'avoid_wafs: Literal\["always", "never", "try_bypasses"\]' "$LF" || ok=0
if [ "$ok" = 1 ]; then pass V08 "lightfuzz preset tables match each preset's avoid_wafs"; else fail V08 "lightfuzz preset tables match each preset's avoid_wafs" "mismatch"; fi
if grep -q '"--custom-nameservers"' bbot/scanner/preset/args.py && [ -f bbot/presets/nuclei/nuclei-recommended.yml ] && [ -f bbot/presets/nuclei/nuclei-new.yml ]; then
pass V09 "--custom-nameservers, nuclei-recommended and nuclei-new exist as documented"
else
fail V09 "--custom-nameservers, nuclei-recommended and nuclei-new exist as documented" "missing"
fi
if N=$(gh api --method GET "repos/blacklanternsecurity/public-dns-servers/contents/nameservers.txt?ref=master" -H "Accept: application/vnd.github.raw+json" 2>/dev/null | grep -cvE '^\s*(#|$)'); then
if [ "$N" -ge 4000 ] && [ "$N" -le 6000 ]; then pass V10 "brute_nameservers list is ~5K servers ($N)"; else fail V10 "brute_nameservers list is ~5K servers" "count $N"; fi
else
skip V10 "brute_nameservers list is ~5K servers" "could not fetch nameservers.txt"
fi
# C13 (A5): generated blocks match their sources
if python3 - <<'PY'
p=open('bbot/presets/subdomain-enum.yml').read().strip(); r=open('README.md').read()
l=open('docs/scanning/presets_list.md').read(); c=open('docs/scanning/configuration.md').read(); d=open('bbot/defaults.yml').read().strip()
import re,sys; a=open('bbot/scanner/preset/args.py').read(); adv=open('docs/scanning/advanced.md').read()
h=re.search(r'"--user-agent",\s*"-ua",\s*help="([^"]+)"', a)
sys.exit(0 if (p in r and "\n".join(" "+x for x in p.split("\n")) in l and d in c and h and h.group(1) in adv) else 1)
PY
then pass V12 "generated README, presets_list, configuration and advanced.md blocks match their sources"
else fail V12 "generated README, presets_list, configuration and advanced.md blocks match their sources" "a generated block is stale"; fi
# C16/C17: lightfuzz-xss table's POST row matches the preset
XSS_TABLE=$(awk '/^### .-p lightfuzz-xss./{s=1} /^### Spider/{s=0} s' docs/modules/lightfuzz.md)
if echo "$XSS_TABLE" | grep -q 'POST fuzzing\*\* | Disabled' && ! grep -qE 'disable_post: *True' bbot/presets/web/lightfuzz-xss.yml; then
fail V13 "lightfuzz-xss table's POST fuzzing row matches the preset" "table says Disabled; preset leaves disable_post False and sets try_get_as_post"
else
pass V13 "lightfuzz-xss table's POST fuzzing row matches the preset"
fi
# C3/C14 (A6): more modules call third-party APIs outside api_request()
MORE=""
for m in azure_tenant docker_pull; do
grep -qE 'self\.helpers\.request\(' "bbot/modules/$m.py" && grep -qE 'https://(azmap\.dev|registry-1\.docker\.io|login\.microsoftonline\.com)' "bbot/modules/$m.py" && MORE="$MORE $m"
done
if ua_unqualified && [ -n "$MORE" ]; then
fail V14 "3.1 migration note holds for azure_tenant and docker_pull" "unqualified claim; still on web.user_agent:$MORE"
else
pass V14 "3.1 migration note holds for azure_tenant and docker_pull"
fi
# ------------------------------------------------------------------------------
summarydiff --git a/bbot/scanner/preset/args.py b/bbot/scanner/preset/args.py
index febde1bc2..f22c565c0 100644
--- a/bbot/scanner/preset/args.py
+++ b/bbot/scanner/preset/args.py
@@ -466,7 +466,7 @@ class BBOTArgs:
misc.add_argument(
"--user-agent",
"-ua",
- help="Set the user-agent for HTTP requests (third-party API calls send BBOT/<version>)",
+ help="Set the user-agent for HTTP requests (most API calls send BBOT/<version>)",
)
misc.add_argument(
"--user-agent-suffix", "-uas", help="Suffix to append to the user-agent", metavar="SUFFIX", default=None
diff --git a/docs/migration/3.1_breaking_changes.md b/docs/migration/3.1_breaking_changes.md
index 7cccb85db..849ffbba8 100644
--- a/docs/migration/3.1_breaking_changes.md
+++ b/docs/migration/3.1_breaking_changes.md
@@ -97,7 +97,7 @@ With `urls: true`, wayback now probes each host once and drops URLs on hosts tha
### API requests identify as BBOT
-Requests that modules make to third-party APIs now send `User-Agent: BBOT/<version>` instead of the scan's user agent, so `web.user_agent` and `web.user_agent_suffix` no longer apply to them. Requests to your targets still use `web.user_agent`.
+Modules that query third-party APIs through BBOT's API request helper now send `User-Agent: BBOT/<version>` instead of the scan's user agent, so `web.user_agent` and `web.user_agent_suffix` no longer apply to those requests. Not every module uses the helper yet, so some third-party requests still carry `web.user_agent`; don't rely on `BBOT/<version>` for allowlisting. Requests to your targets still use `web.user_agent`.
---
diff --git a/docs/modules/lightfuzz.md b/docs/modules/lightfuzz.md
index fe495f6bb..7395d864a 100644
--- a/docs/modules/lightfuzz.md
+++ b/docs/modules/lightfuzz.md
@@ -209,7 +209,7 @@ If the server fetches the injected URL, the resulting DNS or HTTP interaction is
| Value | Behavior |
|---|---|
| `always` | Skip WAF-protected targets entirely |
-| `try_bypasses` (default) | Ask the `nowafpls` helper whether the WAF blocks payloads in the request body, and whether padding the body gets past it. POST body and JSON parameters are fuzzed with padding if it does, unpadded if the WAF isn't blocking them, and skipped otherwise. GET, cookie and header parameters are always skipped, since they have no body to pad. |
+| `try_bypasses` (default) | Ask the `nowafpls` helper whether the WAF blocks payloads in the request body, and whether padding the body gets past it. POST body and JSON parameters are fuzzed with padding if it does, unpadded if the WAF isn't blocking them, and skipped otherwise. Cookie and header parameters are skipped, since they have no body to pad. GET parameters are skipped too, unless `try_get_as_post` is on (as in `lightfuzz-heavy` and `lightfuzz-xss`), in which case they are retested as POST bodies and handled like POST parameters above. |
| `never` | Fuzz WAF-protected targets as-is |
## Presets
@@ -275,7 +275,7 @@ A focused preset for XSS hunting only. Demonstrates how to build a single-submod
|---|---|
| **Submodules** | `xss` only |
| **Companion modules** | `paramminer_getparams`, `reflected_parameters` |
-| **POST fuzzing** | Disabled |
+| **POST fuzzing** | Enabled, and `try_get_as_post` is on (GET params retested as POST) |
| **Query string collapsing** | Off |
| **WAF avoidance** | Try bypasses (`avoid_wafs: try_bypasses`) |
diff --git a/docs/scanning/advanced.md b/docs/scanning/advanced.md
index 2af69f85c..3496cd361 100644
--- a/docs/scanning/advanced.md
+++ b/docs/scanning/advanced.md
@@ -148,7 +148,7 @@ Misc:
--custom-yara-rules, -cy CUSTOM_YARA_RULES
Add custom yara rules to excavate
--user-agent, -ua USER_AGENT
- Set the user-agent for HTTP requests (third-party API calls send BBOT/<version>)
+ Set the user-agent for HTTP requests (most API calls send BBOT/<version>)
--user-agent-suffix, -uas SUFFIX
Suffix to append to the user-agent|
|
||
| ### API requests identify as BBOT | ||
|
|
||
| Requests that modules make to third-party APIs now send `User-Agent: BBOT/<version>` instead of the scan's user agent, so `web.user_agent` and `web.user_agent_suffix` no longer apply to them. Requests to your targets still use `web.user_agent`. |
There was a problem hiding this comment.
🔴 Only api_request() sends BBOT/<version>. urlscan, dockerhub, docker_pull, azure_tenant, postman and wayback's CDX lookup still send web.user_agent. Say the helper does this and that not every module uses it yet.
Contributor
Author
There was a problem hiding this comment.
Fixed in 3e6b6d7. The note now says the change applies to modules that use the API request helper, that not every module does yet, and not to rely on BBOT/<version> for allowlisting.
| misc.add_argument( | ||
| "--user-agent", | ||
| "-ua", | ||
| help="Set the user-agent for HTTP requests (third-party API calls send BBOT/<version>)", |
There was a problem hiding this comment.
🔴 Not all third-party API calls send BBOT/<version> (see the migration note comment). Suggest "most API calls send BBOT/", then rerun docs.py for advanced.md.
Contributor
Author
There was a problem hiding this comment.
Fixed in 3e6b6d7: "most API calls send BBOT/".
| Add custom yara rules to excavate | ||
| --user-agent, -ua USER_AGENT | ||
| Set the user-agent for all HTTP requests | ||
| Set the user-agent for HTTP requests (third-party API calls send BBOT/<version>) |
There was a problem hiding this comment.
🔴 Generated copy of the --user-agent help. It will follow once args.py is fixed and docs.py is rerun.
| | Value | Behavior | | ||
| |---|---| | ||
| | `always` | Skip WAF-protected targets entirely | | ||
| | `try_bypasses` (default) | Ask the `nowafpls` helper whether the WAF blocks payloads in the request body, and whether padding the body gets past it. POST body and JSON parameters are fuzzed with padding if it does, unpadded if the WAF isn't blocking them, and skipped otherwise. GET, cookie and header parameters are always skipped, since they have no body to pad. | |
There was a problem hiding this comment.
🔴 With try_get_as_post on (lightfuzz-xss, lightfuzz-heavy), GET params are retested as POST bodies in bypass mode (lightfuzz.py:392), so they aren't always skipped.
Contributor
Author
There was a problem hiding this comment.
Fixed in 3e6b6d7 with your wording. Confirmed try_get_as_post still runs in bypass mode (lightfuzz.py:393).
| @@ -267,6 +277,7 @@ A focused preset for XSS hunting only. Demonstrates how to build a single-submod | |||
| | **Companion modules** | `paramminer_getparams`, `reflected_parameters` | | |||
| | **POST fuzzing** | Disabled | | |||
There was a problem hiding this comment.
🔴 lightfuzz-xss leaves disable_post at False and sets try_get_as_post: True, so this row should say POST fuzzing is enabled, with GET params retested as POST.
Contributor
Author
There was a problem hiding this comment.
Fixed in 3e6b6d7. The preset doesn't include lightfuzz-light, so disable_post stays False.
| config: | ||
| dns: | ||
| threads: 25 | ||
| brute_threads: 1000 |
There was a problem hiding this comment.
🔴 For the removed threads: 25 above: please paste before/after bbot -p subdomain-enum --current-preset-full dns output showing threads go 25 to 50.
Contributor
Author
There was a problem hiding this comment.
Added to the PR body under Validation: threads goes from 25 on dev to 50 on this branch, and it's the only value that changes in the full output.
pjhartlieb
requested changes
Nov 9, 2026
pjhartlieb
left a comment
pjhartlieb
left a comment
Contributor
There was a problem hiding this comment.
Reviewed at fedf360.
🔴 The 3.1.0 speed highlight does not match the measurement it cites
release_history.md:16 says a typical subdomain-enum scan finishes in about 40% less time with the same subdomains. The only timing among the cited PRs is #3606's: one -f subdomain-enum -m http scan of a large wildcard target, crt and crt_db excluded, 22m 50s on 3.0.2 against 11m 51s, 510 against 513 subdomains. #3612 is not part of that number: it merged after the timing, and its own table shows brute-forcing slower at its 600-resolver default. Suggested: keep #3457 and #3612 on the BlastDNS sentence, then "With the higher default DNS concurrency from #3606, a -f subdomain-enum -m http scan of a large wildcard target (crt and crt_db excluded) took 11m 51s against 22m 50s on 3.0.2, and found 513 in-scope subdomains against 510."
🔴 The User-Agent migration note and --user-agent help overstate the change
Resolved in 3e6b6d7: the note scopes BBOT/<version> to modules using the API request helper and warns against allowlisting on it, the help says "most API calls", and advanced.md is regenerated.
🔴 lightfuzz WAF Handling says GET parameters are always skipped
Resolved in 3e6b6d7 with the try_get_as_post wording.
🔴 No validation output in the PR body
Resolved: the body shows the dns section of --current-preset-full on dev and on this branch, threads 25 to 50.
🔴 lightfuzz-xss table says POST fuzzing is Disabled
Resolved in 3e6b6d7.
🟢 Every cited PR is merged and the Dependencies line matches this head
All 54 PRs cited in the 3.1.0 section are merged, apart from the release PR and this one. Every version in the Dependencies line matches this head's pins, lock and module deps. The nuclei budget figure (134 to 15 requests per host) is the one #3479 measured.
Reproduce this review
# save the script below as verify-3613.sh and the patch as control-3613.patch, then:
bash verify-3613.sh blacklanternsecurity/bbot fedf360cf6c86a91446505227a93da22860c57f1
# expected: SUMMARY 16 PASS, 2 FAIL, 0 SKIP, 0 BROKEN (V15 the speed highlight; V18 dev now locks pycryptodome 4.3.0 and radixtarget 4.3.1, informational)
# control: bash verify-3613.sh blacklanternsecurity/bbot fedf360cf6c86a91446505227a93da22860c57f1 "$PWD/control-3613.patch"
# expected: SUMMARY 12 PASS, 6 FAIL, 0 SKIP, 0 BROKEN (the patch reverts the round-1 fixes, so V01 V02 V03 V13 V14 fail, and applies the suggested wording, so V15 passes)#!/usr/bin/env bash
# verify-3613.sh: reproducible checks for blacklanternsecurity/bbot#3613
# Usage: bash verify-3613.sh <owner/repo> <sha> [<control-patch>]
# Changes nothing outside a mktemp directory and nothing on GitHub (PR-VF-02).
#
# Round 2. The control patch reverts the round-1 fixes (args.py, the migration note,
# lightfuzz.md, advanced.md back to 88fef73a) and applies the suggested 3.1.0 speed wording.
# CONTROL: patch control-3613.patch
# CONTROL-EXPECT: V01=FAIL V02=FAIL V03=FAIL V13=FAIL V14=FAIL V15=PASS
set -uo pipefail
REPO="${1:?owner/repo}"; SHA="${2:?sha}"; PATCH="${3:-}"
PASS=0; FAIL=0; SKIP=0; BROKEN=0
pass() { echo "PASS $1 $2"; PASS=$((PASS+1)); }
fail() { echo "FAIL $1 $2 -- $3"; FAIL=$((FAIL+1)); }
skip() { echo "SKIP $1 $2 -- $3"; SKIP=$((SKIP+1)); }
broken() { echo "BROKEN $1 $2 -- $3"; BROKEN=$((BROKEN+1)); }
summary() { echo "SUMMARY $PASS PASS, $FAIL FAIL, $SKIP SKIP, $BROKEN BROKEN"; [ "$FAIL" -eq 0 ] && [ "$BROKEN" -eq 0 ]; }
echo "script_sha256 $(shasum -a 256 "$0" | cut -d' ' -f1)"
echo "git $(git --version | cut -d' ' -f3)"
echo "gh $(gh --version | head -1 | cut -d' ' -f3)"
WORK="$(mktemp -d)"; trap 'rm -rf "$WORK"' EXIT
export GIT_CONFIG_GLOBAL=/dev/null GIT_CONFIG_NOSYSTEM=1 GIT_TERMINAL_PROMPT=0
GIT=(git -c credential.helper= -c 'credential.helper=!gh auth git-credential')
"${GIT[@]}" clone --quiet --filter=blob:none --no-checkout "https://github.com/$REPO.git" "$WORK/src" || { broken V00 "clone" "clone failed"; summary; exit 1; }
cd "$WORK/src"
"${GIT[@]}" cat-file -e "$SHA^{commit}" 2>/dev/null || "${GIT[@]}" fetch --quiet origin "+refs/pull/*/head:refs/remotes/pr/*" || true
"${GIT[@]}" checkout --quiet --detach "$SHA" || { broken V00 "checkout $SHA" "checkout failed"; summary; exit 1; }
[ "$(git rev-parse HEAD)" = "$SHA" ] || { broken V00 "HEAD is $SHA" "HEAD mismatch"; summary; exit 1; }
if [ -n "$PATCH" ]; then git apply "$PATCH" || { broken V00 "apply control patch" "git apply failed"; summary; exit 1; }; fi
# ---- checks: each line states the desired property --------------------------
# Modules that call a third-party API with self.helpers.request() rather than api_request(),
# so they send the scan's web.user_agent, not BBOT/<version>.
OFFENDERS=""
for f in bbot/modules/urlscan.py bbot/modules/dockerhub.py bbot/modules/templates/postman.py bbot/modules/wayback.py; do
[ -f "$f" ] || continue
if grep -qE '(base_url|api_url|site_url)\s*=\s*f?"https?://' "$f" && grep -qE 'self\.helpers\.request\(' "$f"; then
OFFENDERS="$OFFENDERS $(basename "$f" .py)"
fi
done
if grep -q 'kwargs\["headers"\]\["User-Agent"\] = f"BBOT/{__version__}"' bbot/modules/base.py; then
pass V11 "api_request() sends User-Agent BBOT/<version>"
else
fail V11 "api_request() sends User-Agent BBOT/<version>" "base.py pattern not found"
fi
MIG=docs/migration/3.1_breaking_changes.md
UA_SECTION=$(awk '/^### API requests identify as BBOT/{s=1;next} /^(---|### )/{s=0} s' "$MIG")
QUAL='not every module|some third-party|api_request|API request helper|not all'
ua_unqualified() { [ -n "$UA_SECTION" ] && echo "$UA_SECTION" | grep -q 'BBOT/<version>' && ! echo "$UA_SECTION" | grep -qiE "$QUAL"; }
if ua_unqualified && [ -n "$OFFENDERS" ]; then
fail V01 "3.1 migration note's User-Agent claim matches the modules" "claims all third-party API requests send BBOT/<version>; still on web.user_agent:$OFFENDERS"
else
pass V01 "3.1 migration note's User-Agent claim matches the modules"
fi
help_unqualified() { grep -E 'BBOT/<version>' "$1" | grep -i 'user-agent' | grep -viqE '\((most|some)[^)]*BBOT/<version>'; }
if [ -n "$OFFENDERS" ] && { help_unqualified bbot/scanner/preset/args.py || help_unqualified docs/scanning/advanced.md; }; then
fail V02 "--user-agent help text matches the modules" "args.py or advanced.md says API calls send BBOT/<version> without qualification; still on web.user_agent:$OFFENDERS"
else
pass V02 "--user-agent help text matches the modules"
fi
LF=bbot/modules/lightfuzz/lightfuzz.py
WAF_ROW=$(grep -F '`try_bypasses` (default)' docs/modules/lightfuzz.md)
if echo "$WAF_ROW" | grep -q 'GET' && ! echo "$WAF_ROW" | grep -q 'try_get_as_post' \
&& grep -q 'if self.try_get_as_post and original_type == "GETPARAM":' "$LF" \
&& grep -q 'try_get_as_post: True' bbot/presets/web/lightfuzz-xss.yml; then
fail V03 "lightfuzz WAF Handling describes GET params under try_get_as_post" "doc says GET params are always skipped; lightfuzz.py retests them as POST in bypass mode and lightfuzz-xss turns that on"
else
pass V03 "lightfuzz WAF Handling describes GET params under try_get_as_post"
fi
if grep -q 'requires-python = ">=3.11' pyproject.toml && grep -q 'badge/python-3.11+' README.md && grep -q 'BBOT requires Python 3.11+' docs/troubleshooting.md; then
pass V04 "README badge and troubleshooting say 3.11+, matching requires-python"
else
fail V04 "README badge and troubleshooting say 3.11+, matching requires-python" "mismatch"
fi
if grep -qE '^ threads: 50$' bbot/defaults.yml && grep -qE '^ max_concurrency: 256$' bbot/defaults.yml \
&& grep -q 'default 50 queries per resolver' docs/scanning/tips_and_tricks.md && grep -q '256 total' docs/scanning/tips_and_tricks.md; then
pass V05 "DNS speed tip matches dns.threads=50 and dns.max_concurrency=256"
else
fail V05 "DNS speed tip matches dns.threads=50 and dns.max_concurrency=256" "mismatch"
fi
if ! grep -qE '^\s+threads:' bbot/presets/subdomain-enum.yml && ! grep -q 'threads: 25' README.md && ! grep -q 'threads: 25' docs/scanning/presets_list.md; then
pass V06 "subdomain-enum no longer lowers dns.threads, and README/presets_list match"
else
fail V06 "subdomain-enum no longer lowers dns.threads, and README/presets_list match" "threads still set somewhere"
fi
CC=bbot/modules/internal/cloudcheck.py
if grep -q 'event.add_tag(provider_type)' "$CC" && grep -q 'event.add_tag(provider_name)' "$CC" \
&& ! grep -rqE 'cdn-github|cloud-azure|cdn-akamai|cdn-ip' docs/scanning/output.md docs/scanning/events.md docs/modules/custom_yara_rules.md; then
pass V07 "tag examples use cloudcheck's separate type and provider tags"
else
fail V07 "tag examples use cloudcheck's separate type and provider tags" "old cdn-x tags remain or code differs"
fi
ok=1
grep -q 'avoid_wafs: always' bbot/presets/web/lightfuzz-light.yml || ok=0
grep -q 'avoid_wafs: try_bypasses' bbot/presets/web/lightfuzz.yml || ok=0
grep -q 'avoid_wafs: never' bbot/presets/web/lightfuzz-max.yml || ok=0
grep -q 'avoid_wafs: try_bypasses' bbot/presets/web/lightfuzz-xss.yml || ok=0
grep -q 'avoid_wafs: Literal\["always", "never", "try_bypasses"\]' "$LF" || ok=0
if [ "$ok" = 1 ]; then pass V08 "lightfuzz preset tables match each preset's avoid_wafs"; else fail V08 "lightfuzz preset tables match each preset's avoid_wafs" "mismatch"; fi
if grep -q '"--custom-nameservers"' bbot/scanner/preset/args.py && [ -f bbot/presets/nuclei/nuclei-recommended.yml ] && [ -f bbot/presets/nuclei/nuclei-new.yml ]; then
pass V09 "--custom-nameservers, nuclei-recommended and nuclei-new exist as documented"
else
fail V09 "--custom-nameservers, nuclei-recommended and nuclei-new exist as documented" "missing"
fi
if N=$(gh api --method GET "repos/blacklanternsecurity/public-dns-servers/contents/nameservers.txt?ref=master" -H "Accept: application/vnd.github.raw+json" 2>/dev/null | grep -cvE '^\s*(#|$)'); then
if [ "$N" -ge 4000 ] && [ "$N" -le 6000 ]; then pass V10 "brute_nameservers list is ~5K servers ($N)"; else fail V10 "brute_nameservers list is ~5K servers" "count $N"; fi
else
skip V10 "brute_nameservers list is ~5K servers" "could not fetch nameservers.txt"
fi
# C13 (A5): generated blocks match their sources
if python3 - <<'PY'
p=open('bbot/presets/subdomain-enum.yml').read().strip(); r=open('README.md').read()
l=open('docs/scanning/presets_list.md').read(); c=open('docs/scanning/configuration.md').read(); d=open('bbot/defaults.yml').read().strip()
import re,sys; a=open('bbot/scanner/preset/args.py').read(); adv=open('docs/scanning/advanced.md').read()
h=re.search(r'"--user-agent",\s*"-ua",\s*help="([^"]+)"', a)
sys.exit(0 if (p in r and "\n".join(" "+x for x in p.split("\n")) in l and d in c and h and h.group(1) in adv) else 1)
PY
then pass V12 "generated README, presets_list, configuration and advanced.md blocks match their sources"
else fail V12 "generated README, presets_list, configuration and advanced.md blocks match their sources" "a generated block is stale"; fi
# C16/C17: lightfuzz-xss table's POST row matches the preset
XSS_TABLE=$(awk '/^### .-p lightfuzz-xss./{s=1} /^### Spider/{s=0} s' docs/modules/lightfuzz.md)
if echo "$XSS_TABLE" | grep -q 'POST fuzzing\*\* | Disabled' && ! grep -qE 'disable_post: *True' bbot/presets/web/lightfuzz-xss.yml; then
fail V13 "lightfuzz-xss table's POST fuzzing row matches the preset" "table says Disabled; preset leaves disable_post False and sets try_get_as_post"
else
pass V13 "lightfuzz-xss table's POST fuzzing row matches the preset"
fi
# C3/C14 (A6): more modules call third-party APIs outside api_request()
MORE=""
for m in azure_tenant docker_pull; do
grep -qE 'self\.helpers\.request\(' "bbot/modules/$m.py" && grep -qE 'https://(azmap\.dev|registry-1\.docker\.io|login\.microsoftonline\.com)' "bbot/modules/$m.py" && MORE="$MORE $m"
done
if ua_unqualified && [ -n "$MORE" ]; then
fail V14 "3.1 migration note holds for azure_tenant and docker_pull" "unqualified claim; still on web.user_agent:$MORE"
else
pass V14 "3.1 migration note holds for azure_tenant and docker_pull"
fi
# ---- round 2: the 3.1.0 release history (fedf360c) ----------------------------
RH=docs/release_history.md
SEC=$(awk '/^### 3\.1\.0 /{s=1;print;next} /^### [0-9]/{s=0} s' "$RH")
# V15: the speed claim matches the measurement it cites (#3606 "Effect on a scan").
B3606=$(gh api --method GET repos/blacklanternsecurity/bbot/pulls/3606 --jq .body 2>/dev/null)
if [ -z "$SEC" ] || [ -z "$B3606" ]; then
skip V15 "3.1.0 speed claim matches #3606's measurement" "no 3.1.0 section or #3606 unreadable"
elif ! echo "$B3606" | grep -q '| wall time | 22m 50s | 11m 51s |' || ! echo "$B3606" | grep -q 'against a large wildcard target'; then
broken V15 "3.1.0 speed claim matches #3606's measurement" "#3606 no longer carries the measurement this check reads"
else
# C4/R2C1/R2C2: every speed claim in the section must carry #3606's measurement and its target:
# its times (22m 50s, 11m 51s) or its ratio (48%, half, halved), plus "large wildcard"; no other figure, no "typical".
BAD=""
while IFS= read -r L; do
[ -z "$L" ] && continue
M=0; { echo "$L" | grep -q '22m 50s' && echo "$L" | grep -q '11m 51s'; } && M=1
echo "$L" | grep -qiE '48 ?%|half|halved' && M=1
PCT=$(echo "$L" | grep -oE '[0-9]+ ?%' | grep -vE '^48 ?%$')
if [ "$M" = 0 ] || ! echo "$L" | grep -qi 'large wildcard' || [ -n "$PCT" ] || echo "$L" | grep -qi 'typical'; then BAD="$BAD | ${L:0:120}"; fi
done <<< "$(echo "$SEC" | grep -iE 'faster|slower|less time|as long|wall time|halved|took|[0-9]+m [0-9]+s|[0-9]+ ?% (less|more|quicker)|finish(es)? in' | grep -iE 'scan')"
if [ -z "$BAD" ]; then pass V15 "3.1.0 speed claims match #3606's measurement"
else fail V15 "3.1.0 speed claims match #3606's measurement" "unmeasured speed claim:$BAD; #3606: one large wildcard target, 22m 50s to 11m 51s (48% less), 510 to 513 subdomains"; fi
fi
# V16: every PR the 3.1.0 section cites is merged, except the release PR (#3584) and this one.
if [ -z "$SEC" ]; then
skip V16 "every PR cited in 3.1.0 is merged" "no 3.1.0 section"
else
BAD=""; N=0
for n in $(echo "$SEC" | grep -oE 'bbot/pull/[0-9]+' | cut -d/ -f3 | sort -un); do
N=$((N+1)); [ "$n" = 3584 ] || [ "$n" = 3613 ] && continue
m=$(gh api --method GET "repos/blacklanternsecurity/bbot/pulls/$n" --jq '.merged_at // "no"' 2>/dev/null) || m=err
[ "$m" = no ] || [ "$m" = err ] && BAD="$BAD #$n($m)"
done
if [ -z "$BAD" ]; then pass V16 "every PR cited in 3.1.0 is merged ($N cited)"; else fail V16 "every PR cited in 3.1.0 is merged" "not merged:$BAD"; fi
fi
# V17: the Dependencies line matches what this head pins or locks.
lockv() { awk -v n="$1" '$0=="name = \""n"\""{getline; gsub(/version = |"/,""); print; exit}' uv.lock; }
ok=1; why=""
for pair in "blastdns:blastdns>=2.0.3" "blasthttp:blasthttp>=0.10.1" "cloudcheck:cloudcheck>=11.2.0" "radixtarget:radixtarget>=4.3.0"; do
grep -q "\"${pair#*:}" pyproject.toml || { ok=0; why="$why ${pair%%:*}"; }
done
[ "$(lockv pycryptodome)" = 4.2.0 ] || { ok=0; why="$why pycryptodome-lock=$(lockv pycryptodome)"; }
grep -q 'baddns~=2.5.0' bbot/modules/baddns.py || { ok=0; why="$why baddns"; }
grep -q 'badsecrets~=1.3.0' bbot/modules/badsecrets.py || { ok=0; why="$why badsecrets"; }
grep -q 'Field("3.99.8"' bbot/modules/trufflehog.py || { ok=0; why="$why trufflehog"; }
for nv in "blastdns 2.0.3" "blasthttp 0.10.1" "baddns 2.5.0" "badsecrets 1.3.0" "cloudcheck 11.2.0" "radixtarget 4.3.0" "pycryptodome 4.2.0" "3.99.8"; do
echo "$SEC" | grep -q "$nv" || { ok=0; why="$why doc-missing:$nv"; }
done
if [ "$ok" = 1 ]; then pass V17 "3.1.0 Dependencies line matches this head"; else fail V17 "3.1.0 Dependencies line matches this head" "mismatch:$why"; fi
# V18: ... and still matches dev, which this PR merges into (dev at e8b3ecf6, captured 2026-11-09).
DEV=e8b3ecf623c05e58bdfbc01c94c58d72f2555dd0
"${GIT[@]}" fetch --quiet origin "$DEV" 2>/dev/null || true
if ! git cat-file -e "$DEV^{commit}" 2>/dev/null; then
skip V18 "3.1.0 Dependencies line matches dev's lock" "could not fetch dev $DEV"
else
DP=$(git show "$DEV:uv.lock" | awk '$0=="name = \"pycryptodome\""{getline; gsub(/version = |"/,""); print; exit}')
DR=$(git show "$DEV:uv.lock" | awk '$0=="name = \"radixtarget\""{getline; gsub(/version = |"/,""); print; exit}')
if echo "$SEC" | grep -q "pycryptodome $DP" && echo "$SEC" | grep -q "radixtarget $DR"; then
pass V18 "3.1.0 Dependencies line matches dev's lock"
else
fail V18 "3.1.0 Dependencies line matches dev's lock" "dev locks pycryptodome $DP and radixtarget $DR"
fi
fi
# ------------------------------------------------------------------------------
summarydiff --git a/bbot/scanner/preset/args.py b/bbot/scanner/preset/args.py
index f22c565c0..febde1bc2 100644
--- a/bbot/scanner/preset/args.py
+++ b/bbot/scanner/preset/args.py
@@ -466,7 +466,7 @@ class BBOTArgs:
misc.add_argument(
"--user-agent",
"-ua",
- help="Set the user-agent for HTTP requests (most API calls send BBOT/<version>)",
+ help="Set the user-agent for HTTP requests (third-party API calls send BBOT/<version>)",
)
misc.add_argument(
"--user-agent-suffix", "-uas", help="Suffix to append to the user-agent", metavar="SUFFIX", default=None
diff --git a/docs/migration/3.1_breaking_changes.md b/docs/migration/3.1_breaking_changes.md
index d594d1426..7cccb85db 100644
--- a/docs/migration/3.1_breaking_changes.md
+++ b/docs/migration/3.1_breaking_changes.md
@@ -97,7 +97,7 @@ With `urls: true`, wayback now probes each host once and drops URLs on hosts tha
### API requests identify as BBOT
-Modules that query third-party APIs through BBOT's API request helper now send `User-Agent: BBOT/<version>` instead of the scan's user agent, so `web.user_agent` and `web.user_agent_suffix` no longer apply to those requests. Not every module uses the helper yet, so some third-party requests still send `web.user_agent`; don't rely on `BBOT/<version>` for allowlisting. Requests to your targets still use `web.user_agent`.
+Requests that modules make to third-party APIs now send `User-Agent: BBOT/<version>` instead of the scan's user agent, so `web.user_agent` and `web.user_agent_suffix` no longer apply to them. Requests to your targets still use `web.user_agent`.
---
diff --git a/docs/modules/lightfuzz.md b/docs/modules/lightfuzz.md
index 7395d864a..fe495f6bb 100644
--- a/docs/modules/lightfuzz.md
+++ b/docs/modules/lightfuzz.md
@@ -209,7 +209,7 @@ If the server fetches the injected URL, the resulting DNS or HTTP interaction is
| Value | Behavior |
|---|---|
| `always` | Skip WAF-protected targets entirely |
-| `try_bypasses` (default) | Ask the `nowafpls` helper whether the WAF blocks payloads in the request body, and whether padding the body gets past it. POST body and JSON parameters are fuzzed with padding if it does, unpadded if the WAF isn't blocking them, and skipped otherwise. Cookie and header parameters are skipped, since they have no body to pad. GET parameters are skipped too, unless `try_get_as_post` is on (as in `lightfuzz-heavy` and `lightfuzz-xss`), in which case they are retested as POST bodies and handled like POST parameters above. |
+| `try_bypasses` (default) | Ask the `nowafpls` helper whether the WAF blocks payloads in the request body, and whether padding the body gets past it. POST body and JSON parameters are fuzzed with padding if it does, unpadded if the WAF isn't blocking them, and skipped otherwise. GET, cookie and header parameters are always skipped, since they have no body to pad. |
| `never` | Fuzz WAF-protected targets as-is |
## Presets
@@ -275,7 +275,7 @@ A focused preset for XSS hunting only. Demonstrates how to build a single-submod
|---|---|
| **Submodules** | `xss` only |
| **Companion modules** | `paramminer_getparams`, `reflected_parameters` |
-| **POST fuzzing** | Enabled, and `try_get_as_post` is on (GET params retested as POST) |
+| **POST fuzzing** | Disabled |
| **Query string collapsing** | Off |
| **WAF avoidance** | Try bypasses (`avoid_wafs: try_bypasses`) |
diff --git a/docs/scanning/advanced.md b/docs/scanning/advanced.md
index 3496cd361..2af69f85c 100644
--- a/docs/scanning/advanced.md
+++ b/docs/scanning/advanced.md
@@ -148,7 +148,7 @@ Misc:
--custom-yara-rules, -cy CUSTOM_YARA_RULES
Add custom yara rules to excavate
--user-agent, -ua USER_AGENT
- Set the user-agent for HTTP requests (most API calls send BBOT/<version>)
+ Set the user-agent for HTTP requests (third-party API calls send BBOT/<version>)
--user-agent-suffix, -uas SUFFIX
Suffix to append to the user-agent
diff --git a/docs/release_history.md b/docs/release_history.md
index d6487f684..818d4c5c8 100644
--- a/docs/release_history.md
+++ b/docs/release_history.md
@@ -13,7 +13,7 @@
#### Highlights
-- **DNS brute-forcing moves to BlastDNS:** massdns is gone, and brute-forcing runs through blastdns 2.0 with adaptive rate control. With higher default DNS concurrency, a typical `subdomain-enum` scan finishes in about 40% less time than on 3.0.2, with the same subdomains found ([#3457](https://github.com/blacklanternsecurity/bbot/pull/3457), [#3606](https://github.com/blacklanternsecurity/bbot/pull/3606), [#3612](https://github.com/blacklanternsecurity/bbot/pull/3612))
+- **DNS brute-forcing moves to BlastDNS:** massdns is gone, and brute-forcing runs through blastdns 2.0 with adaptive rate control. ([#3457](https://github.com/blacklanternsecurity/bbot/pull/3457), [#3612](https://github.com/blacklanternsecurity/bbot/pull/3612)). With the higher default DNS concurrency from [#3606](https://github.com/blacklanternsecurity/bbot/pull/3606), a `-f subdomain-enum -m http` scan of a large wildcard target (crt and crt_db excluded) took 11m 51s against 22m 50s on 3.0.2, and found 513 in-scope subdomains against 510
- **New `--custom-nameservers` flag** for choosing the resolvers BBOT uses for normal resolution ([#3457](https://github.com/blacklanternsecurity/bbot/pull/3457))
- **baddns 2.5.0**, with a new `DELEGATION` submodule that is on by default ([#3457](https://github.com/blacklanternsecurity/bbot/pull/3457))
- **CDN/WAF port filtering in core, on by default** ([#3534](https://github.com/blacklanternsecurity/bbot/pull/3534))|
|
||
| #### Highlights | ||
|
|
||
| - **DNS brute-forcing moves to BlastDNS:** massdns is gone, and brute-forcing runs through blastdns 2.0 with adaptive rate control. With higher default DNS concurrency, a typical `subdomain-enum` scan finishes in about 40% less time than on 3.0.2, with the same subdomains found ([#3457](https://github.com/blacklanternsecurity/bbot/pull/3457), [#3606](https://github.com/blacklanternsecurity/bbot/pull/3606), [#3612](https://github.com/blacklanternsecurity/bbot/pull/3612)) |
Contributor
There was a problem hiding this comment.
en0f
requested changes
Nov 9, 2026
en0f
left a comment
en0f
left a comment
There was a problem hiding this comment.
Reviewed at fedf360.
🔴 The 3.1.0 speed highlight still doesn't match its measurement
Still open from the round-2 review at this head. #3606 is the only cited PR that timed a scan: one large wildcard target, -f subdomain-enum -m http, crt and crt_db excluded, 22m 50s on 3.0.2 against 11m 51s, 510 against 513 subdomains, measured before #3612 merged. Dev with BlastDNS but before #3606 had a 14m 03s main phase against 11m 01s on 3.0.2, so the speedup belongs to #3606's concurrency change. The wording already suggested on line 16 fixes it.
🔴 The migration guide doesn't say lightfuzz now fuzzes WAF-protected hosts
In 3.0.2, lightfuzz read a top-level avoid_wafs that defaulted to True, so it skipped every WAF-tagged host in every preset. In 3.1, modules.lightfuzz.avoid_wafs defaults to try_bypasses, and the lightfuzz presets also enable nowafpls, which sends every WAF-tagged host a POST carrying <script>alert(1)</script>, padded up to 1 MiB. Anyone relying on 3.0's skip sends more invasive traffic toward WAF-fronted targets after upgrading, and neither the guide nor the "Check your setup" box says so. Suggested entry under Modules: 3.0 skipped WAF-tagged hosts; 3.1 tries bypasses by default (link lightfuzz.md § WAF Handling), lightfuzz-max fuzzes them as-is, and a top-level avoid_wafs is no longer read. For the old behaviour, set modules.lightfuzz.avoid_wafs: always and pass -em nowafpls. While there, lightfuzz.yml and lightfuzz-heavy.yml still describe themselves as skipping confirmed WAFs.
🟢 The WAF Handling table matches the code mode for mode
Its three rows are the three values of the avoid_wafs Literal in lightfuzz.py, with try_bypasses as the default, and each preset's WAF row names the value its YAML sets. Putting the config value next to the behaviour lets a reader check it against their own preset.
Reproduce this review
# save the script below as verify-3613.sh and the patch as control-3613.patch, then:
bash verify-3613.sh blacklanternsecurity/bbot fedf360cf6c86a91446505227a93da22860c57f1
# expected: SUMMARY 17 PASS, 4 FAIL, 0 SKIP, 0 BROKEN (V15 speed highlight; V21 migration guide; V19 preset description, informational; V18 dev lock drift, informational)
# control: bash verify-3613.sh blacklanternsecurity/bbot fedf360cf6c86a91446505227a93da22860c57f1 "$PWD/control-3613.patch"
# expected: SUMMARY 15 PASS, 6 FAIL, 0 SKIP, 0 BROKEN (the patch reverts the round-1 fixes, so V01 V02 V03 V13 V14 fail; it applies the suggested wording, the lightfuzz-heavy description and a migration entry, so V15 V19 V21 pass)#!/usr/bin/env bash
# verify-3613.sh: reproducible checks for blacklanternsecurity/bbot#3613
# Usage: bash verify-3613.sh <owner/repo> <sha> [<control-patch>]
# Changes nothing outside a mktemp directory and nothing on GitHub (PR-VF-02).
#
# Round 3, at the same head as round 2 (fedf360c). The control patch reverts the round-1
# fixes (args.py, the migration note, lightfuzz.md, advanced.md back to 88fef73a), applies the
# suggested 3.1.0 speed wording, rewrites the lightfuzz-heavy preset description, and adds a lightfuzz WAF entry to the migration guide.
# CONTROL: patch control-3613.patch
# CONTROL-EXPECT: V01=FAIL V02=FAIL V03=FAIL V13=FAIL V14=FAIL V15=PASS V19=PASS V21=PASS
set -uo pipefail
REPO="${1:?owner/repo}"; SHA="${2:?sha}"; PATCH="${3:-}"
PASS=0; FAIL=0; SKIP=0; BROKEN=0
pass() { echo "PASS $1 $2"; PASS=$((PASS+1)); }
fail() { echo "FAIL $1 $2 -- $3"; FAIL=$((FAIL+1)); }
skip() { echo "SKIP $1 $2 -- $3"; SKIP=$((SKIP+1)); }
broken() { echo "BROKEN $1 $2 -- $3"; BROKEN=$((BROKEN+1)); }
summary() { echo "SUMMARY $PASS PASS, $FAIL FAIL, $SKIP SKIP, $BROKEN BROKEN"; [ "$FAIL" -eq 0 ] && [ "$BROKEN" -eq 0 ]; }
echo "script_sha256 $(shasum -a 256 "$0" | cut -d' ' -f1)"
echo "git $(git --version | cut -d' ' -f3)"
echo "gh $(gh --version | head -1 | cut -d' ' -f3)"
WORK="$(mktemp -d)"; trap 'rm -rf "$WORK"' EXIT
export GIT_CONFIG_GLOBAL=/dev/null GIT_CONFIG_NOSYSTEM=1 GIT_TERMINAL_PROMPT=0
GIT=(git -c credential.helper= -c 'credential.helper=!gh auth git-credential')
"${GIT[@]}" clone --quiet --filter=blob:none --no-checkout "https://github.com/$REPO.git" "$WORK/src" || { broken V00 "clone" "clone failed"; summary; exit 1; }
cd "$WORK/src"
"${GIT[@]}" cat-file -e "$SHA^{commit}" 2>/dev/null || "${GIT[@]}" fetch --quiet origin "+refs/pull/*/head:refs/remotes/pr/*" || true
"${GIT[@]}" checkout --quiet --detach "$SHA" || { broken V00 "checkout $SHA" "checkout failed"; summary; exit 1; }
[ "$(git rev-parse HEAD)" = "$SHA" ] || { broken V00 "HEAD is $SHA" "HEAD mismatch"; summary; exit 1; }
if [ -n "$PATCH" ]; then git apply "$PATCH" || { broken V00 "apply control patch" "git apply failed"; summary; exit 1; }; fi
# ---- checks: each line states the desired property --------------------------
# Modules that call a third-party API with self.helpers.request() rather than api_request(),
# so they send the scan's web.user_agent, not BBOT/<version>.
OFFENDERS=""
for f in bbot/modules/urlscan.py bbot/modules/dockerhub.py bbot/modules/templates/postman.py bbot/modules/wayback.py; do
[ -f "$f" ] || continue
if grep -qE '(base_url|api_url|site_url)\s*=\s*f?"https?://' "$f" && grep -qE 'self\.helpers\.request\(' "$f"; then
OFFENDERS="$OFFENDERS $(basename "$f" .py)"
fi
done
if grep -q 'kwargs\["headers"\]\["User-Agent"\] = f"BBOT/{__version__}"' bbot/modules/base.py; then
pass V11 "api_request() sends User-Agent BBOT/<version>"
else
fail V11 "api_request() sends User-Agent BBOT/<version>" "base.py pattern not found"
fi
MIG=docs/migration/3.1_breaking_changes.md
UA_SECTION=$(awk '/^### API requests identify as BBOT/{s=1;next} /^(---|### )/{s=0} s' "$MIG")
QUAL='not every module|some third-party|api_request|API request helper|not all'
ua_unqualified() { [ -n "$UA_SECTION" ] && echo "$UA_SECTION" | grep -q 'BBOT/<version>' && ! echo "$UA_SECTION" | grep -qiE "$QUAL"; }
if ua_unqualified && [ -n "$OFFENDERS" ]; then
fail V01 "3.1 migration note's User-Agent claim matches the modules" "claims all third-party API requests send BBOT/<version>; still on web.user_agent:$OFFENDERS"
else
pass V01 "3.1 migration note's User-Agent claim matches the modules"
fi
help_unqualified() { grep -E 'BBOT/<version>' "$1" | grep -i 'user-agent' | grep -viqE '\((most|some)[^)]*BBOT/<version>'; }
if [ -n "$OFFENDERS" ] && { help_unqualified bbot/scanner/preset/args.py || help_unqualified docs/scanning/advanced.md; }; then
fail V02 "--user-agent help text matches the modules" "args.py or advanced.md says API calls send BBOT/<version> without qualification; still on web.user_agent:$OFFENDERS"
else
pass V02 "--user-agent help text matches the modules"
fi
LF=bbot/modules/lightfuzz/lightfuzz.py
WAF_ROW=$(grep -F '`try_bypasses` (default)' docs/modules/lightfuzz.md)
if echo "$WAF_ROW" | grep -q 'GET' && ! echo "$WAF_ROW" | grep -q 'try_get_as_post' \
&& grep -q 'if self.try_get_as_post and original_type == "GETPARAM":' "$LF" \
&& grep -q 'try_get_as_post: True' bbot/presets/web/lightfuzz-xss.yml; then
fail V03 "lightfuzz WAF Handling describes GET params under try_get_as_post" "doc says GET params are always skipped; lightfuzz.py retests them as POST in bypass mode and lightfuzz-xss turns that on"
else
pass V03 "lightfuzz WAF Handling describes GET params under try_get_as_post"
fi
if grep -q 'requires-python = ">=3.11' pyproject.toml && grep -q 'badge/python-3.11+' README.md && grep -q 'BBOT requires Python 3.11+' docs/troubleshooting.md; then
pass V04 "README badge and troubleshooting say 3.11+, matching requires-python"
else
fail V04 "README badge and troubleshooting say 3.11+, matching requires-python" "mismatch"
fi
if grep -qE '^ threads: 50$' bbot/defaults.yml && grep -qE '^ max_concurrency: 256$' bbot/defaults.yml \
&& grep -q 'default 50 queries per resolver' docs/scanning/tips_and_tricks.md && grep -q '256 total' docs/scanning/tips_and_tricks.md; then
pass V05 "DNS speed tip matches dns.threads=50 and dns.max_concurrency=256"
else
fail V05 "DNS speed tip matches dns.threads=50 and dns.max_concurrency=256" "mismatch"
fi
if ! grep -qE '^\s+threads:' bbot/presets/subdomain-enum.yml && ! grep -q 'threads: 25' README.md && ! grep -q 'threads: 25' docs/scanning/presets_list.md; then
pass V06 "subdomain-enum no longer lowers dns.threads, and README/presets_list match"
else
fail V06 "subdomain-enum no longer lowers dns.threads, and README/presets_list match" "threads still set somewhere"
fi
CC=bbot/modules/internal/cloudcheck.py
if grep -q 'event.add_tag(provider_type)' "$CC" && grep -q 'event.add_tag(provider_name)' "$CC" \
&& ! grep -rqE 'cdn-github|cloud-azure|cdn-akamai|cdn-ip' docs/scanning/output.md docs/scanning/events.md docs/modules/custom_yara_rules.md; then
pass V07 "tag examples use cloudcheck's separate type and provider tags"
else
fail V07 "tag examples use cloudcheck's separate type and provider tags" "old cdn-x tags remain or code differs"
fi
ok=1
grep -q 'avoid_wafs: always' bbot/presets/web/lightfuzz-light.yml || ok=0
grep -q 'avoid_wafs: try_bypasses' bbot/presets/web/lightfuzz.yml || ok=0
grep -q 'avoid_wafs: never' bbot/presets/web/lightfuzz-max.yml || ok=0
grep -q 'avoid_wafs: try_bypasses' bbot/presets/web/lightfuzz-xss.yml || ok=0
grep -q 'avoid_wafs: Literal\["always", "never", "try_bypasses"\]' "$LF" || ok=0
if [ "$ok" = 1 ]; then pass V08 "lightfuzz preset tables match each preset's avoid_wafs"; else fail V08 "lightfuzz preset tables match each preset's avoid_wafs" "mismatch"; fi
if grep -q '"--custom-nameservers"' bbot/scanner/preset/args.py && [ -f bbot/presets/nuclei/nuclei-recommended.yml ] && [ -f bbot/presets/nuclei/nuclei-new.yml ]; then
pass V09 "--custom-nameservers, nuclei-recommended and nuclei-new exist as documented"
else
fail V09 "--custom-nameservers, nuclei-recommended and nuclei-new exist as documented" "missing"
fi
if N=$(gh api --method GET "repos/blacklanternsecurity/public-dns-servers/contents/nameservers.txt?ref=master" -H "Accept: application/vnd.github.raw+json" 2>/dev/null | grep -cvE '^\s*(#|$)'); then
if [ "$N" -ge 4000 ] && [ "$N" -le 6000 ]; then pass V10 "brute_nameservers list is ~5K servers ($N)"; else fail V10 "brute_nameservers list is ~5K servers" "count $N"; fi
else
skip V10 "brute_nameservers list is ~5K servers" "could not fetch nameservers.txt"
fi
# C13 (A5): generated blocks match their sources
if python3 - <<'PY'
p=open('bbot/presets/subdomain-enum.yml').read().strip(); r=open('README.md').read()
l=open('docs/scanning/presets_list.md').read(); c=open('docs/scanning/configuration.md').read(); d=open('bbot/defaults.yml').read().strip()
import re,sys; a=open('bbot/scanner/preset/args.py').read(); adv=open('docs/scanning/advanced.md').read()
h=re.search(r'"--user-agent",\s*"-ua",\s*help="([^"]+)"', a)
sys.exit(0 if (p in r and "\n".join(" "+x for x in p.split("\n")) in l and d in c and h and h.group(1) in adv) else 1)
PY
then pass V12 "generated README, presets_list, configuration and advanced.md blocks match their sources"
else fail V12 "generated README, presets_list, configuration and advanced.md blocks match their sources" "a generated block is stale"; fi
# C16/C17: lightfuzz-xss table's POST row matches the preset
XSS_TABLE=$(awk '/^### .-p lightfuzz-xss./{s=1} /^### Spider/{s=0} s' docs/modules/lightfuzz.md)
if echo "$XSS_TABLE" | grep -q 'POST fuzzing\*\* | Disabled' && ! grep -qE 'disable_post: *True' bbot/presets/web/lightfuzz-xss.yml; then
fail V13 "lightfuzz-xss table's POST fuzzing row matches the preset" "table says Disabled; preset leaves disable_post False and sets try_get_as_post"
else
pass V13 "lightfuzz-xss table's POST fuzzing row matches the preset"
fi
# C3/C14 (A6): more modules call third-party APIs outside api_request()
MORE=""
for m in azure_tenant docker_pull; do
grep -qE 'self\.helpers\.request\(' "bbot/modules/$m.py" && grep -qE 'https://(azmap\.dev|registry-1\.docker\.io|login\.microsoftonline\.com)' "bbot/modules/$m.py" && MORE="$MORE $m"
done
if ua_unqualified && [ -n "$MORE" ]; then
fail V14 "3.1 migration note holds for azure_tenant and docker_pull" "unqualified claim; still on web.user_agent:$MORE"
else
pass V14 "3.1 migration note holds for azure_tenant and docker_pull"
fi
# ---- round 2: the 3.1.0 release history (fedf360c) ----------------------------
RH=docs/release_history.md
SEC=$(awk '/^### 3\.1\.0 /{s=1;print;next} /^### [0-9]/{s=0} s' "$RH")
# V15: the speed claim matches the measurement it cites (#3606 "Effect on a scan").
B3606=$(gh api --method GET repos/blacklanternsecurity/bbot/pulls/3606 --jq .body 2>/dev/null)
if [ -z "$SEC" ] || [ -z "$B3606" ]; then
skip V15 "3.1.0 speed claim matches #3606's measurement" "no 3.1.0 section or #3606 unreadable"
elif ! echo "$B3606" | grep -q '| wall time | 22m 50s | 11m 51s |' || ! echo "$B3606" | grep -q 'against a large wildcard target'; then
broken V15 "3.1.0 speed claim matches #3606's measurement" "#3606 no longer carries the measurement this check reads"
else
# C4/R2C1/R2C2: every speed claim in the section must carry #3606's measurement and its target:
# its times (22m 50s, 11m 51s) or its ratio (48%, half, halved), plus "large wildcard"; no other figure, no "typical".
BAD=""
while IFS= read -r L; do
[ -z "$L" ] && continue
M=0; { echo "$L" | grep -q '22m 50s' && echo "$L" | grep -q '11m 51s'; } && M=1
echo "$L" | grep -qiE '48 ?%|half|halved' && M=1
PCT=$(echo "$L" | grep -oE '[0-9]+ ?%' | grep -vE '^48 ?%$')
if [ "$M" = 0 ] || ! echo "$L" | grep -qi 'large wildcard' || [ -n "$PCT" ] || echo "$L" | grep -qi 'typical'; then BAD="$BAD | ${L:0:120}"; fi
done <<< "$(echo "$SEC" | grep -iE 'faster|slower|less time|as long|wall time|halved|took|[0-9]+m [0-9]+s|[0-9]+ ?% (less|more|quicker)|finish(es)? in' | grep -iE 'scan')"
if [ -z "$BAD" ]; then pass V15 "3.1.0 speed claims match #3606's measurement"
else fail V15 "3.1.0 speed claims match #3606's measurement" "unmeasured speed claim:$BAD; #3606: one large wildcard target, 22m 50s to 11m 51s (48% less), 510 to 513 subdomains"; fi
fi
# V16: every PR the 3.1.0 section cites is merged, except the release PR (#3584) and this one.
if [ -z "$SEC" ]; then
skip V16 "every PR cited in 3.1.0 is merged" "no 3.1.0 section"
else
BAD=""; N=0
for n in $(echo "$SEC" | grep -oE 'bbot/pull/[0-9]+' | cut -d/ -f3 | sort -un); do
N=$((N+1)); [ "$n" = 3584 ] || [ "$n" = 3613 ] && continue
m=$(gh api --method GET "repos/blacklanternsecurity/bbot/pulls/$n" --jq '.merged_at // "no"' 2>/dev/null) || m=err
[ "$m" = no ] || [ "$m" = err ] && BAD="$BAD #$n($m)"
done
if [ -z "$BAD" ]; then pass V16 "every PR cited in 3.1.0 is merged ($N cited)"; else fail V16 "every PR cited in 3.1.0 is merged" "not merged:$BAD"; fi
fi
# V17: the Dependencies line matches what this head pins or locks.
lockv() { awk -v n="$1" '$0=="name = \""n"\""{getline; gsub(/version = |"/,""); print; exit}' uv.lock; }
ok=1; why=""
for pair in "blastdns:blastdns>=2.0.3" "blasthttp:blasthttp>=0.10.1" "cloudcheck:cloudcheck>=11.2.0" "radixtarget:radixtarget>=4.3.0"; do
grep -q "\"${pair#*:}" pyproject.toml || { ok=0; why="$why ${pair%%:*}"; }
done
[ "$(lockv pycryptodome)" = 4.2.0 ] || { ok=0; why="$why pycryptodome-lock=$(lockv pycryptodome)"; }
grep -q 'baddns~=2.5.0' bbot/modules/baddns.py || { ok=0; why="$why baddns"; }
grep -q 'badsecrets~=1.3.0' bbot/modules/badsecrets.py || { ok=0; why="$why badsecrets"; }
grep -q 'Field("3.99.8"' bbot/modules/trufflehog.py || { ok=0; why="$why trufflehog"; }
for nv in "blastdns 2.0.3" "blasthttp 0.10.1" "baddns 2.5.0" "badsecrets 1.3.0" "cloudcheck 11.2.0" "radixtarget 4.3.0" "pycryptodome 4.2.0" "3.99.8"; do
echo "$SEC" | grep -q "$nv" || { ok=0; why="$why doc-missing:$nv"; }
done
if [ "$ok" = 1 ]; then pass V17 "3.1.0 Dependencies line matches this head"; else fail V17 "3.1.0 Dependencies line matches this head" "mismatch:$why"; fi
# V18: ... and still matches dev, which this PR merges into (dev at e8b3ecf6, captured 2026-11-09).
DEV=e8b3ecf623c05e58bdfbc01c94c58d72f2555dd0
"${GIT[@]}" fetch --quiet origin "$DEV" 2>/dev/null || true
if ! git cat-file -e "$DEV^{commit}" 2>/dev/null; then
skip V18 "3.1.0 Dependencies line matches dev's lock" "could not fetch dev $DEV"
else
DP=$(git show "$DEV:uv.lock" | awk '$0=="name = \"pycryptodome\""{getline; gsub(/version = |"/,""); print; exit}')
DR=$(git show "$DEV:uv.lock" | awk '$0=="name = \"radixtarget\""{getline; gsub(/version = |"/,""); print; exit}')
if echo "$SEC" | grep -q "pycryptodome $DP" && echo "$SEC" | grep -q "radixtarget $DR"; then
pass V18 "3.1.0 Dependencies line matches dev's lock"
else
fail V18 "3.1.0 Dependencies line matches dev's lock" "dev locks pycryptodome $DP and radixtarget $DR"
fi
fi
# ---- round 3 -------------------------------------------------------------------
# V19: lightfuzz-heavy (includes lightfuzz -> avoid_wafs: try_bypasses; disable_post False, try_get_as_post True)
# does not describe itself as skipping WAFs in its preset file (what `bbot -lp` prints; presets_list.md is generated from it).
H=bbot/presets/web/lightfuzz-heavy.yml
eff=$(grep -oE 'avoid_wafs: *[a-z_]+' "$H" | awk '{print $2}')
[ -z "$eff" ] && grep -qE '^ - lightfuzz$' "$H" && eff=$(grep -oE 'avoid_wafs: *[a-z_]+' bbot/presets/web/lightfuzz.yml | awk '{print $2}')
if [ -z "$eff" ]; then
broken V19 "lightfuzz-heavy's description matches its avoid_wafs" "could not resolve avoid_wafs for lightfuzz-heavy"
elif [ "$eff" = try_bypasses ] && grep -q 'disable_post: False' "$H" && grep -m1 '^description:' "$H" | grep -qi 'skips confirmed WAFs'; then
fail V19 "lightfuzz-heavy's description matches its avoid_wafs" "says it still skips confirmed WAFs; it runs try_bypasses with POST fuzzing on"
else
pass V19 "lightfuzz-heavy's description matches its avoid_wafs"
fi
# V20: each lightfuzz preset's "WAF avoidance" row in lightfuzz.md names the avoid_wafs value its YAML sets.
LFMD=docs/modules/lightfuzz.md; BAD=""; N=0
for p in lightfuzz-light lightfuzz lightfuzz-max lightfuzz-xss; do
eff=$(grep -oE 'avoid_wafs: *[a-z_]+' "bbot/presets/web/$p.yml" | awk '{print $2}')
row=$(awk -v h="### \`-p $p\`" '$0==h{s=1;next} /^### /{s=0} s && /\*\*WAF avoidance\*\*/' "$LFMD")
N=$((N+1))
if [ -z "$eff" ] || [ -z "$row" ] || ! echo "$row" | grep -q "avoid_wafs: $eff"; then BAD="$BAD $p(yaml=$eff)"; fi
done
if [ -z "$BAD" ]; then pass V20 "lightfuzz.md WAF avoidance rows match each preset's avoid_wafs ($N presets)"
else fail V20 "lightfuzz.md WAF avoidance rows match each preset's avoid_wafs" "row missing or wrong:$BAD"; fi
# V21: v3.0.2 lightfuzz skipped every WAF-tagged host (top-level avoid_wafs, default True); this head defaults
# to try_bypasses. The 3.1 migration guide's Modules section says so (C6: any heading).
"${GIT[@]}" fetch --quiet origin tag v3.0.2 2>/dev/null || true
OLD=$(git show v3.0.2:bbot/modules/lightfuzz/lightfuzz.py 2>/dev/null | grep -c 'self.avoid_wafs = self.scan.config.get("avoid_wafs", True)')
NEW=$(grep -c '"try_bypasses",' bbot/modules/lightfuzz/lightfuzz.py)
if [ "$OLD" != 1 ] || [ "$NEW" -lt 1 ]; then
broken V21 "3.1 migration guide documents lightfuzz's WAF default change" "could not read v3.0.2 or head lightfuzz.py"
elif awk '/^## /{m=($0 ~ /^## Modules/)} m && /avoid_wafs/' docs/migration/3.1_breaking_changes.md | grep -q .; then
pass V21 "3.1 migration guide documents lightfuzz's WAF default change"
else
fail V21 "3.1 migration guide documents lightfuzz's WAF default change" "3.0.2 skipped every WAF host; head defaults to try_bypasses; no lightfuzz/avoid_wafs entry in the guide"
fi
# ------------------------------------------------------------------------------
summarydiff --git a/bbot/presets/web/lightfuzz-heavy.yml b/bbot/presets/web/lightfuzz-heavy.yml
index c67474af1..5eed2fde6 100644
--- a/bbot/presets/web/lightfuzz-heavy.yml
+++ b/bbot/presets/web/lightfuzz-heavy.yml
@@ -1,4 +1,4 @@
-description: "Aggressive fuzzing: everything in lightfuzz, plus paramminer brute-force parameter discovery (headers, GET params, cookies), POST request fuzzing enabled, try_get_as_post enabled (GET params retested as POST), and robots.txt parsing. Still skips confirmed WAFs."
+description: "Aggressive fuzzing: everything in lightfuzz, plus paramminer brute-force parameter discovery (headers, GET params, cookies), POST request fuzzing enabled, try_get_as_post enabled (GET params retested as POST), and robots.txt parsing. On confirmed WAFs, POST bodies (and GET params retested as POST) are fuzzed with nowafpls padding where it gets through."
include:
- lightfuzz
diff --git a/bbot/scanner/preset/args.py b/bbot/scanner/preset/args.py
index f22c565c0..febde1bc2 100644
--- a/bbot/scanner/preset/args.py
+++ b/bbot/scanner/preset/args.py
@@ -466,7 +466,7 @@ class BBOTArgs:
misc.add_argument(
"--user-agent",
"-ua",
- help="Set the user-agent for HTTP requests (most API calls send BBOT/<version>)",
+ help="Set the user-agent for HTTP requests (third-party API calls send BBOT/<version>)",
)
misc.add_argument(
"--user-agent-suffix", "-uas", help="Suffix to append to the user-agent", metavar="SUFFIX", default=None
diff --git a/docs/migration/3.1_breaking_changes.md b/docs/migration/3.1_breaking_changes.md
index d594d1426..ff78cc12a 100644
--- a/docs/migration/3.1_breaking_changes.md
+++ b/docs/migration/3.1_breaking_changes.md
@@ -95,9 +95,13 @@ dnsdumpster no longer has a keyless mode. Set `modules.dnsdumpster.api_key`, or
With `urls: true`, wayback now probes each host once and drops URLs on hosts that don't respond. Expect fewer, but live, URLs from wayback.
+### lightfuzz tries WAF bypasses by default
+
+In 3.0, lightfuzz skipped every WAF-tagged host. In 3.1, `modules.lightfuzz.avoid_wafs` defaults to `try_bypasses`. Set it to `always` for the old behaviour.
+
### API requests identify as BBOT
-Modules that query third-party APIs through BBOT's API request helper now send `User-Agent: BBOT/<version>` instead of the scan's user agent, so `web.user_agent` and `web.user_agent_suffix` no longer apply to those requests. Not every module uses the helper yet, so some third-party requests still send `web.user_agent`; don't rely on `BBOT/<version>` for allowlisting. Requests to your targets still use `web.user_agent`.
+Requests that modules make to third-party APIs now send `User-Agent: BBOT/<version>` instead of the scan's user agent, so `web.user_agent` and `web.user_agent_suffix` no longer apply to them. Requests to your targets still use `web.user_agent`.
---
diff --git a/docs/modules/lightfuzz.md b/docs/modules/lightfuzz.md
index 7395d864a..fe495f6bb 100644
--- a/docs/modules/lightfuzz.md
+++ b/docs/modules/lightfuzz.md
@@ -209,7 +209,7 @@ If the server fetches the injected URL, the resulting DNS or HTTP interaction is
| Value | Behavior |
|---|---|
| `always` | Skip WAF-protected targets entirely |
-| `try_bypasses` (default) | Ask the `nowafpls` helper whether the WAF blocks payloads in the request body, and whether padding the body gets past it. POST body and JSON parameters are fuzzed with padding if it does, unpadded if the WAF isn't blocking them, and skipped otherwise. Cookie and header parameters are skipped, since they have no body to pad. GET parameters are skipped too, unless `try_get_as_post` is on (as in `lightfuzz-heavy` and `lightfuzz-xss`), in which case they are retested as POST bodies and handled like POST parameters above. |
+| `try_bypasses` (default) | Ask the `nowafpls` helper whether the WAF blocks payloads in the request body, and whether padding the body gets past it. POST body and JSON parameters are fuzzed with padding if it does, unpadded if the WAF isn't blocking them, and skipped otherwise. GET, cookie and header parameters are always skipped, since they have no body to pad. |
| `never` | Fuzz WAF-protected targets as-is |
## Presets
@@ -275,7 +275,7 @@ A focused preset for XSS hunting only. Demonstrates how to build a single-submod
|---|---|
| **Submodules** | `xss` only |
| **Companion modules** | `paramminer_getparams`, `reflected_parameters` |
-| **POST fuzzing** | Enabled, and `try_get_as_post` is on (GET params retested as POST) |
+| **POST fuzzing** | Disabled |
| **Query string collapsing** | Off |
| **WAF avoidance** | Try bypasses (`avoid_wafs: try_bypasses`) |
diff --git a/docs/release_history.md b/docs/release_history.md
index d6487f684..818d4c5c8 100644
--- a/docs/release_history.md
+++ b/docs/release_history.md
@@ -13,7 +13,7 @@
#### Highlights
-- **DNS brute-forcing moves to BlastDNS:** massdns is gone, and brute-forcing runs through blastdns 2.0 with adaptive rate control. With higher default DNS concurrency, a typical `subdomain-enum` scan finishes in about 40% less time than on 3.0.2, with the same subdomains found ([#3457](https://github.com/blacklanternsecurity/bbot/pull/3457), [#3606](https://github.com/blacklanternsecurity/bbot/pull/3606), [#3612](https://github.com/blacklanternsecurity/bbot/pull/3612))
+- **DNS brute-forcing moves to BlastDNS:** massdns is gone, and brute-forcing runs through blastdns 2.0 with adaptive rate control. ([#3457](https://github.com/blacklanternsecurity/bbot/pull/3457), [#3612](https://github.com/blacklanternsecurity/bbot/pull/3612)). With the higher default DNS concurrency from [#3606](https://github.com/blacklanternsecurity/bbot/pull/3606), a `-f subdomain-enum -m http` scan of a large wildcard target (crt and crt_db excluded) took 11m 51s against 22m 50s on 3.0.2, and found 513 in-scope subdomains against 510
- **New `--custom-nameservers` flag** for choosing the resolvers BBOT uses for normal resolution ([#3457](https://github.com/blacklanternsecurity/bbot/pull/3457))
- **baddns 2.5.0**, with a new `DELEGATION` submodule that is on by default ([#3457](https://github.com/blacklanternsecurity/bbot/pull/3457))
- **CDN/WAF port filtering in core, on by default** ([#3534](https://github.com/blacklanternsecurity/bbot/pull/3534))
diff --git a/docs/scanning/advanced.md b/docs/scanning/advanced.md
index 3496cd361..2af69f85c 100644
--- a/docs/scanning/advanced.md
+++ b/docs/scanning/advanced.md
@@ -148,7 +148,7 @@ Misc:
--custom-yara-rules, -cy CUSTOM_YARA_RULES
Add custom yara rules to excavate
--user-agent, -ua USER_AGENT
- Set the user-agent for HTTP requests (most API calls send BBOT/<version>)
+ Set the user-agent for HTTP requests (third-party API calls send BBOT/<version>)
--user-agent-suffix, -uas SUFFIX
Suffix to append to the user-agent
|
|
||
| #### Highlights | ||
|
|
||
| - **DNS brute-forcing moves to BlastDNS:** massdns is gone, and brute-forcing runs through blastdns 2.0 with adaptive rate control. With higher default DNS concurrency, a typical `subdomain-enum` scan finishes in about 40% less time than on 3.0.2, with the same subdomains found ([#3457](https://github.com/blacklanternsecurity/bbot/pull/3457), [#3606](https://github.com/blacklanternsecurity/bbot/pull/3606), [#3612](https://github.com/blacklanternsecurity/bbot/pull/3612)) |
|
|
||
| With `urls: true`, wayback now probes each host once and drops URLs on hosts that don't respond. Expect fewer, but live, URLs from wayback. | ||
|
|
||
| ### API requests identify as BBOT |
There was a problem hiding this comment.
🔴 Add an entry here: 3.0 lightfuzz skipped every WAF-tagged host; 3.1 defaults modules.lightfuzz.avoid_wafs to try_bypasses and the lightfuzz presets add nowafpls probes. Old behaviour: avoid_wafs: always plus -em nowafpls.
Contributor
Author
There was a problem hiding this comment.
Added under Modules in 52bc5f0, with the WAF Handling link and a "Check your setup" bullet in cd2bf7f. It covers the try_bypasses default (every WAF-tagged host is probed, but only fuzzed if the probe gets past the WAF), the nowafpls module in the lightfuzz presets, lightfuzz-max fuzzing WAF-tagged hosts without padding, and the top-level avoid_wafs no longer being read. It also gives avoid_wafs: always plus -em nowafpls for the old behavior. The lightfuzz.yml and lightfuzz-heavy.yml descriptions are updated too.
en0f
requested changes
Nov 10, 2026
en0f
left a comment
en0f
left a comment
There was a problem hiding this comment.
Reviewed at c068b08.
🔴 The 3.1.0 speed highlight still doesn't match its measurement
Still open from rounds 2 and 3: line 17 is unchanged and still says a typical subdomain-enum scan takes about 40% less time. #3606 is the only cited PR that timed a scan: one large wildcard target, -f subdomain-enum -m http, crt and crt_db excluded, 22m 50s on 3.0.2 against 11m 51s, 510 against 513 subdomains. The wording suggested on line 17 in round 2 still fixes it. #3584's sentence works too: on a large wildcard subdomain-enum scan, #3606's higher DNS concurrency halved wall time compared with 3.0.2.
🔴 never still pads request bodies, but the docs say it fuzzes without padding
The migration table says never means "fuzz without padding" and that lightfuzz-max "fuzzes them without padding" (lines 93-94), and the two never rows in lightfuzz.md say "as-is". With never, prepare_request still pads every POST and JSON probe to a WAF-tagged host through nowafpls.pad_json whenever padding gets past the WAF (submodules/base.py:168 skips only on always), with 128 KB to 1 MB of padding, and Test_Nowafpls_never_still_pads asserts it. So lightfuzz-max sends padded bodies to WAF-fronted hosts. Say that never drops the probe gate but still pads request bodies where padding gets through, in all four places. Optionally fix the avoid_wafs Field description in lightfuzz.py ("fuzzes them raw") and rerun docs.py so configuration.md agrees.
🔴 The migration guide doesn't say lightfuzz now fuzzes WAF-protected hosts
Resolved in 52bc5f0 and cd2bf7f, and unchanged by c068b08. The entry, the old-behaviour recipe (avoid_wafs=always plus -em nowafpls), the "Check your setup" bullet and both preset descriptions hold against v3.0.2 and this head, apart from the never wording above.
Reproduce this review
# save the script below as verify-3613.sh and the patch as control-3613.patch, then:
bash verify-3613.sh blacklanternsecurity/bbot c068b085a2dad9e5422ddbdf2df31d534c43d12e
# expected: SUMMARY 24 PASS, 3 FAIL, 0 SKIP, 0 BROKEN (V15 speed highlight; V28 never wording; V30 Dependencies vs #3627, informational)
# control: bash verify-3613.sh blacklanternsecurity/bbot c068b085a2dad9e5422ddbdf2df31d534c43d12e "$PWD/control-3613.patch"
# expected: SUMMARY 20 PASS, 7 FAIL, 0 SKIP, 0 BROKEN (the patch puts fedf360c back for the lightfuzz presets, migration guide and release history, so V19 V21 V22 V23 V25 V27 fail and V30 still fails; it applies the suggested speed wording and rewords the never rows in lightfuzz.md, so V15 V28 pass)#!/usr/bin/env bash
# verify-3613.sh: reproducible checks for blacklanternsecurity/bbot#3613
# Usage: bash verify-3613.sh <owner/repo> <sha> [<control-patch>]
# Changes nothing outside a mktemp directory and nothing on GitHub (PR-VF-02).
#
# Round 5, head c068b085 (round 4 at cd2bf7ff was never posted: the head moved). The control patch puts the round-3 head (fedf360c) back for the
# two lightfuzz preset files, the migration guide and the release history, then applies the
# 3.1.0 speed wording suggested in rounds 2 and 3, and rewrites lightfuzz.md's two `never` rows to
# say bodies are still padded (challenge C7). So the F14 checks must fail there and the
# F10 check must pass there.
# CONTROL: patch control-3613.patch
# CONTROL-EXPECT: V15=PASS V19=FAIL V21=FAIL V22=FAIL V23=FAIL V25=FAIL V27=FAIL V28=PASS
set -uo pipefail
REPO="${1:?owner/repo}"; SHA="${2:?sha}"; PATCH="${3:-}"
PASS=0; FAIL=0; SKIP=0; BROKEN=0
pass() { echo "PASS $1 $2"; PASS=$((PASS+1)); }
fail() { echo "FAIL $1 $2 -- $3"; FAIL=$((FAIL+1)); }
skip() { echo "SKIP $1 $2 -- $3"; SKIP=$((SKIP+1)); }
broken() { echo "BROKEN $1 $2 -- $3"; BROKEN=$((BROKEN+1)); }
summary() { echo "SUMMARY $PASS PASS, $FAIL FAIL, $SKIP SKIP, $BROKEN BROKEN"; [ "$FAIL" -eq 0 ] && [ "$BROKEN" -eq 0 ]; }
echo "script_sha256 $(shasum -a 256 "$0" | cut -d' ' -f1)"
echo "git $(git --version | cut -d' ' -f3)"
echo "gh $(gh --version | head -1 | cut -d' ' -f3)"
WORK="$(mktemp -d)"; trap 'rm -rf "$WORK"' EXIT
export GIT_CONFIG_GLOBAL=/dev/null GIT_CONFIG_NOSYSTEM=1 GIT_TERMINAL_PROMPT=0
GIT=(git -c credential.helper= -c 'credential.helper=!gh auth git-credential')
"${GIT[@]}" clone --quiet --filter=blob:none --no-checkout "https://github.com/$REPO.git" "$WORK/src" || { broken V00 "clone" "clone failed"; summary; exit 1; }
cd "$WORK/src"
"${GIT[@]}" cat-file -e "$SHA^{commit}" 2>/dev/null || "${GIT[@]}" fetch --quiet origin "+refs/pull/*/head:refs/remotes/pr/*" || true
"${GIT[@]}" checkout --quiet --detach "$SHA" || { broken V00 "checkout $SHA" "checkout failed"; summary; exit 1; }
[ "$(git rev-parse HEAD)" = "$SHA" ] || { broken V00 "HEAD is $SHA" "HEAD mismatch"; summary; exit 1; }
if [ -n "$PATCH" ]; then git apply "$PATCH" || { broken V00 "apply control patch" "git apply failed"; summary; exit 1; }; fi
# ---- checks: each line states the desired property --------------------------
# Modules that call a third-party API with self.helpers.request() rather than api_request(),
# so they send the scan's web.user_agent, not BBOT/<version>.
OFFENDERS=""
for f in bbot/modules/urlscan.py bbot/modules/dockerhub.py bbot/modules/templates/postman.py bbot/modules/wayback.py; do
[ -f "$f" ] || continue
if grep -qE '(base_url|api_url|site_url)\s*=\s*f?"https?://' "$f" && grep -qE 'self\.helpers\.request\(' "$f"; then
OFFENDERS="$OFFENDERS $(basename "$f" .py)"
fi
done
if grep -q 'kwargs\["headers"\]\["User-Agent"\] = f"BBOT/{__version__}"' bbot/modules/base.py; then
pass V11 "api_request() sends User-Agent BBOT/<version>"
else
fail V11 "api_request() sends User-Agent BBOT/<version>" "base.py pattern not found"
fi
MIG=docs/migration/3.1_breaking_changes.md
UA_SECTION=$(awk '/^### API requests identify as BBOT/{s=1;next} /^(---|### )/{s=0} s' "$MIG")
QUAL='not every module|some third-party|api_request|API request helper|not all'
ua_unqualified() { [ -n "$UA_SECTION" ] && echo "$UA_SECTION" | grep -q 'BBOT/<version>' && ! echo "$UA_SECTION" | grep -qiE "$QUAL"; }
if ua_unqualified && [ -n "$OFFENDERS" ]; then
fail V01 "3.1 migration note's User-Agent claim matches the modules" "claims all third-party API requests send BBOT/<version>; still on web.user_agent:$OFFENDERS"
else
pass V01 "3.1 migration note's User-Agent claim matches the modules"
fi
help_unqualified() { grep -E 'BBOT/<version>' "$1" | grep -i 'user-agent' | grep -viqE '\((most|some)[^)]*BBOT/<version>'; }
if [ -n "$OFFENDERS" ] && { help_unqualified bbot/scanner/preset/args.py || help_unqualified docs/scanning/advanced.md; }; then
fail V02 "--user-agent help text matches the modules" "args.py or advanced.md says API calls send BBOT/<version> without qualification; still on web.user_agent:$OFFENDERS"
else
pass V02 "--user-agent help text matches the modules"
fi
LF=bbot/modules/lightfuzz/lightfuzz.py
WAF_ROW=$(grep -F '`try_bypasses` (default)' docs/modules/lightfuzz.md)
if echo "$WAF_ROW" | grep -q 'GET' && ! echo "$WAF_ROW" | grep -q 'try_get_as_post' \
&& grep -q 'if self.try_get_as_post and original_type == "GETPARAM":' "$LF" \
&& grep -q 'try_get_as_post: True' bbot/presets/web/lightfuzz-xss.yml; then
fail V03 "lightfuzz WAF Handling describes GET params under try_get_as_post" "doc says GET params are always skipped; lightfuzz.py retests them as POST in bypass mode and lightfuzz-xss turns that on"
else
pass V03 "lightfuzz WAF Handling describes GET params under try_get_as_post"
fi
if grep -q 'requires-python = ">=3.11' pyproject.toml && grep -q 'badge/python-3.11+' README.md && grep -q 'BBOT requires Python 3.11+' docs/troubleshooting.md; then
pass V04 "README badge and troubleshooting say 3.11+, matching requires-python"
else
fail V04 "README badge and troubleshooting say 3.11+, matching requires-python" "mismatch"
fi
if grep -qE '^ threads: 50$' bbot/defaults.yml && grep -qE '^ max_concurrency: 256$' bbot/defaults.yml \
&& grep -q 'default 50 queries per resolver' docs/scanning/tips_and_tricks.md && grep -q '256 total' docs/scanning/tips_and_tricks.md; then
pass V05 "DNS speed tip matches dns.threads=50 and dns.max_concurrency=256"
else
fail V05 "DNS speed tip matches dns.threads=50 and dns.max_concurrency=256" "mismatch"
fi
if ! grep -qE '^\s+threads:' bbot/presets/subdomain-enum.yml && ! grep -q 'threads: 25' README.md && ! grep -q 'threads: 25' docs/scanning/presets_list.md; then
pass V06 "subdomain-enum no longer lowers dns.threads, and README/presets_list match"
else
fail V06 "subdomain-enum no longer lowers dns.threads, and README/presets_list match" "threads still set somewhere"
fi
CC=bbot/modules/internal/cloudcheck.py
if grep -q 'event.add_tag(provider_type)' "$CC" && grep -q 'event.add_tag(provider_name)' "$CC" \
&& ! grep -rqE 'cdn-github|cloud-azure|cdn-akamai|cdn-ip' docs/scanning/output.md docs/scanning/events.md docs/modules/custom_yara_rules.md; then
pass V07 "tag examples use cloudcheck's separate type and provider tags"
else
fail V07 "tag examples use cloudcheck's separate type and provider tags" "old cdn-x tags remain or code differs"
fi
ok=1
grep -q 'avoid_wafs: always' bbot/presets/web/lightfuzz-light.yml || ok=0
grep -q 'avoid_wafs: try_bypasses' bbot/presets/web/lightfuzz.yml || ok=0
grep -q 'avoid_wafs: never' bbot/presets/web/lightfuzz-max.yml || ok=0
grep -q 'avoid_wafs: try_bypasses' bbot/presets/web/lightfuzz-xss.yml || ok=0
grep -q 'avoid_wafs: Literal\["always", "never", "try_bypasses"\]' "$LF" || ok=0
if [ "$ok" = 1 ]; then pass V08 "lightfuzz preset tables match each preset's avoid_wafs"; else fail V08 "lightfuzz preset tables match each preset's avoid_wafs" "mismatch"; fi
if grep -q '"--custom-nameservers"' bbot/scanner/preset/args.py && [ -f bbot/presets/nuclei/nuclei-recommended.yml ] && [ -f bbot/presets/nuclei/nuclei-new.yml ]; then
pass V09 "--custom-nameservers, nuclei-recommended and nuclei-new exist as documented"
else
fail V09 "--custom-nameservers, nuclei-recommended and nuclei-new exist as documented" "missing"
fi
if N=$(gh api --method GET "repos/blacklanternsecurity/public-dns-servers/contents/nameservers.txt?ref=master" -H "Accept: application/vnd.github.raw+json" 2>/dev/null | grep -cvE '^\s*(#|$)'); then
if [ "$N" -ge 4000 ] && [ "$N" -le 6000 ]; then pass V10 "brute_nameservers list is ~5K servers ($N)"; else fail V10 "brute_nameservers list is ~5K servers" "count $N"; fi
else
skip V10 "brute_nameservers list is ~5K servers" "could not fetch nameservers.txt"
fi
# C13 (A5): generated blocks match their sources
if python3 - <<'PY'
p=open('bbot/presets/subdomain-enum.yml').read().strip(); r=open('README.md').read()
l=open('docs/scanning/presets_list.md').read(); c=open('docs/scanning/configuration.md').read(); d=open('bbot/defaults.yml').read().strip()
import re,sys; a=open('bbot/scanner/preset/args.py').read(); adv=open('docs/scanning/advanced.md').read()
h=re.search(r'"--user-agent",\s*"-ua",\s*help="([^"]+)"', a)
sys.exit(0 if (p in r and "\n".join(" "+x for x in p.split("\n")) in l and d in c and h and h.group(1) in adv) else 1)
PY
then pass V12 "generated README, presets_list, configuration and advanced.md blocks match their sources"
else fail V12 "generated README, presets_list, configuration and advanced.md blocks match their sources" "a generated block is stale"; fi
# C16/C17: lightfuzz-xss table's POST row matches the preset
XSS_TABLE=$(awk '/^### .-p lightfuzz-xss./{s=1} /^### Spider/{s=0} s' docs/modules/lightfuzz.md)
if echo "$XSS_TABLE" | grep -q 'POST fuzzing\*\* | Disabled' && ! grep -qE 'disable_post: *True' bbot/presets/web/lightfuzz-xss.yml; then
fail V13 "lightfuzz-xss table's POST fuzzing row matches the preset" "table says Disabled; preset leaves disable_post False and sets try_get_as_post"
else
pass V13 "lightfuzz-xss table's POST fuzzing row matches the preset"
fi
# C3/C14 (A6): more modules call third-party APIs outside api_request()
MORE=""
for m in azure_tenant docker_pull; do
grep -qE 'self\.helpers\.request\(' "bbot/modules/$m.py" && grep -qE 'https://(azmap\.dev|registry-1\.docker\.io|login\.microsoftonline\.com)' "bbot/modules/$m.py" && MORE="$MORE $m"
done
if ua_unqualified && [ -n "$MORE" ]; then
fail V14 "3.1 migration note holds for azure_tenant and docker_pull" "unqualified claim; still on web.user_agent:$MORE"
else
pass V14 "3.1 migration note holds for azure_tenant and docker_pull"
fi
# ---- round 2: the 3.1.0 release history (fedf360c) ----------------------------
RH=docs/release_history.md
SEC=$(awk '/^### 3\.1\.0 /{s=1;print;next} /^### [0-9]/{s=0} s' "$RH")
# V15: the speed claim matches the measurement it cites (#3606 "Effect on a scan").
B3606=$(gh api --method GET repos/blacklanternsecurity/bbot/pulls/3606 --jq .body 2>/dev/null)
if [ -z "$SEC" ] || [ -z "$B3606" ]; then
skip V15 "3.1.0 speed claim matches #3606's measurement" "no 3.1.0 section or #3606 unreadable"
elif ! echo "$B3606" | grep -q '| wall time | 22m 50s | 11m 51s |' || ! echo "$B3606" | grep -q 'against a large wildcard target'; then
broken V15 "3.1.0 speed claim matches #3606's measurement" "#3606 no longer carries the measurement this check reads"
else
# C4/R2C1/R2C2: every speed claim in the section must carry #3606's measurement and its target:
# its times (22m 50s, 11m 51s) or its ratio (48%, half, halved), plus "large wildcard"; no other figure, no "typical".
BAD=""
while IFS= read -r L; do
[ -z "$L" ] && continue
M=0; { echo "$L" | grep -q '22m 50s' && echo "$L" | grep -q '11m 51s'; } && M=1
echo "$L" | grep -qiE '48 ?%|half|halve[sd]?|twice as fast|(^|[^0-9.])2 ?x (faster|quicker)' && M=1 # C3 (r5): accept accurate rewordings
PCT=$(echo "$L" | grep -oE '[0-9]+ ?%' | grep -vE '^48 ?%$')
if [ "$M" = 0 ] || ! echo "$L" | grep -qi 'large wildcard' || [ -n "$PCT" ] || echo "$L" | grep -qi 'typical'; then BAD="$BAD | ${L:0:120}"; fi
done <<< "$(echo "$SEC" | grep -iE 'faster|quicker|slower|less time|as long|wall time|halve[sd]?|twice|double[sd]?|took|[0-9]+m [0-9]+s|[0-9]+ ?% (less|more|quicker)|finish(es)? in' | grep -iE 'scan')"
if [ -z "$BAD" ]; then pass V15 "3.1.0 speed claims match #3606's measurement"
else fail V15 "3.1.0 speed claims match #3606's measurement" "unmeasured speed claim:$BAD; #3606: one large wildcard target, 22m 50s to 11m 51s (48% less), 510 to 513 subdomains"; fi
fi
# ---- round 3 -------------------------------------------------------------------
# V19: lightfuzz-heavy (includes lightfuzz -> avoid_wafs: try_bypasses; disable_post False, try_get_as_post True)
# does not describe itself as skipping WAFs in its preset file (what `bbot -lp` prints; presets_list.md is generated from it).
H=bbot/presets/web/lightfuzz-heavy.yml
eff=$(grep -oE 'avoid_wafs: *[a-z_]+' "$H" | awk '{print $2}')
[ -z "$eff" ] && grep -qE '^ - lightfuzz$' "$H" && eff=$(grep -oE 'avoid_wafs: *[a-z_]+' bbot/presets/web/lightfuzz.yml | awk '{print $2}')
if [ -z "$eff" ]; then
broken V19 "lightfuzz-heavy's description matches its avoid_wafs" "could not resolve avoid_wafs for lightfuzz-heavy"
elif [ "$eff" = try_bypasses ] && grep -q 'disable_post: False' "$H" && grep -m1 '^description:' "$H" | grep -qi 'skips confirmed WAFs'; then
fail V19 "lightfuzz-heavy's description matches its avoid_wafs" "says it still skips confirmed WAFs; it runs try_bypasses with POST fuzzing on"
else
pass V19 "lightfuzz-heavy's description matches its avoid_wafs"
fi
# V20: each lightfuzz preset's "WAF avoidance" row in lightfuzz.md names the avoid_wafs value its YAML sets.
LFMD=docs/modules/lightfuzz.md; BAD=""; N=0
for p in lightfuzz-light lightfuzz lightfuzz-max lightfuzz-xss; do
eff=$(grep -oE 'avoid_wafs: *[a-z_]+' "bbot/presets/web/$p.yml" | awk '{print $2}')
row=$(awk -v h="### \`-p $p\`" '$0==h{s=1;next} /^### /{s=0} s && /\*\*WAF avoidance\*\*/' "$LFMD")
N=$((N+1))
if [ -z "$eff" ] || [ -z "$row" ] || ! echo "$row" | grep -q "avoid_wafs: $eff"; then BAD="$BAD $p(yaml=$eff)"; fi
done
if [ -z "$BAD" ]; then pass V20 "lightfuzz.md WAF avoidance rows match each preset's avoid_wafs ($N presets)"
else fail V20 "lightfuzz.md WAF avoidance rows match each preset's avoid_wafs" "row missing or wrong:$BAD"; fi
# V21: v3.0.2 lightfuzz skipped every WAF-tagged host (top-level avoid_wafs, default True); this head defaults
# to try_bypasses. The 3.1 migration guide's Modules section says so (C6: any heading).
"${GIT[@]}" fetch --quiet origin tag v3.0.2 2>/dev/null || true
OLD=$(git show v3.0.2:bbot/modules/lightfuzz/lightfuzz.py 2>/dev/null | grep -c 'self.avoid_wafs = self.scan.config.get("avoid_wafs", True)')
NEW=$(grep -c '"try_bypasses",' bbot/modules/lightfuzz/lightfuzz.py)
if [ "$OLD" != 1 ] || [ "$NEW" -lt 1 ]; then
broken V21 "3.1 migration guide documents lightfuzz's WAF default change" "could not read v3.0.2 or head lightfuzz.py"
elif awk '/^## /{m=($0 ~ /^## Modules/)} m && /avoid_wafs/' docs/migration/3.1_breaking_changes.md | grep -q .; then
pass V21 "3.1 migration guide documents lightfuzz's WAF default change"
else
fail V21 "3.1 migration guide documents lightfuzz's WAF default change" "3.0.2 skipped every WAF host; head defaults to try_bypasses; no lightfuzz/avoid_wafs entry in the guide"
fi
# ---- round 4 (cd2bf7ff) ----------------------------------------------------------
LFSEC=$(awk '/^### `lightfuzz` probes WAF/{s=1;next} /^### /{s=0} s' "$MIG")
# V22: the entry's 3.0 -> 3.1 table matches v3.0.2 and this head.
O_LF=$(git show v3.0.2:bbot/modules/lightfuzz/lightfuzz.py 2>/dev/null)
O_MAX=$(git show v3.0.2:bbot/presets/web/lightfuzz-max.yml 2>/dev/null)
if [ -z "$O_LF" ] || [ -z "$O_MAX" ]; then
broken V22 "migration WAF table matches v3.0.2 and this head" "could not read v3.0.2"
else
ok=1; why=""
echo "$O_LF" | grep -q 'self.avoid_wafs = self.scan.config.get("avoid_wafs", True)' || { ok=0; why="$why v3.0.2-not-top-level"; }
echo "$O_MAX" | grep -qE '^ avoid_wafs: False' || { ok=0; why="$why v3.0.2-max-no-False"; }
grep -qE '^ avoid_wafs: never' bbot/presets/web/lightfuzz-max.yml || { ok=0; why="$why head-max-not-never"; }
for s in 'no longer read' '`lightfuzz-max` still skipped WAF-tagged hosts' '`lightfuzz-max` sets `never`' '`true` / `false` still work as `always` / `never`' 'defaults to `try_bypasses`'; do
echo "$LFSEC" | grep -qF "$s" || { ok=0; why="$why doc-missing:[$s]"; }
done
if [ "$ok" = 1 ]; then pass V22 "migration WAF table matches v3.0.2 and this head"
else fail V22 "migration WAF table matches v3.0.2 and this head" "$why"; fi
fi
# V23: the old-behaviour recipe works, and the entry names exactly the presets that pull in nowafpls.
incl() { awk '/^include:/{s=1;next} /^[a-z]/{s=0} s && /^ - /{print $2}' "bbot/presets/web/$1.yml"; }
mods() { awk '/^modules:/{s=1;next} /^[a-z]/{s=0} s && /^ - /{print $2}' "bbot/presets/web/$1.yml"; }
has_nowafpls() { local p="$1" d="${2:-0}"; [ "$d" -gt 5 ] && return 1; [ -f "bbot/presets/web/$p.yml" ] || return 1
{ incl "$p"; mods "$p"; } | grep -qx nowafpls && return 0
for i in $(incl "$p"); do has_nowafpls "$i" $((d+1)) && return 0; done; return 1; }
ACT=$(for f in bbot/presets/web/lightfuzz*.yml; do p=$(basename "$f" .yml); has_nowafpls "$p" && echo "$p"; done | sort | tr '\n' ' ')
DOC=$(echo "$LFSEC" | grep -F 'also enable the `nowafpls` module' | grep -oE '`lightfuzz[a-z-]*`' | tr -d '`' | sort -u | tr '\n' ' ')
ok=1; why=""
[ -n "$DOC" ] && [ "$ACT" = "$DOC" ] || { ok=0; why="$why presets-with-nowafpls=[$ACT] doc=[$DOC]"; }
echo "$LFSEC" | grep -qF -- '-c modules.lightfuzz.avoid_wafs=always -em nowafpls' || { ok=0; why="$why recipe-missing"; }
grep -q 'if self.avoid_wafs == "always":' "$LF" && [ -f bbot/modules/nowafpls.py ] || { ok=0; why="$why code"; }
grep -qE '^ avoid_wafs: always' bbot/presets/web/lightfuzz-light.yml && echo "$LFSEC" | grep -qF '`lightfuzz-light` still skips WAF-tagged hosts' || { ok=0; why="$why light"; }
if [ "$ok" = 1 ]; then pass V23 "old-behaviour recipe and nowafpls preset list match the presets"
else fail V23 "old-behaviour recipe and nowafpls preset list match the presets" "$why"; fi
# V24: avoid_wafs still accepts the boolean form, as the entry says.
if awk '/def _normalize_avoid_wafs/{s=1} s && n++<6' "$LF" | grep -q 'return "always" if v else "never"'; then
pass V24 "modules.lightfuzz.avoid_wafs maps true/false to always/never"
else fail V24 "modules.lightfuzz.avoid_wafs maps true/false to always/never" "validator not found"; fi
# V25: lightfuzz.yml and lightfuzz-heavy.yml (try_bypasses) don't say they skip WAFs, and presets_list.md
# carries every lightfuzz preset description verbatim (it is generated from them).
ok=1; why=""
for p in lightfuzz lightfuzz-heavy; do grep -m1 '^description:' "bbot/presets/web/$p.yml" | grep -qiE 'skips confirmed WAFs' && { ok=0; why="$why $p-says-skips"; }; done
for f in bbot/presets/web/lightfuzz*.yml; do
d=$(grep -m1 '^description:' "$f" | sed -E 's/^description: "(.*)"$/\1/')
grep -qF -- "$d" docs/scanning/presets_list.md || { ok=0; why="$why stale:$(basename "$f" .yml)"; }
done
if [ "$ok" = 1 ]; then pass V25 "lightfuzz preset descriptions match try_bypasses and presets_list.md"
else fail V25 "lightfuzz preset descriptions match try_bypasses and presets_list.md" "$why"; fi
CHANGED="$WORK/changed.txt"
printf '%s\n' README.md docs/migration/3.1_breaking_changes.md docs/modules/custom_yara_rules.md docs/modules/lightfuzz.md docs/modules/nuclei.md docs/modules/wayback.md docs/release_history.md docs/scanning/advanced.md docs/scanning/configuration.md docs/scanning/events.md docs/scanning/output.md docs/scanning/presets.md docs/scanning/presets_list.md docs/scanning/tips_and_tricks.md docs/troubleshooting.md > "$CHANGED"; export CHANGED
# V26: relative links in the changed docs resolve, including #anchors to headings (static stand-in for a docs build).
BAD=$(python3 - <<'PY'
import re, os, subprocess
files = [l for l in open(os.environ["CHANGED"]).read().split() if l.startswith("docs/") and l.endswith(".md")]
def slug(h): return re.sub(r"[^\w\- ]", "", h.strip().lower()).replace(" ", "-")
bad = []
for f in files:
for m in re.finditer(r"\]\(([^)\s]+)\)", open(f).read()):
t = m.group(1)
# skip schemes, and the generator's placeholder [0]("") links (37 of them at base too)
if re.match(r"^[a-z]+:", t) or t.strip('"') == "": continue
path, _, anc = t.partition("#")
tgt = os.path.normpath(os.path.join(os.path.dirname(f), path)) if path else f
if not os.path.exists(tgt): bad.append(f"{f}->{t}"); continue
if anc and tgt.endswith(".md"):
heads = {slug(re.sub(r"^#+\s*", "", l)) for l in open(tgt) if l.startswith("#")}
if anc not in heads: bad.append(f"{f}->{t}")
print(" ".join(bad))
PY
)
if [ -z "$BAD" ]; then pass V26 "relative links and anchors in the changed docs resolve"
else fail V26 "relative links and anchors in the changed docs resolve" "$BAD"; fi
# V27: the 3.1.0 'Check your setup' box flags the lightfuzz WAF change with the same recipe as the guide.
BOX=$(awk '/^!!! warning "Check your setup before upgrading"/{s=1;next} /^#### /{s=0} s' "$RH")
if echo "$BOX" | grep -qi 'lightfuzz' && echo "$BOX" | grep -qF 'modules.lightfuzz.avoid_wafs: always' && echo "$BOX" | grep -qF -- '-em nowafpls'; then
pass V27 "3.1.0 Check-your-setup box flags the lightfuzz WAF change"
else fail V27 "3.1.0 Check-your-setup box flags the lightfuzz WAF change" "no lightfuzz WAF bullet with the recipe"; fi
# V28 (challenge C7): no doc row says avoid_wafs=never fuzzes unpadded / as-is, since under never
# prepare_request still pads POST/JSON bodies on WAF hosts (submodules/base.py) and the repo's
# Test_Nowafpls_never_still_pads asserts it.
BASEPY=bbot/modules/lightfuzz/submodules/base.py
if ! grep -q 'if "waf" in self.event.tags and self.lightfuzz.avoid_wafs != "always":' "$BASEPY" \
|| ! grep -q 'class Test_Nowafpls_never_still_pads' bbot/test/test_step_2/module_tests/test_module_lightfuzz.py; then
pass V28 "docs describe avoid_wafs=never the way the code pads" "(code no longer pads under never)"
else
BAD=""
echo "$LFSEC" | grep -iE '`never`' | grep -qiE 'without padding|as-is|\braw\b|unpadded' && BAD="$BAD migration-guide"
grep -E '^\| `never` \|' "$LFMD" | grep -qiE 'as-is|raw|without padding' && BAD="$BAD lightfuzz.md-WAF-table"
grep -F 'avoid_wafs: never' "$LFMD" | grep -qiE 'as-is|raw|without padding' && BAD="$BAD lightfuzz.md-max-preset-row"
if [ -z "$BAD" ]; then pass V28 "docs describe avoid_wafs=never the way the code pads"
else fail V28 "docs describe avoid_wafs=never the way the code pads" "says unpadded/as-is:$BAD; base.py pads POST/JSON bodies on WAF hosts unless avoid_wafs=always"; fi
fi
# V29 (round 5): the #3629 lines in 3.1.0 match #3629 as merged to dev: FINDING gains an optional port
# rendered host:port, legba and medusa pass it, and nuclei correlation is fixed.
M3629=3085baacf1f812fcf446d73f673aa2454b199ce4
"${GIT[@]}" fetch --quiet origin "$M3629" 2>/dev/null || true
MA=$(gh api --method GET repos/blacklanternsecurity/bbot/pulls/3629 --jq '.merged_at // "no"' 2>/dev/null) || MA=err
if ! git cat-file -e "$M3629^{commit}" 2>/dev/null || [ "$MA" = err ]; then
skip V29 "3.1.0 #3629 lines match #3629" "could not read #3629 or its merge commit"
else
ok=1; why=""
[ "$MA" != no ] || { ok=0; why="$why not-merged"; }
git show "$M3629:bbot/core/event/base.py" | grep -q 'port: Optional\[int\] = None' || { ok=0; why="$why no-FINDING-port"; }
git show "$M3629:bbot/core/event/base.py" | grep -q 'make_netloc(host, port) if port else host' || { ok=0; why="$why no-host:port"; }
for m in legba medusa; do git show "$M3629:bbot/modules/$m.py" | grep -q '"port": str(event.port)' || { ok=0; why="$why $m"; }; done
git show "$M3629^1:bbot/modules/nuclei.py" | grep -q 'def correlate_event' && ! git show "$M3629:bbot/modules/nuclei.py" | grep -q 'def correlate_event' || { ok=0; why="$why nuclei-correlate"; }
grep -q 'FINDING has an optional `port`.*legba and medusa.*pull/3629' "$RH" && grep -q 'nuclei:.*Results are parented.*pull/3629' "$RH" || { ok=0; why="$why doc-lines-missing"; }
if [ "$ok" = 1 ]; then pass V29 "3.1.0 #3629 lines match #3629 as merged"; else fail V29 "3.1.0 #3629 lines match #3629 as merged" "$why"; fi
fi
# V30 (challenge C17, round 5): the 3.1.0 Dependencies section reflects #3627 (merged to dev before #3629,
# whose notes c068b085 added): pyjwt, lxml and socksio dropped, xmltojson replaced by xmltodict.
M3627=5d968b1c421eb0568f19abaaccb0c7fc92130a32
"${GIT[@]}" fetch --quiet origin "$M3627" 2>/dev/null || true
if ! git cat-file -e "$M3627^{commit}" 2>/dev/null; then
skip V30 "3.1.0 Dependencies reflects #3627" "could not fetch #3627's merge"
elif ! git diff "$M3627^1" "$M3627" -- pyproject.toml | grep -q '^+ "xmltodict'; then
broken V30 "3.1.0 Dependencies reflects #3627" "#3627 no longer shows the xmltodict swap"
else
DEP=$(awk '/^### 3\.1\.0 /{s=1} /^### [0-9]/ && !/3\.1\.0/{s=0} s' "$RH" | awk '/^#### Dependencies/{s=1;next} /^#### /{s=0} s')
if echo "$DEP" | grep -qiE 'xmltodict|pyjwt|lxml|socksio|3627'; then pass V30 "3.1.0 Dependencies reflects #3627"
else fail V30 "3.1.0 Dependencies reflects #3627" "no mention of #3627's removals (pyjwt, lxml, socksio) or the xmltojson -> xmltodict swap"; fi
fi
# ------------------------------------------------------------------------------
summarydiff --git a/bbot/presets/web/lightfuzz-heavy.yml b/bbot/presets/web/lightfuzz-heavy.yml
index 3bc6dac1d..c67474af1 100644
--- a/bbot/presets/web/lightfuzz-heavy.yml
+++ b/bbot/presets/web/lightfuzz-heavy.yml
@@ -1,4 +1,4 @@
-description: "Aggressive fuzzing: everything in lightfuzz, plus paramminer brute-force parameter discovery (headers, GET params, cookies), POST request fuzzing enabled, try_get_as_post enabled (GET params retested as POST), and robots.txt parsing. On WAF-tagged hosts, only request-body parameters are fuzzed, and only if the WAF doesn't block them or nowafpls body padding gets them past it."
+description: "Aggressive fuzzing: everything in lightfuzz, plus paramminer brute-force parameter discovery (headers, GET params, cookies), POST request fuzzing enabled, try_get_as_post enabled (GET params retested as POST), and robots.txt parsing. Still skips confirmed WAFs."
include:
- lightfuzz
diff --git a/bbot/presets/web/lightfuzz.yml b/bbot/presets/web/lightfuzz.yml
index d3d8259e6..3ae97c8c7 100644
--- a/bbot/presets/web/lightfuzz.yml
+++ b/bbot/presets/web/lightfuzz.yml
@@ -1,4 +1,4 @@
-description: "Default fuzzing: all 9 submodules (cmdi, crypto, path, serial, sqli, ssti, xss, esi, ssrf) plus companion modules (badsecrets, hunt, reflected_parameters). POST fuzzing disabled but try_post_as_get enabled, so POST params are retested as GET. On WAF-tagged hosts, only request-body parameters are fuzzed, and only if the WAF doesn't block them or nowafpls body padding gets them past it."
+description: "Default fuzzing: all 9 submodules (cmdi, crypto, path, serial, sqli, ssti, xss, esi, ssrf) plus companion modules (badsecrets, hunt, reflected_parameters). POST fuzzing disabled but try_post_as_get enabled, so POST params are retested as GET. Skips confirmed WAFs."
include:
- lightfuzz-light
diff --git a/docs/migration/3.1_breaking_changes.md b/docs/migration/3.1_breaking_changes.md
index f7b3d33af..d594d1426 100644
--- a/docs/migration/3.1_breaking_changes.md
+++ b/docs/migration/3.1_breaking_changes.md
@@ -83,22 +83,6 @@ By default, open ports other than 80 and 443 are dropped on hosts tagged `cdn` o
## Modules
-### `lightfuzz` probes WAF-protected hosts instead of skipping them
-
-In 3.0, lightfuzz skipped every host tagged `waf`. In 3.1, `modules.lightfuzz.avoid_wafs` defaults to `try_bypasses`: lightfuzz sends each WAF-tagged host a test payload in a POST body, first unpadded and then padded (the nowafpls technique). It fuzzes the host only if that probe gets past the WAF: either the padding gets the payload through, or the WAF didn't block it even unpadded. Even then, only request-body parameters are fuzzed. The `lightfuzz`, `lightfuzz-heavy`, `lightfuzz-max`, and `lightfuzz-xss` presets also enable the `nowafpls` module, which runs the same probe on every WAF-tagged host and emits a `FINDING` when padding gets through. Expect more traffic to WAF-fronted targets after upgrading. See [WAF Handling](../modules/lightfuzz.md#waf-handling).
-
-| 3.0 | 3.1 |
-|-----|-----|
-| the top-level `avoid_wafs` (default `true`) controlled lightfuzz | no longer read |
-| `modules.lightfuzz.avoid_wafs` was ignored | controls lightfuzz: `always` (skip), `never` (fuzz without padding), or `try_bypasses` (default); `true` / `false` still work as `always` / `never` |
-| `lightfuzz-max` still skipped WAF-tagged hosts | `lightfuzz-max` sets `never` and fuzzes them without padding |
-
-`lightfuzz-light` still skips WAF-tagged hosts. To keep the 3.0 behavior with the other presets, set `avoid_wafs` to `always` and exclude nowafpls:
-
-```bash
-bbot -t evilcorp.com -p lightfuzz -c modules.lightfuzz.avoid_wafs=always -em nowafpls
-```
-
### `dnsdumpster` requires an API key
dnsdumpster no longer has a keyless mode. Set `modules.dnsdumpster.api_key`, or the module is disabled at setup.
diff --git a/docs/release_history.md b/docs/release_history.md
index cce12b8..85adf39 100644
--- a/docs/release_history.md
+++ b/docs/release_history.md
@@ -10,11 +10,10 @@
- `module_dirs` only works as a top-level preset key, and BBOT asks before loading each custom module folder. Runs without a terminal need `-y`, and `-S` no longer implies it.
- Preset `conditions` are read-only: they can warn or abort, but can't modify the preset.
- dnsdumpster requires an API key, and waf_bypass only tests scan targets by default.
- - lightfuzz now probes WAF-tagged hosts with nowafpls instead of skipping them, and fuzzes one only if the probe gets past the WAF. For the old behavior, set `modules.lightfuzz.avoid_wafs: always` and pass `-em nowafpls`.
#### Highlights
-- **DNS brute-forcing moves to BlastDNS:** massdns is gone, and brute-forcing runs through blastdns 2.0 with adaptive rate control. With higher default DNS concurrency, a typical `subdomain-enum` scan finishes in about 40% less time than on 3.0.2, with the same subdomains found ([#3457](https://github.com/blacklanternsecurity/bbot/pull/3457), [#3606](https://github.com/blacklanternsecurity/bbot/pull/3606), [#3612](https://github.com/blacklanternsecurity/bbot/pull/3612))
+- **DNS brute-forcing moves to BlastDNS:** massdns is gone, and brute-forcing runs through blastdns 2.0 with adaptive rate control. With the higher default DNS concurrency from [#3606](https://github.com/blacklanternsecurity/bbot/pull/3606), a `-f subdomain-enum -m http` scan of a large wildcard target (crt and crt_db excluded) took 11m 51s against 22m 50s on 3.0.2, and found 513 in-scope subdomains against 510 ([#3457](https://github.com/blacklanternsecurity/bbot/pull/3457), [#3606](https://github.com/blacklanternsecurity/bbot/pull/3606), [#3612](https://github.com/blacklanternsecurity/bbot/pull/3612))
- **New `--custom-nameservers` flag** for choosing the resolvers BBOT uses for normal resolution ([#3457](https://github.com/blacklanternsecurity/bbot/pull/3457))
- **baddns 2.5.0**, with a new `DELEGATION` submodule that is on by default ([#3457](https://github.com/blacklanternsecurity/bbot/pull/3457))
- **CDN/WAF port filtering in core, on by default** ([#3534](https://github.com/blacklanternsecurity/bbot/pull/3534))
diff --git a/docs/modules/lightfuzz.md b/docs/modules/lightfuzz.md
index 7395d86..063d304 100644
--- a/docs/modules/lightfuzz.md
+++ b/docs/modules/lightfuzz.md
@@ -210,7 +210,7 @@ If the server fetches the injected URL, the resulting DNS or HTTP interaction is
|---|---|
| `always` | Skip WAF-protected targets entirely |
| `try_bypasses` (default) | Ask the `nowafpls` helper whether the WAF blocks payloads in the request body, and whether padding the body gets past it. POST body and JSON parameters are fuzzed with padding if it does, unpadded if the WAF isn't blocking them, and skipped otherwise. Cookie and header parameters are skipped, since they have no body to pad. GET parameters are skipped too, unless `try_get_as_post` is on (as in `lightfuzz-heavy` and `lightfuzz-xss`), in which case they are retested as POST bodies and handled like POST parameters above. |
-| `never` | Fuzz WAF-protected targets as-is |
+| `never` | Fuzz WAF-protected targets without the probe gate. POST and JSON bodies are still padded where nowafpls finds padding gets through |
## Presets
@@ -260,7 +260,7 @@ Everything in `lightfuzz-heavy`, plus:
| Added setting | Value |
|---|---|
-| **WAF avoidance** | Off (`avoid_wafs: never`): WAF-protected targets are fuzzed as-is |
+| **WAF avoidance** | Off (`avoid_wafs: never`): WAF-protected targets are fuzzed, with request bodies padded where that gets past the WAF |
| **Query string collapsing** | Off — each unique parameter-value pair is fuzzed individually instead of deduplicating by parameter name |
| **Force common headers** | On — headers like `X-Forwarded-For` are fuzzed even if not observed on the target |
| **Speculate params** | On — potential parameters are extracted from JSON/XML response bodies |
Comment on lines
+93
to
+94
| | `modules.lightfuzz.avoid_wafs` was ignored | controls lightfuzz: `always` (skip), `never` (fuzz without padding), or `try_bypasses` (default); `true` / `false` still work as `always` / `never` | | ||
| | `lightfuzz-max` still skipped WAF-tagged hosts | `lightfuzz-max` sets `never` and fuzzes them without padding | |
There was a problem hiding this comment.
🔴 never still pads POST and JSON bodies on WAF-tagged hosts when padding gets through (submodules/base.py:168, Test_Nowafpls_never_still_pads). Say it drops the probe gate but still pads bodies, here and for lightfuzz-max.
Contributor
Author
There was a problem hiding this comment.
Fixed in 0506155. Both rows now say never fuzzes whatever the probe finds and still pads POST and JSON bodies where padding gets through.
| |---|---| | ||
| | `always` | Skip WAF-protected targets entirely | | ||
| | `try_bypasses` (default) | Ask the `nowafpls` helper whether the WAF blocks payloads in the request body, and whether padding the body gets past it. POST body and JSON parameters are fuzzed with padding if it does, unpadded if the WAF isn't blocking them, and skipped otherwise. Cookie and header parameters are skipped, since they have no body to pad. GET parameters are skipped too, unless `try_get_as_post` is on (as in `lightfuzz-heavy` and `lightfuzz-xss`), in which case they are retested as POST bodies and handled like POST parameters above. | | ||
| | `never` | Fuzz WAF-protected targets as-is | |
There was a problem hiding this comment.
🔴 Not as-is: under never, POST and JSON bodies are still padded where nowafpls finds padding gets past the WAF.
Contributor
Author
There was a problem hiding this comment.
Fixed in 0506155 with the same wording. The avoid_wafs Field description and configuration.md are updated to match.
| | Added setting | Value | | ||
| |---|---| | ||
| | **WAF avoidance** | Off — WAF-protected targets are fuzzed | | ||
| | **WAF avoidance** | Off (`avoid_wafs: never`): WAF-protected targets are fuzzed as-is | |
There was a problem hiding this comment.
🔴 Same as the never row above: lightfuzz-max still pads request bodies where padding gets through.
Merged
pjhartlieb
requested changes
Nov 10, 2026
pjhartlieb
left a comment
pjhartlieb
left a comment
Contributor
There was a problem hiding this comment.
Reviewed at 5427051.
🔴 3.1.0 Dependencies lists versions dev doesn't ship
The Dependencies line has pycryptodome 4.2.0 and radixtarget 4.3.0, but dev already locks 4.3.0 (#3620) and 4.3.1 (#3617), both merged on 11-09. It also has blastdns 2.0.4 from #3631, which is still open, while dev locks 2.0.3. To fix it, bump those two versions and merge #3631 before this. Optionally, note that #3627 made requests and urllib3 direct runtime dependencies.
🔴 never still pads request bodies, but the docs say it fuzzes without padding
Resolved in 0506155. All four places, plus the avoid_wafs Field description and configuration.md, now say never fuzzes whatever the probe finds and still pads POST and JSON bodies where padding gets through. That matches filter_event, prepare_request and nowafpls.pad_json.
Reproduce this review
# save the script below as verify-3613.sh, then:
bash verify-3613.sh blacklanternsecurity/bbot 542705158a69209ef3245d6a588b860938813e71
# expected: SUMMARY 27 PASS, 3 FAIL, 0 SKIP, 0 BROKEN (V15 speed wording, not blocking; V31 blastdns vs #3631; V33 pycryptodome and radixtarget vs dev's lock)
# control: bash verify-3613.sh blacklanternsecurity/bbot c068b085a2dad9e5422ddbdf2df31d534c43d12e
# expected: SUMMARY 25 PASS, 5 FAIL, 0 SKIP, 0 BROKEN (V28 V30 V32 fail on the round-5 wording; V31 passes on 2.0.3; V15 and V33 fail at both)#!/usr/bin/env bash
# verify-3613.sh: reproducible checks for blacklanternsecurity/bbot#3613
# Usage: bash verify-3613.sh <owner/repo> <sha> [<control-patch>]
# Changes nothing outside a mktemp directory and nothing on GitHub (PR-VF-02).
#
# Round 6, head 54270515. Re-review: the control is c068b085, the head round 5 requested
# changes on. There the `never` wording (V28, V32) and the #3627 Dependencies line (V30) must
# fail, and the blastdns line (V31) must pass, since c068b085 still listed 2.0.3, the version dev pins.
# V15 (the speed highlight) is unchanged between the two, so it fails at both and is not declared;
# round 5's control shows it passing on the suggested wording. V33 fails at both (pycryptodome,
# radixtarget) and is not declared either. V31 reads #3631's live state; V33 is pinned.
# CONTROL: sha c068b085a2dad9e5422ddbdf2df31d534c43d12e
# CONTROL-EXPECT: V28=FAIL V30=FAIL V31=PASS V32=FAIL
set -uo pipefail
REPO="${1:?owner/repo}"; SHA="${2:?sha}"; PATCH="${3:-}"
PASS=0; FAIL=0; SKIP=0; BROKEN=0
pass() { echo "PASS $1 $2"; PASS=$((PASS+1)); }
fail() { echo "FAIL $1 $2 -- $3"; FAIL=$((FAIL+1)); }
skip() { echo "SKIP $1 $2 -- $3"; SKIP=$((SKIP+1)); }
broken() { echo "BROKEN $1 $2 -- $3"; BROKEN=$((BROKEN+1)); }
summary() { echo "SUMMARY $PASS PASS, $FAIL FAIL, $SKIP SKIP, $BROKEN BROKEN"; [ "$FAIL" -eq 0 ] && [ "$BROKEN" -eq 0 ]; }
echo "script_sha256 $(shasum -a 256 "$0" | cut -d' ' -f1)"
echo "git $(git --version | cut -d' ' -f3)"
echo "gh $(gh --version | head -1 | cut -d' ' -f3)"
WORK="$(mktemp -d)"; trap 'rm -rf "$WORK"' EXIT
export GIT_CONFIG_GLOBAL=/dev/null GIT_CONFIG_NOSYSTEM=1 GIT_TERMINAL_PROMPT=0
GIT=(git -c credential.helper= -c 'credential.helper=!gh auth git-credential')
"${GIT[@]}" clone --quiet --filter=blob:none --no-checkout "https://github.com/$REPO.git" "$WORK/src" || { broken V00 "clone" "clone failed"; summary; exit 1; }
cd "$WORK/src"
"${GIT[@]}" cat-file -e "$SHA^{commit}" 2>/dev/null || "${GIT[@]}" fetch --quiet origin "+refs/pull/*/head:refs/remotes/pr/*" || true
"${GIT[@]}" checkout --quiet --detach "$SHA" || { broken V00 "checkout $SHA" "checkout failed"; summary; exit 1; }
[ "$(git rev-parse HEAD)" = "$SHA" ] || { broken V00 "HEAD is $SHA" "HEAD mismatch"; summary; exit 1; }
if [ -n "$PATCH" ]; then git apply "$PATCH" || { broken V00 "apply control patch" "git apply failed"; summary; exit 1; }; fi
# ---- checks: each line states the desired property --------------------------
# Modules that call a third-party API with self.helpers.request() rather than api_request(),
# so they send the scan's web.user_agent, not BBOT/<version>.
OFFENDERS=""
for f in bbot/modules/urlscan.py bbot/modules/dockerhub.py bbot/modules/templates/postman.py bbot/modules/wayback.py; do
[ -f "$f" ] || continue
if grep -qE '(base_url|api_url|site_url)\s*=\s*f?"https?://' "$f" && grep -qE 'self\.helpers\.request\(' "$f"; then
OFFENDERS="$OFFENDERS $(basename "$f" .py)"
fi
done
if grep -q 'kwargs\["headers"\]\["User-Agent"\] = f"BBOT/{__version__}"' bbot/modules/base.py; then
pass V11 "api_request() sends User-Agent BBOT/<version>"
else
fail V11 "api_request() sends User-Agent BBOT/<version>" "base.py pattern not found"
fi
MIG=docs/migration/3.1_breaking_changes.md
UA_SECTION=$(awk '/^### API requests identify as BBOT/{s=1;next} /^(---|### )/{s=0} s' "$MIG")
QUAL='not every module|some third-party|api_request|API request helper|not all'
ua_unqualified() { [ -n "$UA_SECTION" ] && echo "$UA_SECTION" | grep -q 'BBOT/<version>' && ! echo "$UA_SECTION" | grep -qiE "$QUAL"; }
if ua_unqualified && [ -n "$OFFENDERS" ]; then
fail V01 "3.1 migration note's User-Agent claim matches the modules" "claims all third-party API requests send BBOT/<version>; still on web.user_agent:$OFFENDERS"
else
pass V01 "3.1 migration note's User-Agent claim matches the modules"
fi
help_unqualified() { grep -E 'BBOT/<version>' "$1" | grep -i 'user-agent' | grep -viqE '\((most|some)[^)]*BBOT/<version>'; }
if [ -n "$OFFENDERS" ] && { help_unqualified bbot/scanner/preset/args.py || help_unqualified docs/scanning/advanced.md; }; then
fail V02 "--user-agent help text matches the modules" "args.py or advanced.md says API calls send BBOT/<version> without qualification; still on web.user_agent:$OFFENDERS"
else
pass V02 "--user-agent help text matches the modules"
fi
LF=bbot/modules/lightfuzz/lightfuzz.py
WAF_ROW=$(grep -F '`try_bypasses` (default)' docs/modules/lightfuzz.md)
if echo "$WAF_ROW" | grep -q 'GET' && ! echo "$WAF_ROW" | grep -q 'try_get_as_post' \
&& grep -q 'if self.try_get_as_post and original_type == "GETPARAM":' "$LF" \
&& grep -q 'try_get_as_post: True' bbot/presets/web/lightfuzz-xss.yml; then
fail V03 "lightfuzz WAF Handling describes GET params under try_get_as_post" "doc says GET params are always skipped; lightfuzz.py retests them as POST in bypass mode and lightfuzz-xss turns that on"
else
pass V03 "lightfuzz WAF Handling describes GET params under try_get_as_post"
fi
if grep -q 'requires-python = ">=3.11' pyproject.toml && grep -q 'badge/python-3.11+' README.md && grep -q 'BBOT requires Python 3.11+' docs/troubleshooting.md; then
pass V04 "README badge and troubleshooting say 3.11+, matching requires-python"
else
fail V04 "README badge and troubleshooting say 3.11+, matching requires-python" "mismatch"
fi
if grep -qE '^ threads: 50$' bbot/defaults.yml && grep -qE '^ max_concurrency: 256$' bbot/defaults.yml \
&& grep -q 'default 50 queries per resolver' docs/scanning/tips_and_tricks.md && grep -q '256 total' docs/scanning/tips_and_tricks.md; then
pass V05 "DNS speed tip matches dns.threads=50 and dns.max_concurrency=256"
else
fail V05 "DNS speed tip matches dns.threads=50 and dns.max_concurrency=256" "mismatch"
fi
if ! grep -qE '^\s+threads:' bbot/presets/subdomain-enum.yml && ! grep -q 'threads: 25' README.md && ! grep -q 'threads: 25' docs/scanning/presets_list.md; then
pass V06 "subdomain-enum no longer lowers dns.threads, and README/presets_list match"
else
fail V06 "subdomain-enum no longer lowers dns.threads, and README/presets_list match" "threads still set somewhere"
fi
CC=bbot/modules/internal/cloudcheck.py
if grep -q 'event.add_tag(provider_type)' "$CC" && grep -q 'event.add_tag(provider_name)' "$CC" \
&& ! grep -rqE 'cdn-github|cloud-azure|cdn-akamai|cdn-ip' docs/scanning/output.md docs/scanning/events.md docs/modules/custom_yara_rules.md; then
pass V07 "tag examples use cloudcheck's separate type and provider tags"
else
fail V07 "tag examples use cloudcheck's separate type and provider tags" "old cdn-x tags remain or code differs"
fi
ok=1
grep -q 'avoid_wafs: always' bbot/presets/web/lightfuzz-light.yml || ok=0
grep -q 'avoid_wafs: try_bypasses' bbot/presets/web/lightfuzz.yml || ok=0
grep -q 'avoid_wafs: never' bbot/presets/web/lightfuzz-max.yml || ok=0
grep -q 'avoid_wafs: try_bypasses' bbot/presets/web/lightfuzz-xss.yml || ok=0
grep -q 'avoid_wafs: Literal\["always", "never", "try_bypasses"\]' "$LF" || ok=0
if [ "$ok" = 1 ]; then pass V08 "lightfuzz preset tables match each preset's avoid_wafs"; else fail V08 "lightfuzz preset tables match each preset's avoid_wafs" "mismatch"; fi
if grep -q '"--custom-nameservers"' bbot/scanner/preset/args.py && [ -f bbot/presets/nuclei/nuclei-recommended.yml ] && [ -f bbot/presets/nuclei/nuclei-new.yml ]; then
pass V09 "--custom-nameservers, nuclei-recommended and nuclei-new exist as documented"
else
fail V09 "--custom-nameservers, nuclei-recommended and nuclei-new exist as documented" "missing"
fi
if N=$(gh api --method GET "repos/blacklanternsecurity/public-dns-servers/contents/nameservers.txt?ref=master" -H "Accept: application/vnd.github.raw+json" 2>/dev/null | grep -cvE '^\s*(#|$)'); then
if [ "$N" -ge 4000 ] && [ "$N" -le 6000 ]; then pass V10 "brute_nameservers list is ~5K servers ($N)"; else fail V10 "brute_nameservers list is ~5K servers" "count $N"; fi
else
skip V10 "brute_nameservers list is ~5K servers" "could not fetch nameservers.txt"
fi
# C13 (A5): generated blocks match their sources
if python3 - <<'PY'
p=open('bbot/presets/subdomain-enum.yml').read().strip(); r=open('README.md').read()
l=open('docs/scanning/presets_list.md').read(); c=open('docs/scanning/configuration.md').read(); d=open('bbot/defaults.yml').read().strip()
import re,sys; a=open('bbot/scanner/preset/args.py').read(); adv=open('docs/scanning/advanced.md').read()
h=re.search(r'"--user-agent",\s*"-ua",\s*help="([^"]+)"', a)
sys.exit(0 if (p in r and "\n".join(" "+x for x in p.split("\n")) in l and d in c and h and h.group(1) in adv) else 1)
PY
then pass V12 "generated README, presets_list, configuration and advanced.md blocks match their sources"
else fail V12 "generated README, presets_list, configuration and advanced.md blocks match their sources" "a generated block is stale"; fi
# C16/C17: lightfuzz-xss table's POST row matches the preset
XSS_TABLE=$(awk '/^### .-p lightfuzz-xss./{s=1} /^### Spider/{s=0} s' docs/modules/lightfuzz.md)
if echo "$XSS_TABLE" | grep -q 'POST fuzzing\*\* | Disabled' && ! grep -qE 'disable_post: *True' bbot/presets/web/lightfuzz-xss.yml; then
fail V13 "lightfuzz-xss table's POST fuzzing row matches the preset" "table says Disabled; preset leaves disable_post False and sets try_get_as_post"
else
pass V13 "lightfuzz-xss table's POST fuzzing row matches the preset"
fi
# C3/C14 (A6): more modules call third-party APIs outside api_request()
MORE=""
for m in azure_tenant docker_pull; do
grep -qE 'self\.helpers\.request\(' "bbot/modules/$m.py" && grep -qE 'https://(azmap\.dev|registry-1\.docker\.io|login\.microsoftonline\.com)' "bbot/modules/$m.py" && MORE="$MORE $m"
done
if ua_unqualified && [ -n "$MORE" ]; then
fail V14 "3.1 migration note holds for azure_tenant and docker_pull" "unqualified claim; still on web.user_agent:$MORE"
else
pass V14 "3.1 migration note holds for azure_tenant and docker_pull"
fi
# ---- round 2: the 3.1.0 release history (fedf360c) ----------------------------
RH=docs/release_history.md
SEC=$(awk '/^### 3\.1\.0 /{s=1;print;next} /^### [0-9]/{s=0} s' "$RH")
# V15: the speed claim matches the measurement it cites (#3606 "Effect on a scan").
B3606=$(gh api --method GET repos/blacklanternsecurity/bbot/pulls/3606 --jq .body 2>/dev/null)
if [ -z "$SEC" ] || [ -z "$B3606" ]; then
skip V15 "3.1.0 speed claim matches #3606's measurement" "no 3.1.0 section or #3606 unreadable"
elif ! echo "$B3606" | grep -q '| wall time | 22m 50s | 11m 51s |' || ! echo "$B3606" | grep -q 'against a large wildcard target'; then
broken V15 "3.1.0 speed claim matches #3606's measurement" "#3606 no longer carries the measurement this check reads"
else
# C4/R2C1/R2C2: every speed claim in the section must carry #3606's measurement and its target:
# its times (22m 50s, 11m 51s) or its ratio (48%, half, halved), plus "large wildcard"; no other figure, no "typical".
BAD=""
while IFS= read -r L; do
[ -z "$L" ] && continue
M=0; { echo "$L" | grep -q '22m 50s' && echo "$L" | grep -q '11m 51s'; } && M=1
echo "$L" | grep -qiE '48 ?%|half|halve[sd]?|twice as fast|(^|[^0-9.])2 ?x (faster|quicker)' && M=1 # C3 (r5): accept accurate rewordings
# R6 R2C1: a figure at or below #3606's 48% on a line scoped to 'large wildcard' is supported too
echo "$L" | grep -oE '[0-9]+ ?%' | tr -d ' %' | awk '$1<=48{f=1} END{exit !f}' && M=1
PCT=$(echo "$L" | grep -oE '[0-9]+ ?%' | tr -d ' %' | awk '$1>48')
if [ "$M" = 0 ] || ! echo "$L" | grep -qi 'large wildcard' || [ -n "$PCT" ] || echo "$L" | grep -qi 'typical'; then BAD="$BAD | ${L:0:120}"; fi
done <<< "$(echo "$SEC" | grep -iE 'faster|quicker|slower|less time|as long|wall time|halve[sd]?|twice|double[sd]?|took|[0-9]+m [0-9]+s|[0-9]+ ?% (less|more|quicker)|finish(es)? in' | grep -iE 'scan')"
if [ -z "$BAD" ]; then pass V15 "3.1.0 speed claims match #3606's measurement"
else fail V15 "3.1.0 speed claims match #3606's measurement" "unmeasured speed claim:$BAD; #3606: one large wildcard target, 22m 50s to 11m 51s (48% less), 510 to 513 subdomains"; fi
fi
# ---- round 3 -------------------------------------------------------------------
# V19: lightfuzz-heavy (includes lightfuzz -> avoid_wafs: try_bypasses; disable_post False, try_get_as_post True)
# does not describe itself as skipping WAFs in its preset file (what `bbot -lp` prints; presets_list.md is generated from it).
H=bbot/presets/web/lightfuzz-heavy.yml
eff=$(grep -oE 'avoid_wafs: *[a-z_]+' "$H" | awk '{print $2}')
[ -z "$eff" ] && grep -qE '^ - lightfuzz$' "$H" && eff=$(grep -oE 'avoid_wafs: *[a-z_]+' bbot/presets/web/lightfuzz.yml | awk '{print $2}')
if [ -z "$eff" ]; then
broken V19 "lightfuzz-heavy's description matches its avoid_wafs" "could not resolve avoid_wafs for lightfuzz-heavy"
elif [ "$eff" = try_bypasses ] && grep -q 'disable_post: False' "$H" && grep -m1 '^description:' "$H" | grep -qi 'skips confirmed WAFs'; then
fail V19 "lightfuzz-heavy's description matches its avoid_wafs" "says it still skips confirmed WAFs; it runs try_bypasses with POST fuzzing on"
else
pass V19 "lightfuzz-heavy's description matches its avoid_wafs"
fi
# V20: each lightfuzz preset's "WAF avoidance" row in lightfuzz.md names the avoid_wafs value its YAML sets.
LFMD=docs/modules/lightfuzz.md; BAD=""; N=0
for p in lightfuzz-light lightfuzz lightfuzz-max lightfuzz-xss; do
eff=$(grep -oE 'avoid_wafs: *[a-z_]+' "bbot/presets/web/$p.yml" | awk '{print $2}')
row=$(awk -v h="### \`-p $p\`" '$0==h{s=1;next} /^### /{s=0} s && /\*\*WAF avoidance\*\*/' "$LFMD")
N=$((N+1))
if [ -z "$eff" ] || [ -z "$row" ] || ! echo "$row" | grep -q "avoid_wafs: $eff"; then BAD="$BAD $p(yaml=$eff)"; fi
done
if [ -z "$BAD" ]; then pass V20 "lightfuzz.md WAF avoidance rows match each preset's avoid_wafs ($N presets)"
else fail V20 "lightfuzz.md WAF avoidance rows match each preset's avoid_wafs" "row missing or wrong:$BAD"; fi
# V21: v3.0.2 lightfuzz skipped every WAF-tagged host (top-level avoid_wafs, default True); this head defaults
# to try_bypasses. The 3.1 migration guide's Modules section says so (C6: any heading).
"${GIT[@]}" fetch --quiet origin tag v3.0.2 2>/dev/null || true
OLD=$(git show v3.0.2:bbot/modules/lightfuzz/lightfuzz.py 2>/dev/null | grep -c 'self.avoid_wafs = self.scan.config.get("avoid_wafs", True)')
NEW=$(grep -c '"try_bypasses",' bbot/modules/lightfuzz/lightfuzz.py)
if [ "$OLD" != 1 ] || [ "$NEW" -lt 1 ]; then
broken V21 "3.1 migration guide documents lightfuzz's WAF default change" "could not read v3.0.2 or head lightfuzz.py"
elif awk '/^## /{m=($0 ~ /^## Modules/)} m && /avoid_wafs/' docs/migration/3.1_breaking_changes.md | grep -q .; then
pass V21 "3.1 migration guide documents lightfuzz's WAF default change"
else
fail V21 "3.1 migration guide documents lightfuzz's WAF default change" "3.0.2 skipped every WAF host; head defaults to try_bypasses; no lightfuzz/avoid_wafs entry in the guide"
fi
# ---- round 4 (cd2bf7ff) ----------------------------------------------------------
LFSEC=$(awk '/^### `lightfuzz` probes WAF/{s=1;next} /^### /{s=0} s' "$MIG")
# V22: the entry's 3.0 -> 3.1 table matches v3.0.2 and this head.
O_LF=$(git show v3.0.2:bbot/modules/lightfuzz/lightfuzz.py 2>/dev/null)
O_MAX=$(git show v3.0.2:bbot/presets/web/lightfuzz-max.yml 2>/dev/null)
if [ -z "$O_LF" ] || [ -z "$O_MAX" ]; then
broken V22 "migration WAF table matches v3.0.2 and this head" "could not read v3.0.2"
else
ok=1; why=""
echo "$O_LF" | grep -q 'self.avoid_wafs = self.scan.config.get("avoid_wafs", True)' || { ok=0; why="$why v3.0.2-not-top-level"; }
echo "$O_MAX" | grep -qE '^ avoid_wafs: False' || { ok=0; why="$why v3.0.2-max-no-False"; }
grep -qE '^ avoid_wafs: never' bbot/presets/web/lightfuzz-max.yml || { ok=0; why="$why head-max-not-never"; }
for s in 'no longer read' '`lightfuzz-max` still skipped WAF-tagged hosts' '`lightfuzz-max` sets `never`' '`true` / `false` still work as `always` / `never`' 'defaults to `try_bypasses`'; do
echo "$LFSEC" | grep -qF "$s" || { ok=0; why="$why doc-missing:[$s]"; }
done
if [ "$ok" = 1 ]; then pass V22 "migration WAF table matches v3.0.2 and this head"
else fail V22 "migration WAF table matches v3.0.2 and this head" "$why"; fi
fi
# V23: the old-behaviour recipe works, and the entry names exactly the presets that pull in nowafpls.
incl() { awk '/^include:/{s=1;next} /^[a-z]/{s=0} s && /^ - /{print $2}' "bbot/presets/web/$1.yml"; }
mods() { awk '/^modules:/{s=1;next} /^[a-z]/{s=0} s && /^ - /{print $2}' "bbot/presets/web/$1.yml"; }
has_nowafpls() { local p="$1" d="${2:-0}"; [ "$d" -gt 5 ] && return 1; [ -f "bbot/presets/web/$p.yml" ] || return 1
{ incl "$p"; mods "$p"; } | grep -qx nowafpls && return 0
for i in $(incl "$p"); do has_nowafpls "$i" $((d+1)) && return 0; done; return 1; }
ACT=$(for f in bbot/presets/web/lightfuzz*.yml; do p=$(basename "$f" .yml); has_nowafpls "$p" && echo "$p"; done | sort | tr '\n' ' ')
DOC=$(echo "$LFSEC" | grep -F 'also enable the `nowafpls` module' | grep -oE '`lightfuzz[a-z-]*`' | tr -d '`' | sort -u | tr '\n' ' ')
ok=1; why=""
[ -n "$DOC" ] && [ "$ACT" = "$DOC" ] || { ok=0; why="$why presets-with-nowafpls=[$ACT] doc=[$DOC]"; }
echo "$LFSEC" | grep -qF -- '-c modules.lightfuzz.avoid_wafs=always -em nowafpls' || { ok=0; why="$why recipe-missing"; }
grep -q 'if self.avoid_wafs == "always":' "$LF" && [ -f bbot/modules/nowafpls.py ] || { ok=0; why="$why code"; }
grep -qE '^ avoid_wafs: always' bbot/presets/web/lightfuzz-light.yml && echo "$LFSEC" | grep -qF '`lightfuzz-light` still skips WAF-tagged hosts' || { ok=0; why="$why light"; }
if [ "$ok" = 1 ]; then pass V23 "old-behaviour recipe and nowafpls preset list match the presets"
else fail V23 "old-behaviour recipe and nowafpls preset list match the presets" "$why"; fi
# V24: avoid_wafs still accepts the boolean form, as the entry says.
if awk '/def _normalize_avoid_wafs/{s=1} s && n++<6' "$LF" | grep -q 'return "always" if v else "never"'; then
pass V24 "modules.lightfuzz.avoid_wafs maps true/false to always/never"
else fail V24 "modules.lightfuzz.avoid_wafs maps true/false to always/never" "validator not found"; fi
# V25: lightfuzz.yml and lightfuzz-heavy.yml (try_bypasses) don't say they skip WAFs, and presets_list.md
# carries every lightfuzz preset description verbatim (it is generated from them).
ok=1; why=""
for p in lightfuzz lightfuzz-heavy; do grep -m1 '^description:' "bbot/presets/web/$p.yml" | grep -qiE 'skips confirmed WAFs' && { ok=0; why="$why $p-says-skips"; }; done
for f in bbot/presets/web/lightfuzz*.yml; do
d=$(grep -m1 '^description:' "$f" | sed -E 's/^description: "(.*)"$/\1/')
grep -qF -- "$d" docs/scanning/presets_list.md || { ok=0; why="$why stale:$(basename "$f" .yml)"; }
done
if [ "$ok" = 1 ]; then pass V25 "lightfuzz preset descriptions match try_bypasses and presets_list.md"
else fail V25 "lightfuzz preset descriptions match try_bypasses and presets_list.md" "$why"; fi
CHANGED="$WORK/changed.txt"
printf '%s\n' README.md docs/migration/3.1_breaking_changes.md docs/modules/custom_yara_rules.md docs/modules/lightfuzz.md docs/modules/nuclei.md docs/modules/wayback.md docs/release_history.md docs/scanning/advanced.md docs/scanning/configuration.md docs/scanning/events.md docs/scanning/output.md docs/scanning/presets.md docs/scanning/presets_list.md docs/scanning/tips_and_tricks.md docs/troubleshooting.md > "$CHANGED"; export CHANGED
# V26: relative links in the changed docs resolve, including #anchors to headings (static stand-in for a docs build).
BAD=$(python3 - <<'PY'
import re, os, subprocess
files = [l for l in open(os.environ["CHANGED"]).read().split() if l.startswith("docs/") and l.endswith(".md")]
def slug(h): return re.sub(r"[^\w\- ]", "", h.strip().lower()).replace(" ", "-")
bad = []
for f in files:
for m in re.finditer(r"\]\(([^)\s]+)\)", open(f).read()):
t = m.group(1)
# skip schemes, and the generator's placeholder [0]("") links (37 of them at base too)
if re.match(r"^[a-z]+:", t) or t.strip('"') == "": continue
path, _, anc = t.partition("#")
tgt = os.path.normpath(os.path.join(os.path.dirname(f), path)) if path else f
if not os.path.exists(tgt): bad.append(f"{f}->{t}"); continue
if anc and tgt.endswith(".md"):
heads = {slug(re.sub(r"^#+\s*", "", l)) for l in open(tgt) if l.startswith("#")}
if anc not in heads: bad.append(f"{f}->{t}")
print(" ".join(bad))
PY
)
if [ -z "$BAD" ]; then pass V26 "relative links and anchors in the changed docs resolve"
else fail V26 "relative links and anchors in the changed docs resolve" "$BAD"; fi
# V27: the 3.1.0 'Check your setup' box flags the lightfuzz WAF change with the same recipe as the guide.
BOX=$(awk '/^!!! warning "Check your setup before upgrading"/{s=1;next} /^#### /{s=0} s' "$RH")
if echo "$BOX" | grep -qi 'lightfuzz' && echo "$BOX" | grep -qF 'modules.lightfuzz.avoid_wafs: always' && echo "$BOX" | grep -qF -- '-em nowafpls'; then
pass V27 "3.1.0 Check-your-setup box flags the lightfuzz WAF change"
else fail V27 "3.1.0 Check-your-setup box flags the lightfuzz WAF change" "no lightfuzz WAF bullet with the recipe"; fi
# V28 (challenge C7): no doc row says avoid_wafs=never fuzzes unpadded / as-is, since under never
# prepare_request still pads POST/JSON bodies on WAF hosts (submodules/base.py) and the repo's
# Test_Nowafpls_never_still_pads asserts it.
BASEPY=bbot/modules/lightfuzz/submodules/base.py
if ! grep -q 'if "waf" in self.event.tags and self.lightfuzz.avoid_wafs != "always":' "$BASEPY" \
|| ! grep -q 'class Test_Nowafpls_never_still_pads' bbot/test/test_step_2/module_tests/test_module_lightfuzz.py; then
pass V28 "docs describe avoid_wafs=never the way the code pads" "(code no longer pads under never)"
else
BAD=""
echo "$LFSEC" | grep -iE '`never`' | grep -qiE 'without padding|as-is|\braw\b|unpadded' && BAD="$BAD migration-guide"
grep -E '^\| `never` \|' "$LFMD" | grep -qiE 'as-is|raw|without padding' && BAD="$BAD lightfuzz.md-WAF-table"
grep -F 'avoid_wafs: never' "$LFMD" | grep -qiE 'as-is|raw|without padding' && BAD="$BAD lightfuzz.md-max-preset-row"
if [ -z "$BAD" ]; then pass V28 "docs describe avoid_wafs=never the way the code pads"
else fail V28 "docs describe avoid_wafs=never the way the code pads" "says unpadded/as-is:$BAD; base.py pads POST/JSON bodies on WAF hosts unless avoid_wafs=always"; fi
fi
# V29 (round 5): the #3629 lines in 3.1.0 match #3629 as merged to dev: FINDING gains an optional port
# rendered host:port, legba and medusa pass it, and nuclei correlation is fixed.
M3629=3085baacf1f812fcf446d73f673aa2454b199ce4
"${GIT[@]}" fetch --quiet origin "$M3629" 2>/dev/null || true
MA=$(gh api --method GET repos/blacklanternsecurity/bbot/pulls/3629 --jq '.merged_at // "no"' 2>/dev/null) || MA=err
if ! git cat-file -e "$M3629^{commit}" 2>/dev/null || [ "$MA" = err ]; then
skip V29 "3.1.0 #3629 lines match #3629" "could not read #3629 or its merge commit"
else
ok=1; why=""
[ "$MA" != no ] || { ok=0; why="$why not-merged"; }
git show "$M3629:bbot/core/event/base.py" | grep -q 'port: Optional\[int\] = None' || { ok=0; why="$why no-FINDING-port"; }
git show "$M3629:bbot/core/event/base.py" | grep -q 'make_netloc(host, port) if port else host' || { ok=0; why="$why no-host:port"; }
for m in legba medusa; do git show "$M3629:bbot/modules/$m.py" | grep -q '"port": str(event.port)' || { ok=0; why="$why $m"; }; done
git show "$M3629^1:bbot/modules/nuclei.py" | grep -q 'def correlate_event' && ! git show "$M3629:bbot/modules/nuclei.py" | grep -q 'def correlate_event' || { ok=0; why="$why nuclei-correlate"; }
grep -q 'FINDING has an optional `port`.*legba and medusa.*pull/3629' "$RH" && grep -q 'nuclei:.*Results are parented.*pull/3629' "$RH" || { ok=0; why="$why doc-lines-missing"; }
if [ "$ok" = 1 ]; then pass V29 "3.1.0 #3629 lines match #3629 as merged"; else fail V29 "3.1.0 #3629 lines match #3629 as merged" "$why"; fi
fi
# V30 (challenge C17, round 5): the 3.1.0 Dependencies section reflects #3627 (merged to dev before #3629,
# whose notes c068b085 added): pyjwt, lxml and socksio dropped, xmltojson replaced by xmltodict.
M3627=5d968b1c421eb0568f19abaaccb0c7fc92130a32
"${GIT[@]}" fetch --quiet origin "$M3627" 2>/dev/null || true
if ! git cat-file -e "$M3627^{commit}" 2>/dev/null; then
skip V30 "3.1.0 Dependencies reflects #3627" "could not fetch #3627's merge"
elif ! git diff "$M3627^1" "$M3627" -- pyproject.toml | grep -q '^+ "xmltodict'; then
broken V30 "3.1.0 Dependencies reflects #3627" "#3627 no longer shows the xmltodict swap"
else
DEP=$(awk '/^### 3\.1\.0 /{s=1} /^### [0-9]/ && !/3\.1\.0/{s=0} s' "$RH" | awk '/^#### Dependencies/{s=1;next} /^#### /{s=0} s')
if echo "$DEP" | grep -qiE 'xmltodict|pyjwt|lxml|socksio|3627'; then pass V30 "3.1.0 Dependencies reflects #3627"
else fail V30 "3.1.0 Dependencies reflects #3627" "no mention of #3627's removals (pyjwt, lxml, socksio) or the xmltojson -> xmltodict swap"; fi
fi
# ------------------------------------------------------------------------------
# V31 (round 6): each dependency version in 3.1.0 Dependencies is either the one dev pins, or comes
# from a cited PR that has merged. 54270515 lists blastdns 2.0.4 citing #3631.
DEP=$(awk '/^### 3\.1\.0 /{s=1} /^### [0-9]/ && !/3\.1\.0/{s=0} s' "$RH" | awk '/^#### Dependencies/{s=1;next} /^#### /{s=0} s')
"${GIT[@]}" fetch --quiet origin dev 2>/dev/null || true
DEVPIN=$(git show FETCH_HEAD:pyproject.toml 2>/dev/null | grep -oE '"blastdns>=[0-9.]+' | grep -oE '[0-9]+\.[0-9]+\.[0-9]+')
NOTED=$(echo "$DEP" | grep -oE 'blastdns [0-9]+\.[0-9]+\.[0-9]+' | grep -oE '[0-9.]+$' | head -1)
if [ -z "$DEVPIN" ] || [ -z "$NOTED" ]; then
skip V31 "3.1.0 blastdns version is dev's pin or comes from a merged PR" "could not read dev's pin or the notes' blastdns line"
elif [ "$NOTED" = "$DEVPIN" ]; then
pass V31 "3.1.0 blastdns version is dev's pin or comes from a merged PR" "(notes $NOTED, dev pins >=$DEVPIN)"
else
M3631=$(gh api --method GET repos/blacklanternsecurity/bbot/pulls/3631 --jq '.merged_at // "no"' 2>/dev/null) || M3631=err
if [ "$M3631" = err ]; then skip V31 "3.1.0 blastdns version is dev's pin or comes from a merged PR" "could not read #3631"
elif [ "$M3631" != no ] && echo "$DEP" | grep -q 'pull/3631'; then pass V31 "3.1.0 blastdns version is dev's pin or comes from a merged PR" "(#3631 merged $M3631)"
else fail V31 "3.1.0 blastdns version is dev's pin or comes from a merged PR" "notes say blastdns $NOTED, dev pins >=$DEVPIN, and #3631 (the 2.0.4 bump) is not merged"; fi
fi
# V32 (round 6): the avoid_wafs Field description and the generated configuration.md row don't say
# `never` fuzzes raw / without padding, and the two carry the same description.
LFPY=bbot/modules/lightfuzz/lightfuzz.py; CFG=docs/scanning/configuration.md
FIELD=$(python3 - "$LFPY" <<'PY'
import ast, sys
src = open(sys.argv[1]).read()
for node in ast.walk(ast.parse(src)):
if isinstance(node, ast.AnnAssign) and getattr(node.target, "id", "") == "avoid_wafs":
for kw in node.value.keywords:
if kw.arg == "description":
print(ast.literal_eval(kw.value))
PY
)
CFGROW=$(grep -E '^\| modules\.lightfuzz\.avoid_wafs ' "$CFG" | awk -F'|' '{print $4}' | sed 's/^ *//; s/ *$//')
if [ -z "$FIELD" ] || [ -z "$CFGROW" ]; then
broken V32 "avoid_wafs description says never still pads, and configuration.md matches it" "could not read the Field description or the configuration.md row"
else
BAD=""
echo "$FIELD" | grep -qiE "'never' fuzzes them raw|never[^;]*(without padding|unpadded|as-is)" && BAD="$BAD Field-says-raw"
[ "$FIELD" = "$CFGROW" ] || BAD="$BAD configuration.md-differs"
if [ -z "$BAD" ]; then pass V32 "avoid_wafs description says never still pads, and configuration.md matches it"
else fail V32 "avoid_wafs description says never still pads, and configuration.md matches it" "$BAD"; fi
fi
# V33 (round 6, challenges C6/C7): each version (blastdns aside) in the 3.1.0 Dependencies line matches dev's uv.lock
# at 5d737af6 (dev's tip when round 6 was collected; pinned so a replay gets the same answer).
DEV=5d737af6f40f40ccb6a32d222ae24861b3c20baa
"${GIT[@]}" fetch --quiet origin "$DEV" 2>/dev/null || true
if ! git cat-file -e "$DEV^{commit}" 2>/dev/null; then
skip V33 "3.1.0 Dependencies versions match dev's uv.lock at 5d737af6" "could not fetch $DEV"
else
LOCK=$(git show "$DEV:uv.lock")
DEP=$(awk '/^### 3\.1\.0 /{s=1} /^### [0-9]/ && !/3\.1\.0/{s=0} s' "$RH" | awk '/^#### Dependencies/{s=1;next} /^#### /{s=0} s')
BAD=""
# R6 R2C4: blastdns is V31's (it depends on whether #3631 merges first), not V33's
for p in blasthttp baddns cloudcheck radixtarget pycryptodome; do
N=$(echo "$DEP" | grep -oE "$p [0-9]+\.[0-9]+\.[0-9]+" | head -1 | cut -d' ' -f2)
L=$(echo "$LOCK" | awk -v n="name = \"$p\"" '$0==n{getline; gsub(/"/,"",$3); print $3; exit}')
[ -n "$N" ] && [ "$N" != "$L" ] && BAD="$BAD $p:notes=$N,lock=$L"
done
if [ -z "$BAD" ]; then pass V33 "3.1.0 Dependencies versions match dev's uv.lock at 5d737af6"
else fail V33 "3.1.0 Dependencies versions match dev's uv.lock at 5d737af6" "$BAD"; fi
fi
# ------------------------------------------------------------------------------
summary|
|
||
| #### Dependencies | ||
|
|
||
| - blastdns 2.0.4 ([#3612](https://github.com/blacklanternsecurity/bbot/pull/3612), [#3631](https://github.com/blacklanternsecurity/bbot/pull/3631)), blasthttp 0.10.1, baddns 2.5.0, badsecrets 1.3.0 ([#3526](https://github.com/blacklanternsecurity/bbot/pull/3526)), cloudcheck 11.2.0 ([#3559](https://github.com/blacklanternsecurity/bbot/pull/3559)), radixtarget 4.3.0 ([#3558](https://github.com/blacklanternsecurity/bbot/pull/3558)), pycryptodome 4.2.0 ([#3542](https://github.com/blacklanternsecurity/bbot/pull/3542), [#3597](https://github.com/blacklanternsecurity/bbot/pull/3597)). |
Contributor
Contributor
Author
en0f
approved these changes
Nov 10, 2026
pjhartlieb
approved these changes
Nov 10, 2026
pjhartlieb
left a comment
pjhartlieb
left a comment
Contributor
There was a problem hiding this comment.
Reviewed at 74f897b.
🔴 3.1.0 Dependencies lists versions dev doesn't ship
Resolved in 74f897b. radixtarget 4.3.1 and pycryptodome 4.3.0 now match dev, and with #3631 merged, blastdns 2.0.4 is what dev ships. Every version on the line matches dev's uv.lock at 72ebe17.
Reproduce this review
# save the script below as verify-3613.sh, then:
bash verify-3613.sh blacklanternsecurity/bbot 74f897b5c9d012b747c8092b912c92be0282f8d1
# expected: SUMMARY 30 PASS, 1 FAIL, 0 SKIP, 0 BROKEN (V15 speed wording, not blocking)
# control: bash verify-3613.sh blacklanternsecurity/bbot 542705158a69209ef3245d6a588b860938813e71
# expected: SUMMARY 28 PASS, 3 FAIL, 0 SKIP, 0 BROKEN (V33 and V34 fail on pycryptodome 4.2.0 and radixtarget 4.3.0; V15 fails at both)#!/usr/bin/env bash
# verify-3613.sh: reproducible checks for blacklanternsecurity/bbot#3613
# Usage: bash verify-3613.sh <owner/repo> <sha> [<control-patch>]
# Changes nothing outside a mktemp directory and nothing on GitHub (PR-VF-02).
#
# Round 7, head 74f897b5. Re-review: the control is 54270515, the head round 6 requested
# changes on. There the Dependencies line still lists pycryptodome 4.2.0 and radixtarget 4.3.0,
# so V33 (pinned to dev 5d737af6) and V34 (pinned to dev 72ebe17c, which includes #3631) must fail.
# V15 (the speed highlight) is unchanged since fedf360c, so it fails at both and is not declared
# (F10, a note since round 2). V31 reads #3631's live state; #3631 merged 2026-11-10T22:40:32Z,
# so V31 passes at both and is not declared. Every other check is carried from round 6 unchanged.
# CONTROL: sha 542705158a69209ef3245d6a588b860938813e71
# CONTROL-EXPECT: V33=FAIL V34=FAIL
set -uo pipefail
REPO="${1:?owner/repo}"; SHA="${2:?sha}"; PATCH="${3:-}"
PASS=0; FAIL=0; SKIP=0; BROKEN=0
pass() { echo "PASS $1 $2"; PASS=$((PASS+1)); }
fail() { echo "FAIL $1 $2 -- $3"; FAIL=$((FAIL+1)); }
skip() { echo "SKIP $1 $2 -- $3"; SKIP=$((SKIP+1)); }
broken() { echo "BROKEN $1 $2 -- $3"; BROKEN=$((BROKEN+1)); }
summary() { echo "SUMMARY $PASS PASS, $FAIL FAIL, $SKIP SKIP, $BROKEN BROKEN"; [ "$FAIL" -eq 0 ] && [ "$BROKEN" -eq 0 ]; }
echo "script_sha256 $(shasum -a 256 "$0" | cut -d' ' -f1)"
echo "git $(git --version | cut -d' ' -f3)"
echo "gh $(gh --version | head -1 | cut -d' ' -f3)"
WORK="$(mktemp -d)"; trap 'rm -rf "$WORK"' EXIT
export GIT_CONFIG_GLOBAL=/dev/null GIT_CONFIG_NOSYSTEM=1 GIT_TERMINAL_PROMPT=0
GIT=(git -c credential.helper= -c 'credential.helper=!gh auth git-credential')
"${GIT[@]}" clone --quiet --filter=blob:none --no-checkout "https://github.com/$REPO.git" "$WORK/src" || { broken V00 "clone" "clone failed"; summary; exit 1; }
cd "$WORK/src"
"${GIT[@]}" cat-file -e "$SHA^{commit}" 2>/dev/null || "${GIT[@]}" fetch --quiet origin "+refs/pull/*/head:refs/remotes/pr/*" || true
"${GIT[@]}" checkout --quiet --detach "$SHA" || { broken V00 "checkout $SHA" "checkout failed"; summary; exit 1; }
[ "$(git rev-parse HEAD)" = "$SHA" ] || { broken V00 "HEAD is $SHA" "HEAD mismatch"; summary; exit 1; }
if [ -n "$PATCH" ]; then git apply "$PATCH" || { broken V00 "apply control patch" "git apply failed"; summary; exit 1; }; fi
# ---- checks: each line states the desired property --------------------------
# Modules that call a third-party API with self.helpers.request() rather than api_request(),
# so they send the scan's web.user_agent, not BBOT/<version>.
OFFENDERS=""
for f in bbot/modules/urlscan.py bbot/modules/dockerhub.py bbot/modules/templates/postman.py bbot/modules/wayback.py; do
[ -f "$f" ] || continue
if grep -qE '(base_url|api_url|site_url)\s*=\s*f?"https?://' "$f" && grep -qE 'self\.helpers\.request\(' "$f"; then
OFFENDERS="$OFFENDERS $(basename "$f" .py)"
fi
done
if grep -q 'kwargs\["headers"\]\["User-Agent"\] = f"BBOT/{__version__}"' bbot/modules/base.py; then
pass V11 "api_request() sends User-Agent BBOT/<version>"
else
fail V11 "api_request() sends User-Agent BBOT/<version>" "base.py pattern not found"
fi
MIG=docs/migration/3.1_breaking_changes.md
UA_SECTION=$(awk '/^### API requests identify as BBOT/{s=1;next} /^(---|### )/{s=0} s' "$MIG")
QUAL='not every module|some third-party|api_request|API request helper|not all'
ua_unqualified() { [ -n "$UA_SECTION" ] && echo "$UA_SECTION" | grep -q 'BBOT/<version>' && ! echo "$UA_SECTION" | grep -qiE "$QUAL"; }
if ua_unqualified && [ -n "$OFFENDERS" ]; then
fail V01 "3.1 migration note's User-Agent claim matches the modules" "claims all third-party API requests send BBOT/<version>; still on web.user_agent:$OFFENDERS"
else
pass V01 "3.1 migration note's User-Agent claim matches the modules"
fi
help_unqualified() { grep -E 'BBOT/<version>' "$1" | grep -i 'user-agent' | grep -viqE '\((most|some)[^)]*BBOT/<version>'; }
if [ -n "$OFFENDERS" ] && { help_unqualified bbot/scanner/preset/args.py || help_unqualified docs/scanning/advanced.md; }; then
fail V02 "--user-agent help text matches the modules" "args.py or advanced.md says API calls send BBOT/<version> without qualification; still on web.user_agent:$OFFENDERS"
else
pass V02 "--user-agent help text matches the modules"
fi
LF=bbot/modules/lightfuzz/lightfuzz.py
WAF_ROW=$(grep -F '`try_bypasses` (default)' docs/modules/lightfuzz.md)
if echo "$WAF_ROW" | grep -q 'GET' && ! echo "$WAF_ROW" | grep -q 'try_get_as_post' \
&& grep -q 'if self.try_get_as_post and original_type == "GETPARAM":' "$LF" \
&& grep -q 'try_get_as_post: True' bbot/presets/web/lightfuzz-xss.yml; then
fail V03 "lightfuzz WAF Handling describes GET params under try_get_as_post" "doc says GET params are always skipped; lightfuzz.py retests them as POST in bypass mode and lightfuzz-xss turns that on"
else
pass V03 "lightfuzz WAF Handling describes GET params under try_get_as_post"
fi
if grep -q 'requires-python = ">=3.11' pyproject.toml && grep -q 'badge/python-3.11+' README.md && grep -q 'BBOT requires Python 3.11+' docs/troubleshooting.md; then
pass V04 "README badge and troubleshooting say 3.11+, matching requires-python"
else
fail V04 "README badge and troubleshooting say 3.11+, matching requires-python" "mismatch"
fi
if grep -qE '^ threads: 50$' bbot/defaults.yml && grep -qE '^ max_concurrency: 256$' bbot/defaults.yml \
&& grep -q 'default 50 queries per resolver' docs/scanning/tips_and_tricks.md && grep -q '256 total' docs/scanning/tips_and_tricks.md; then
pass V05 "DNS speed tip matches dns.threads=50 and dns.max_concurrency=256"
else
fail V05 "DNS speed tip matches dns.threads=50 and dns.max_concurrency=256" "mismatch"
fi
if ! grep -qE '^\s+threads:' bbot/presets/subdomain-enum.yml && ! grep -q 'threads: 25' README.md && ! grep -q 'threads: 25' docs/scanning/presets_list.md; then
pass V06 "subdomain-enum no longer lowers dns.threads, and README/presets_list match"
else
fail V06 "subdomain-enum no longer lowers dns.threads, and README/presets_list match" "threads still set somewhere"
fi
CC=bbot/modules/internal/cloudcheck.py
if grep -q 'event.add_tag(provider_type)' "$CC" && grep -q 'event.add_tag(provider_name)' "$CC" \
&& ! grep -rqE 'cdn-github|cloud-azure|cdn-akamai|cdn-ip' docs/scanning/output.md docs/scanning/events.md docs/modules/custom_yara_rules.md; then
pass V07 "tag examples use cloudcheck's separate type and provider tags"
else
fail V07 "tag examples use cloudcheck's separate type and provider tags" "old cdn-x tags remain or code differs"
fi
ok=1
grep -q 'avoid_wafs: always' bbot/presets/web/lightfuzz-light.yml || ok=0
grep -q 'avoid_wafs: try_bypasses' bbot/presets/web/lightfuzz.yml || ok=0
grep -q 'avoid_wafs: never' bbot/presets/web/lightfuzz-max.yml || ok=0
grep -q 'avoid_wafs: try_bypasses' bbot/presets/web/lightfuzz-xss.yml || ok=0
grep -q 'avoid_wafs: Literal\["always", "never", "try_bypasses"\]' "$LF" || ok=0
if [ "$ok" = 1 ]; then pass V08 "lightfuzz preset tables match each preset's avoid_wafs"; else fail V08 "lightfuzz preset tables match each preset's avoid_wafs" "mismatch"; fi
if grep -q '"--custom-nameservers"' bbot/scanner/preset/args.py && [ -f bbot/presets/nuclei/nuclei-recommended.yml ] && [ -f bbot/presets/nuclei/nuclei-new.yml ]; then
pass V09 "--custom-nameservers, nuclei-recommended and nuclei-new exist as documented"
else
fail V09 "--custom-nameservers, nuclei-recommended and nuclei-new exist as documented" "missing"
fi
if N=$(gh api --method GET "repos/blacklanternsecurity/public-dns-servers/contents/nameservers.txt?ref=master" -H "Accept: application/vnd.github.raw+json" 2>/dev/null | grep -cvE '^\s*(#|$)'); then
if [ "$N" -ge 4000 ] && [ "$N" -le 6000 ]; then pass V10 "brute_nameservers list is ~5K servers ($N)"; else fail V10 "brute_nameservers list is ~5K servers" "count $N"; fi
else
skip V10 "brute_nameservers list is ~5K servers" "could not fetch nameservers.txt"
fi
# C13 (A5): generated blocks match their sources
if python3 - <<'PY'
p=open('bbot/presets/subdomain-enum.yml').read().strip(); r=open('README.md').read()
l=open('docs/scanning/presets_list.md').read(); c=open('docs/scanning/configuration.md').read(); d=open('bbot/defaults.yml').read().strip()
import re,sys; a=open('bbot/scanner/preset/args.py').read(); adv=open('docs/scanning/advanced.md').read()
h=re.search(r'"--user-agent",\s*"-ua",\s*help="([^"]+)"', a)
sys.exit(0 if (p in r and "\n".join(" "+x for x in p.split("\n")) in l and d in c and h and h.group(1) in adv) else 1)
PY
then pass V12 "generated README, presets_list, configuration and advanced.md blocks match their sources"
else fail V12 "generated README, presets_list, configuration and advanced.md blocks match their sources" "a generated block is stale"; fi
# C16/C17: lightfuzz-xss table's POST row matches the preset
XSS_TABLE=$(awk '/^### .-p lightfuzz-xss./{s=1} /^### Spider/{s=0} s' docs/modules/lightfuzz.md)
if echo "$XSS_TABLE" | grep -q 'POST fuzzing\*\* | Disabled' && ! grep -qE 'disable_post: *True' bbot/presets/web/lightfuzz-xss.yml; then
fail V13 "lightfuzz-xss table's POST fuzzing row matches the preset" "table says Disabled; preset leaves disable_post False and sets try_get_as_post"
else
pass V13 "lightfuzz-xss table's POST fuzzing row matches the preset"
fi
# C3/C14 (A6): more modules call third-party APIs outside api_request()
MORE=""
for m in azure_tenant docker_pull; do
grep -qE 'self\.helpers\.request\(' "bbot/modules/$m.py" && grep -qE 'https://(azmap\.dev|registry-1\.docker\.io|login\.microsoftonline\.com)' "bbot/modules/$m.py" && MORE="$MORE $m"
done
if ua_unqualified && [ -n "$MORE" ]; then
fail V14 "3.1 migration note holds for azure_tenant and docker_pull" "unqualified claim; still on web.user_agent:$MORE"
else
pass V14 "3.1 migration note holds for azure_tenant and docker_pull"
fi
# ---- round 2: the 3.1.0 release history (fedf360c) ----------------------------
RH=docs/release_history.md
SEC=$(awk '/^### 3\.1\.0 /{s=1;print;next} /^### [0-9]/{s=0} s' "$RH")
# V15: the speed claim matches the measurement it cites (#3606 "Effect on a scan").
B3606=$(gh api --method GET repos/blacklanternsecurity/bbot/pulls/3606 --jq .body 2>/dev/null)
if [ -z "$SEC" ] || [ -z "$B3606" ]; then
skip V15 "3.1.0 speed claim matches #3606's measurement" "no 3.1.0 section or #3606 unreadable"
elif ! echo "$B3606" | grep -q '| wall time | 22m 50s | 11m 51s |' || ! echo "$B3606" | grep -q 'against a large wildcard target'; then
broken V15 "3.1.0 speed claim matches #3606's measurement" "#3606 no longer carries the measurement this check reads"
else
# C4/R2C1/R2C2: every speed claim in the section must carry #3606's measurement and its target:
# its times (22m 50s, 11m 51s) or its ratio (48%, half, halved), plus "large wildcard"; no other figure, no "typical".
BAD=""
while IFS= read -r L; do
[ -z "$L" ] && continue
M=0; { echo "$L" | grep -q '22m 50s' && echo "$L" | grep -q '11m 51s'; } && M=1
echo "$L" | grep -qiE '48 ?%|half|halve[sd]?|twice as fast|(^|[^0-9.])2 ?x (faster|quicker)' && M=1 # C3 (r5): accept accurate rewordings
# R6 R2C1: a figure at or below #3606's 48% on a line scoped to 'large wildcard' is supported too
echo "$L" | grep -oE '[0-9]+ ?%' | tr -d ' %' | awk '$1<=48{f=1} END{exit !f}' && M=1
PCT=$(echo "$L" | grep -oE '[0-9]+ ?%' | tr -d ' %' | awk '$1>48')
if [ "$M" = 0 ] || ! echo "$L" | grep -qi 'large wildcard' || [ -n "$PCT" ] || echo "$L" | grep -qi 'typical'; then BAD="$BAD | ${L:0:120}"; fi
done <<< "$(echo "$SEC" | grep -iE 'faster|quicker|slower|less time|as long|wall time|halve[sd]?|twice|double[sd]?|took|[0-9]+m [0-9]+s|[0-9]+ ?% (less|more|quicker)|finish(es)? in' | grep -iE 'scan')"
if [ -z "$BAD" ]; then pass V15 "3.1.0 speed claims match #3606's measurement"
else fail V15 "3.1.0 speed claims match #3606's measurement" "unmeasured speed claim:$BAD; #3606: one large wildcard target, 22m 50s to 11m 51s (48% less), 510 to 513 subdomains"; fi
fi
# ---- round 3 -------------------------------------------------------------------
# V19: lightfuzz-heavy (includes lightfuzz -> avoid_wafs: try_bypasses; disable_post False, try_get_as_post True)
# does not describe itself as skipping WAFs in its preset file (what `bbot -lp` prints; presets_list.md is generated from it).
H=bbot/presets/web/lightfuzz-heavy.yml
eff=$(grep -oE 'avoid_wafs: *[a-z_]+' "$H" | awk '{print $2}')
[ -z "$eff" ] && grep -qE '^ - lightfuzz$' "$H" && eff=$(grep -oE 'avoid_wafs: *[a-z_]+' bbot/presets/web/lightfuzz.yml | awk '{print $2}')
if [ -z "$eff" ]; then
broken V19 "lightfuzz-heavy's description matches its avoid_wafs" "could not resolve avoid_wafs for lightfuzz-heavy"
elif [ "$eff" = try_bypasses ] && grep -q 'disable_post: False' "$H" && grep -m1 '^description:' "$H" | grep -qi 'skips confirmed WAFs'; then
fail V19 "lightfuzz-heavy's description matches its avoid_wafs" "says it still skips confirmed WAFs; it runs try_bypasses with POST fuzzing on"
else
pass V19 "lightfuzz-heavy's description matches its avoid_wafs"
fi
# V20: each lightfuzz preset's "WAF avoidance" row in lightfuzz.md names the avoid_wafs value its YAML sets.
LFMD=docs/modules/lightfuzz.md; BAD=""; N=0
for p in lightfuzz-light lightfuzz lightfuzz-max lightfuzz-xss; do
eff=$(grep -oE 'avoid_wafs: *[a-z_]+' "bbot/presets/web/$p.yml" | awk '{print $2}')
row=$(awk -v h="### \`-p $p\`" '$0==h{s=1;next} /^### /{s=0} s && /\*\*WAF avoidance\*\*/' "$LFMD")
N=$((N+1))
if [ -z "$eff" ] || [ -z "$row" ] || ! echo "$row" | grep -q "avoid_wafs: $eff"; then BAD="$BAD $p(yaml=$eff)"; fi
done
if [ -z "$BAD" ]; then pass V20 "lightfuzz.md WAF avoidance rows match each preset's avoid_wafs ($N presets)"
else fail V20 "lightfuzz.md WAF avoidance rows match each preset's avoid_wafs" "row missing or wrong:$BAD"; fi
# V21: v3.0.2 lightfuzz skipped every WAF-tagged host (top-level avoid_wafs, default True); this head defaults
# to try_bypasses. The 3.1 migration guide's Modules section says so (C6: any heading).
"${GIT[@]}" fetch --quiet origin tag v3.0.2 2>/dev/null || true
OLD=$(git show v3.0.2:bbot/modules/lightfuzz/lightfuzz.py 2>/dev/null | grep -c 'self.avoid_wafs = self.scan.config.get("avoid_wafs", True)')
NEW=$(grep -c '"try_bypasses",' bbot/modules/lightfuzz/lightfuzz.py)
if [ "$OLD" != 1 ] || [ "$NEW" -lt 1 ]; then
broken V21 "3.1 migration guide documents lightfuzz's WAF default change" "could not read v3.0.2 or head lightfuzz.py"
elif awk '/^## /{m=($0 ~ /^## Modules/)} m && /avoid_wafs/' docs/migration/3.1_breaking_changes.md | grep -q .; then
pass V21 "3.1 migration guide documents lightfuzz's WAF default change"
else
fail V21 "3.1 migration guide documents lightfuzz's WAF default change" "3.0.2 skipped every WAF host; head defaults to try_bypasses; no lightfuzz/avoid_wafs entry in the guide"
fi
# ---- round 4 (cd2bf7ff) ----------------------------------------------------------
LFSEC=$(awk '/^### `lightfuzz` probes WAF/{s=1;next} /^### /{s=0} s' "$MIG")
# V22: the entry's 3.0 -> 3.1 table matches v3.0.2 and this head.
O_LF=$(git show v3.0.2:bbot/modules/lightfuzz/lightfuzz.py 2>/dev/null)
O_MAX=$(git show v3.0.2:bbot/presets/web/lightfuzz-max.yml 2>/dev/null)
if [ -z "$O_LF" ] || [ -z "$O_MAX" ]; then
broken V22 "migration WAF table matches v3.0.2 and this head" "could not read v3.0.2"
else
ok=1; why=""
echo "$O_LF" | grep -q 'self.avoid_wafs = self.scan.config.get("avoid_wafs", True)' || { ok=0; why="$why v3.0.2-not-top-level"; }
echo "$O_MAX" | grep -qE '^ avoid_wafs: False' || { ok=0; why="$why v3.0.2-max-no-False"; }
grep -qE '^ avoid_wafs: never' bbot/presets/web/lightfuzz-max.yml || { ok=0; why="$why head-max-not-never"; }
for s in 'no longer read' '`lightfuzz-max` still skipped WAF-tagged hosts' '`lightfuzz-max` sets `never`' '`true` / `false` still work as `always` / `never`' 'defaults to `try_bypasses`'; do
echo "$LFSEC" | grep -qF "$s" || { ok=0; why="$why doc-missing:[$s]"; }
done
if [ "$ok" = 1 ]; then pass V22 "migration WAF table matches v3.0.2 and this head"
else fail V22 "migration WAF table matches v3.0.2 and this head" "$why"; fi
fi
# V23: the old-behaviour recipe works, and the entry names exactly the presets that pull in nowafpls.
incl() { awk '/^include:/{s=1;next} /^[a-z]/{s=0} s && /^ - /{print $2}' "bbot/presets/web/$1.yml"; }
mods() { awk '/^modules:/{s=1;next} /^[a-z]/{s=0} s && /^ - /{print $2}' "bbot/presets/web/$1.yml"; }
has_nowafpls() { local p="$1" d="${2:-0}"; [ "$d" -gt 5 ] && return 1; [ -f "bbot/presets/web/$p.yml" ] || return 1
{ incl "$p"; mods "$p"; } | grep -qx nowafpls && return 0
for i in $(incl "$p"); do has_nowafpls "$i" $((d+1)) && return 0; done; return 1; }
ACT=$(for f in bbot/presets/web/lightfuzz*.yml; do p=$(basename "$f" .yml); has_nowafpls "$p" && echo "$p"; done | sort | tr '\n' ' ')
DOC=$(echo "$LFSEC" | grep -F 'also enable the `nowafpls` module' | grep -oE '`lightfuzz[a-z-]*`' | tr -d '`' | sort -u | tr '\n' ' ')
ok=1; why=""
[ -n "$DOC" ] && [ "$ACT" = "$DOC" ] || { ok=0; why="$why presets-with-nowafpls=[$ACT] doc=[$DOC]"; }
echo "$LFSEC" | grep -qF -- '-c modules.lightfuzz.avoid_wafs=always -em nowafpls' || { ok=0; why="$why recipe-missing"; }
grep -q 'if self.avoid_wafs == "always":' "$LF" && [ -f bbot/modules/nowafpls.py ] || { ok=0; why="$why code"; }
grep -qE '^ avoid_wafs: always' bbot/presets/web/lightfuzz-light.yml && echo "$LFSEC" | grep -qF '`lightfuzz-light` still skips WAF-tagged hosts' || { ok=0; why="$why light"; }
if [ "$ok" = 1 ]; then pass V23 "old-behaviour recipe and nowafpls preset list match the presets"
else fail V23 "old-behaviour recipe and nowafpls preset list match the presets" "$why"; fi
# V24: avoid_wafs still accepts the boolean form, as the entry says.
if awk '/def _normalize_avoid_wafs/{s=1} s && n++<6' "$LF" | grep -q 'return "always" if v else "never"'; then
pass V24 "modules.lightfuzz.avoid_wafs maps true/false to always/never"
else fail V24 "modules.lightfuzz.avoid_wafs maps true/false to always/never" "validator not found"; fi
# V25: lightfuzz.yml and lightfuzz-heavy.yml (try_bypasses) don't say they skip WAFs, and presets_list.md
# carries every lightfuzz preset description verbatim (it is generated from them).
ok=1; why=""
for p in lightfuzz lightfuzz-heavy; do grep -m1 '^description:' "bbot/presets/web/$p.yml" | grep -qiE 'skips confirmed WAFs' && { ok=0; why="$why $p-says-skips"; }; done
for f in bbot/presets/web/lightfuzz*.yml; do
d=$(grep -m1 '^description:' "$f" | sed -E 's/^description: "(.*)"$/\1/')
grep -qF -- "$d" docs/scanning/presets_list.md || { ok=0; why="$why stale:$(basename "$f" .yml)"; }
done
if [ "$ok" = 1 ]; then pass V25 "lightfuzz preset descriptions match try_bypasses and presets_list.md"
else fail V25 "lightfuzz preset descriptions match try_bypasses and presets_list.md" "$why"; fi
CHANGED="$WORK/changed.txt"
printf '%s\n' README.md docs/migration/3.1_breaking_changes.md docs/modules/custom_yara_rules.md docs/modules/lightfuzz.md docs/modules/nuclei.md docs/modules/wayback.md docs/release_history.md docs/scanning/advanced.md docs/scanning/configuration.md docs/scanning/events.md docs/scanning/output.md docs/scanning/presets.md docs/scanning/presets_list.md docs/scanning/tips_and_tricks.md docs/troubleshooting.md > "$CHANGED"; export CHANGED
# V26: relative links in the changed docs resolve, including #anchors to headings (static stand-in for a docs build).
BAD=$(python3 - <<'PY'
import re, os, subprocess
files = [l for l in open(os.environ["CHANGED"]).read().split() if l.startswith("docs/") and l.endswith(".md")]
def slug(h): return re.sub(r"[^\w\- ]", "", h.strip().lower()).replace(" ", "-")
bad = []
for f in files:
for m in re.finditer(r"\]\(([^)\s]+)\)", open(f).read()):
t = m.group(1)
# skip schemes, and the generator's placeholder [0]("") links (37 of them at base too)
if re.match(r"^[a-z]+:", t) or t.strip('"') == "": continue
path, _, anc = t.partition("#")
tgt = os.path.normpath(os.path.join(os.path.dirname(f), path)) if path else f
if not os.path.exists(tgt): bad.append(f"{f}->{t}"); continue
if anc and tgt.endswith(".md"):
heads = {slug(re.sub(r"^#+\s*", "", l)) for l in open(tgt) if l.startswith("#")}
if anc not in heads: bad.append(f"{f}->{t}")
print(" ".join(bad))
PY
)
if [ -z "$BAD" ]; then pass V26 "relative links and anchors in the changed docs resolve"
else fail V26 "relative links and anchors in the changed docs resolve" "$BAD"; fi
# V27: the 3.1.0 'Check your setup' box flags the lightfuzz WAF change with the same recipe as the guide.
BOX=$(awk '/^!!! warning "Check your setup before upgrading"/{s=1;next} /^#### /{s=0} s' "$RH")
if echo "$BOX" | grep -qi 'lightfuzz' && echo "$BOX" | grep -qF 'modules.lightfuzz.avoid_wafs: always' && echo "$BOX" | grep -qF -- '-em nowafpls'; then
pass V27 "3.1.0 Check-your-setup box flags the lightfuzz WAF change"
else fail V27 "3.1.0 Check-your-setup box flags the lightfuzz WAF change" "no lightfuzz WAF bullet with the recipe"; fi
# V28 (challenge C7): no doc row says avoid_wafs=never fuzzes unpadded / as-is, since under never
# prepare_request still pads POST/JSON bodies on WAF hosts (submodules/base.py) and the repo's
# Test_Nowafpls_never_still_pads asserts it.
BASEPY=bbot/modules/lightfuzz/submodules/base.py
if ! grep -q 'if "waf" in self.event.tags and self.lightfuzz.avoid_wafs != "always":' "$BASEPY" \
|| ! grep -q 'class Test_Nowafpls_never_still_pads' bbot/test/test_step_2/module_tests/test_module_lightfuzz.py; then
pass V28 "docs describe avoid_wafs=never the way the code pads" "(code no longer pads under never)"
else
BAD=""
echo "$LFSEC" | grep -iE '`never`' | grep -qiE 'without padding|as-is|\braw\b|unpadded' && BAD="$BAD migration-guide"
grep -E '^\| `never` \|' "$LFMD" | grep -qiE 'as-is|raw|without padding' && BAD="$BAD lightfuzz.md-WAF-table"
grep -F 'avoid_wafs: never' "$LFMD" | grep -qiE 'as-is|raw|without padding' && BAD="$BAD lightfuzz.md-max-preset-row"
if [ -z "$BAD" ]; then pass V28 "docs describe avoid_wafs=never the way the code pads"
else fail V28 "docs describe avoid_wafs=never the way the code pads" "says unpadded/as-is:$BAD; base.py pads POST/JSON bodies on WAF hosts unless avoid_wafs=always"; fi
fi
# V29 (round 5): the #3629 lines in 3.1.0 match #3629 as merged to dev: FINDING gains an optional port
# rendered host:port, legba and medusa pass it, and nuclei correlation is fixed.
M3629=3085baacf1f812fcf446d73f673aa2454b199ce4
"${GIT[@]}" fetch --quiet origin "$M3629" 2>/dev/null || true
MA=$(gh api --method GET repos/blacklanternsecurity/bbot/pulls/3629 --jq '.merged_at // "no"' 2>/dev/null) || MA=err
if ! git cat-file -e "$M3629^{commit}" 2>/dev/null || [ "$MA" = err ]; then
skip V29 "3.1.0 #3629 lines match #3629" "could not read #3629 or its merge commit"
else
ok=1; why=""
[ "$MA" != no ] || { ok=0; why="$why not-merged"; }
git show "$M3629:bbot/core/event/base.py" | grep -q 'port: Optional\[int\] = None' || { ok=0; why="$why no-FINDING-port"; }
git show "$M3629:bbot/core/event/base.py" | grep -q 'make_netloc(host, port) if port else host' || { ok=0; why="$why no-host:port"; }
for m in legba medusa; do git show "$M3629:bbot/modules/$m.py" | grep -q '"port": str(event.port)' || { ok=0; why="$why $m"; }; done
git show "$M3629^1:bbot/modules/nuclei.py" | grep -q 'def correlate_event' && ! git show "$M3629:bbot/modules/nuclei.py" | grep -q 'def correlate_event' || { ok=0; why="$why nuclei-correlate"; }
grep -q 'FINDING has an optional `port`.*legba and medusa.*pull/3629' "$RH" && grep -q 'nuclei:.*Results are parented.*pull/3629' "$RH" || { ok=0; why="$why doc-lines-missing"; }
if [ "$ok" = 1 ]; then pass V29 "3.1.0 #3629 lines match #3629 as merged"; else fail V29 "3.1.0 #3629 lines match #3629 as merged" "$why"; fi
fi
# V30 (challenge C17, round 5): the 3.1.0 Dependencies section reflects #3627 (merged to dev before #3629,
# whose notes c068b085 added): pyjwt, lxml and socksio dropped, xmltojson replaced by xmltodict.
M3627=5d968b1c421eb0568f19abaaccb0c7fc92130a32
"${GIT[@]}" fetch --quiet origin "$M3627" 2>/dev/null || true
if ! git cat-file -e "$M3627^{commit}" 2>/dev/null; then
skip V30 "3.1.0 Dependencies reflects #3627" "could not fetch #3627's merge"
elif ! git diff "$M3627^1" "$M3627" -- pyproject.toml | grep -q '^+ "xmltodict'; then
broken V30 "3.1.0 Dependencies reflects #3627" "#3627 no longer shows the xmltodict swap"
else
DEP=$(awk '/^### 3\.1\.0 /{s=1} /^### [0-9]/ && !/3\.1\.0/{s=0} s' "$RH" | awk '/^#### Dependencies/{s=1;next} /^#### /{s=0} s')
if echo "$DEP" | grep -qiE 'xmltodict|pyjwt|lxml|socksio|3627'; then pass V30 "3.1.0 Dependencies reflects #3627"
else fail V30 "3.1.0 Dependencies reflects #3627" "no mention of #3627's removals (pyjwt, lxml, socksio) or the xmltojson -> xmltodict swap"; fi
fi
# ------------------------------------------------------------------------------
# V31 (round 6): each dependency version in 3.1.0 Dependencies is either the one dev pins, or comes
# from a cited PR that has merged. 54270515 lists blastdns 2.0.4 citing #3631.
DEP=$(awk '/^### 3\.1\.0 /{s=1} /^### [0-9]/ && !/3\.1\.0/{s=0} s' "$RH" | awk '/^#### Dependencies/{s=1;next} /^#### /{s=0} s')
"${GIT[@]}" fetch --quiet origin dev 2>/dev/null || true
DEVPIN=$(git show FETCH_HEAD:pyproject.toml 2>/dev/null | grep -oE '"blastdns>=[0-9.]+' | grep -oE '[0-9]+\.[0-9]+\.[0-9]+')
NOTED=$(echo "$DEP" | grep -oE 'blastdns [0-9]+\.[0-9]+\.[0-9]+' | grep -oE '[0-9.]+$' | head -1)
if [ -z "$DEVPIN" ] || [ -z "$NOTED" ]; then
skip V31 "3.1.0 blastdns version is dev's pin or comes from a merged PR" "could not read dev's pin or the notes' blastdns line"
elif [ "$NOTED" = "$DEVPIN" ]; then
pass V31 "3.1.0 blastdns version is dev's pin or comes from a merged PR" "(notes $NOTED, dev pins >=$DEVPIN)"
else
M3631=$(gh api --method GET repos/blacklanternsecurity/bbot/pulls/3631 --jq '.merged_at // "no"' 2>/dev/null) || M3631=err
if [ "$M3631" = err ]; then skip V31 "3.1.0 blastdns version is dev's pin or comes from a merged PR" "could not read #3631"
elif [ "$M3631" != no ] && echo "$DEP" | grep -q 'pull/3631'; then pass V31 "3.1.0 blastdns version is dev's pin or comes from a merged PR" "(#3631 merged $M3631)"
else fail V31 "3.1.0 blastdns version is dev's pin or comes from a merged PR" "notes say blastdns $NOTED, dev pins >=$DEVPIN, and #3631 (the 2.0.4 bump) is not merged"; fi
fi
# V32 (round 6): the avoid_wafs Field description and the generated configuration.md row don't say
# `never` fuzzes raw / without padding, and the two carry the same description.
LFPY=bbot/modules/lightfuzz/lightfuzz.py; CFG=docs/scanning/configuration.md
FIELD=$(python3 - "$LFPY" <<'PY'
import ast, sys
src = open(sys.argv[1]).read()
for node in ast.walk(ast.parse(src)):
if isinstance(node, ast.AnnAssign) and getattr(node.target, "id", "") == "avoid_wafs":
for kw in node.value.keywords:
if kw.arg == "description":
print(ast.literal_eval(kw.value))
PY
)
CFGROW=$(grep -E '^\| modules\.lightfuzz\.avoid_wafs ' "$CFG" | awk -F'|' '{print $4}' | sed 's/^ *//; s/ *$//')
if [ -z "$FIELD" ] || [ -z "$CFGROW" ]; then
broken V32 "avoid_wafs description says never still pads, and configuration.md matches it" "could not read the Field description or the configuration.md row"
else
BAD=""
echo "$FIELD" | grep -qiE "'never' fuzzes them raw|never[^;]*(without padding|unpadded|as-is)" && BAD="$BAD Field-says-raw"
[ "$FIELD" = "$CFGROW" ] || BAD="$BAD configuration.md-differs"
if [ -z "$BAD" ]; then pass V32 "avoid_wafs description says never still pads, and configuration.md matches it"
else fail V32 "avoid_wafs description says never still pads, and configuration.md matches it" "$BAD"; fi
fi
# V33 (round 6, challenges C6/C7): each version (blastdns aside) in the 3.1.0 Dependencies line matches dev's uv.lock
# at 5d737af6 (dev's tip when round 6 was collected; pinned so a replay gets the same answer).
DEV=5d737af6f40f40ccb6a32d222ae24861b3c20baa
"${GIT[@]}" fetch --quiet origin "$DEV" 2>/dev/null || true
if ! git cat-file -e "$DEV^{commit}" 2>/dev/null; then
skip V33 "3.1.0 Dependencies versions match dev's uv.lock at 5d737af6" "could not fetch $DEV"
else
LOCK=$(git show "$DEV:uv.lock")
DEP=$(awk '/^### 3\.1\.0 /{s=1} /^### [0-9]/ && !/3\.1\.0/{s=0} s' "$RH" | awk '/^#### Dependencies/{s=1;next} /^#### /{s=0} s')
BAD=""
# R6 R2C4: blastdns is V31's (it depends on whether #3631 merges first), not V33's
for p in blasthttp baddns cloudcheck radixtarget pycryptodome; do
N=$(echo "$DEP" | grep -oE "$p [0-9]+\.[0-9]+\.[0-9]+" | head -1 | cut -d' ' -f2)
L=$(echo "$LOCK" | awk -v n="name = \"$p\"" '$0==n{getline; gsub(/"/,"",$3); print $3; exit}')
[ -n "$N" ] && [ "$N" != "$L" ] && BAD="$BAD $p:notes=$N,lock=$L"
done
if [ -z "$BAD" ]; then pass V33 "3.1.0 Dependencies versions match dev's uv.lock at 5d737af6"
else fail V33 "3.1.0 Dependencies versions match dev's uv.lock at 5d737af6" "$BAD"; fi
fi
# V34 (round 7): every version in the 3.1.0 Dependencies line, blastdns included, matches dev's uv.lock
# at 72ebe17c (dev's tip at round 7's collect: #3631 merged, blastdns 2.0.4; pinned so a replay agrees).
DEV34=72ebe17ca023c191a0b1a0ff1586e124c7f33031
"${GIT[@]}" fetch --quiet origin "$DEV34" 2>/dev/null || true
if ! git cat-file -e "$DEV34^{commit}" 2>/dev/null; then
skip V34 "3.1.0 Dependencies versions match dev's uv.lock at 72ebe17c" "could not fetch $DEV34"
else
LOCK34=$(git show "$DEV34:uv.lock")
DEP34=$(awk '/^### 3\.1\.0 /{s=1} /^### [0-9]/ && !/3\.1\.0/{s=0} s' "$RH" | awk '/^#### Dependencies/{s=1;next} /^#### /{s=0} s')
BAD=""; N_CHECKED=0
for p in blastdns blasthttp baddns cloudcheck radixtarget pycryptodome; do
N=$(echo "$DEP34" | grep -oE "$p [0-9]+\.[0-9]+\.[0-9]+" | head -1 | cut -d' ' -f2)
L=$(echo "$LOCK34" | awk -v n="name = \"$p\"" '$0==n{getline; gsub(/"/,"",$3); print $3; exit}')
[ -n "$N" ] && N_CHECKED=$((N_CHECKED+1))
[ -n "$N" ] && [ "$N" != "$L" ] && BAD="$BAD $p:notes=$N,lock=$L"
done
if [ "$N_CHECKED" -lt 6 ]; then broken V34 "3.1.0 Dependencies versions match dev's uv.lock at 72ebe17c" "read only $N_CHECKED of 6 versions from the notes"
elif [ -z "$BAD" ]; then pass V34 "3.1.0 Dependencies versions match dev's uv.lock at 72ebe17c" "($N_CHECKED versions)"
else fail V34 "3.1.0 Dependencies versions match dev's uv.lock at 72ebe17c" "$BAD"; fi
fi
# ------------------------------------------------------------------------------
summary
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Brings the docs up to date with 3.1.
Migration guide
pip install --upgrade bbotstays on 3.0 instead of failing.User-Agent: BBOT/<version>instead ofweb.user_agent. Not every module uses the helper yet, so the guide says not to rely on it for allowlisting.Docs updated for 3.1
--custom-nameservers.conditionsare described as read-only, not as a way to change the scan.avoid_wafsmodes, including howtry_get_as_postretests GET parameters as padded POST bodies. The preset tables match what each preset sets.nuclei-recommendedandnuclei-newpresets.Small source changes
subdomain-enumpreset: dropsdns.threads: 25. It was a raise over the old default of 10, but now lowers the default of 50 (Raise DNS concurrency defaults and size dnsresolve from them #3606).--user-agenthelp no longer says "all HTTP requests", and notes that most API calls sendBBOT/<version>.dns.brute_nameserverscomment: the list is about 5K servers, not 10K.Older fixes
configuration.mdnamed options that don't exist (max_threads,http_user_agent).output.md,events.mdandcustom_yara_rules.mdusedcdn-github-style tags. cloudcheck tags a host with the type (cdn) and the provider (github).Generated docs were rebuilt with
bbot/scripts/docs.py.Validation
bbot -p subdomain-enum --current-preset-full, dns section, ondev(402f32e):On this branch:
diffof the two full outputs:threadsis the only value that changes. The rest of the diff is the same module config blocks printed in a different order.ruff checkandruff format --checkpass onargs.py.