Skip to content

Commit 91bd6a3

Browse files
SimonSchubertSimonSchubert
committed
Add commands
1 parent 1553492 commit 91bd6a3

2 files changed

Lines changed: 236 additions & 0 deletions

File tree

‎assets/commands/index.txt‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6455,6 +6455,7 @@ prowler-kubernetes.md
64556455
prowler-m365.md
64566456
prowler.md
64576457
proxify.md
6458+
proxy-scraper.md
64586459
proxychains.md
64596460
prqlc.md
64606461
prs.md

‎assets/commands/proxy-scraper.md‎

Lines changed: 235 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,235 @@
1+
# TAGLINE
2+
3+
Find public proxies that pass a live check
4+
5+
# TLDR
6+
7+
**Open the setup wizard**
8+
9+
```proxy-scraper```
10+
11+
**Stop after 50 proxies** that can tunnel HTTPS
12+
13+
```proxy-scraper --want 50 --https-only -y```
14+
15+
**Keep only** Germany, Austria, and Switzerland
16+
17+
```proxy-scraper --country DE,AT,CH -l 20000 -y```
18+
19+
**Recheck the public hourly list** from this network
20+
21+
```proxy-scraper --recheck live -y```
22+
23+
**Serve those hits** as one local rotating proxy
24+
25+
```proxy-scraper --recheck --serve```
26+
27+
**Print matching hits** on standard output
28+
29+
```proxy-scraper --recheck live --want 20 -y -o -```
30+
31+
**Write a proxychains config** next to the results
32+
33+
```proxy-scraper --want 30 -y --export proxychains```
34+
35+
**Rank sources** by hit rate
36+
37+
```proxy-scraper --list-sources```
38+
39+
# SYNOPSIS
40+
41+
**proxy-scraper** [_options_]
42+
43+
# DESCRIPTION
44+
45+
**proxy-scraper** collects publicly listed HTTP, SOCKS4, and SOCKS5 proxies and keeps the ones that pass a live check. A hit has to fetch two independent pages and return a foreign exit address, then a static HTML page has to arrive unchanged. HTTPS is tested through a tunnel with verified TLS. Each hit records latency, anonymity (**elite**, **anonymous**, or **transparent**), country, and whether the exit looks like a datacenter or appears on the SpamCop blocklist.
46+
47+
With no arguments in a terminal, a wizard asks what to look for and prints the matching command. **-y** skips the wizard. **--want** _N_ stops once enough matches are found. **--recheck** checks the previous hits again. **--recheck live** starts from the project's hourly public list.
48+
49+
**--serve** turns the hits into one proxy on **127.0.0.1:8899** (HTTP and SOCKS5 on the same port). Each new connection goes through a different upstream. The proxy username can request **country-XX**, **type-http**, **type-socks4**, **type-socks5**, or **session-NAME**.
50+
51+
Each run writes a folder under **results/**. **results/latest.txt** names the newest folder. On macOS and Linux, **results/latest** is also a symlink. The folder holds **all.txt** (`type://ip:port`, fastest first), one list per protocol, **proxies.json**, and **proxies.csv**.
52+
53+
The PyPI package is **proxy-scraper-cli**. The shell command is **proxy-scraper**, and the same program is also installed as **proxy-scraper-cli**. **proxy-scraper --mcp**, or the separate **proxy-scraper-mcp** command, speaks MCP on standard I/O and needs the **mcp** extra (`pipx install "proxy-scraper-cli[mcp]"`, Python 3.10 or newer). The tool itself needs Python 3.9 or newer. Release **1.8.0**. MIT license.
54+
55+
# PARAMETERS
56+
57+
**-y**, **--yes**
58+
59+
> Skip the wizard and start with the defaults, plus any other flags on the command line.
60+
61+
**-i**, **--interactive**
62+
63+
> Open the wizard even when other arguments are present. Those arguments are the starting values.
64+
65+
**-V**, **--version**
66+
67+
> Print the version and exit.
68+
69+
**--types** _http_ _socks4_ _socks5_
70+
71+
> Protocols to check. The default is all three.
72+
73+
**-l**, **--limit** _N_
74+
75+
> Check only the _N_ most promising candidates, ordered by history and source hit rate. **0** means no cap.
76+
77+
**--want** _N_
78+
79+
> Stop once _N_ proxies match the filters.
80+
81+
**--country** _CC_
82+
83+
> Comma-separated country codes, for example **DE,AT,CH**.
84+
85+
**--https-only**
86+
87+
> Keep only proxies that can tunnel HTTPS.
88+
89+
**--anonymity** _anonymous_|_elite_
90+
91+
> Minimum anonymity. **elite** is stricter than **anonymous**.
92+
93+
**--max-latency** _MS_
94+
95+
> Drop proxies slower than this many milliseconds. The check gives up at that latency instead of waiting out the full timeout.
96+
97+
**--no-datacenter**
98+
99+
> Drop exits that look like cloud or hosting addresses.
100+
101+
**--no-blocklisted**
102+
103+
> Drop exits listed by SpamCop.
104+
105+
**--target** _URL_
106+
107+
> Keep only proxies that reach this site. Repeat the flag for another site.
108+
109+
**--recheck** [_FILE_|**live**]
110+
111+
> Skip collection. With no argument, check the last run plus history. **live** downloads the public hourly list and checks it from this network. A path checks that file.
112+
113+
**--fast**
114+
115+
> Skip the HTTPS test. The confirmation request and the anonymity check still run.
116+
117+
**-c**, **--concurrency** _N_
118+
119+
> Simultaneous checks. Default **2000**.
120+
121+
**-t**, **--timeout** _SECONDS_
122+
123+
> Time limit per proxy. Default **8**.
124+
125+
**--connect-timeout** _SECONDS_
126+
127+
> Time limit for the TCP connect. Default **4**.
128+
129+
**--no-geo**
130+
131+
> Skip the country lookup.
132+
133+
**--no-dnsbl**
134+
135+
> Skip the SpamCop lookup.
136+
137+
**--serve** [_PORT_]
138+
139+
> After the run, listen as a rotating proxy. Default port **8899**, bound to **127.0.0.1**.
140+
141+
**--serve-host** _ADDRESS_
142+
143+
> Bind address. **0.0.0.0** accepts connections from other machines.
144+
145+
**--serve-password** _SECRET_
146+
147+
> Require this password on HTTP and SOCKS5 logins and on the status page. **PROXY_SCRAPER_SERVE_PASSWORD** sets the same value without putting it in the process list.
148+
149+
**--rotate** _weighted_|_random_|_round-robin_|_fastest_
150+
151+
> How the server picks the next upstream. The default is **weighted** (faster, proven proxies more often).
152+
153+
**--sticky** _SEC_
154+
155+
> Keep the same upstream for one target site for this many seconds.
156+
157+
**--serve-refill** _HOURS_
158+
159+
> While serving, check fresh proxies on this interval and add the hits to the pool.
160+
161+
**-o**, **--output** _FILE_
162+
163+
> Also write every hit as `type://ip:port`. **-** prints them on standard output and moves the interface to standard error.
164+
165+
**--export** _FORMATS_
166+
167+
> Extra files in the results folder: **proxychains**, **clash**, **curl**, or **all**. Separate names with commas.
168+
169+
**--list-sources** [_N_]
170+
171+
> Print the source ranking by hit rate and exit. Default **50**.
172+
173+
**--discover**
174+
175+
> Search GitHub for new proxy lists on this run.
176+
177+
**--no-discover**
178+
179+
> Turn off the automatic GitHub search.
180+
181+
**--discover-repos** _N_
182+
183+
> Maximum repositories to inspect during discovery. Default **400**, or **40** without a GitHub token.
184+
185+
**--no-cache**
186+
187+
> Download every source list again. Unchanged lists are normally skipped via ETag.
188+
189+
**--all-sources**
190+
191+
> Also load sources marked dead, outdated, or unreachable.
192+
193+
**--completion** _bash_|_zsh_|_fish_
194+
195+
> Print a completion script for that shell.
196+
197+
**--mcp**
198+
199+
> Run as an MCP server on standard I/O. Requires the **mcp** extra and Python 3.10 or newer.
200+
201+
# CONFIGURATION
202+
203+
An installed copy stores learned state (source hit rates and proxy history) in the user data directory: **~/Library/Application Support/proxy-scraper** on macOS, **$XDG_DATA_HOME/proxy-scraper** or **~/.local/share/proxy-scraper** on Linux, and **%LOCALAPPDATA%\proxy-scraper** on Windows. **PROXY_SCRAPER_HOME** replaces that directory. Result files go to **./results** in the current directory.
204+
205+
A checkout that still contains **pyproject.toml** and **proxy_scraper.py** keeps state in **data/** and results in **results/**, both inside the project.
206+
207+
**GITHUB_TOKEN**, or a logged-in **gh**, raises the limit for **--discover**. Without a token the search stops at 40 repositories.
208+
209+
While **--serve** is running, **http://127.0.0.1:8899/__proxy-scraper/status** returns JSON and **/__proxy-scraper/metrics** returns Prometheus text. A password, when set, is required as Basic auth on those paths.
210+
211+
# CAVEATS
212+
213+
A public proxy is operated by someone else, who can read traffic that is not encrypted. Keep passwords, cookies, and personal data off it.
214+
215+
**--serve-host** set to anything other than a loopback address, with no password, is an open proxy for anyone who can reach the port.
216+
217+
Many networks block outbound proxy connections. When the hit rate falls below 0.2% the tool warns and does not treat that run as evidence that its sources went bad.
218+
219+
Hits go stale within minutes. Run **--recheck** before relying on a saved list. The hourly public list still needs a check from this network.
220+
221+
On PyPI the name **proxy-scraper** belongs to an older package. This command is installed as **proxy-scraper-cli**.
222+
223+
# HISTORY
224+
225+
Written by **Maximilian Feix**. First public release **1.0.0** on 24 September 2026. The installable command arrived in **1.3.0** the same day. **1.8.0** (27 September 2026) is the current release.
226+
227+
# SEE ALSO
228+
229+
[curl](/man/curl)(1), [proxychains](/man/proxychains)(1), [mitmproxy](/man/mitmproxy)(1)
230+
231+
# RESOURCES
232+
233+
```[Source code](https://github.com/maximilianfeix/proxy-scraper)```
234+
235+
<!-- verified: 2026-09-27 -->

0 commit comments

Comments
 (0)