[Bug]: Pooled browsers get ~5 s/navigation slower with use, and the janitor never recycles a busy pool #2231

crawl4ai version

0.9.2 (Docker server, unclecode/crawl4ai:0.9.2)

Expected Behavior

A pooled browser serves the thousandth navigation about as fast as the first, or the pool replaces it when it stops doing so.

Current Behavior

A pooled browser degrades with sustained use until it is roughly 5 s per navigation slower than a browser created fresh in the same container, at the same instant, against the same host. Because janitor() only closes browsers that have been idle past a TTL, a server under continuous load never has an idle browser and therefore never recycles one. The degradation persists until the container restarts.

GET /health stays green throughout, so nothing an orchestrator watches reflects it.

Same workload, same code, same config, on one container 30 minutes apart:

The same 20-URL batch, isolated:

degraded pool      128.2 s   (and 150.2 s -> HTTP 504 on the attempt before)
after restart       10.6 s / 10.4 s / 11.2 s
cold container      16.0 s   (first request after boot - cold start is not the problem)

The penalty is per-navigation and origin-independent. Measured in the same minute on the degraded container, comparing the pooled browser against a browser created fresh inside that same container:

                  degraded pooled browser     fresh browser
site A (Shopify)            6.59 s                2.12 s
example.com                 5.79 s                0.72 s
iana.org                    5.92 s                0.72 s

What it is not - each ruled out by measurement

• The target site. curl TTFB 0.58 s; raw Playwright goto inside the container 0.56 s, while the pooled browser needed 6.59 s.
• Target-side throttling of our IP. A fresh browser was fast at the same instant, from the same container, to the same host.
• DNS / egress. getaddrinfo 0.02 s in the container.
• Concurrency. In-process arun_many of 12 URLs: 5.2 s. Three concurrent rounds back to back left single-page time at 1.0 s.
• context.route handlers accumulating from the block_resources hook (the factory calls context.route("**/*") per page on a shared context). A real suspect, but 30 pages with the hook attached stayed flat at 1.1 s.
• Browser age or page count alone. 60 sequential real pages on one crawler: flat 1.05-1.15 s. 160 inline raw: page creations: flat.

I could not reproduce it in-process. Four attempts failed to trigger it, so something about sustained real page crawling through the server's pooled crawler is responsible and I cannot name it. The threshold is somewhere between 60 pages (clean) and several hundred (badly degraded); I have not bracketed it further. I would rather report the effect with the evidence than guess at the mechanism.

Why it never recovers on its own

crawler_pool.janitor() closes a browser only after it has been idle past cold_ttl / hot_ttl, and skips any crawler with active_requests > 0. Under steady load neither condition is ever met, so the workload that causes the degradation is exactly the workload that prevents recycling. release_crawler is called in a finally, so the count does return to zero - idleness, not leakage, is the blocker.

Two suggestions

1. Recycle on use, not only on idleness. USAGE_COUNT and LAST_USED already exist in crawler_pool.py, so a pool.max_uses / pool.max_age_s sweep in janitor() is a small change: close a browser past either threshold when active_requests == 0, and let get_crawler() re-create it on demand. That is a mitigation, not a fix - it bounds the damage without explaining the cause. Happy to open a PR if you want it.

We run this today from outside the process, on a cron, against /monitor/browsers + POST /monitor/actions/kill_browser, and it works: after a kill the next /crawl completed in 569 ms. Doing it in the janitor would remove the need for anyone to build that.

Two things that do not work as recycling mechanisms, both measured:

• POST /monitor/actions/restart_browser {"sig": "permanent"} wedges the worker - it deadlocks the pool lock (fix(docker): restart_browser on the permanent browser deadlocks the pool #2230).
• gunicorn --max-requests. It fires correctly and then hangs in Waiting for application shutdown. because the ASGI shutdown blocks closing the browser pool; with the image's --timeout 1800 the arbiter will not kill the stuck worker for 30 minutes. Every worker wedged within 7 minutes and the service returned 502 throughout.

2. Make the pool's state visible to /health. Every failure mode here - this one, a dead pooled browser after a Page.goto timeout (#842, PR #1946), and the restart deadlock - is invisible to GET /health, because health never touches the pool. An optional deep/readiness check would let an orchestrator restart a container that is answering nothing, instead of a human noticing a latency graph an hour later.

Is this reproducible?

Yes

Inputs Causing the Bug

Steps to Reproduce

1. Run the 0.9.2 Docker server with a pool config that keeps one browser hot.
2. Drive real (not `raw:`) page crawls through `/crawl` for ~30-60 minutes, several hundred pages, never letting the pool go idle past the TTL.
3. Time a single navigation through the pool, then create a browser directly inside the same container (`playwright` via `docker exec`) and time the same URL. The pooled one is ~5 s slower.
4. Restart the container and repeat step 3 - the gap is gone.

Code snippets

OS

Linux (Docker)

Python version

3.12 (image)

Browser

Chromium (bundled)

Browser version

No response

Error logs & Screenshots (if applicable)

The gunicorn --max-requests attempt at recycling, which is how the shutdown-blocks-on-the-pool behaviour surfaced:

18:16:59 Maximum request limit of 63 exceeded. Terminating process.
18:16:59 Shutting down
18:16:59 Waiting for application shutdown.   <- stops here

/health latency went 1.2 ms -> 8 s and the proxy answered 502 for everything until the containers were restarted.