Super Webcrawler AI (LLM-Only Upgrade)

LLM-native crawler using Firecrawl (scraping to markdown) + Parallel.ai-style parallelism. No RL, driven by LangChain ReAct agents.

Setup

Add API keys in agent.py/crawler_env.py (Firecrawl, HF).
pip install -r requirements.txt
Run: python main.py --goal "your goal" --start_url "https://start.com" --parallel 3

Name		Name	Last commit message	Last commit date
Latest commit History 2 Commits
README.md		README.md
agent.py		agent.py
crawler_env.py		crawler_env.py
logs		logs
main.py		main.py
requirements.txt		requirements.txt
utils.py		utils.py