Skip to content
BranchesTags

Repository files navigation

Web Intelligence Platform

A self-contained Python tool that crawls any website and returns structured business intelligence — services, pricing, contacts, hours, FAQs, team, testimonials, tech stack, and more.

Run one command, open the studio in your browser, paste a URL, and get JSON you can copy or download. No separate frontend repo, no npm, no Next.js.

┌─────────────────────────────────────────────────────────────┐
│  python run.py                                              │
│    → FastAPI REST API (port 8000)                           │
│    → Streamlit Studio UI (port 8501)                        │
└─────────────────────────────────────────────────────────────┘

1. What This Project Does

Capability Description
Website crawling Async BFS on the same domain, robots.txt aware, URL exclusions for login/cart/etc.
Structured extraction Deterministic extractors (JSON-LD, DOM patterns, trafilatura) — no LLM required for core data
Multi-strategy fetch Static HTTP → Playwright headless → headed → stealth → session replay
Retrieval providers Internal crawler, Firecrawl cloud fallback, or Auto (internal first, escalate on blocks)
Output modes minimal, structured, full, debug — same crawl, different JSON shape
Optional LLM enrich Fills gaps only (industry, tagline, empty service descriptions) when API keys are set

The default path is deterministic, reproducible, and offline-after-crawl.

2. Quick Start

git clone <your-repo-url>
cd Crawler

python -m venv .venv
source .venv/bin/activate   # Windows: .venv\Scripts\activate

pip install -r requirements.txt
python -m playwright install chromium

cp .env.example .env        # optional — edit keys if using Firecrawl / LLM

python run.py

Then open:

Service URL
Studio UI http://127.0.0.1:8501
REST API http://127.0.0.1:8000
OpenAPI docs http://127.0.0.1:8000/docs

Equivalent:

make dev

CLI (no UI)

python main.py https://stripe.com --out json/stripe.json
python main.py https://example.com --enrich

3. Architecture

                    ┌──────────────────┐
                    │  Streamlit UI    │  ui/app.py
                    │  (Studio)        │
                    └────────┬─────────┘
                             │ POST /crawl (or direct import)
                             ▼
                    ┌──────────────────┐
                    │  FastAPI         │  api/
                    │  extract_service │
                    └────────┬─────────┘
                             │
              ┌──────────────┼──────────────┐
              ▼              ▼              ▼
       retrieval/      core/           output/
       auto | internal  orchestrator    formatter
       | firecrawl           │
              └──────────────┴──────────────┘
                             ▼
                    extraction/ (deterministic)
                    crawler/ + fetch/

Everything lives in one Python repo:

  • run.py — single entrypoint (API + UI)
  • api/ — FastAPI routes and shared run_extract()
  • ui/ — Streamlit studio
  • crawler/, fetch/ — internal retrieval
  • retrieval/ — provider abstraction (internal / firecrawl / auto)
  • extraction/ — deterministic extractors + optional ai_enricher
  • output/ — mode formatters
  • llm/ — OpenAI / OpenRouter (enrichment only)
  • main.py — Typer CLI for scripting

4. Environment Variables

Copy .env.example to .env. Only Firecrawl and LLM keys are optional for basic crawls.

Required for nothing (defaults work)

Crawler and extraction settings have sensible defaults.

Retrieval providers

Variable Default Purpose
RETRIEVAL_PROVIDER auto auto | internal | firecrawl
ENABLE_FIRECRAWL false Must be true to use Firecrawl
FIRECRAWL_API_KEY — Firecrawl API key
FIRECRAWL_TIMEOUT_SEC 300 Crawl job timeout (poll until done)
FIRECRAWL_POLL_INTERVAL_SEC 2 Seconds between crawl status polls
STRUCTURED_USE_LLM false Internal structured mode: LLM fills same schema as Firecrawl

Optional LLM enrichment

Variable Purpose
ENRICH true to enable gap-fill via LLM
OPENAI_API_KEY Use OpenAI (auto-detected)
OPENROUTER_API_KEY Use OpenRouter (auto-detected)
LLM_PROVIDER Force openai or openrouter
OPENAI_MODEL / OPENROUTER_MODEL Model override

Platform ports

Variable Default
API_HOST 0.0.0.0
API_PORT 8000
UI_PORT 8501

5. Frontend Usage (Streamlit Studio)

  1. Enter a URL, e.g. https://stripe.com
  2. Choose retrieval provider:
    • Auto — internal crawl; Firecrawl if blocked / empty
    • Internal — only this project's crawler
    • Firecrawl — cloud scrape (needs API key)
  3. Choose output mode (see section 7)
  4. Expand Advanced settings for depth / max pages / concurrency
  5. Click Extract Website Intelligence
  6. Use Download JSON or copy from the response panel

The UI calls POST /crawl when the API is up, and falls back to in-process extraction if not.

6. API Usage

Unified crawl + extract

curl -s -X POST http://127.0.0.1:8000/crawl \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "provider": "auto",
    "output_mode": "structured",
    "depth": 2,
    "max_pages": 50
  }' | jq .

Same body at POST /api/v1/crawl.

Legacy routes

# Internal crawler only
curl -X POST http://127.0.0.1:8000/api/v1/internal \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com", "mode": "structured"}'

# Firecrawl only
curl -X POST http://127.0.0.1:8000/api/v1/external \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com", "mode": "full"}'

Health

curl http://127.0.0.1:8000/health

GET / redirects to the Streamlit studio port.

7. Output Modes

Mode Use case
minimal Fast preview — name, description, service names
structured SaaS / booking integrations — strict schema
full Complete canonical WebsiteIntelligence model
debug Engineering — fetch strategies, blocks, timings

Minimal example

{
  "business_name": "Stripe",
  "description": "Financial infrastructure platform for businesses.",
  "services": ["Payments", "Billing", "Checkout"]
}

Structured example

{
  "business_details": {
    "business_name": "Example Spa",
    "city": "Karachi",
    "phone_number": "+92 300 1234567",
    "primary_language": "en",
    "currency": "PKR",
    "time_zone": "Asia/Karachi",
    "short_description": "Premium massage and wellness center."
  },
  "business_type": "Health & Wellness",
  "operating_hours": {
    "monday": { "open": "10:00", "close": "22:00" },
    "tuesday": { "open": "10:00", "close": "22:00" }
  },
  "day_off": ["Sunday"],
  "services": [
    {
      "category": "Massage",
      "items": [
        {
          "service_name": "Swedish Massage",
          "price": 4500,
          "duration_minutes": 60
        }
      ]
    }
  ]
}

Full output

Includes everything from deterministic extraction:

  • business, services, pricing, contacts, social, faq, team, testimonials
  • tech_stack, seo, hours, locations, raw_pages (when enabled)
  • crawl_metadata, blocked_pages, extraction_mode

Debug example

{
  "provider_used": "internal",
  "fallback_triggered": false,
  "crawl_success": true,
  "extraction_ms": 842,
  "pages": [
    {
      "url": "https://example.com/",
      "category": "home",
      "fetch_strategy": "static",
      "block_reason": null
    }
  ],
  "blocked_pages": [],
  "http_errors": [],
  "llm_usage": null
}

8. Retrieval Providers

Provider When to use
Auto Default — try internal multi-strategy crawl; escalate to Firecrawl on heavy blocks or empty results
Internal Full control, no external cost, works offline after Playwright install
Firecrawl Hard targets (Cloudflare, heavy JS SPAs) when you have an API key

Enable Firecrawl in .env:

ENABLE_FIRECRAWL=true
FIRECRAWL_API_KEY=fc-...

9. Anti-Bot & 403 Handling

Many sites return 403 Forbidden to bots. This platform handles that in layers:

  1. Realistic browser User-Agent (Chrome on macOS by default)
  2. Static HTTP first — often enough for marketing sites
  3. Playwright escalation — headless → headed → stealth
  4. Challenge detection — saves HTML/screenshots under debug/ when enabled
  5. Auto → Firecrawl — cloud renderer when internal crawl is blocked

403 Forbidden — what to try

  1. Run with Auto provider and Firecrawl configured
  2. Increase MAX_PAGES / depth only after a successful shallow crawl
  3. Check debug/ artifacts for challenge pages (CAPTCHA, Cloudflare)
  4. Use Debug output mode to see blocked_pages and fetch_strategy per URL

10. Troubleshooting

Empty extraction

  • Site may be JS-only with no server-rendered HTML → try Firecrawl or increase fetch strategies
  • All pages blocked → see blocked_pages in debug output
  • Wrong domain / redirects → verify final URL in debug crawl_pages

Firecrawl errors

  • Confirm ENABLE_FIRECRAWL=true and valid FIRECRAWL_API_KEY
  • Check API quota and FIRECRAWL_TIMEOUT_SEC

LLM enrichment fails

  • Set OPENAI_API_KEY or OPENROUTER_API_KEY
  • Enrichment is optional — deterministic output still returns

Studio cannot reach API

  • Start with python run.py (not Streamlit alone)
  • Or set STUDIO_PREFER_API=false to force in-process extraction

Playwright missing

python -m playwright install chromium

11. Project Layout

Crawler/
├── run.py              # ← start here
├── Makefile
├── main.py             # CLI
├── ui/app.py           # Streamlit studio
├── api/                # FastAPI
├── core/               # orchestrator
├── crawler/
├── fetch/
├── retrieval/          # internal | firecrawl | auto
├── extraction/
├── output/
├── llm/
├── json/               # CLI output samples
└── tests/

12. Screenshots

Placeholder — add docs/screenshots/studio.png after your first run.

Studio home JSON response panel
screenshot screenshot

13. Development

pytest -q
python -m uvicorn api.main:app --reload --port 8000
streamlit run ui/app.py

License

Use and modify for your own projects. Add a license file if you distribute publicly.

About

crawler with internal crawler & firecrawel support

Resources

Readme
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages