🕷️ E-Commerce Web Scraping Agent

A robust Python web scraper built with Playwright and BeautifulSoup for extracting product data from public e-commerce websites. Designed as a professional tool for freelance data collection gigs.

Features

• 🎭 Playwright browser automation — Handles JavaScript-rendered pages
• 🍜 BeautifulSoup HTML parsing — Fast, reliable data extraction
• 🛡️ Anti-bot protection — User-agent rotation, random delays, realistic headers, viewport randomization
• 📄 Dual export — JSON and CSV output formats
• 📋 Site profiles — Configurable CSS selectors per site (easy to add new targets)
• 🎨 Rich CLI — Beautiful terminal output with progress tracking

Quick Start

Install

pip install -r requirements.txt
playwright install chromium

Run

# Scrape 2 pages from books.toscrape.com (default)
python -m scraper.main --site books_toscrape --pages 2 --format both

# List available site profiles
python -m scraper.main --list-sites

# Custom delay range
python -m scraper.main --site books_toscrape --pages 3 --delay 3-7

# Show browser window (debug mode)
python -m scraper.main --site books_toscrape --pages 1 --no-headless

Output

Files are saved to ./output/ by default:

• books_toscrape.json — Pretty-printed JSON with product count
• books_toscrape.csv — Standard CSV with headers

Adding a New Site

Edit scraper/config.py and add a new SiteProfile:

MY_SHOP = SiteProfile(
    name="my_shop",
    base_url="https://example-shop.com/",
    page_url_template="products?page={page}",
    product_container=".product-card",
    name_selector=".product-title",
    price_selector=".product-price",
    stock_selector=".stock-status",
    link_selector=".product-title a",
)

# Register it
SITE_PROFILES["my_shop"] = MY_SHOP

Then run: python -m scraper.main --site my_shop --pages 5

CLI Options

Flag	Description	Default
--site	Site profile name	(required)
--pages	Max pages to scrape	2
--format	json, csv, or both	both
--output	Output directory	./output
--delay	Min-max delay (seconds)	2-5
--no-headless	Show browser window	false
--list-sites	List available profiles	—

Project Structure

scraper/
├── __init__.py      # Package metadata
├── anti_bot.py      # UA rotation, delays, headers, viewport randomization
├── config.py        # Site profiles with CSS selectors
├── parser.py        # BeautifulSoup product extraction & price normalization
├── scraper.py       # Playwright browser automation
├── exporter.py      # JSON/CSV export
└── main.py          # CLI entry point

Legal Notice

This tool is designed for scraping public, legally permissible websites. Always check a site's robots.txt and Terms of Service before scraping. The included profiles target websites specifically built for scraping practice.