🏠 Mudate - home search

Track and compare real estate listings from Zonaprop, Argenprop, MercadoLibre, and Remax.
Filter, review, map, and export properties — all from a single self-hosted web app.

---

Features

• Multi-engine scraping — Zonaprop, Argenprop, MercadoLibre, and Remax via Playwright (stealth mode)
• Tracks price history — every price change is recorded per property
• Geocodes addresses and shows pins on an interactive map (Leaflet + Nominatim)
• Filter & sort by review, type, status, price, m², provider, and more
• Review system — mark properties as Interesting, Contact, Discard, Unsure, or Duplicate
• Deduplication — fuzzy address matching removes cross-engine duplicates; same-engine reactivation detection merges re-listed properties
• Bulk actions — multi-select with batch review and side-by-side comparison
• Export to CSV — respects active filters
• Export/Import DB — full database backup and restore via the admin UI
• Dark mode — toggleable theme with localStorage persistence
• Daily auto-refresh — a scheduler re-scrapes all sessions every morning at 08:00 (Argentina time)
• No build step — frontend is plain HTML/JS (Alpine.js + Tailwind CDN)
• Single-file database — everything stored in db.json

---

Quick Start

Option A — Docker (recommended)

Requires Docker Desktop (Mac/Windows) or Docker Engine (Linux).

git clone https://github.com/YOUR_USERNAME/mudate.git   # replace YOUR_USERNAME with your fork URL
cd mudate
docker compose up

Open http://localhost:8010 — that's it.

Data is saved to ./data/db.json on your machine and survives container restarts.

To run in the background:

docker compose up -d
docker compose logs -f   # follow logs
docker compose down      # stop

---

Option B — Railway (cloud, zero local setup)

Deploy your own instance to Railway with one click:

Note: For persistent storage on Railway, attach a Volume and set DB_PATH to a path inside it (e.g. /data/db.json). Without a volume the database resets on every deploy.

---

Option C — Manual (Python 3.11+, Linux)

1. Install system dependencies

# Debian 11/12, Ubuntu 22.04, Raspberry Pi OS
sudo apt update && sudo apt install -y python3 python3-venv python3-pip git

# Playwright browser libraries (manual fallback — see note below)
sudo apt install -y libnss3 libatk1.0-0 libatk-bridge2.0-0 libcups2 \
  libdrm2 libxkbcommon0 libxcomposite1 libxdamage1 libxfixes3 libxrandr2 \
  libgbm1 libpango-1.0-0 libcairo2 libasound2 libwayland-client0

# Fedora/RHEL
sudo dnf install -y python3 python3-virtualenv git
sudo dnf install -y nss atk at-spi2-atk cups-libs libdrm libxkbcommon \
  libXcomposite libXdamage libXfixes libXrandr mesa-libgbm pango cairo alsa-lib wayland-libclient

# Arch Linux
sudo pacman -S python python-virtualenv git

# Playwright browser dependencies
sudo pacman -S nss atk at-spi2-core cups libdrm libxkbcommon libxcomposite \
  libxdamage libxfixes libxrandr mesa pango cairo alsa-lib

Playwright's browser libraries: the commands above are a manual fallback. Step 2 runs playwright install --with-deps chromium, which detects your distribution and installs the exact packages it needs — prefer that over installing them by hand.

If you do install them yourself, note that the ALSA package was renamed: libasound2 on Debian 11/12, Ubuntu 22.04 and Raspberry Pi OS, but libasound2t64 on Ubuntu 24.04+ and Debian 13. Using the wrong name makes apt fail with "Unable to locate package".

2. Clone and set up the project

git clone https://github.com/YOUR_USERNAME/mudate.git   # replace YOUR_USERNAME with your fork URL
cd mudate/backend

# Create virtual environment
python3 -m venv .venv

# Install Python dependencies
.venv/bin/pip install -r requirements.txt

# Install Playwright Chromium (~180 MB download) and its system libraries.
# --with-deps resolves the right package names for your distro automatically.
.venv/bin/playwright install --with-deps chromium

3. Run the app

.venv/bin/uvicorn main:app --host 0.0.0.0 --port 8010 --reload

Open http://localhost:8010.

4. (Optional) Run as a background service

To keep the app running after closing the terminal:

# Option A: nohup
nohup .venv/bin/uvicorn main:app --host 0.0.0.0 --port 8010 > /tmp/mudate.log 2>&1 &

# Option B: systemd user service
mkdir -p ~/.config/systemd/user
cat > ~/.config/systemd/user/mudate.service << 'EOF'
[Unit]
Description=Mudate
After=network.target

[Service]
Type=simple
WorkingDirectory=%h/mudate/backend
ExecStart=%h/mudate/backend/.venv/bin/uvicorn main:app --host 0.0.0.0 --port 8010
# Uncomment when the machine has no display (headless server, Raspberry Pi, NAS):
# Environment=PLAYWRIGHT_HEADLESS=1
Restart=on-failure

[Install]
WantedBy=default.target
EOF

systemctl --user daemon-reload
systemctl --user enable --now mudate

5. (Optional) Set environment variables

export OPENCAGE_API_KEY=your_key_here   # improves geocoding accuracy
export DB_PATH=/path/to/db.json         # custom database location
export PLAYWRIGHT_HEADLESS=1            # REQUIRED on a server with no display (Raspberry Pi, NAS, VPS)

---

Configuration

All settings are passed as environment variables.

Variable	Default	Description
DB_PATH	../db.json	Path to the JSON database file
OPENCAGE_API_KEY	(empty)	Optional — improves geocoding accuracy. Free tier: 2 500 req/day. Get one at opencagedata.com
PLAYWRIGHT_HEADLESS	(auto)	1 forces headless, 0 forces headed. By default Zonaprop runs headed unless Docker is detected (/.dockerenv). On a server without a display (Raspberry Pi, NAS, VPS) set PLAYWRIGHT_HEADLESS=1 in the service environment, otherwise Chromium tries to open a window and the scrape fails

With Docker, set variables in a .env file next to docker-compose.yml:

OPENCAGE_API_KEY=your_key_here

Manually, prefix the command:

OPENCAGE_API_KEY=your_key_here uvicorn main:app --host 0.0.0.0 --port 8010

---

How to Use

1. Enter a username — no password, just a name to separate your data from others on the same instance.
2. Create a search — paste a filter URL from Zonaprop, Argenprop, MercadoLibre, or Remax (the path after the domain, e.g. /inmuebles-venta-palermo-capital-federal-argentina.html).
3. Click Actualizar — the scraper runs in the background and populates the table.
4. Review listings — use the filter bar, open property details, add notes, mark reviews.
5. Open the map — addresses are geocoded automatically when you switch to map view.
6. Export — use the ⋯ menu to export the filtered table to CSV.

---

Project Structure

mudate/
├── backend/
│   ├── main.py            # FastAPI app + API routes
│   ├── config.py          # Centralized settings (env vars & constants)
│   ├── schemas.py         # Pydantic request/response models
│   ├── storage.py         # Atomic JSON read/write
│   ├── deduplicator.py    # Cross-engine fuzzy deduplication
│   ├── same_engine_dedup.py # Same-engine reactivation detection
│   ├── geocoder.py        # Address geocoding (Nominatim → OpenCage)
│   ├── geocoding_tasks.py # Concurrent batch geocoding
│   ├── scheduler.py       # Daily auto-refresh (APScheduler)
│   ├── scrapers/
│   │   ├── base.py        # BaseScraper ABC + shared utils
│   │   ├── factory.py     # Engine → scraper class mapping
│   │   ├── runner.py      # Scrape run orchestration
│   │   ├── persistence.py # Merge listings into db.json
│   │   ├── zonaprop.py    # Zonaprop scraper
│   │   ├── argenprop.py   # Argenprop scraper
│   │   ├── mercadolibre.py # MercadoLibre scraper
│   │   └── remax.py       # Remax scraper
│   ├── tests/             # Standalone diagnostic test scripts
│   └── requirements.txt
├── frontend/
│   ├── index.html         # SPA shell (Alpine.js + Tailwind + Leaflet)
│   ├── app.js             # Alpine component (state, methods, filters)
│   ├── map.js             # Leaflet map & geocoding methods
│   ├── constants.js       # Review options, pin colors, api() helper
│   └── dark.css           # Dark mode overrides
├── scripts/
│   └── reimage_argenprop.py # One-off script to re-scrape Argenprop images
├── Dockerfile
├── docker-compose.yml
├── README.md
├── DOCS.md
├── AGENTS.md              # Project context for AI sessions
└── data/                  # Created automatically — holds db.json (git-ignored)

---

Updating

git pull
docker compose up --build   # rebuilds the image with the latest code

---

Notes & Limitations

• Single-user friendly — the JSON database works well for personal use or a small group. It is not designed for many concurrent users writing at the same time.
• Scraper fragility — if Zonaprop, Argenprop, MercadoLibre, or Remax update their HTML structure, selectors in the corresponding scraper files may need to be updated.
• Geocoding rate limit — Nominatim enforces 1 request/second. Geocoding a large session takes time. Adding an OPENCAGE_API_KEY improves hit rate but the rate limit stays.
• Argentina only — geocoding is scoped to Argentina. Adapting to other countries requires changes in geocoder.py.

---

Tech Stack

Layer	Technology
Backend	Python 3.11, FastAPI, Uvicorn
Scraping	Playwright, playwright-stealth, BeautifulSoup4
Scheduling	APScheduler
Geocoding	Nominatim, OpenCage (optional)
Frontend	Alpine.js, Tailwind CSS (CDN), Leaflet.js
Database	JSON flat file with file locking

---

Screenshots

---

Troubleshooting

• Cloudflare blocks (HTTP 403 or a "Just a moment…" page): the run banner now detects this and prints the exact profile to delete, e.g. rm -rf /home/<you>/.mudate_browser_headless. Zonaprop stores cf_clearance cookies in a persistent browser profile; once a profile is poisoned with bot-flagged cookies, every retry fails the same way, so clearing it is the fix — not re-running. Delete the profile your run is actually using, then restart the app:

  • headed run (local) → rm -rf ~/.mudate_browser
  • headless run / Docker → rm -rf ~/.mudate_browser_headless

  Confirm with .venv/bin/python tests/test_cloudflare.py before and after.

• Playwright install failures: run playwright install --with-deps chromium so the packages are resolved for your distro. If you install them manually, check the package names above — the ALSA library is libasound2 on Debian 11/12 and Raspberry Pi OS but libasound2t64 on Ubuntu 24.04+ / Debian 13.

• Port conflicts: Mudate defaults to port 8010; if it's already in use, pass a different --port to uvicorn (or change the host port in docker-compose.yml).

• zsh activation errors: If source .venv/bin/activate fails in zsh, use the direct venv path instead: .venv/bin/uvicorn main:app --host 0.0.0.0 --port 8010.

---

Contributing

PRs and issues are welcome. If a scraper breaks, opening an issue with the current HTML structure is the fastest way to get it fixed.

---

License

MIT — do whatever you want with it.