Michigan County Officials Contact Scraper

A production-grade, AI-assisted web scraping pipeline that automatically discovers and enriches contact information for elected and appointed officials across all 83 Michigan counties. Built for civic data and government outreach workflows.

---

What Data Was Scraped

The scraper targets county-level government officials in Michigan. For each official, it attempts to find:

Field	Example
Email	jane.doe@inghamcounty.gov
Phone	517-676-7201
Website	https://www.ingham.org/county-clerk
Office_Address	341 S. Jefferson St, Mason, MI 48854
Instagram	https://instagram.com/janedoeofficial
Twitter_X	https://x.com/janedoe
Facebook	https://facebook.com/janedoeforingham

Positions Covered

• Board of Commissioners Chairman
• County Administrator
• County Clerk
• County Treasurer
• Prosecuting Attorney
• Sheriff
• Register of Deeds
• Chief Executive Officer

The input CSV is expected to already contain the official's name, position, and county — the scraper only fills in the missing contact fields.

---

Source Websites

The scraper draws from multiple source tiers, prioritised by trustworthiness:

Tier 1 — Official county .gov sites (score: 100) Each county's own government website, discovered automatically by discover_counties.py. Examples:

• https://www.inghamcounty.gov
• https://www.waynecounty.com
• https://kentcountymi.gov

Tier 2 — Statewide authoritative directories (score: 85–95)

• Michigan Prosecuting Attorneys Association: michiganprosecutor.org/about-us/prosecutor-directory/
• Michigan Sheriffs' Association: misheriff.org/sheriffs-offices/
• Michigan state portal: michigan.gov

Tier 3 — Reference and social sources (score: 45–60)

• ballotpedia.org — elected official profiles
• facebook.com, instagram.com, x.com — social media handles

Search engine — SerpAPI (Google Search) is used to discover URLs in all tiers when direct path probing fails.

---

Scraping Methodology

The pipeline runs in three scripts executed in sequence:

Script 1: discover_counties.py — County Website Discovery

Run once before scraping to build county_sites.json.

1. Pattern probing — tries ~11 known URL templates for each county (e.g. {slug}county.gov, co.{slug}.mi.us) using HTTP HEAD requests. No API cost.
2. Search fallback — if no pattern resolves, queries SerpAPI for "{county} County Michigan official government website" and scores results by domain and content signals.
3. Confidence verification — fetches the candidate homepage and checks for county name, "Michigan", and government role keywords (clerk, commissioner, sheriff, etc.) in page text. Confidence is scored 0–1.
4. Sub-page discovery — for counties with confidence ≥ 0.5, probes common sub-paths (/clerk, /board-of-commissioners, /sheriff, etc.) to build a position-to-URL map stored alongside the base URL.

Script 2: scraper.py — Contact Enrichment (6-Phase Pipeline)

Processes the input CSV row by row. Already-populated fields are never overwritten.

Phase 1 — Direct county URL discovery Uses county_sites.json to construct position-specific URLs on the county's .gov domain. Position-specific paths (e.g. /prosecuting-attorney) are tried before generic ones (e.g. /staff-directory). Up to 5 direct URLs are collected per official without any search queries.

Phase 2 — Multi-pass web search Three search passes via SerpAPI, each with different query strategies:

• Targeted: "Jane Doe" "Ingham County" Michigan County Clerk email phone
• Broad: Ingham County Michigan County Clerk office phone email address
• Social: "Jane Doe" Ingham County Michigan official Facebook Instagram

Results are ranked by domain trust score plus bonuses for name appearance in title/snippet and county slug in the URL. Garbage sources (archive.org, genealogy sites, outdated government document bundles) are filtered via a blocklist.

Phase 3 — Fetch and extract Each URL is fetched and processed:

• Cloudflare-obfuscated emails (data-cfemail attributes, [email protected] placeholders, AT/DOT substitutions) are decoded before parsing.
• Structured data (JSON-LD, Microdata) is extracted with extruct — if a Person or GovernmentOrganization schema block contains contact fields, those are prioritised.
• PDFs are parsed with pdfplumber (up to 8 pages, 6000 chars).
• Pages returning < 200 chars of text, or HTTP 403/429, are retried with headless Chromium via Playwright.
• A focused context window (2500 chars) is extracted around the official's last name to reduce noise fed to the AI.
• Claude (claude-sonnet-4-6) reads the context and returns structured JSON with extracted fields plus confidence, is_about_correct_person, and is_current flags.

Phase 4 — Stage 2 verification Individual field values from low-trust sources (domain score < 60) or borderline confidence (0.4–0.7) are sent to Claude for a dedicated verification pass. Claude can reject a value or return a corrected version (e.g. fixing phone formatting or resolving an email typo).

Phase 5 — Cross-source reconciliation When multiple sources return data for the same official, Claude compares all results and picks the most trustworthy value per field. Confidence scoring rules:

• Two sources agree → very high confidence
• Single .gov source → high confidence (0.85+)
• Single non-gov source → low confidence (0.50–0.60)

If Claude's reconciliation call fails, the script falls back to taking the result with the highest domain_score × confidence product.

Phase 6 — Write back to CSV Values meeting the confidence threshold are written into the output row. Split thresholds apply: .gov-sourced values need ≥ 0.60 confidence; all other sources need ≥ 0.72. Values below threshold are recorded in _flags for manual review rather than silently discarded.

Script 3: audit.py — Validation and Review Flagging

Post-enrichment quality check. Produces audit_report.csv listing every issue found per row.

---

Libraries Used

Library	Role
anthropic	Claude API client — all AI extraction, verification, and reconciliation calls
httpx	Async-capable HTTP client for page fetching and URL probing
beautifulsoup4 + lxml	HTML parsing and text extraction
pdfplumber	Text extraction from PDF staff directories and meeting minutes
playwright	Headless Chromium fallback for JS-rendered pages and bot-blocked sites
extruct	Structured data extraction (JSON-LD, Microdata) from HTML
phonenumbers	Phone number parsing, validation, and normalisation to NNN-NNN-NNNN
google-search-results	SerpAPI Python client for Google Search queries

---

Challenges Faced

Email obfuscation Many county .gov sites use Cloudflare's email protection (replacing @ with encoded data-cfemail attributes) or plain-text tricks like name [AT] county [DOT] gov. A custom decoder handles both patterns before any parsing occurs.

JavaScript-rendered pages A significant number of county websites render their staff directories entirely in JavaScript, returning near-empty HTML to httpx. Playwright (headless Chromium) is used as a fallback whenever a fetched page returns fewer than 200 characters of visible text, or when the server returns 403/429.

PDF directories Some counties publish their staff contact lists exclusively as PDFs (e.g. board meeting agendas, annual reports). pdfplumber extracts text from these, capped at 8 pages to avoid runaway memory usage.

Stale and outdated data Web search results frequently return pages from previous election cycles listing former officeholders. Claude is explicitly instructed to set is_current: false when it detects terms like "former", "retired", "previous", or "ex-". These results are logged with a warning flag rather than written to the output.

County website URL fragmentation Michigan counties have no standardised web presence. URLs follow at least 11 different patterns ({slug}county.gov, co.{slug}.mi.us, {slug}countymi.gov, etc.), and several counties use entirely non-standard domains. discover_counties.py handles this with a pattern-first, search-fallback approach.

Shared phone numbers County switchboard numbers frequently appear on multiple officials' pages, leading to false duplicate detections. The audit script flags these for human review rather than automatically discarding them.

Rate limiting SerpAPI enforces per-minute query limits. A 0.6-second delay is applied between all HTTP requests, and search queries are batched to stay within limits.

---

Data Cleaning Steps

Cleaning happens at two points in the pipeline:

During scraping (scraper.py)

• Phone normalisation — all phone numbers are parsed with the phonenumbers library and reformatted to NNN-NNN-NNNN. Numbers that fail validation are discarded rather than written.
• Email normalisation — emails are lowercased and mailto: prefixes are stripped.
• Email inference — before searching, the scraper generates likely email candidates from known county domain patterns (e.g. firstname.lastname@{slug}county.gov) and passes them to Claude as anchors, reducing hallucination risk.
• Garbage URL filtering — a blocklist removes archive.org mirrors, genealogy sites, and outdated Congressional Directory PDF packages from search results before any fetching occurs.
• Noise reduction — <script>, <style>, <nav>, <footer>, <header>, <aside>, and <iframe> tags are stripped from HTML before text is extracted. A focused 2500-char window around the official's name is passed to Claude rather than the full page.

During audit (audit.py)

• Email format validation — regex check + rejection of personal domains (gmail, yahoo, hotmail, outlook).
• Phone format validation — regex check for NNN-NNN-NNNN pattern plus phonenumbers library validation.
• Website format validation — must begin with http:// or https://.
• Address validation — must start with a street number and contain MI or Michigan.
• Completeness scoring — each row gets a 0–1 score based on how many of Email, Phone, Website, Office_Address are filled. Rows below 50% are flagged.
• Duplicate detection — rows sharing an identical email or phone are grouped and flagged for investigation.

---

Output Format

Enriched CSV (*_filled.csv)

Same columns as the input, with empty fields populated and three metadata columns appended:

Column	Description
_flags	Semicolon-separated warnings (low confidence, outdated source, missing fields)
_overall_conf	Float 0–1 representing the pipeline's confidence in the result
_sources_used	Up to two source URLs that contributed data for this row

Audit Report (audit_report.csv)

Column	Description
row_num	1-indexed row number in the filled CSV (including header)
#	Original record ID from input
county	County name
position	Official's position
name	Official's name
completeness	Percentage of required fields filled (e.g. 75%)
needs_review	YES if any issue was found, blank otherwise
issues	Pipe-separated list of specific issues

County Sites Index (county_sites.json)

Produced by discover_counties.py. Maps each county to its base URL, confidence score, discovery method, and known sub-page URLs per position type.

---

How to Run

1. Install dependencies

pip install -r requirements.txt
playwright install chromium

2. Set API keys

In both discover_counties.py and scraper.py, replace the placeholder values:

ANTHROPIC_API_KEY = "your-anthropic-key"
SERP_API_KEY      = "your-serpapi-key"

3. Discover county websites (run once)

# All 83 Michigan counties
python discover_counties.py --all-michigan --output county_sites.json

# Or a subset
python discover_counties.py --counties "Ingham,Wayne,Oakland" --output county_sites.json

4. Run the scraper

python scraper.py --input michigan_officials.csv --output michigan_officials_filled.csv

The script saves a checkpoint after every row. If interrupted, re-run the same command to resume from where it stopped.

5. Audit the results

# Fast audit (no network calls)
python audit.py --input michigan_officials_filled.csv --report audit_report.csv

# With website liveness checks (slower — one HTTP request per row)
python audit.py --input michigan_officials_filled.csv --report audit_report.csv --check-urls

---

Assumptions and Limitations

Assumptions

• The input CSV already contains accurate Name, Position, and County values. The scraper does not validate or correct these — it only fills contact fields.
• Officials are current at time of scraping. The pipeline flags potentially outdated records but cannot guarantee freshness if a county website has not been updated.
• County email domains follow one of the eight known patterns used for inference. Officials at counties with non-standard email formats may have lower fill rates.

Limitations

• No SMTP validation — inferred email candidates are passed to Claude as textual hints, not verified by probing mail servers. An inferred email that matches a plausible pattern but is not found on any source page will not be written to the output.
• Social media coverage is incomplete — many county officials do not maintain official social accounts, and personal accounts are excluded. Fill rates for Instagram, Twitter_X, and Facebook are significantly lower than for Email and Phone.
• Playwright requires a local Chromium install — in headless server environments, ensure playwright install chromium has been run and that a display server or DISPLAY variable is not required (Playwright's headless mode does not need one).
• SerpAPI query budget — each official consumes up to ~9 search queries across three passes. For a full 83-county dataset with 5–8 officials per county, budget approximately 3,000–6,000 queries.
• Rate of change — government contact information changes with elections (typically every 2–4 years) and administrative appointments. This dataset should be re-scraped after major election cycles.
• Non-English pages — not expected for Michigan county sites, but the pipeline has no multilingual handling. ENDOFFILE