List casetext.com in the known-dead reference URL guard

[MaxGhenis]

Summary

Adds casetext.com to the known-dead reference URL guard (policyengine_us/tests/code_health/known_dead_reference_urls/casetext_com.toml), so the offline code-health test fails if a casetext link reappears in the package or docs/. #9627 removed every casetext link: 197 links to 55 distinct URLs, counted on the removed lines of its diff. main has none.

The prefix is the whole host, https://casetext.com/; the guard also matches it over http://, with www., and in any letter case. On 2026-10-06, 70 casetext URLs returned HTTP 410 Gone with a "This service is no longer available" notice. They were every casetext URL tried except the root:

• 59 statute, regulation and case pages
• the 3 index paths
• 4 scheme and www. variants
• 4 made-up paths

The root redirects to a Thomson Reuters product page with no legal text. The evidence is below.

The entry's replacement text points to the sources #9627 used: the official legislature or code site, then law.justia.com, then Cornell LII for state regulations.

Changes in the 2026-10-06 fix round

This round answers an independent agent review that returned REQUEST_CHANGES; the review was not posted on GitHub. It confirmed 410 for a statute page and a case page, but its regulation requests returned access errors, a browser-UA curl -L failed DNS, and its full package and docs scan did not finish.

1. Redirect-following GET evidence for statute, regulation and case URLs (below).
2. Tests that pin what the entry covers. test_dead_urls_are_flagged gains a statute URL over http://www., a regulation URL and a case URL inside a markdown link. test_archived_and_live_urls_are_not_flagged gains a Wayback copy of a casetext case page.
3. Entry metadata. verified moves to 2026-10-06, the date of the check below. The comment now quotes the 410 notice's opening words and corrects the count to 197 links (55 distinct URLs). The earlier body of this PR and Replace dead casetext.com references with live sources #9627's body said 199.
4. Merged main (3bed4d0) without conflicts. The branch still changes only the changelog fragment, the TOML file and the test file.

Evidence

Run 2026-10-06 03:09–03:15 UTC with /usr/bin/curl, the only curl on PATH. The run did not log the curl version; curl --version reported 8.7.1 afterwards, and the binary is dated before the run. Each URL got one GET -L --max-redirs 10 with curl's default user agent and one with a desktop Safari user agent: 146 requests, every curl exit code 0. The URLs are:

• every distinct casetext URL Replace dead casetext.com references with live sources #9627 removed (46 statute, 9 regulation)
• four case pages (Replace dead casetext.com references with live sources #9627 removed none)
• the three index paths and the root in three forms
• http:// and https://www. variants of one statute and one regulation URL
• four made-up control paths

kind	URLs	requests	hop statuses	final status
statute	46	92	410	410
regulation	9	18	410	410
case	4	8	410	410
index (/statute, /regulation, /case)	3	6	410	410
http:// variant	2	4	410	410
https://www. variant	2	4	301 > 410 (to the bare host)	410
made-up control path	4	8	410	410
root (https://, http://)	2	4	301 > 200	200 at www.thomsonreuters.com/en/cocounsel
root (https://www.)	1	2	301 > 301 > 200	200 at www.thomsonreuters.com/en/cocounsel

All 140 responses that ended in 410 have the same 640-byte body (one distinct MD5):

This service is no longer available, but we appreciate you being a part of it. For legal research, please visit Westlaw, and if you're curious about legal AI, check out CoCounsel. Thanks for stopping by!

The made-up paths get the same response. So the 410 is a catch-all shutdown response for every non-root path tried, not a removal of individual pages. For example, the N.J.A.C. 18:35-2.9 regulation page returned HTTP/2 410 with content-length: 640 and server: cloudflare, and no redirect.

The artifacts are saved on the hub host under ~/reviews/us-hub/fixes/9768-evidence/:

• probe.sh: the exact script
• url_list.tsv: its input
• run/results.tsv: one row per request, with final status, redirect count and final URL
• run/hops/: every hop's status line and headers
• run/bodies/: every final response body
• SUMMARY.md: a row for each regulation, case, index, variant, root and control request
• curl_version.txt: the curl version, recorded after the run

Tests

These ran locally on Python 3.13.9 with pytest 9.1.1, one process at a time.

• Before. With casetext_com.toml removed, as on main, pytest test_known_dead_reference_urls.py -k casetext gives 3 failed and 1 passed (exit 1). The statute, regulation and case examples fail; the Wayback example passes.
• After, at 864967e: the whole file passes, 39 tests (exit 0).
• Full package and docs scan at 864967e, using the test module's own scanned_files() and find_dead_urls_in_file(). It loads 362 entries across 119 hosts and scans 17,975 files under policyengine_us/ and docs/: 6,567 .py, 11,065 .yaml, 3 .yml, 293 .md and 47 .ipynb. That is every tracked file with a scanned suffix except the guard's own test file. It finds 0 known-dead URLs, so 0 casetext ones.
• Planted links. Two temporary files carried casetext links: a statute link in a docs/ markdown file and a regulation URL split across two Python string literals in a .py file. With them in place, test_package_and_docs_have_no_known_dead_reference_urls fails (exit 1). The failure names both files and lines and gives the entry's replacement text. The files were removed afterwards.
• ruff format --check and ruff check pass on the test file, and the TOML parses.

Impact

No model output change. The PR touches a TOML file that only test_known_dead_reference_urls.py reads, that test file and a changelog fragment. Searching for known_dead_reference_urls finds no other reader. No variable, parameter or formula changes, so household and microsimulation outputs are unchanged, and no microsimulation was run.

axiom: n/a: test data only (code-health guard; no policy provision changes)

🤖 Generated with Claude Code

[MaxGhenis]

Merged by the US + core hub on its merge gates. Audit:

• Head: 864967ee9d.
• Independent review: Opus 5.5, ~/reviews/us-hub/reviews-gen/us-9768/review-r1.md. APPROVE at this head with no P1 or P2.
  • The earlier review requested changes for missing regulation-URL evidence.
  • The fix round added redirect-following GETs for 73 URLs (146 requests). Every non-root path returned 410 with the same shutdown notice; the root redirects to Thomson Reuters.
  • It also added a guard scan over all 17,975 tracked files, which finds 0 dead links, and a regression test that fails without the entry.
  • The reviewer hand-checked the prefix pattern against the archive-URL and subdomain edge cases.
• CI: every check passed at this head.
• Impact: none on model outputs. The PR changes only a TOML data file, its test and a changelog fragment.
• Follow-up nits (P3):
  • assert the matched entry is casetext in test_dead_urls_are_flagged;
  • subdomains of casetext.com are not covered.