[tomayac]

The ≥100 threshold in public_hash_list counts distinct hostnames of the resource URL, so a font used only on 100 $city.$vulnerable-group.com landing pages passes, even though it identifies a single site, and any origin could then probe Cross-Origin Storage for its hash to infer that the user belongs to that group. This PR counts distinct sites (eTLD+1 of the embedding page) using the full Public Suffix List, including the private section that BigQuery's NET.PUBLIC_SUFFIX() skips, so alice.github.io and bob.github.io still count as two sites, renames num_origins to num_sites, and adds a script plus a monthly workflow that keep the private suffix rules current. @max-ostapenko, could you please review?

[tomayac]

This is a follow-up to #324.

[tunetheweb]

TIL NET.REG_DOMAIN only considers ICANN domains from the PSL and not Private PSL domains! That's sad... 😔

[max-ostapenko]

Hi @tomayac,
To avoid maintaining a 3,400-line static JavaScript file in the repo and a recurring GitHub Actions workflow to update it, I have automated the PSL ingestion in our pipeline:

1. We downloads the complete Public Suffix List (both ICANN and Private domains) directly into BigQuery: httparchive.urls.public_suffix_list.

   • Columns: suffix STRING, is_wildcard BOOLEAN, is_exception BOOLEAN, is_private BOOLEAN, section STRING.
   • This runs automatically on every monthly crawl before the Dataform pipeline triggers.

2. The table is declared in Dataform as urls.public_suffix_list.

Could you please:

1. Rebase your PR on fancy-warbler.
2. Update the private_suffixes CTE in definitions/output/performance/public_hash_list.js to query the table for the private rules:

WITH private_suffixes AS (
  SELECT suffix, is_wildcard
  FROM ${ctx.ref('urls', 'public_suffix_list')}
  WHERE is_private AND NOT is_exception
),

(and remove the sqlStringList helper).

3. Remove the other files from your PR.

Your core SQL logic for site counting in public_hash_list.js will stay intact, and the suffix list will stay automatically up to date!

[tomayac]

Thanks, @max-ostapenko, that's cleaner. Rebased on fancy-warbler, switched private_suffixes to read from urls.public_suffix_list, removed sqlStringList, and dropped the static list, script, workflow, and config changes. The PR now only touches public_hash_list.js. PTAL, thank you!