[Global-PGM S1] Global metadata bundle via interim GAUL→ISO3 crosswalk (#231) - #238
Merged
Merged
Conversation
…swalk (#231) The viewser pgm loa is still Africa+ME-bounded, but the platform's data supply (views-datafactory, which the models already consume) is global and GAUL-coded. Rebuild the bundled priogrid table from the factory's harvested GAUL-2024 admin assets: iso3_code → VIEWS isoab → country_id, duplicate isoab resolved to the most-recently-observed entity. 13,110 → 66,205 cells; rusty_bucket global forecast coverage 20.2% → 99.5%. Declared, not silent: the stamp's new priogrid_source block records source sha256s, the crosswalk rule, and exactly which codes could not map (disputed/non-VIEWS territories, ~0.5% of land cells). Explicitly an INTERIM handover (decision on #231): runtime accessors depend only on the bundle schema, so the eventual platform coding standard lands as a build-script-only change (register C-211). Build script gains the source seam (--priogrid-source datafactory|viewser, legacy queryset kept); pure crosswalk functions unit-tested; sample-scale canary decoupled from bundle size (fixed 13,110-gid subset, pinned budget). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This was referenced Jul 20, 2026
Closed
Declare the GAUL-2024 Kosovo→Serbia absorption in the metadata bundle stamp and C-211 narrative
#249
Closed
Contributor
Author
|
Max-effort review follow-up: all 15 findings are now tracked as actionable issues (13 in-repo: #239 #240 #241 #242 #243 #244 #245 #246 #247 #248 #249 #250 #251; 2 cross-repo: views-platform/views-datafactory#341, views-platform/views-postprocessing#123). #239–#250 are candidates to fix on this branch pre-merge; #251 is a standalone efficiency follow-up. |
…e nets (#239–#250) Executes 12 of the 13 issues from the PR #238 max-effort review, in dependency order, with one bundle regeneration: Crosswalk behavior: iso3 normalized once — bucket, map, and histogram read the same stripped value (#241); read_gaul_cells validates gid uniqueness, set-identity across the two harvest files, and full 259,200-cell completeness at cause (#240, #239 — the 50k output floor is demoted to defence-in-depth); active-active isoab collisions raise instead of silently relabelling a country (#242); unmatched_iso3 is deterministically ordered by (-count, code) so the committed stamp is byte-reproducible (#243); harvest filenames live in one GAUL_HARVEST_FILES constant consumed by both the reader and the sha256 provenance (#244). Stamp contract: querysets reported per actual branch; per-source inner-key contract validated at write time; max_month_id.priogrid nullability documented and tested; coverage accounting pinned by the 259,200 partition identity (#246). The GAUL Kosovo→Serbia absorption (10 cells, VIEWS 232 → 233) is declared in known_gaul_absorptions and validated against the built table every regen (#249). Nets: priogrid row floor (>60k) + continental canaries (#245); sample-scale canary comment corrected to the measured 235,440 cell-frames and its scope boundary documented (#247). Words: coverage prose now states both denominators — 99.50% of the 64,818- cell forecast region (322 unmatched) vs 29,373 of 95,578 GAUL land cells grid-wide, Antarctica-dominated (#248); register C-112/C-22 no longer contradict C-211 and the header date is current (#250). Parquet bytes unchanged (normalization is a no-op on the clean harvest); stamp regenerated once with the new ordering + declarations. 378 tests green (13 new). Closes #239, closes #240, closes #241, closes #242, closes #243, closes #244, closes #245, closes #246, closes #247, closes #248, closes #249, closes #250. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Follow-up medium review of the hardening batch surfaced 8 findings; all addressed: The absorption net now checks BOTH directions: forward (declared Kosovo gids map to Serbia, Kosovo empty) and reverse — every active, assignable country with zero cells must be declared. First real run of the reverse check immediately caught what the one-way net could not see: Luxembourg, Singapore, and The Gambia are active countries GAUL majority-coding folds entirely into neighbours, and the SUN/YUG codes are dead in GAUL 2024 — all now declared in the stamp's new declared_zero_cell_countries block (with four unique-ISO retired states), each with its reason. Also: Kosovo stamp test gains the datafactory skip-guard its siblings had and now binds the absorbed gids to the declared absorber (SRB), not merely to 'one uniform country'; the absorption 'cells' count is derived from the gid list (stamped_absorptions) instead of hand-maintained; null month_id rows are refused at the resolver door (they would bypass the collision guard yet win the sort — latent, now closed); the coverage test reads the required-key contract and PRIO_GRID_N_CELLS from the script instead of re-typing them; the full-harvest test reuses _write_harvest; _load_script is lru_cached (13 re-execs -> 1, ~245ms CI saved, verified mutation-free). Bundle regenerated: parquets byte-identical, stamp gains the zero-cell declarations. 382 tests green (4 new). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This was referenced Jul 21, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #231. Part of epic #230 (tracker #237).
What
The viewser
pg_metadataqueryset is still Africa+ME at the source (verified: 11,169,720 rows = 13,110 cells × 852 months), while the platform's data supply — views-datafactory, which the models already consume — is global and GAUL-coded. Per the decision on #231, the bundledpriogrid.parquetis now built from the factory's harvested GAUL-2024 admin assets via an explicitly interim crosswalk: GAULiso3_code→ VIEWSisoab→country_id, duplicate isoab resolved to the most-recently-observed entity.Numbers
priogrid_sourceblock (source sha256s, crosswalk rule, per-code unmatched histogram).Changes
scripts/build_entity_metadata.py: source seam (--priogrid-source datafactory|viewser, legacy queryset kept); pureread_gaul_cells/active_country_by_isoab/crosswalk_priogrid; ≥50k-cell floor guard; no machine-local paths in committed provenance.test_sample_scale.pydecoupled from bundle size (fixed 13,110-gid subset, pinned budget); stamp required-fields updated.Ritual
review-diff: REVIEW → 1 warning (local path in stamp) + 2 suggestions addressed; re-run verified. ship-it: ruff clean, 365 passed (2 skipped, 1 xfailed).
🤖 Generated with Claude Code