S4 — enrichment.py: numpy/pyarrow keyed gather (drop pandas merge)

[Polichinel]

Epic: #85 · S4 · [UNILATERAL — low priority]

Background

GaulLookupEnricher is OWN-CHOICE pandas: it loads the GAUL lookup via pd.read_parquet (enrichment.py:48) and attaches metadata with base.merge(left_on=pg_id_col, right_index=True, how="left") (enrichment.py:117-119). This is a keyed metadata-attach join, not frame algebra. It does not block samples (it attaches geo columns to whatever rows exist), so this is low priority.

Work

• Replace pd.read_parquet(lookup) + .merge with a pandas-free keyed gather: load the lookup once into a priogrid_gid → row numpy/dict structure (via pyarrow.parquet, already imported for _read_version), then gather metadata by cell id, producing NaN/sentinel for misses to preserve the fail-loud null behaviour.
• Explicitly do NOT push this into views-frames — PredictionFrame carries values + a SpatioTemporalIndex, not arbitrary GAUL columns; modelling a 9-column geographic attach as a frame op would distort the value-object contract. The target is plain numpy/arrow keyed gather.

Acceptance criteria

• Enricher no longer imports/uses pandas; output is column-for-column identical to the merge path, including NaN positions for unmapped cells.
• The country_iso_a3.isna() warning path fires identically.

Parity / validation

tests/test_enrichment.py + tests/test_append_metadata.py as the oracle: feed identical inputs to old merge vs new gather, assert equality incl. NaN positions and dtypes.

Dependencies

Independent of S1–S3 (can land any time). Keep gaul_schema.py (the 9-column contract) as the single source of truth; coordinate with S5.

Files

views_postprocessing/unfao/enrichment.py, tests/test_enrichment.py, tests/test_append_metadata.py.

[Polichinel]

Scope update (2026-07-27): the historical half of this issue is delivered by #126/PR #129 — geography attaches via a pyarrow searchsorted gather from the ADR-011 lookup (unfao/historical.py), no pandas merge. Remaining scope here: the legacy forecast path's enrichment.py merge only, which is slated for deletion with that path post-run-0 — at which point this issue likely closes as moot rather than implemented. Re-assess after run 0.

🤖 Generated with Claude Code