Skip to content

Export CPS parent pointers as person-level parent ids - #1032

Merged
MaxGhenis merged 7 commits into
mainfrom
export-parent-ids-884
Sep 27, 2026
Merged

MaxGhenis merged 7 commits into
mainfrom
export-parent-ids-884

Conversation

@MaxGhenis

@MaxGhenis MaxGhenis commented Sep 26, 2026 •

Copy link
Copy Markdown
Contributor

Closes microcosm#884.

What this does

Exports the CPS ASEC parent pointers as two person columns, parent_1_id and parent_2_id: each is the person_id of the co-resident member that the person's PEPAR1/PEPAR2 pointer names within PH_SEQ, and 0 when the pointer is absent or names no co-resident line. Until now Microcosm exported only the count behind them (own_children_in_household), so a rules engine could ask how many children an adult has but not whose child a person is. policyengine-us#9406 consumes the ids for Medicaid MAGI household composition (42 CFR 435.603(f)(3)).

  • Eligibility stage (eligibility_inputs.py): one pointer normalization behind both the count and the ids; int64 storage. Where relationships are measured (the stage's own derivation) the count must equal the number of ids naming each person, exactly.
  • Release gate: every nonzero id names a co-resident person (exact int64 household comparison), nobody is their own parent, nobody is named by more people than own_children_in_household counts (ACS rows impute the count with pointers declared unknown, so the gate checks this bound rather than equality), and every person_id in a household carrying a link fits PolicyEngine-US's int32 storage. Parent ids may be all zero ("unknown"), so they are not in the non-constant release set.
  • Existing frames: a frame that already carries the pre-Export CPS parent pointers (PEPAR1/PEPAR2) as person-level parent ids #884 outputs with signal (a --base-h5 input, an existing pool) passes through untouched, without parent ids. The ids are optional end to end: the release gate asserts their identities only when both are present, so existing pools still feed releases, and PolicyEngine-US uses its count-based proxy where they are absent.
  • Stored-input register (microcosm-data stored_inputs.py): parent_1_id/parent_2_id are registered stored non-variable columns (the pinned engine, policyengine-us 2.2.1, does not define them), naming their producer and the removal condition: drop the entries when the pin moves to a policyengine-us release with #9406. Without this, main's stored-input contract (Refuse US releases that store model inputs the certified engine does not define (#1026) #1031) refuses every release that carries them.
  • ACS spine (acs_inputs.py, operator_boundary.py): declares the pointers unknown (0) natively.
  • Multispine pool (multispine_pool.py): the ids are native-complete on both arms and never enter the QRF transfer plan.
  • Clones (puf_support.py, puf_capital_gains_tail.py): the support clone and the capital-gains tail clone shift parent ids with person_id, leaving 0 alone; rows that never had a pointer carry 0.
  • Input mass (input_mass.py, shared): PERSON_REFERENCE_COLUMNS are skipped as structural, like entity ids.
  • Pool resume (tools/build_us_multispine_pool.py): a transferred/simulated checkpoint whose recorded person schema lacks an eligibility output is ignored as stale, since the byte-stable legacy identity cannot see new outputs.

How this branch was built

The work was done on 9/7 on parent-ids-export (pushed unchanged as a backup at 7ec9eafc1), which fell ~797 commits behind main. This branch is fresh from origin/main (31315687d) with the three substantive commits cherry-picked (ccce48cc9, 806f780a4, 29a7b71fd); the lane-journal commits (ade4b5cb4, 2361549cb, 7ec9eafc1) and the journal/checkpoint files are dropped.

Conflicts were only digest pins. Re-pinned at observed values after the rebuild, each for the stated reason:

pin why it moves
source_stages.json asset sha256 (sources.yaml, generate_us_bundle_from_constants.py, test_us_bundle_core_contracts.py, test_us_spec_bundle.py) the eligibility stage gains two outputs
US bundle spec_sha256 → d3469f47… (test_us_multispine_pool_tool.py, us-f0-coverage.json) follows from the stage asset
field-usage counts 42,180 → 42,184 and 32,400 → 32,404; source_stages pointer claim; pointer inventory four new authored fields (two outputs, two nonnegative outputs)
seed-protocol, seed-map, resource-semantics, source-manifest and full-checkpoint identity digests (inventory_coverage.py); the loader golden vector puf_support and puf_capital_gains_tail are attested kernel modules (spec_engine/seeds.py _DIRECT_KERNEL_MODULES)

tools/generate_us_bundle_from_constants.py --check and tools/spec_engine_coverage.py --check pass (42,184/42,184 fields; 41/41 inventory checks). docs/evidence/spec-engine/us-f0-coverage.json was regenerated by the tool, not edited.

Effect on certified builds

A build from this code gains two person columns. For existing columns the intent is no change, and the round-1 review's one counterexample (a --base-h5 frame whose carried veterans_benefits was overwritten by re-derivation) is fixed by passing such frames through untouched (as main does), covered by test_legacy_frame_passes_through_untouched. Draw sites seed from seed/salt/stable-key material; the moved digests attest source and are not seed inputs. The QRF transfer targets are unchanged.

A PolicyEngine-US version without these variables loads such a build unchanged: policyengine_core skips dataset columns that match no variable and logs a warning. The latest published populace_us_2024.h5 has person_id 4–1,432,502, inside int32; the gate now enforces that for linked households instead of assuming it.

Publishing a build with these columns is a separate, deliberate release step and is not part of this PR.

Verification

  • Rebased on origin/main 2bce66156. tools/generate_us_bundle_from_constants.py --check and tools/spec_engine_coverage.py --check pass (42,184/42,184 fields; 41/41 inventory checks); every pin was recomputed from the tools, not hand-edited.
  • Local: the eligibility, input-mass, ACS, capital-gains-tail, L0 export, stored-input register, spec-engine and affected multispine-pool tests, and the whole microcosm-data test package, pass; tools/ci_test_groups.py --verify ok; ruff check clean. The full suite runs in CI.
  • Independent review: round 1 request-changes (7 findings), all addressed in 808ee8feb. Round 2 confirmed those resolved, answered "no" on whether a completing build changes existing columns, and found the stored-input register gap and the existing-frame key problem, addressed in 813978feb. Round 3 confirmed those and asked that the register entries rest on their producer in the microcosm-data tests (the parent ids postdate the H5 files the register was built from), addressed in dd372e109. Round 4 approved (no new correctness bug; existing output columns unchanged), with nits carried under known limits.
  • Known limits, documented rather than fixed: after derivation the gate's count check is an upper bound (a remap that wrongly zeroes a pointer cannot be seen), and the int32 check covers only households that carry a link.

🤖 Generated with Claude Code

claude and others added 4 commits September 26, 2026 09:21
Resolve PEPAR1/PEPAR2 against (PH_SEQ, A_LINENO) to the co-resident
member's person_id and export them as parent_1_id / parent_2_id, the
relation behind own_children_in_household rather than only its
cardinality (microcosm#884, consumed by policyengine-us#9404).

- eligibility_inputs: one shared pointer normalization behind both the
  count and the ids, int64 storage (a clone-remapped person_id runs past
  2**53), and gate invariants asserting co-residence, no self-parenting,
  and that own_children_in_household equals the number of people whose
  parent ids name that person.
- acs_inputs / operator_boundary: the ACS spine declares the pointers
  unknown (0) natively. Its synthesized PEPAR values are a
  reference-person link, not the ASEC measured pointer, and an ACS-minted
  person_id is renumbered by the assembly offset after that mapping runs.
- multispine_pool: the parent ids are native-complete on both arms, so
  they never enter the QRF transfer plan. They are not PolicyEngine-US
  variables, so _target_encoding would treat an id as continuous and hand
  an ACS child a fractional interpolation between two ASEC person ids.
  The transfer surface is unchanged at 118 targets.
- puf_support: the support clone now shifts person-referencing columns
  with person_id, preserving the 0 sentinel; without it a clone's child
  pointed at the other arm's parent.
- source_stages.json + regenerated sources.yaml, with the frozen
  generation-0 digest re-pinned.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
- Stage tests: two-parent, single-parent, three-generation (the
  grandchild names only the parent), unresolved and absent pointers,
  int64 dtype, person_id-not-line-number, missing person_id, and a
  repeated (PH_SEQ, A_LINENO).
- Clone tests: each support arm's child names that arm's parent, and the
  0 sentinel is not shifted into a valid-looking id.
- Gate tests: a pointer naming no person, one outside the household,
  self-parenting, a count disagreeing with the pointers, a null id, and
  the person_id-0 case, which cannot be named by a 0-means-unknown
  pointer and is reported rather than failed.
- ACS tests: the native mapping writes the declared unknown sentinel,
  the receipt satisfies the operator boundary, and the columns stay
  absent when the raw pointers are.
- us_input_mass_totals now drops the person-referencing id columns: the
  weighted sum of an arbitrary integer label is not a mass, and it moves
  whenever person ids are renumbered, so comparing it against a
  reference release on a different id space would fail for no reason.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…sign

The new eligibility-stage outputs move the source_stages asset digest,
the bundle spec_sha256 (now d3469f47...), and, because puf_support is an
attested kernel module, the seed-protocol, seed-map, resource-semantics,
source-manifest and full-checkpoint identity digests. Re-pinned each at
its observed value, bumped the field-usage counts by the four new
authored fields, re-pinned the source_stages pointer claim and pointer
inventory, and regenerated docs/evidence/spec-engine/us-f0-coverage.json
with tools/spec_engine_coverage.py. The operator-ordering doc's
eligibility row now says why the stage is pre-clone and CPS-scoped, and
that its outputs name a person_id.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…sumers

Independent review (2026-09-25) returned request-changes with seven findings
and the rebased branch failed CI in four lanes. Changes:

- Capital-gains tail clone shifts parent ids with person_id, keeping the 0
  sentinel (_shift_person_references); the preassembled PUF tax-detail rows,
  which never had a pointer, carry 0 rather than null through the support
  clone's reference remap.
- The release gate no longer demands own_children_in_household equal the
  pointers naming a person. ACS rows impute the count with pointers declared
  unknown, and a clone moves the unnameable id-0 parent, so the gate asserts
  pointed_at_by <= count; the exact equality is asserted where the
  relationships are measured, when the stage derives them.
- A frame that already carries the pre-#884 outputs with signal (a --base-h5
  input) gets only parent_1_id/parent_2_id appended, resolved from the raw
  pointers when present and 0 otherwise; no other column is recomputed.
  Parent ids leave the non-constant release set: all-zero means unknown.
- Co-residence compares exact int64 household ids (float64 conflated ids past
  2**53), and a new check requires every person_id in a household carrying a
  parent link to fit PolicyEngine-US's int32 storage.
- Pool resume ignores a transferred/simulated checkpoint whose recorded person
  schema lacks an eligibility output, since the byte-stable legacy identity
  cannot see new outputs.
- The input-mass exclusion moves into the shared helper as
  PERSON_REFERENCE_COLUMNS, restoring main's exact US alias.
- Re-pinned at observed values after the rebase and these edits: spec_sha256
  1cf1f44d..., the five inventory digests, field-usage mode/effect counts,
  and the loader golden vector (attested kernel sources moved).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@MaxGhenis
MaxGhenis force-pushed the export-parent-ids-884 branch from 11cad61 to 808ee8f Compare September 27, 2026 06:19
MaxGhenis and others added 3 commits September 27, 2026 02:33
…ional

Round-2 review of #1032:

- Main's stored-input contract (#1031) refuses a stored column that is
  neither a variable of the certified engine (policyengine-us 2.2.1) nor
  registered. parent_1_id/parent_2_id now have register entries naming
  their producer and the removal condition (the pinned engine defining
  them, which policyengine-us#9406 does); the producer-binding tests
  cover them.
- Existing frames with the pre-#884 outputs (a --base-h5 input, an
  existing pool) pass through untouched instead of having ids appended:
  resolving raw pointers there keyed on (PH_SEQ, A_LINENO) pairs that
  repeat after cloning and could export ACS reference-person links. The
  release gate treats the ids as optional and asserts their identities
  only when both are present, so existing pools still feed releases.
- A vacuous test assertion now asserts the gate passes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…d file

Round-3 review of #1032: two microcosm-data tests require every register
entry to be stored by an H5 file examined for #1026, and no examined file
stores the parent ids yet. Following the *_spine precedent, the tests name
them as producer-bound entries (bound to eligibility_inputs by
test_us_stored_input_register.py), and the register docstring says so.
Also fixes a gate assertion that filtered on text no message contains.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The guard treated any transferred/simulated checkpoint lacking an
eligibility output as stale, so the checkpoint round-trip tests' synthetic
fixtures (which carry no eligibility columns) were rebuilt instead of
resumed, and it still required the parent ids after they became optional.
It now marks stale only a checkpoint that carries the pre-#884 eligibility
outputs but not the parent ids: the shape main writes, where resuming
would build without ids that a fresh run produces. Anything else is left
to the existing identity and schema checks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@MaxGhenis
MaxGhenis marked this pull request as ready for review September 27, 2026 09:07
@MaxGhenis
MaxGhenis merged commit bd92673 into main Sep 27, 2026
24 checks passed
@MaxGhenis
MaxGhenis deleted the export-parent-ids-884 branch September 27, 2026 09:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Export CPS parent pointers (PEPAR1/PEPAR2) as person-level parent ids

2 participants