Skip to content

Exported input DataFrame no longer round-trips: main CI fails test_an_exported_frame_loads_back_and_exports_itself #2333

Closed

Description

@MaxGhenis

policyengine_uk/tests/test_dataframe_dataset.py::test_an_exported_frame_loads_back_and_exports_itself fails on main. It failed in the "Code changes" run on 7343f94 (run 38143054188) and fails locally on 41e970b:

DataFrame shape mismatch
[left]:  (1, 38)
[right]: (1, 18)

Repro. The falsifying example is one person, one benunit and one household in LONDON, with zero inputs:

exported = Simulation(dataset=frame(population, "both")).to_input_dataframe()
again = Simulation(dataset=exported).to_input_dataframe()

Cause. exported already holds uc_receives_new_claimant_health_element__2026 … __2029 next to the 2025 inputs. That variable came in with #2251, fix-uc-health-claim-status. Loading that frame back adds the entity id columns for 2026-2029: person_id, benunit_id, household_id, person_benunit_id and person_household_id, __2026 to __2029. So the second export has 20 more columns than the first, and the round-trip identity fails.

Fix options.

  • Don't export values the simulation set for years other than the dataset year.
  • Or export the id columns for every year that holds inputs, so the export is a fixed point.

The test's invariant (export ∘ load ∘ export = export) is the right one, so keep it.

Activity

  1. MaxGhenis commented on Oct 11, 2026

    @MaxGhenis
    CollaboratorAuthor

    Tracked by the UK hub, which already has a fix in progress (the conflict between #2251 and #2270; uc_reform.py:121 records uc_receives_new_claimant_health_element as an input for 2026–2029). The fix PR should close this issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions