Skip to content

Simulation(dataset=DataFrame) fails when a household or benefit unit has more than one person #2266

Closed

Description

@MaxGhenis

What happens

Simulation(dataset=<pandas DataFrame>) raises ValueError when any benefit unit or household has more than one person and the frame holds a benefit unit or household column (including the identifier columns it needs).

Reproduction

On main (2.125.2, policyengine-core 3.32.27):

import pandas as pd
from policyengine_uk import Simulation

frame = pd.DataFrame(
    {
        "person_id__2025": [1, 2],
        "person_benunit_id__2025": [1, 1],
        "person_household_id__2025": [1, 1],
        "benunit_id__2025": [1, 1],
        "household_id__2025": [1, 1],
        "age__2025": [40, 38],
        "council_tax__2025": [1_500.0, 1_500.0],
    }
)
Simulation(dataset=frame)
ValueError: Unable to set value "[1 1]" for variable "benunit_id", as its length is 2 while there are 1 benunits in the simulation.

The same happens for a frame exported with to_input_dataframe() from a simulation built on a dataset with multi-person households (Unable to set value ... for variable "rent", as its length is 12 while there are 6 households).

Cause, read in the code

Simulation.build_from_dataframe (policyengine_uk/simulation.py) takes a frame with one row per person and columns named variable__period, the format to_input_dataframe() exports. It builds the entities with build_from_ids, then calls self.set_input(variable, time_period, df[column]) for every column. For a benefit unit or household variable the column still has one value per person, so its length is wrong unless every benefit unit and household has exactly one person.

policyengine-core's own loader for this format takes each group's value from its first member (population.value_from_first_person).

A second, smaller problem in the same function: get_first_array finds a column with df.columns.str.contains(variable_name + "__"), so household_id__ also matches person_household_id__2025 and benunit_id__ matches person_benunit_id__2025. build_from_ids passes the result through np.unique, so today either column gives the same entities, but the match should be exact.

Suggested fix

For a variable whose entity is not the person, set population.value_from_first_person(values), and match column names exactly (column.split("__")[0] == variable_name). Add a test that a frame with a multi-person household builds and that a dataset-built simulation's to_input_dataframe() loads back with the same results.

Found while fixing #2254 (PR #2265 does not depend on it).

Activity

  1. added a commit that references this issue on Oct 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions