3.32.12 gives every holder's storage an empty set: about 1.3 MB more per policyengine-us simulation #577

What happens

Since 3.32.12, every InMemoryStorage is created with an empty set of its own:

def __init__(self, is_eternal: bool):
    self._arrays = {}
    self._shared = set()

_shared was added in #556 to record which arrays a branch still shares with the simulation it was created from. Only a storage made by clone(share_arrays=True) ever puts anything in it.

A simulation has one holder, and so one storage, for every variable in the tax-benefit system, whether or not the variable is ever read: Simulation.__init__ and SimulationBuilder build input_variables by calling get_holder for each variable, which creates the holder. An empty set is 216 bytes on CPython 3.11 to 3.14, so the new attribute costs about 224 bytes per variable per simulation, branches included. policyengine-us has 6,194 variables, which makes it about 1.3 MB per simulation.

That is small for one microsimulation. It is not small in the YAML test runner, which keeps every case's simulation alive until the session ends (#569 addresses that): a batch of policyengine-us tests ends with 1,500 to 2,000 live simulations.

Evidence

policyengine-us CI, same tests, 3.32.11 against 3.32.12. policyengine-us's PR workflow resolves the latest core before it runs. Its main did not change between the last 3.32.11 run (2026-10-02 02:01Z) and the first 3.32.12 runs (from 09:25Z; 3.32.12 was published at 04:58Z). The Linux peak RSS that test_batched.py prints for the four batches of Full Suite - Baseline (states-shard-3):

Jobs: 110390394947, 110597412824, 110671921673 (3.32.11); 110841371769, 110880078069, 110973714235 (3.32.12). The uv lock --upgrade-package policyengine-core step of each log shows the core version, and the two environments differ in nothing else.

Batch 3 and the single USDA batch (14,872 to 14,900 MB on 3.32.11) had no room for the increase. From 09:25Z both jobs die with "The runner has received a shutdown signal" / exit 143 on most pull requests and on pushes to main: PolicyEngine/policyengine-us#9757.

Object counts. One process over the same 457 policyengine-us baseline YAML files on master (b78b0ba) and on the commit before #556 (7950c01), counting live objects every 50 cases:

Every simulation has 6,194 holders, and master has one more gc-tracked object per holder: the set.

In isolation. 6,194 objects shaped like a storage take 385 bytes each with a set of their own and 161 bytes each without one (tracemalloc, CPython 3.14.7): 2.27 MB against 0.95 MB.

Fix

Keep one empty, immutable object for every storage that shares nothing, and give a storage a set of its own only while at least one of its keys is shared.