Repository navigation
Give a storage a set of shared keys only while it shares an array - #578
Merged
Merged
Conversation
3.32.12 (#556) created an empty set in every InMemoryStorage. A simulation has a storage for every variable of the tax-benefit system, so that was about 224 bytes per variable per simulation: 1.3 MB for each policyengine-us simulation, branches included. The YAML test runner keeps every case's simulation, and policyengine-us's largest CI batches grew by 2.1 to 2.6 GB and ran out of memory on 16 GB runners. A storage that shares nothing now refers to one empty frozenset, and has a set of its own only while at least one key is shared. Fixes #577 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ld pickles A storage that shares nothing now has no _shared attribute of its own and reads the class's shared-nothing object, so a deep copy or a pickle of it does too (review finding). Adds tests for deepcopy and pickle round trips, for a storage pickled by 3.32.12 with an empty set, and the property that a storage has the attribute exactly when it shares something. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Two threads making first reads of one storage at once, which clone's docstring allows at the cost of an extra copy, could raise AttributeError: one released the set (del self._shared) while the other was between its membership check and its discard, or between its truth test and intersection_update. Each method now reads the set once into a local and works on it, and the release is a single dict pop that never raises. Regression tests run the other thread's read inside the membership check and inside the truth test, and 2,000 threaded trials; all three fail on the previous head (found in review round 2). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review round 2's mutation run found that a source storage keeping a reference to each clone's set (a leak, with no change in behaviour) survived every test. This test kills it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The comments said clone's docstring allows overlapping first reads at the cost of an extra copy. It says a storage is not safe to read from several threads at once, and review round 3 reproduced overlapping-read races that 3.32.12 already has. What this code guarantees is narrower: overlapping first reads do not raise. Comment-only change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
Author
|
Merged on the gates in Max's merge rules, pinned to the reviewed head c9fdfa6:
|
MaxGhenis
added a commit
that referenced
this pull request
Oct 3, 2026
…a set per storage #578 gave a storage a set of shared keys only while it shares an array: an empty set in every InMemoryStorage cost 216 bytes per variable per simulation, about 1.3 MB per policyengine-us simulation, and ran PE-US's CI batches out of memory (#577). This branch's derived marks added a second empty set to every storage, and its __setstate__ gave every unpickled storage its own _shared set, which #578's pickle tests reject. Resolve by keeping #578's code and giving _derived the same treatment: _NOTHING_DERIVED, a class-level empty frozenset, until a storage holds a derived value; its own set from the first derived put; released with one dict pop when the last derived key is replaced by an input or deleted. The class attributes cover pickles from before either set existed, so __setstate__ goes. A clone copies only the marks of keys it holds. apply_reform's wipe releases the marks instead of assigning an empty set. Tests: #578's differential now also stores derived values and compares the effective marks and is_derived with the eager storage; new test_storage_derived_index.py pins the lazy marks (inputs never need a set, release on replace/delete/wipe, clones, copies, old pickles, simulation and apply_reform). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MaxGhenis
added a commit
that referenced
this pull request
Oct 3, 2026
…ly for read branches Review round 3: - Holder._set no longer drops a record entry when a calculated value is stored over an input. A calculate_add at a variable's own period stores the unchanged input and lost its record; a clone, which shares its source's record until #561, deleted the source's entries; and ETERNITY variables keep entries under several periods. The case the cleanup was for, calculate_add caching a sum over an input, is what #571 stops. - Master now gives a storage that shares nothing a frozenset for _shared (#578). The drop step deleted from it directly; it now leaves that to InMemoryStorage._stop_sharing_dropped_keys. - The helpers evicted calculate's fast cache for a sub-period stored under any branch name, including one the simulation does not read. They now evict only for branches it reads, as #576's holder-write property requires. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MaxGhenis
added a commit
that referenced
this pull request
Oct 3, 2026
…h clones Measured on a policyengine-us household (policyengine-us 2.18.3, one household_net_income calculation, 6,185 storages, 30,925 holder clones across 5 branches), master with #578 retains 20.94 MB per simulation. With derived marks as a set per storage that holds a derived value, the merge retained 24.35 MB: 1,514 sets in the simulation and a copy of its source's marks in every holder clone. That is 2.6 times the 1.3 MB per simulation that ran policyengine-us's CI out of memory (#577). 5,201 of the 5,204 stored values are derived, so record the inputs instead: is_derived is "stored and not an input". The 3 storages holding inputs have a set (648 bytes); the rest read _NO_INPUTS. A storage and its clones share one frozenset of inputs until one of them changes its own. Retained memory is now 21.25 MB per simulation, +0.31 MB on master: one more 8-byte attribute slot in each of the 37,110 storage objects. A pickle records that inputs were recorded; a state from before (no marker) counts every stored value as an input, as it did then. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Oct 3, 2026
MaxGhenis
added a commit
that referenced
this pull request
Oct 3, 2026
…562) * Carry over only inputs, the latest at or before the requested period Auto-carry-over took the latest-starting stored period of any kind and returned the default if it started after the requested period. Any later stored period therefore hid an earlier input: a later input (inputs for 2012 and 2014 gave the default for 2013), or a later period the simulation had already calculated (2014 calculated first made 2013 the default). And values the simulation calculated carried like inputs: a twelfth cached at a month by calculate_divide (the policyengine-us monthly_age bug #557 fixes), a value masked by defined_for, a default cached where defined_for was false everywhere, or a formula result from before the formula's end. Results depended on which periods were calculated first. The rule now: a period takes the input stored for the latest-starting period that starts no later than it (on a tie, the one ending last), preferring the variable's own definition-period unit; with none, the default. Values the simulation calculates are stored with derived=True: the mark lives in the storage with the value (InMemoryStorage and OnDiskStorage keep it per key, every put sets or clears it, it counts only while the key is stored, and clones copy it), so deletions, direct writes and #556's shared arrays cannot leave it stale. Holder.is_derived(period, branch_name) reports the mark of the value get_array reads, so a value one branch calculated never hides another branch's input. A derived value never replaces an input the branch reads (calculate_add/calculate_divide used to overwrite inputs stored in another unit), calculate_add caches only sums over several sub-periods, and dump/restore keeps the marks. Storages gain has(), which neither reads nor copies an array. Supersedes #557's carry-over change: its 253 tests pass here. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Keep uprating, input helpers, dumps and deletions consistent with input-only carry-over Round-2 review findings: - A default cached for a period with no earlier input became the base the uprating path uprated later periods from; the default is not cached for variables with uprating (as before). - A value calculated inside a set_input helper was written to the input's branch and recorded as a user input, bypassing the guard that keeps inputs; derived writes now stay on the branch they were calculated on. - dump_simulation reads each value's mark from the same branch and period as the value it dumps, so #552's branch dumps keep matching marks. - Carry-over checks candidates latest first and stops at the first input, instead of resolving the storing branch for every known period. - Storages drop the marks of deleted keys. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Test that deleting a whole branch drops its derived marks Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Cache the default only where master cached a carried value With no input at or before the period but a later period stored, master returned the default without caching it; caching it changed what formulas that test whether a value is stored see (policyengine-uk's maintenance loan and current_education formulas check get_array(period.last_year)), and gave the uprating path a calculated base. Return it uncached there, as master did; cache it (derived) otherwise, as master cached the value it carried. This replaces the uprating-only special case. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Read input periods in one pass over readable keys; clear marks with the caches Round-3 review findings: - A period stored only under a branch this one cannot read was taken for an input (own-unit preference ranked it first) and read back as NaN. Holder.get_input_periods(branch_name) now lists, in one pass over the stored keys, the periods whose readable value is an input, taking for each period the key get_array reads first; carry-over picks from those. This also replaces the per-period walk up the branch chain. - apply_reform's cache wipe kept the storages' marks, and rebuilding a disk index brought a stale mark back onto a new input; the wipe now clears the marks and OnDiskStorage.restore starts without any. - Storages pickled before the marks existed failed to clone, put or delete; __setstate__ now defaults them (and #556's shared-key set). - The public carry-over rule now states the own-unit preference and that an input for the period itself is read back as stored. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Test memory-over-disk precedence, in-memory mark clearing and index rebuilds Closes three mutants the suite did not kill. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Break equal-extent carry-over ties by unit, then period Two inputs covering the same days in units other than the variable's (month:2013-01:2 and day:2013-01-01:59) resolved to whichever was stored first. Prefer the larger unit, then the period's string form, so the choice depends only on the inputs (round-4 review). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Record a storage's inputs, not its derived values, and share them with clones Measured on a policyengine-us household (policyengine-us 2.18.3, one household_net_income calculation, 6,185 storages, 30,925 holder clones across 5 branches), master with #578 retains 20.94 MB per simulation. With derived marks as a set per storage that holds a derived value, the merge retained 24.35 MB: 1,514 sets in the simulation and a copy of its source's marks in every holder clone. That is 2.6 times the 1.3 MB per simulation that ran policyengine-us's CI out of memory (#577). 5,201 of the 5,204 stored values are derived, so record the inputs instead: is_derived is "stored and not an input". The 3 storages holding inputs have a set (648 bytes); the rest read _NO_INPUTS. A storage and its clones share one frozenset of inputs until one of them changes its own. Retained memory is now 21.25 MB per simulation, +0.31 MB on master: one more 8-byte attribute slot in each of the 37,110 storage objects. A pickle records that inputs were recorded; a state from before (no marker) counts every stored value as an input, as it did then. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Carry over an input that ends after the last representable date The carry-over sort key evaluated period.stop for every candidate input. A period that ends after 9999-12-31 (day:9999-12-30:3, or day:2012-01-01:4000000) has no stop: computing it raises OverflowError. Master carried such inputs ([10, 20]); this branch raised. Sort by _end_order, which puts such a period after every period that has a stop, so it still ends last. Found by the review of #582, which uses the same key for uprating. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Order period ends by day number, so ends after 9999-12-31 compare too 6ddbaf9 sorted every period whose stop cannot be computed after those that have one, but equal to each other, so of two inputs ending after 9999-12-31 the string form decided: day:9999-12-30:9 was carried over day:9999-12-30:10, and 24 months over 800 days from the same day (delta review, P2). Month and year periods past 9999 also raised ValueError, or returned a stop no date can hold, rather than OverflowError. _end_order is now the number of the day after the period's last day, counted as date.toordinal counts, by integer arithmetic on the same calendar. It equals stop + 1 day wherever stop has a value and orders every other period by its true end. Tests: the review's two cases in both set orders; a property that _end_order agrees with Period.stop where it has a value and with numpy's calendar everywhere (1,000 examples, sizes to 5,000,000); a longer period from the same day ends later. Also asserts that a pickled state without a set of shared keys gets none (a surviving mutant). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
MaxGhenis
added a commit
that referenced
this pull request
Oct 6, 2026
Brings in #562 (carry over only inputs), #563 and #582 (uprate only from inputs), #578 (shared-key sets), #574 (per-call parameter tracing) and #584. One input record instead of two: the storages' `_inputs` (memory) and `_derived` (disk) from #562 replace this branch's `_input_keys`, so an input is a value stored without `derived=True`, as on master. Sequence numbers stay alongside. Disk storage takes master's `_path_to_write` (a clone never writes over a file it shares) in place of this branch's per-store file names, so the process tokens and restore ordering go. Dumps write master's `derived_periods.txt` (branch-aware) and restore inputs first, then calculated values under one later number. `apply_reform` marks every value set_input did not store as derived and drops it, keeping master's semantics without wiping and replaying. Tests that expected a calculated result to carry over now check that the result is cached, since only inputs carry. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MaxGhenis
added a commit
to MaxGhenis/policyengine-core
that referenced
this pull request
Oct 6, 2026
Brings in PolicyEngine#562, PolicyEngine#563, PolicyEngine#582 (input/derived storage marks, input-only carry-over and uprating), PolicyEngine#574, PolicyEngine#584 (parameter tracing and copy guards), PolicyEngine#578 (shared keys only while sharing) and PolicyEngine#585 (temporary storage directories). Conflict in holders/holder.py, resolved by keeping master's bookkeeping: - _set keeps master's ``derived`` mark (stored with the value, and a derived value is never redirected to the running set_input's branch) and this branch's ``is_input``, which decides what the record names. - put_in_cache keeps master's guard (a derived value never replaces a readable input) and passes ``is_input=False``. - master's ``Holder._stores`` (storage ``has``) replaces this branch's equivalent ``_stores``. simulation.py merged cleanly but copied the record in ``clone`` twice (PolicyEngine#562 added the same copy); kept master's copy and added the ``_user_input_contexts`` reset next to it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MaxGhenis
added a commit
that referenced
this pull request
Oct 9, 2026
…ger period (#581) * Count only inputs as already set when a set_input helper splits a longer period set_input_divide_by_period and set_input_dispatch_by_period treated any value stored for a sub-period as already set, including values the simulation had calculated (a cached default or formula result). The same annual input then gave different months depending on what was calculated first: a divided input skipped calculated months and shared the rest among the others, or failed as inconsistent after the year had been read; a dispatched input reused a calculated month for every later month. The helpers now read the simulation's record of inputs (_user_input_keys): a recorded sub-period keeps its input (and, for dispatch, passes it on to the later sub-periods, as before), and a calculated one is replaced. After storing, they drop the variable's calculated values over overlapping periods (the sums calculate_add caches, the twelfths calculate_divide caches) under the input's branch and the branches it reads through, record what they store as inputs, and evict calculate's fast cache for those periods. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Evict calculate's fast cache for the periods a set_input helper drops calculate does not put sums or twelfths in its fast cache today, so this is defensive: a value held there for a period whose stored value the helper drops goes with it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Restore dumped values as inputs; test nested branches and rolling years A dump does not record which values were inputs, and restore_simulation stored every value with put_in_cache, so a restored simulation had an empty input record and the helpers replaced restored inputs (review finding 1). Restore now stores each value as a recorded input, the rule #576 applies to dumps without an inputs.txt. Tests: a restored month keeps its value under a yearly input (divide and dispatch); an input on a nested branch drops the sum its parent branch calculated; an input stored for twelve months from March is kept. The order-independence property now creates up to two nested branches, with calculations at each level. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Keep the property test off disk file names Windows rejects On disk, a value calculated for a period like year:2013:2 is stored in a file named after the period, and Windows rejects ":" in file names (policyengine-core#526). The Windows CI jobs failed on such a read in the on-disk examples. In disk mode the property now skips those periods. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Record which dumped values were inputs; forget an input a calculation replaces Review round 2: 1. Restoring every dumped value as an input froze calculated values through a later reform. The dumper now writes, next to each variable's arrays, the periods whose value was an input (inputs.txt), and restore records exactly those; a dump without the file restores every value as an input. This is policyengine-core#576's dumper change, taken byte for byte so the two PRs merge without conflict. 2. A value stored by put_in_cache over an input (say a calculate_add sum over an input set for twelve months) left the input's record entry in place, so the helpers and apply_reform kept treating the calculated value as an input. Holder._set now drops the entry, in both of its forms for twelve months from the first of a month, when it stores a value outside set_input. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Test a calculated value replacing an input with put_in_cache, not calculate_add With policyengine-core#571, calculate_add no longer caches a sum over an input a plain read finds, so the input is kept and the test's premise (master's calculate_add overwriting it) did not hold. Storing the calculated value with put_in_cache replaces the input with or without #571. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Drop the record cleanup on calculated writes; evict the fast cache only for read branches Review round 3: - Holder._set no longer drops a record entry when a calculated value is stored over an input. A calculate_add at a variable's own period stores the unchanged input and lost its record; a clone, which shares its source's record until #561, deleted the source's entries; and ETERNITY variables keep entries under several periods. The case the cleanup was for, calculate_add caching a sum over an input, is what #571 stops. - Master now gives a storage that shares nothing a frozenset for _shared (#578). The drop step deleted from it directly; it now leaves that to InMemoryStorage._stop_sharing_dropped_keys. - The helpers evicted calculate's fast cache for a sub-period stored under any branch name, including one the simulation does not read. They now evict only for branches it reads, as #576's holder-write property requires. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #577.
Problem
3.32.12 (#556) gave every
InMemoryStoragean empty set,_shared, to record the arrays a branch still shares with the simulation it came from. A simulation has a storage for every variable of the tax-benefit system, read or not:Simulation.__init__andSimulationBuilder.buildcallget_holderon every variable to listinput_variables. So the set cost 216 bytes per variable per simulation, branches included: about 1.3 MB for each policyengine-us simulation (6,194 variables).The YAML test runner keeps every case's simulation until the session ends, so policyengine-us's CI batches grew by about 2 GB. Since 3.32.12 was published, its two largest batches have died (counts as of 2026-10-02 20:40Z) on 16 GB runners with "The runner has received a shutdown signal" (PolicyEngine/policyengine-us#9757):
Full Suite - Baseline (ssa-usda): 0 passed of 30 completed runs.states-shard-3: 4 passed of 34.Change
frozenset(_NOTHING_SHARED), the class default.clone(share_arrays=True)gives the clone a set of its own only when at least one array is shared.self._sharedonce and works on that set, and the release is a singledict.popwith a default, so overlapping first reads never raise, as on 3.32.12.AttributeErrorthere; review round 2 found it.clone's docstring says. Review round 3 reproduced overlapping-read races that 3.32.12 already has, and this PR leaves them as they were. One: a read returns the shared read-only view. Another: adeleteracing a first read can leave a shared view unmarked.Nothing a storage returns changes.
Measured
policyengine-us
main615a0f2d, onepolicyengine-core test <paths> -c policyengine_usprocess per row, CPython 3.14.7 on macOS arm64, CI's itemization setting, core imported from a snapshot of each commit (this PR: 15a1118, whose storage code is identical to the head's). The six batch runs ran back to back, so they shared one host-load window. Peak memory is the process's peak physical footprint (/usr/bin/time -l). CPU is user plus system time. The host was shared, so CPU and pytest's time vary by about 15% between runs; this PR changes memory, not speed.The two batches that die in CI. The CI batch with the Oklahoma files is NY + OH + OK (
states --batches 16, shard 3/4, batch 3). The WIC files are in the single USDA batch.An earlier run, at 17:30-18:10 EDT the same day, gave the same peaks within 65 MB: NY/OH/OK 16,282 / 18,156 / 16,287 MB, USDA 17,076 / 18,935 / 17,055 MB.
The same batches on Linux CI, peak RSS printed by
test_batched.py:Live objects, every 100 cases over
states/ok(343 cases, 347 simulations, 2,150,359 holders at the end):One file at a time (3.32.8, the version policyengine-us locks, against 3.32.12):
ok_count_exemptions3,788 vs 3,798 MB;ok_exemptions2,009 vs 2,054;ok_income_tax_before_credits2,085 vs 2,187;ok_standard_deduction2,075 vs 2,083;ok_stc2,654 vs 2,785;wic/wic.yaml3,282 vs 3,223. Pytest time was within a second each. No single case got slower or heavier. The batches die from accumulation across thousands of kept simulations.Per storage: 6,194 empty storages take 161 bytes each on 3.32.11, 385 on 3.32.12 and 169 with this PR (tracemalloc).
This PR only takes the batches back to their 3.32.11 size, which was already close to the runner's limit. #569 with #570 halves them (NY/OH/OK 7.0 GB, USDA 6.4 GB; numbers in the #569 thread).
Invariants (tested)
test_storage_shared_index_differential.py(Hypothesis, 500 examples of up to 60 operations; the reference is the 3.32.12 class) runs sequences ofput,get, in-place writes into whatgetreturned,deleteby period and by branch, sharing and copying clones, and emptying_arraysfrom outside. They cover plain, read-only and masked arrays, in eternal and dated storages. After every step both storages hold the same arrays and the same shared keys, and at the end every read agrees.Simulation.clone()have no per-storage sets. A branch has one for each storage that inherited an array.Hand mutation testing: 17 mutants of the storage (
~/reviews/ci-shutdown-core-3-32-12-2026-10-02/scripts/mutants.py), all killed. Review round 2 ran its own mutants. Its only non-equivalent survivor was the source keeping a reference to each clone's set, and c8e024d kills it.Reviews
_sharedin__slots__. Nothing in policyengine-core, -us or -uk subclassesInMemoryStorage. Its retained-set P3 is fixed in c8e024d.Simulationrace trials with and without the GIL, with no errors, and its 22 non-equivalent mutants were all killed. Its P3 on the comment wording is fixed in c9fdfa6 (comments only).Tests
uv run pytest tests/core: 1,165 passed, 1 skipped, 1 xfailed.🤖 Generated with Claude Code