Skip to content

Give a storage a set of shared keys only while it shares an array - #578

Merged
MaxGhenis merged 5 commits into
masterfrom
lazy-shared-array-index
Oct 3, 2026
Merged

MaxGhenis merged 5 commits into
masterfrom
lazy-shared-array-index

Conversation

@MaxGhenis

@MaxGhenis MaxGhenis commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #577.

Problem

3.32.12 (#556) gave every InMemoryStorage an empty set, _shared, to record the arrays a branch still shares with the simulation it came from. A simulation has a storage for every variable of the tax-benefit system, read or not: Simulation.__init__ and SimulationBuilder.build call get_holder on every variable to list input_variables. So the set cost 216 bytes per variable per simulation, branches included: about 1.3 MB for each policyengine-us simulation (6,194 variables).

The YAML test runner keeps every case's simulation until the session ends, so policyengine-us's CI batches grew by about 2 GB. Since 3.32.12 was published, its two largest batches have died (counts as of 2026-10-02 20:40Z) on 16 GB runners with "The runner has received a shutdown signal" (PolicyEngine/policyengine-us#9757):

  • Full Suite - Baseline (ssa-usda): 0 passed of 30 completed runs.
  • states-shard-3: 4 passed of 34.

Change

  • A storage that shares nothing reads one module-level empty frozenset (_NOTHING_SHARED), the class default.
  • clone(share_arrays=True) gives the clone a set of its own only when at least one array is shared.
  • When the last shared key is read, replaced or deleted, the storage's own attribute is removed, so it, and any copy or pickle of it, reads the class default again.
  • Each method reads self._shared once and works on that set, and the release is a single dict.pop with a default, so overlapping first reads never raise, as on 3.32.12.
    • An earlier head of this PR (028ad74) could raise AttributeError there; review round 2 found it.
    • A storage is still not safe to read from several threads at once, as clone's docstring says. Review round 3 reproduced overlapping-read races that 3.32.12 already has, and this PR leaves them as they were. One: a read returns the shared read-only view. Another: a delete racing a first read can leave a shared view unmarked.

Nothing a storage returns changes.

Measured

policyengine-us main 615a0f2d, one policyengine-core test <paths> -c policyengine_us process per row, CPython 3.14.7 on macOS arm64, CI's itemization setting, core imported from a snapshot of each commit (this PR: 15a1118, whose storage code is identical to the head's). The six batch runs ran back to back, so they shared one host-load window. Peak memory is the process's peak physical footprint (/usr/bin/time -l). CPU is user plus system time. The host was shared, so CPU and pytest's time vary by about 15% between runs; this PR changes memory, not speed.

The two batches that die in CI. The CI batch with the Oklahoma files is NY + OH + OK (states --batches 16, shard 3/4, batch 3). The WIC files are in the single USDA batch.

Core NY/OH/OK (1,328 cases): peak CPU pytest USDA incl. WIC (1,373 cases): peak CPU pytest
3.32.11 16,277 MB 230 s 112 s 17,052 MB 241 s 142 s
3.32.12 18,156 MB 288 s 147 s 18,884 MB 277 s 153 s
this PR 16,288 MB 248 s 147 s 17,116 MB 275 s 175 s

An earlier run, at 17:30-18:10 EDT the same day, gave the same peaks within 65 MB: NY/OH/OK 16,282 / 18,156 / 16,287 MB, USDA 17,076 / 18,935 / 17,055 MB.

The same batches on Linux CI, peak RSS printed by test_batched.py:

  • NY/OH/OK was 15.0-15.1 GB on 3.32.11 and reached the 16 GB limit on 3.32.12.
  • USDA was 14.9 GB on 3.32.11 and dies on 3.32.12.

Live objects, every 100 cases over states/ok (343 cases, 347 simulations, 2,150,359 holders at the end):

Core gc-tracked objects at the end
3.32.11 37,139,122
3.32.12 39,289,489
this PR 37,140,534

One file at a time (3.32.8, the version policyengine-us locks, against 3.32.12): ok_count_exemptions 3,788 vs 3,798 MB; ok_exemptions 2,009 vs 2,054; ok_income_tax_before_credits 2,085 vs 2,187; ok_standard_deduction 2,075 vs 2,083; ok_stc 2,654 vs 2,785; wic/wic.yaml 3,282 vs 3,223. Pytest time was within a second each. No single case got slower or heavier. The batches die from accumulation across thousands of kept simulations.

Per storage: 6,194 empty storages take 161 bytes each on 3.32.11, 385 on 3.32.12 and 169 with this PR (tracemalloc).

This PR only takes the batches back to their 3.32.11 size, which was already close to the runner's limit. #569 with #570 halves them (NY/OH/OK 7.0 GB, USDA 6.4 GB; numbers in the #569 thread).

Invariants (tested)

  • Same behaviour as the 3.32.12 storage. test_storage_shared_index_differential.py (Hypothesis, 500 examples of up to 60 operations; the reference is the 3.32.12 class) runs sequences of put, get, in-place writes into what get returned, delete by period and by branch, sharing and copying clones, and emptying _arrays from outside. They cover plain, read-only and masked arrays, in eternal and dated storages. After every step both storages hold the same arrays and the same shared keys, and at the end every read agrees.
  • A storage reads the shared-nothing object exactly when it shares nothing, and has an attribute of its own exactly when it shares something.
  • A set belongs to one storage: no two storages hold the same set, and nothing else, the source included, refers to it.
  • An array under a shared key cannot be written through.
  • A simulation and Simulation.clone() have no per-storage sets. A branch has one for each storage that inherited an array.
  • Deep copies and pickles keep these rules, and a storage pickled by 3.32.12 with an empty set of its own still works.
  • Concurrent first reads never raise. Two tests run one thread's whole read inside the other's membership check and inside its truth test. A third runs 2,000 threaded trials. All three fail on 028ad74.

Hand mutation testing: 17 mutants of the storage (~/reviews/ci-shutdown-core-3-32-12-2026-10-02/scripts/mutants.py), all killed. Review round 2 ran its own mutants. Its only non-equivalent survivor was the source keeping a reference to each clone's set, and c8e024d kills it.

Reviews

  • Round 1 (read only): approve. Its pickle and annotation suggestions were applied in 028ad74.
  • Round 2 (GPT-6.1 Sol, executed, on 028ad74): request changes for the first-read race, fixed in 15a1118. Its slot-subclass P3 is not addressed: a subclass redeclaring the private _shared in __slots__. Nothing in policyengine-core, -us or -uk subclasses InMemoryStorage. Its retained-set P3 is fixed in c8e024d.
  • Round 3 (GPT-6.1 Sol, executed, on c8e024d): approve. It ran the full core suite (1,166 passed) and 10,000 two-key, 10,000 same-key and 10,000 Simulation race trials with and without the GIL, with no errors, and its 22 non-equivalent mutants were all killed. Its P3 on the comment wording is fixed in c9fdfa6 (comments only).

Tests

  • uv run pytest tests/core: 1,165 passed, 1 skipped, 1 xfailed.
  • The new modules fail on master: they import the shared-nothing object, which master does not have.

🤖 Generated with Claude Code

3.32.12 (#556) created an empty set in every InMemoryStorage. A simulation
has a storage for every variable of the tax-benefit system, so that was
about 224 bytes per variable per simulation: 1.3 MB for each policyengine-us
simulation, branches included. The YAML test runner keeps every case's
simulation, and policyengine-us's largest CI batches grew by 2.1 to 2.6 GB
and ran out of memory on 16 GB runners.

A storage that shares nothing now refers to one empty frozenset, and has a
set of its own only while at least one key is shared.

Fixes #577

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MaxGhenis and others added 2 commits October 2, 2026 17:36
…ld pickles

A storage that shares nothing now has no _shared attribute of its own and
reads the class's shared-nothing object, so a deep copy or a pickle of it
does too (review finding). Adds tests for deepcopy and pickle round trips,
for a storage pickled by 3.32.12 with an empty set, and the property that a
storage has the attribute exactly when it shares something.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Two threads making first reads of one storage at once, which clone's
docstring allows at the cost of an extra copy, could raise AttributeError:
one released the set (del self._shared) while the other was between its
membership check and its discard, or between its truth test and
intersection_update. Each method now reads the set once into a local and
works on it, and the release is a single dict pop that never raises.

Regression tests run the other thread's read inside the membership check
and inside the truth test, and 2,000 threaded trials; all three fail on the
previous head (found in review round 2).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MaxGhenis and others added 2 commits October 2, 2026 23:07
Review round 2's mutation run found that a source storage keeping a
reference to each clone's set (a leak, with no change in behaviour) survived
every test. This test kills it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The comments said clone's docstring allows overlapping first reads at the
cost of an extra copy. It says a storage is not safe to read from several
threads at once, and review round 3 reproduced overlapping-read races that
3.32.12 already has. What this code guarantees is narrower: overlapping
first reads do not raise. Comment-only change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@MaxGhenis
MaxGhenis marked this pull request as ready for review October 3, 2026 11:29
@MaxGhenis
MaxGhenis merged commit ac1acc8 into master Oct 3, 2026
18 checks passed
@MaxGhenis
MaxGhenis deleted the lazy-shared-array-index branch October 3, 2026 11:29
@MaxGhenis

Copy link
Copy Markdown
Contributor Author

Merged on the gates in Max's merge rules, pinned to the reviewed head c9fdfa6:

  • gh pr checks exited 0 with all 18 checks passing on that head.
  • GitHub reported the PR MERGEABLE, and no review requested changes.
  • Review round 3 (GPT-6.1 Sol, which ran the code) approved c8e024d.
  • A delta review approved c9fdfa6. That commit changes comments only: both files' Python syntax trees are identical to c8e024d's.

MaxGhenis added a commit that referenced this pull request Oct 3, 2026
…a set per storage

#578 gave a storage a set of shared keys only while it shares an array: an
empty set in every InMemoryStorage cost 216 bytes per variable per
simulation, about 1.3 MB per policyengine-us simulation, and ran PE-US's CI
batches out of memory (#577). This branch's derived marks added a second
empty set to every storage, and its __setstate__ gave every unpickled
storage its own _shared set, which #578's pickle tests reject.

Resolve by keeping #578's code and giving _derived the same treatment:
_NOTHING_DERIVED, a class-level empty frozenset, until a storage holds a
derived value; its own set from the first derived put; released with one
dict pop when the last derived key is replaced by an input or deleted. The
class attributes cover pickles from before either set existed, so
__setstate__ goes. A clone copies only the marks of keys it holds.
apply_reform's wipe releases the marks instead of assigning an empty set.

Tests: #578's differential now also stores derived values and compares the
effective marks and is_derived with the eager storage; new
test_storage_derived_index.py pins the lazy marks (inputs never need a set,
release on replace/delete/wipe, clones, copies, old pickles, simulation and
apply_reform).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MaxGhenis added a commit that referenced this pull request Oct 3, 2026
…ly for read branches

Review round 3:

- Holder._set no longer drops a record entry when a calculated value is
  stored over an input. A calculate_add at a variable's own period stores
  the unchanged input and lost its record; a clone, which shares its
  source's record until #561, deleted the source's entries; and ETERNITY
  variables keep entries under several periods. The case the cleanup was
  for, calculate_add caching a sum over an input, is what #571 stops.
- Master now gives a storage that shares nothing a frozenset for _shared
  (#578). The drop step deleted from it directly; it now leaves that to
  InMemoryStorage._stop_sharing_dropped_keys.
- The helpers evicted calculate's fast cache for a sub-period stored under
  any branch name, including one the simulation does not read. They now
  evict only for branches it reads, as #576's holder-write property
  requires.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MaxGhenis added a commit that referenced this pull request Oct 3, 2026
…h clones

Measured on a policyengine-us household (policyengine-us 2.18.3, one
household_net_income calculation, 6,185 storages, 30,925 holder clones
across 5 branches), master with #578 retains 20.94 MB per simulation.
With derived marks as a set per storage that holds a derived value, the
merge retained 24.35 MB: 1,514 sets in the simulation and a copy of its
source's marks in every holder clone. That is 2.6 times the 1.3 MB per
simulation that ran policyengine-us's CI out of memory (#577).

5,201 of the 5,204 stored values are derived, so record the inputs
instead: is_derived is "stored and not an input". The 3 storages holding
inputs have a set (648 bytes); the rest read _NO_INPUTS. A storage and its
clones share one frozenset of inputs until one of them changes its own.
Retained memory is now 21.25 MB per simulation, +0.31 MB on master: one
more 8-byte attribute slot in each of the 37,110 storage objects.

A pickle records that inputs were recorded; a state from before (no
marker) counts every stored value as an input, as it did then.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MaxGhenis added a commit that referenced this pull request Oct 3, 2026
…562)

* Carry over only inputs, the latest at or before the requested period

Auto-carry-over took the latest-starting stored period of any kind and
returned the default if it started after the requested period. Any later
stored period therefore hid an earlier input: a later input (inputs for
2012 and 2014 gave the default for 2013), or a later period the simulation
had already calculated (2014 calculated first made 2013 the default). And
values the simulation calculated carried like inputs: a twelfth cached at a
month by calculate_divide (the policyengine-us monthly_age bug #557 fixes),
a value masked by defined_for, a default cached where defined_for was false
everywhere, or a formula result from before the formula's end. Results
depended on which periods were calculated first.

The rule now: a period takes the input stored for the latest-starting
period that starts no later than it (on a tie, the one ending last),
preferring the variable's own definition-period unit; with none, the
default. Values the simulation calculates are stored with derived=True:
the mark lives in the storage with the value (InMemoryStorage and
OnDiskStorage keep it per key, every put sets or clears it, it counts only
while the key is stored, and clones copy it), so deletions, direct writes
and #556's shared arrays cannot leave it stale. Holder.is_derived(period,
branch_name) reports the mark of the value get_array reads, so a value one
branch calculated never hides another branch's input. A derived value never
replaces an input the branch reads (calculate_add/calculate_divide used to
overwrite inputs stored in another unit), calculate_add caches only sums
over several sub-periods, and dump/restore keeps the marks. Storages gain
has(), which neither reads nor copies an array.

Supersedes #557's carry-over change: its 253 tests pass here.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Keep uprating, input helpers, dumps and deletions consistent with input-only carry-over

Round-2 review findings:
- A default cached for a period with no earlier input became the base the
  uprating path uprated later periods from; the default is not cached for
  variables with uprating (as before).
- A value calculated inside a set_input helper was written to the input's
  branch and recorded as a user input, bypassing the guard that keeps
  inputs; derived writes now stay on the branch they were calculated on.
- dump_simulation reads each value's mark from the same branch and period
  as the value it dumps, so #552's branch dumps keep matching marks.
- Carry-over checks candidates latest first and stops at the first input,
  instead of resolving the storing branch for every known period.
- Storages drop the marks of deleted keys.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Test that deleting a whole branch drops its derived marks

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Cache the default only where master cached a carried value

With no input at or before the period but a later period stored, master
returned the default without caching it; caching it changed what formulas
that test whether a value is stored see (policyengine-uk's maintenance
loan and current_education formulas check get_array(period.last_year)),
and gave the uprating path a calculated base. Return it uncached there, as
master did; cache it (derived) otherwise, as master cached the value it
carried. This replaces the uprating-only special case.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Read input periods in one pass over readable keys; clear marks with the caches

Round-3 review findings:
- A period stored only under a branch this one cannot read was taken for
  an input (own-unit preference ranked it first) and read back as NaN.
  Holder.get_input_periods(branch_name) now lists, in one pass over the
  stored keys, the periods whose readable value is an input, taking for
  each period the key get_array reads first; carry-over picks from those.
  This also replaces the per-period walk up the branch chain.
- apply_reform's cache wipe kept the storages' marks, and rebuilding a
  disk index brought a stale mark back onto a new input; the wipe now
  clears the marks and OnDiskStorage.restore starts without any.
- Storages pickled before the marks existed failed to clone, put or
  delete; __setstate__ now defaults them (and #556's shared-key set).
- The public carry-over rule now states the own-unit preference and that
  an input for the period itself is read back as stored.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Test memory-over-disk precedence, in-memory mark clearing and index rebuilds

Closes three mutants the suite did not kill.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Break equal-extent carry-over ties by unit, then period

Two inputs covering the same days in units other than the variable's
(month:2013-01:2 and day:2013-01-01:59) resolved to whichever was stored
first. Prefer the larger unit, then the period's string form, so the
choice depends only on the inputs (round-4 review).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Record a storage's inputs, not its derived values, and share them with clones

Measured on a policyengine-us household (policyengine-us 2.18.3, one
household_net_income calculation, 6,185 storages, 30,925 holder clones
across 5 branches), master with #578 retains 20.94 MB per simulation.
With derived marks as a set per storage that holds a derived value, the
merge retained 24.35 MB: 1,514 sets in the simulation and a copy of its
source's marks in every holder clone. That is 2.6 times the 1.3 MB per
simulation that ran policyengine-us's CI out of memory (#577).

5,201 of the 5,204 stored values are derived, so record the inputs
instead: is_derived is "stored and not an input". The 3 storages holding
inputs have a set (648 bytes); the rest read _NO_INPUTS. A storage and its
clones share one frozenset of inputs until one of them changes its own.
Retained memory is now 21.25 MB per simulation, +0.31 MB on master: one
more 8-byte attribute slot in each of the 37,110 storage objects.

A pickle records that inputs were recorded; a state from before (no
marker) counts every stored value as an input, as it did then.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Carry over an input that ends after the last representable date

The carry-over sort key evaluated period.stop for every candidate input.
A period that ends after 9999-12-31 (day:9999-12-30:3, or
day:2012-01-01:4000000) has no stop: computing it raises OverflowError.
Master carried such inputs ([10, 20]); this branch raised. Sort by
_end_order, which puts such a period after every period that has a stop,
so it still ends last. Found by the review of #582, which uses the same
key for uprating.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Order period ends by day number, so ends after 9999-12-31 compare too

6ddbaf9 sorted every period whose stop cannot be computed after those
that have one, but equal to each other, so of two inputs ending after
9999-12-31 the string form decided: day:9999-12-30:9 was carried over
day:9999-12-30:10, and 24 months over 800 days from the same day (delta
review, P2). Month and year periods past 9999 also raised ValueError, or
returned a stop no date can hold, rather than OverflowError.

_end_order is now the number of the day after the period's last day,
counted as date.toordinal counts, by integer arithmetic on the same
calendar. It equals stop + 1 day wherever stop has a value and orders
every other period by its true end.

Tests: the review's two cases in both set orders; a property that
_end_order agrees with Period.stop where it has a value and with numpy's
calendar everywhere (1,000 examples, sizes to 5,000,000); a longer period
from the same day ends later. Also asserts that a pickled state without
a set of shared keys gets none (a surviving mutant).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
MaxGhenis added a commit that referenced this pull request Oct 6, 2026
Brings in #562 (carry over only inputs), #563 and #582 (uprate only from
inputs), #578 (shared-key sets), #574 (per-call parameter tracing) and #584.

One input record instead of two: the storages' `_inputs` (memory) and
`_derived` (disk) from #562 replace this branch's `_input_keys`, so an input
is a value stored without `derived=True`, as on master. Sequence numbers stay
alongside. Disk storage takes master's `_path_to_write` (a clone never writes
over a file it shares) in place of this branch's per-store file names, so the
process tokens and restore ordering go. Dumps write master's
`derived_periods.txt` (branch-aware) and restore inputs first, then calculated
values under one later number. `apply_reform` marks every value set_input did
not store as derived and drops it, keeping master's semantics without wiping
and replaying. Tests that expected a calculated result to carry over now check
that the result is cached, since only inputs carry.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MaxGhenis added a commit to MaxGhenis/policyengine-core that referenced this pull request Oct 6, 2026
Brings in PolicyEngine#562, PolicyEngine#563, PolicyEngine#582 (input/derived storage marks, input-only
carry-over and uprating), PolicyEngine#574, PolicyEngine#584 (parameter tracing and copy guards),
PolicyEngine#578 (shared keys only while sharing) and PolicyEngine#585 (temporary storage
directories).

Conflict in holders/holder.py, resolved by keeping master's bookkeeping:
- _set keeps master's ``derived`` mark (stored with the value, and a
  derived value is never redirected to the running set_input's branch)
  and this branch's ``is_input``, which decides what the record names.
- put_in_cache keeps master's guard (a derived value never replaces a
  readable input) and passes ``is_input=False``.
- master's ``Holder._stores`` (storage ``has``) replaces this branch's
  equivalent ``_stores``.

simulation.py merged cleanly but copied the record in ``clone`` twice
(PolicyEngine#562 added the same copy); kept master's copy and added the
``_user_input_contexts`` reset next to it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MaxGhenis added a commit that referenced this pull request Oct 9, 2026
…ger period (#581)

* Count only inputs as already set when a set_input helper splits a longer period

set_input_divide_by_period and set_input_dispatch_by_period treated any value
stored for a sub-period as already set, including values the simulation had
calculated (a cached default or formula result). The same annual input then
gave different months depending on what was calculated first: a divided
input skipped calculated months and shared the rest among the others, or
failed as inconsistent after the year had been read; a dispatched input
reused a calculated month for every later month.

The helpers now read the simulation's record of inputs (_user_input_keys):
a recorded sub-period keeps its input (and, for dispatch, passes it on to
the later sub-periods, as before), and a calculated one is replaced. After
storing, they drop the variable's calculated values over overlapping periods
(the sums calculate_add caches, the twelfths calculate_divide caches) under
the input's branch and the branches it reads through, record what they store
as inputs, and evict calculate's fast cache for those periods.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Evict calculate's fast cache for the periods a set_input helper drops

calculate does not put sums or twelfths in its fast cache today, so this is
defensive: a value held there for a period whose stored value the helper
drops goes with it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Restore dumped values as inputs; test nested branches and rolling years

A dump does not record which values were inputs, and restore_simulation
stored every value with put_in_cache, so a restored simulation had an empty
input record and the helpers replaced restored inputs (review finding 1).
Restore now stores each value as a recorded input, the rule #576 applies to
dumps without an inputs.txt.

Tests: a restored month keeps its value under a yearly input (divide and
dispatch); an input on a nested branch drops the sum its parent branch
calculated; an input stored for twelve months from March is kept. The
order-independence property now creates up to two nested branches, with
calculations at each level.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Keep the property test off disk file names Windows rejects

On disk, a value calculated for a period like year:2013:2 is stored in a
file named after the period, and Windows rejects ":" in file names
(policyengine-core#526). The Windows CI jobs failed on such a read in the
on-disk examples. In disk mode the property now skips those periods.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Record which dumped values were inputs; forget an input a calculation replaces

Review round 2:

1. Restoring every dumped value as an input froze calculated values through
   a later reform. The dumper now writes, next to each variable's arrays,
   the periods whose value was an input (inputs.txt), and restore records
   exactly those; a dump without the file restores every value as an input.
   This is policyengine-core#576's dumper change, taken byte for byte so the
   two PRs merge without conflict.

2. A value stored by put_in_cache over an input (say a calculate_add sum over
   an input set for twelve months) left the input's record entry in place,
   so the helpers and apply_reform kept treating the calculated value as an
   input. Holder._set now drops the entry, in both of its forms for twelve
   months from the first of a month, when it stores a value outside
   set_input.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Test a calculated value replacing an input with put_in_cache, not calculate_add

With policyengine-core#571, calculate_add no longer caches a sum over an input
a plain read finds, so the input is kept and the test's premise (master's
calculate_add overwriting it) did not hold. Storing the calculated value with
put_in_cache replaces the input with or without #571.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Drop the record cleanup on calculated writes; evict the fast cache only for read branches

Review round 3:

- Holder._set no longer drops a record entry when a calculated value is
  stored over an input. A calculate_add at a variable's own period stores
  the unchanged input and lost its record; a clone, which shares its
  source's record until #561, deleted the source's entries; and ETERNITY
  variables keep entries under several periods. The case the cleanup was
  for, calculate_add caching a sum over an input, is what #571 stops.
- Master now gives a storage that shares nothing a frozenset for _shared
  (#578). The drop step deleted from it directly; it now leaves that to
  InMemoryStorage._stop_sharing_dropped_keys.
- The helpers evicted calculate's fast cache for a sub-period stored under
  any branch name, including one the simulation does not read. They now
  evict only for branches it reads, as #576's holder-write property
  requires.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3.32.12 gives every holder's storage an empty set: about 1.3 MB more per policyengine-us simulation

1 participant