Repository navigation
Let vectorial parameter nodes be copied and pickled - #584
Merged
Merged
Conversation
VectorialParameterNodeAtInstant.__getattr__ delegated every missing attribute to its recarray. copy and pickle probe a new, empty instance for __setstate__, so the lookup of self.vector recursed; deepcopy found the vector's __deepcopy__ and returned a bare recarray. A reform that calls modify_parameters deep-copies the baseline tree, cached fancy-indexing nodes included, and got a recarray in their place. Mirror #574's guard: answer the copy and pickle protocol names (and __slots__) and any lookup on an unfilled instance with AttributeError, still delegating NumPy's array protocol. Build a child read by name with the node's name and instant (it passed one of three required arguments), and leave the id-keyed enum lookup cache out of copied state. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
Author
|
Merge audit: squash-merged at reviewed head 98d37ea as a013316.
|
MaxGhenis
added a commit
that referenced
this pull request
Oct 6, 2026
Brings in #562 (carry over only inputs), #563 and #582 (uprate only from inputs), #578 (shared-key sets), #574 (per-call parameter tracing) and #584. One input record instead of two: the storages' `_inputs` (memory) and `_derived` (disk) from #562 replace this branch's `_input_keys`, so an input is a value stored without `derived=True`, as on master. Sequence numbers stay alongside. Disk storage takes master's `_path_to_write` (a clone never writes over a file it shares) in place of this branch's per-store file names, so the process tokens and restore ordering go. Dumps write master's `derived_periods.txt` (branch-aware) and restore inputs first, then calculated values under one later number. `apply_reform` marks every value set_input did not store as derived and drops it, keeping master's semantics without wiping and replaying. Tests that expected a calculated result to carry over now check that the result is cached, since only inputs carry. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MaxGhenis
added a commit
to MaxGhenis/policyengine-core
that referenced
this pull request
Oct 6, 2026
Brings in PolicyEngine#562, PolicyEngine#563, PolicyEngine#582 (input/derived storage marks, input-only carry-over and uprating), PolicyEngine#574, PolicyEngine#584 (parameter tracing and copy guards), PolicyEngine#578 (shared keys only while sharing) and PolicyEngine#585 (temporary storage directories). Conflict in holders/holder.py, resolved by keeping master's bookkeeping: - _set keeps master's ``derived`` mark (stored with the value, and a derived value is never redirected to the running set_input's branch) and this branch's ``is_input``, which decides what the record names. - put_in_cache keeps master's guard (a derived value never replaces a readable input) and passes ``is_input=False``. - master's ``Holder._stores`` (storage ``has``) replaces this branch's equivalent ``_stores``. simulation.py merged cleanly but copied the record in ``clone`` twice (PolicyEngine#562 added the same copy); kept master's copy and added the ``_user_input_contexts`` reset next to it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
VectorialParameterNodeAtInstant.__getattr__hands every attribute the node lacks toself.vector, anumpy.recarray. On master, withv = parameters("2015-01-01").rate.single[np.array(["owner", "tenant"])]built fromtests/core/parameters_fancy_indexing:copy.copy(v)andpickle.loads(pickle.dumps(v))raiseRecursionError. Both create the copy with__new__and then look up__setstate__on it before restoring its__dict__. The empty instance has novector, so__getattr__readingself.vectorcalls itself without end.copy.deepcopy(v)returns a barenumpy.recarray. Deepcopy looks__deepcopy__up on the instance, and the vector's answered it.node.ownerandnode["owner"]raiseTypeErrorwhenownerhas children, as onparameters("2015-01-01").rate[np.array(["single", "couple"])].__getattr__built the child node asVectorialParameterNodeAtInstant(result), but__init__needs all three ofname,vectorandinstant_str.ParameterNodeAtInstant.__getitem__caches its vectorial node in_vectorial_node, andParameterNode._at_instant_cachekeeps those nodes.Reform.modify_parametersdeep-copiesself.baseline.parameters, caches included. So once the baseline had fancy-indexed a node at an instant, a reform that calledmodify_parametersheld arecarraywhere the node should be. On master, fancy indexing it in the country template (general[np.array(["age_of_retirement", "age_of_majority", "age_of_retirement"])]at 2017-01-01) raisedValueError: duplicate field of name 'age_of_retirement'. With distinct keys it returned a field selection of the recarray instead of the values.Fix
This mirrors the guard that #574 adds to the tracing wrappers.
_COPY_PROTOCOLand_wrappedhave the same names, the same nine protocol names (as of #574's head 43c2b5b) and the same behaviour.__getattr__raisesAttributeErrorfor the copy and pickle protocol names:__copy__,__deepcopy__,__getstate__,__setstate__,__reduce__,__reduce_ex__,__getnewargs__,__getnewargs_ex__and__slots__. It does the same for any lookup on an instance whosevectoris not yet set. Every other name is still delegated, special names included. NumPy reads__array_struct__through it (probed), sonp.asarray(v)returns the same array as on master: the same dtype, shape and bytes._nameand_instant_str, as__getitem__already does for array keys. The tracer files reads from a vectorial node under that name too (get_traced_child).__getstate__leaves out_enum_lut_cache. That cache is keyed byid(enum), and in another process the same id can belong to a different enum, so a copy rebuilds it on first use.__init__and round trips.__init__has no optional arguments. Copy and pickle never call it: they rebuild the node with__new__and restore its__dict__. Every argument it needs (vector,_name,_instant_str) therefore comes back from the state. The one place that called it with fewer arguments was the__getattr__child above.__slots__. Protocols 0 and 1 pickle throughcopyreg._reduce_ex, which readsgetattr(self, "__slots__", None)only when the class has no__getstate__of its own. Read in this session from the 3.11 and 3.14 sources. Because this class now defines__getstate__, it cannot reach that check. The__slots__entry is a second guard that keeps the set identical to #574's. A mutation run confirms both halves: removing__getstate__and the__slots__entry makes a node holding a slotted vector fail at protocols 0 and 1, while removing either alone does not.Overlap with open PRs
__getattr__. It blocks onlyvectorand__deepcopy__, and it keeps the one-argument child constructor. This PR covers everything Let simulations be deep-copied and unpickled #567's hunk does. Whichever of the two merges second should resolve that hunk by taking this PR's version. Let simulations be deep-copied and unpickled #567's copy and pickle tests pass on Let simulations be deep-copied and unpickled #567's head with this PR's version of the file in place of its own (see Tests)._COPY_PROTOCOLand_wrappedintracers/tracing_parameter_node_at_instant.py, so master holds two copies once this lands. A follow-up should keep one.parameter_node.pyalready imports from tracers, and the tracing module importspolicyengine_core.parameters, so the follow-up has to pick a direction that leaves no import cycle and check it in fresh interpreters. On top of current master (27e3003) this commit applies cleanly, the two sets are equal, and the tests below plus Trace parameter reads per call and stop test reruns leaking module fixtures #574's tracing-isolation tests pass (260 passed).Invariants
These are checked for every input by
test_vectorial_parameter_node_copy_property.py(300 Hypothesis examples). Each example is a random path throughrate.<family>.<housing>.<zone>. Each level is taken by name (attribute or item) or by keys (strings, anEnumArrayor enum items), and the node is copied partway along, with or without lookup caches filled first.copy.copy,copy.deepcopyand pickle protocols 0 to 5, the copy is aVectorialParameterNodeAtInstantwith the same_name,_instant_strand vector dtype and bytes.RATE).np.asarray(copy)has the dtype, shape and bytes ofnp.asarray(original), whose bytes are the vector's.The example tests pin these and the reform path:
test_vectorial_parameter_node_copy.py, 65 tests.Tests
New:
tests/core/parameters/test_vectorial_parameter_node_copy.py,tests/core/parameters/test_vectorial_parameter_node_copy_property.py(behindpytest.importorskip("hypothesis")for the smoke job), and helpers intests/fixtures/vectorial_parameter_nodes.py.On master: 58 of the 65 example and property tests fail, with
RecursionError, arecarraywhere a node should be,TypeErrorfrom the child constructor, orValueErrorfrom the reform. The 7 that pass are deliberate guards: names a recarray also lacks, methodsobjectalready provides, and NumPy parity.Mutation checks, each piece removed in turn:
__dict__read (back togetattr)__getstate____getstate__and__slots____slots__onlyPython 3.11: these tests plus
tests/core/parameters_fancy_indexing,tests/core/parameters,test_parameters.py,test_reforms.py,test_reform_parameter_isolation.pyandtest_tracers.pygive 210 passed. Also run on Python 3.14 (free-threaded build).Full suite on Python 3.11: country-template YAML tests 39 passed, then
pytest testsgave 1231 passed, 4 skipped and 1 xfailed.Let simulations be deep-copied and unpickled #567's
test_simulation_copy_pickle.pyandtest_simulation_copy_pickle_property.py, run on Let simulations be deep-copied and unpickled #567's head (b6c68eb) with this PR's vectorial file in place of Let simulations be deep-copied and unpickled #567's: 40 passed.ruff format --check .passes.Review
Independent review by GPT-6.1 Sol (Subfleet
20261003-101534-core584-review): APPROVE at 98d37ea, with two non-blocking notes. It re-ran the master comparison (58 failures, 7 passes) and the mutation table (11, 4, 2, 8, 10, 0), and confirmed NumPy parity and the choice to keep the parent's name._COPY_PROTOCOLsays copying looks these names up on an instance;copy.copyreads__copy__from the class. Guarding the name is harmless. The wording goes to the follow-up that merges the two copies.parameter_node.pyimports from tracers) and is corrected above.axiom: n/a: core infrastructure (copying parameter nodes), no policy change
🤖 Generated with Claude Code