Skip to content

Let vectorial parameter nodes be copied and pickled - #584

Merged
MaxGhenis merged 1 commit into
masterfrom
fix-vectorial-node-copy-pickle
Oct 3, 2026
Merged

MaxGhenis merged 1 commit into
masterfrom
fix-vectorial-node-copy-pickle

Conversation

@MaxGhenis

@MaxGhenis MaxGhenis commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor

Summary

VectorialParameterNodeAtInstant.__getattr__ hands every attribute the node lacks to self.vector, a numpy.recarray. On master, with v = parameters("2015-01-01").rate.single[np.array(["owner", "tenant"])] built from tests/core/parameters_fancy_indexing:

  • copy.copy(v) and pickle.loads(pickle.dumps(v)) raise RecursionError. Both create the copy with __new__ and then look up __setstate__ on it before restoring its __dict__. The empty instance has no vector, so __getattr__ reading self.vector calls itself without end.
  • copy.deepcopy(v) returns a bare numpy.recarray. Deepcopy looks __deepcopy__ up on the instance, and the vector's answered it.
  • node.owner and node["owner"] raise TypeError when owner has children, as on parameters("2015-01-01").rate[np.array(["single", "couple"])]. __getattr__ built the child node as VectorialParameterNodeAtInstant(result), but __init__ needs all three of name, vector and instant_str.

ParameterNodeAtInstant.__getitem__ caches its vectorial node in _vectorial_node, and ParameterNode._at_instant_cache keeps those nodes. Reform.modify_parameters deep-copies self.baseline.parameters, caches included. So once the baseline had fancy-indexed a node at an instant, a reform that called modify_parameters held a recarray where the node should be. On master, fancy indexing it in the country template (general[np.array(["age_of_retirement", "age_of_majority", "age_of_retirement"])] at 2017-01-01) raised ValueError: duplicate field of name 'age_of_retirement'. With distinct keys it returned a field selection of the recarray instead of the values.

Fix

This mirrors the guard that #574 adds to the tracing wrappers. _COPY_PROTOCOL and _wrapped have the same names, the same nine protocol names (as of #574's head 43c2b5b) and the same behaviour.

  • __getattr__ raises AttributeError for the copy and pickle protocol names: __copy__, __deepcopy__, __getstate__, __setstate__, __reduce__, __reduce_ex__, __getnewargs__, __getnewargs_ex__ and __slots__. It does the same for any lookup on an instance whose vector is not yet set. Every other name is still delegated, special names included. NumPy reads __array_struct__ through it (probed), so np.asarray(v) returns the same array as on master: the same dtype, shape and bytes.
  • A child node read by name keeps the parent's _name and _instant_str, as __getitem__ already does for array keys. The tracer files reads from a vectorial node under that name too (get_traced_child).
  • __getstate__ leaves out _enum_lut_cache. That cache is keyed by id(enum), and in another process the same id can belong to a different enum, so a copy rebuilds it on first use.

__init__ and round trips. __init__ has no optional arguments. Copy and pickle never call it: they rebuild the node with __new__ and restore its __dict__. Every argument it needs (vector, _name, _instant_str) therefore comes back from the state. The one place that called it with fewer arguments was the __getattr__ child above.

__slots__. Protocols 0 and 1 pickle through copyreg._reduce_ex, which reads getattr(self, "__slots__", None) only when the class has no __getstate__ of its own. Read in this session from the 3.11 and 3.14 sources. Because this class now defines __getstate__, it cannot reach that check. The __slots__ entry is a second guard that keeps the set identical to #574's. A mutation run confirms both halves: removing __getstate__ and the __slots__ entry makes a node holding a slotted vector fail at protocols 0 and 1, while removing either alone does not.

Overlap with open PRs

Invariants

These are checked for every input by test_vectorial_parameter_node_copy_property.py (300 Hypothesis examples). Each example is a random path through rate.<family>.<housing>.<zone>. Each level is taken by name (attribute or item) or by keys (strings, an EnumArray or enum items), and the node is copied partway along, with or without lookup caches filled first.

  1. Type and state round-trip. For copy.copy, copy.deepcopy and pickle protocols 0 to 5, the copy is a VectorialParameterNodeAtInstant with the same _name, _instant_str and vector dtype and bytes.
  2. Differential. The rest of the path read from the copy equals the same read from the original, and both equal the YAML values (RATE).
  3. NumPy parity. np.asarray(copy) has the dtype, shape and bytes of np.asarray(original), whose bytes are the vector's.
  4. Ownership. A deep copy or a pickle round trip shares no memory with the original. A shallow copy shares the vector.

The example tests pin these and the reform path: test_vectorial_parameter_node_copy.py, 65 tests.

Tests

  • New: tests/core/parameters/test_vectorial_parameter_node_copy.py, tests/core/parameters/test_vectorial_parameter_node_copy_property.py (behind pytest.importorskip("hypothesis") for the smoke job), and helpers in tests/fixtures/vectorial_parameter_nodes.py.

  • On master: 58 of the 65 example and property tests fail, with RecursionError, a recarray where a node should be, TypeError from the child constructor, or ValueError from the reform. The 7 that pass are deliberate guards: names a recarray also lacks, methods object already provides, and NumPy parity.

  • Mutation checks, each piece removed in turn:

    Piece removed Tests that fail
    protocol-name guard 11
    __dict__ read (back to getattr) 4, recursion
    three-argument child 2
    __getstate__ 8, enum cache
    __getstate__ and __slots__ 10, adding pickle 0/1 with a slotted vector
    __slots__ only none (second guard, as intended)
  • Python 3.11: these tests plus tests/core/parameters_fancy_indexing, tests/core/parameters, test_parameters.py, test_reforms.py, test_reform_parameter_isolation.py and test_tracers.py give 210 passed. Also run on Python 3.14 (free-threaded build).

  • Full suite on Python 3.11: country-template YAML tests 39 passed, then pytest tests gave 1231 passed, 4 skipped and 1 xfailed.

  • Let simulations be deep-copied and unpickled #567's test_simulation_copy_pickle.py and test_simulation_copy_pickle_property.py, run on Let simulations be deep-copied and unpickled #567's head (b6c68eb) with this PR's vectorial file in place of Let simulations be deep-copied and unpickled #567's: 40 passed.

  • ruff format --check . passes.

Review

Independent review by GPT-6.1 Sol (Subfleet 20261003-101534-core584-review): APPROVE at 98d37ea, with two non-blocking notes. It re-ran the master comparison (58 failures, 7 passes) and the mutation table (11, 4, 2, 8, 10, 0), and confirmed NumPy parity and the choice to keep the parent's name.

  1. The comment above _COPY_PROTOCOL says copying looks these names up on an instance; copy.copy reads __copy__ from the class. Guarding the name is harmless. The wording goes to the follow-up that merges the two copies.
  2. An earlier version of this description said the parameters package must not import tracers. That was wrong (parameter_node.py imports from tracers) and is corrected above.

axiom: n/a: core infrastructure (copying parameter nodes), no policy change

🤖 Generated with Claude Code

VectorialParameterNodeAtInstant.__getattr__ delegated every missing
attribute to its recarray. copy and pickle probe a new, empty instance
for __setstate__, so the lookup of self.vector recursed; deepcopy found
the vector's __deepcopy__ and returned a bare recarray. A reform that
calls modify_parameters deep-copies the baseline tree, cached
fancy-indexing nodes included, and got a recarray in their place.

Mirror #574's guard: answer the copy and pickle protocol names (and
__slots__) and any lookup on an unfilled instance with AttributeError,
still delegating NumPy's array protocol. Build a child read by name with
the node's name and instant (it passed one of three required arguments),
and leave the id-keyed enum lookup cache out of copied state.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@MaxGhenis
MaxGhenis merged commit a013316 into master Oct 3, 2026
18 checks passed
@MaxGhenis
MaxGhenis deleted the fix-vectorial-node-copy-pickle branch October 3, 2026 18:32
@MaxGhenis

Copy link
Copy Markdown
Contributor Author

Merge audit: squash-merged at reviewed head 98d37ea as a013316.

MaxGhenis added a commit that referenced this pull request Oct 6, 2026
Brings in #562 (carry over only inputs), #563 and #582 (uprate only from
inputs), #578 (shared-key sets), #574 (per-call parameter tracing) and #584.

One input record instead of two: the storages' `_inputs` (memory) and
`_derived` (disk) from #562 replace this branch's `_input_keys`, so an input
is a value stored without `derived=True`, as on master. Sequence numbers stay
alongside. Disk storage takes master's `_path_to_write` (a clone never writes
over a file it shares) in place of this branch's per-store file names, so the
process tokens and restore ordering go. Dumps write master's
`derived_periods.txt` (branch-aware) and restore inputs first, then calculated
values under one later number. `apply_reform` marks every value set_input did
not store as derived and drops it, keeping master's semantics without wiping
and replaying. Tests that expected a calculated result to carry over now check
that the result is cached, since only inputs carry.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MaxGhenis added a commit to MaxGhenis/policyengine-core that referenced this pull request Oct 6, 2026
Brings in PolicyEngine#562, PolicyEngine#563, PolicyEngine#582 (input/derived storage marks, input-only
carry-over and uprating), PolicyEngine#574, PolicyEngine#584 (parameter tracing and copy guards),
PolicyEngine#578 (shared keys only while sharing) and PolicyEngine#585 (temporary storage
directories).

Conflict in holders/holder.py, resolved by keeping master's bookkeeping:
- _set keeps master's ``derived`` mark (stored with the value, and a
  derived value is never redirected to the running set_input's branch)
  and this branch's ``is_input``, which decides what the record names.
- put_in_cache keeps master's guard (a derived value never replaces a
  readable input) and passes ``is_input=False``.
- master's ``Holder._stores`` (storage ``has``) replaces this branch's
  equivalent ``_stores``.

simulation.py merged cleanly but copied the record in ``clone`` twice
(PolicyEngine#562 added the same copy); kept master's copy and added the
``_user_input_contexts`` reset next to it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant