Compile each variable file once per process

[MaxGhenis]

Fixes #607

Change

add_variables_from_file used spec.loader.exec_module, which compiles (or unmarshals from .pyc) every variable file again for each new system. It now runs a code object from a module-level cache in the new system's own module. The cache is keyed by the file's absolute path (the loader's path, as exec_module uses) and holds the SHA-256 of the source it compiled and the code object. It is filled under a lock, from the same bytes that were hashed (loader.source_to_code).

• Each system still gets its own module in sys.modules, and its own classes, functions and globals. Only the immutable code object is shared.
• Any change to a file's content is compiled again, including an edit that keeps the size and modification time.
• Only the latest version of each file is kept, for the life of the process.
• The lock is re-created in a forked child, so a fork while another thread compiles cannot deadlock the child.

Why: coverage's sys.monitoring core (coverage's default on Python 3.14) keeps every code object it has seen alive. So it held a new copy of every variable file's code for each system a process built. In policyengine-uk's test shards that grew the four workers to 15.3 of 15.6 GiB (PolicyEngine/policyengine-uk#2330). PolicyEngine/policyengine-uk#2331 works around it in the tests until this ships.

Cost

Measured on the 1,110 policyengine-uk variable files (1.2 MB), loader work only, CPU medians. The Opus review measured these on a loaded host.

Case	Before	After
A process's first system, valid .pyc files	0.044 s	0.118 s (read, hash, compile)
A process's first system, no .pyc files	0.137 s	0.118 s
Each later system, valid .pyc files	0.044 s	0.018 s (read, hash, lookup)
Each later system, no .pyc files	0.137 s	0.018 s

So a process that builds one system pays about 0.07 s more CPU at start-up, because variable files' .pyc files are no longer used. A process that builds several pays less per later system.

My earlier figure of "0.180 s instead of 0.289 s per UK system" was measured with PYTHONDONTWRITEBYTECODE=1 in a checkout with no __pycache__ for the variable files, so it is the no-.pyc case.

This doesn't change #520: the variable modules still stay in sys.modules.

Invariants (tests)

In tests/core/test_variable_code_sharing.py:

1. Two systems built from the same files have distinct variable classes, modules, formula functions and formula globals, and identical formula code objects.
2. The cached code for every loaded file equals compile() of its current source, under its absolute path (co_filename checked separately, since code equality ignores it).
3. A file loaded by a relative path compiles under its absolute path, and a later load by the absolute path gets the same code object.
4. A file edited to new content with the same size and modification time is compiled again. Systems built before the edit compute the old formula, and one built after computes the new one. Only the new version stays cached.
5. Eight threads loading one file at once compile it once and get the same code object. A loader that sleeps while compiling makes the threads overlap even under the GIL, and thread errors are collected.
6. A child forked while the lock is held gets a free lock (POSIX only).

In tests/core/test_variable_code_sharing_property.py, a Hypothesis test: for any sequence of writes and loads, every load returns the current content's code under the absolute path, the same object until the next write, and only the latest version is cached.

Review

1. GPT-6.1 Sol reviewed ef07ed6 and requested changes (4 findings); db9182b fixed them.

2. An Opus 5.5 delta review confirmed those fixes and requested 5 smaller changes:

   • co_filename spelling;
   • a vacuous thread test;
   • an eviction assertion that could not fail;
   • cost attribution;
   • fork safety.

   ba633db makes all five.

3. An Opus confirmation review approved ba633db (review-core-608-opus-final-out.md). Its nits are in f52aefa: the counter reset, the property test's finally, a fork test, and the comment wording.

4. The same reviewer approved f52aefa (review-core-608-opus-f52-out.md) and asked for one hardening: the forked child exits in a finally, as in test_data_storage_dir.py. That is 847fe46, which the reviewer said keeps the approval.

Reviews are in ~/reviews/uk-ci-test-memory-2026-10-11/review-core-608-*.

Checks

• uv run --with hypothesis pytest tests: 2,509 passed, 5 skipped, 3 xfailed (Python 3.14t, local, on master 8fdf59e).
• policyengine-core test policyengine_core/country_template/tests -c policyengine_core.country_template: 39 passed.
• Mutants checked:
  • (path, mtime, size) key: invariant 4 fails;
  • no lock: invariant 5 fails, 8 compilations;
  • caller's path spelling: invariant 3 fails;
  • no fork reset: invariant 6 fails.
• ruff format --check .: clean.

axiom: n/a: core infrastructure

🤖 Generated with Claude Code

[MaxGhenis]

Hand-off to the US + core hub (session "Cut peak memory of pe-uk CI test shards").

• Head: 847fe46, on master 8fdf59e.
• Review: three rounds.
  • GPT-6.1 Sol requested changes at ef07ed6 (4 findings), fixed in db9182b.
  • An Opus 5.5 delta review requested 5 smaller changes, fixed in ba633db. Its confirmation approved ba633db, and its nits are in f52aefa.
  • It then approved f52aefa and asked for one hardening, which is 847fe46; it said that commit keeps the approval.
  • Files: ~/reviews/uk-ci-test-memory-2026-10-11/review-core-608-out.md, -opus-delta-out.md, -opus-final-out.md and -opus-f52-out.md.
• Passed locally:
  • uv run --with hypothesis pytest tests: 2,509 passed, 5 skipped, 3 xfailed (Python 3.14t);
  • the country template YAML tests: 39 passed;
  • ruff clean.
  • Mutants: the (path, mtime, size) key, no lock, the caller's path spelling, and no fork reset are each caught by a test.
• CI: ef07ed6's run passed all 12 checks. Later runs were cancelled by newer pushes, and db9182b had passed Windows 3.13 and 3.14 before cancellation. The run at 847fe46 is queued.
• What it changes for users:
  • Later systems in a process skip recompiling or unmarshalling the variable files.
  • A process that builds a single system pays about 0.07 s more CPU at start-up, because variable .pyc files are no longer used.
  • Coverage's sysmon core stops retaining a copy per system.
  • The cost table is in the PR body.
• Downstream: policyengine-uk CI installs the latest core. After a core release with this, Stop the test shards' memory growing with every Simulation under sysmon coverage policyengine-uk#2331's tests of its loader wrapper skip with a note to remove the wrapper; its module release keeps working.
• No Max decision needed: infrastructure, no published numbers.