Skip to content

Pin the depletion-cut outputs to the generator's own code, not all of src/ - #514

Merged
MaxGhenis merged 6 commits into
masterfrom
scope-depletion-artifact-code-check
Oct 4, 2026
Merged

MaxGhenis merged 6 commits into
masterfrom
scope-depletion-artifact-code-check

Conversation

@MaxGhenis

@MaxGhenis MaxGhenis commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor

What this fixes

tests/bridge/test_pe_us_depletion_cut_artifact.py::test__artifact__then_the_recorded_commit_is_this_code came in with #506. It ran git diff --quiet <recorded commit> HEAD -- src scripts, so it failed as soon as any later PR changed anything under src/ or scripts/. #509 hit it on its first CI run after #506 merged: AssertionError: src/ or scripts/ changed since the outputs were generated at 5ac5ab53…. #511, #512 and every future code PR would fail the same way.

The change

The check now pins only the generator's own code:

  • the generator script, scripts/pe_us_depletion_cut_sample_households.py;
  • every script it imports (today scripts/pe_us_minimum_benefit_sample_households.py);
  • the populace_dynamics modules any of those scripts import directly.

generator_sources() reads these from the scripts' import statements, so the set follows the code. Today it is 13 files. None of #509, #511 or #512 touches any of them.

What the check doesn't cover. None of these fail this test, and the docstring lists them:

  • a change to a module the scripts don't import by name (for example something bridge/policyengine_us.py itself imports);
  • the package __init__.py files Python runs on the way to a dotted import (ss/__init__.py for populace_dynamics.ss.params), which today only re-export names;
  • data files that imported code reads (for example data/external/ssa_cola_history.json, read by estimates/parameters.py). The old check over src/ and scripts/ didn't pin these either.

The oracle test, test_pe_us_depletion_cut_oracle.py, still reruns the households against policyengine-us when that interpreter is available.

Invariants, and where they are tested

Twelve new tests, on an invented repository built in tmp_path. In it, the generator imports a helper script, a module via from ... import, a name that isn't a module, and (inside a function) a dotted import populace_dynamics.deep.inner. The helper imports a second script, which imports a module no other file names.

  • The pinned set is exactly the named code. Each import form resolves, a name that isn't a module resolves to nothing, and the never-imported unrelated.py and deep/__init__.py stay out.
  • With no change since the recorded commit, the generator is unchanged.
  • New and edited code the generator never imports leaves it unchanged. That covers a new module, an edited unrelated module, an edited unimported package __init__, and a new script.
  • A change to any pinned file counts as changed (eight parametrized cases):
    • the generator;
    • the helper;
    • a direct import and its package __init__;
    • a module only the helper imports;
    • the second-level script and its own import, which is the recursion;
    • the dotted import inside a function.
  • On the real repository, the pinned set is sorted and unique, every path exists, and it includes the two scripts and both bridge modules.

Mutation check. I ran four deliberate breaks of the resolver, and the new tests fail under each one:

  • recursion removed (pending.append deleted): 2 fail;
  • only top-level statements walked: 2 fail;
  • from package import submodule not resolved: 4 fail;
  • no package __init__ fallback: 2 fail.

The earlier one-level fixture passed under the recursion break, which is why the second script was added.

Tests

  • tests/bridge/test_pe_us_depletion_cut_artifact.py: 39 passed (27 before, 12 new).
  • Tier counts recounted with pytest --collect-only -m <tier>: unit 5,828 → 5,840; total 11,252. Every tier matches tests/tier_counts.json.
  • Black 26.5.1 at -l 79 and ruff check pass on every changed file. These are CI's lint commands.

axiom: n/a: test scope only

🤖 Generated with Claude Code

The artifact test compared all of src/ and scripts/ with the commit the
outputs record, so every later change to unrelated code failed it. It
blocked #509 as soon as #506 merged.

The check now covers the generator script, the scripts it imports, and
the populace_dynamics modules any of those scripts import directly (13
files today, found from the import statements). Seven new tests cover
the helper on an invented repository: no change and unrelated changes
leave the generator unchanged; a change to the generator, a script it
imports, or a direct import counts.

Tier counts: unit 5,828 to 5,835.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@vercel

vercel Bot commented Oct 3, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
social-security-model Ready Ready Preview Oct 4, 2026 2:13am UTC

Request Review

Review r1 on #514 (approve, nonblocking): the helper's import was already
pinned by the generator, so the recursion test proved nothing. The
invented repository now has the helper import a module the generator
never names, a dotted import inside a function, and a name that is not a
module. One new test pins the exact expected set; two more parametrized
cases cover the helper-only module and the dotted import. The fixture's
git calls set core.hooksPath=/dev/null. The docstring now says data files
read by imported code are not pinned (nor were they before).

Tier counts: unit 5,835 to 5,838.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The earlier ruff format pass rewrote four existing asserts into a
style Black 26 rejects. CI checks with black -l 79.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
They only re-export names today; pinning them would bring back
false alarms the narrower check removes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The helper now imports a second script, and only that script imports
extra_two. With pending.append removed, the old fixture still passed
because the final loop pins each imported script's direct imports;
the new cases fail under that mutation. Unit tier 5,840.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
deep/__init__.py runs on the dotted import; it is never pinned, not
never imported. One docstring shortened to fit 79 columns.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@MaxGhenis
MaxGhenis merged commit 7a99c38 into master Oct 4, 2026
12 checks passed
@MaxGhenis

Copy link
Copy Markdown
Contributor Author

Merge record:

@MaxGhenis
MaxGhenis deleted the scope-depletion-artifact-code-check branch October 4, 2026 15:33
MaxGhenis added a commit that referenced this pull request Oct 6, 2026
…lation branch

Takes master's version of PR #506's artifact test (#514 narrowed its pin
to the generator's own code). Tier counts are master's plus this branch's
additions: unit 6,166; artifact 3,536; integration_psid 1,341;
reproduction_legacy 520; oracle_policyengine 232 (11,795).

The registered commit 3ba0d02 (tag registration-19-sa1) is unchanged;
this merge only brings the branch up to date for PR #511.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch was successfully deployed

1 active deployment
Preview — c52f81ed Deployed Oct 4, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant