Leave untested changed sources out of Quick Feedback coverage

[MaxGhenis]

Summary

Quick Feedback measures coverage only for changed source files whose tests actually run in that job. A changed .py file is left out of coverage when it maps to test targets and none of them are in the run, because the job deferred them or narrowed them away. Files with no mapped tests are measured as before.

Why

limit_test_paths drops a changed file's tests in two ways:

• Narrowing: when a change maps to more than SELECTIVE_TEST_MAX_TARGETS targets or SELECTIVE_TEST_MAX_FILES test files, it runs only the directly changed tests.
• Deferral: it always defers slow directories (gov/ssa, household, reform, microsimulation, contrib/congress) to the full suite.

run_tests still passed every changed non-test .py file to coverage run --include. The report then counted those files' formula bodies as uncovered, and Codecov scored the PR's project coverage from that report alone.

Observed:

• Replace dead Northeast and California local reference URLs #9766 narrowed from "8 targets / ~746 test files" to gov/hud/integration.yaml. Coverage over its 26 changed variable files was 69%, so codecov/project failed (-30.91%) while codecov/patch was 100%. Log: run 37144819774, job 111267711769.
• Replace dead federal and miscellaneous reference URLs #9764 narrowed to its changed guard test. Coverage was 51%, so codecov/project failed at 54.27% (-45.73%).
• Count only taxable Social Security and gross gains in the qualifying-relative income test #9691 merged with codecov/project failing (96.96%, -3.04%).

Change

• New SelectiveTestRunner.get_untested_sources(source_files, test_paths) returns the source files whose map_files_to_tests({file}) is non-empty and shares no target with the run.
• run_tests drops those files from --include and lists them. When none are left, it runs without coverage, so coverage xml writes nothing, has_coverage=false skips the Codecov upload, and Codecov posts no project status. That is already what happens today for changes with no Python source and for runs deferred entirely.

The whole selector is unchanged when no targets are dropped.

Replaying the real changed-file lists through the new code, with the CI limits of 25 targets and 100 files:

PR	changed .py sources	targets run	coverage commands
#9766	26	1	0 (all 26 left out)
#9764	15	1	0
#9769	59	2	0

Invariant

A changed source is in the coverage --include list exactly when one of these holds:

• it maps to no test targets, or
• at least one of its mapped targets is in the run.

Four tests in policyengine_us/tests/core/test_run_selective_tests.py:

• a narrowed run issues no coverage command;
• a deferred gov/ssa source drops out while the gov/irs source whose tests run stays in;
• a source with running tests and an unmapped source both stay in, as today;
• a direct check of get_untested_sources.

On upstream/main's selector, the first, second and fourth fail on their assertions and the third passes. Locally the file has 18 passed.

Microsimulation impact

None. The PR changes only CI test selection (policyengine_us/tests/run_selective_tests.py and its tests). No variable, parameter or dataset changes, so no model output can change. Its effect is on which changed files the Quick Feedback job counts for coverage.

axiom: n/a: CI test-selection infrastructure

🤖 Generated with Claude Code

[MaxGhenis]

Merged by the US + core hub on its merge gates. Audit:

• Head: 31d73fac0b24b1eb3c0ab9677029ed8e91d5e6ff.
• Independent review: ~/reviews/dead-reference-urls-2026-09-30/outputs/review-9778-hard.md approved head e905bdb4db with no blockers and one wording nit. The only commit since is a merge of main (31d73fac0b). The hub confirmed the PR's own diff is identical before and after that merge (3 files, +151/−2), and that it merges cleanly with current main.
• CI: every check passed at this head.
• Impact: CI test selection only, so no model output changes. The hub added that statement to the body.
• Decisions: none.