Pin the October 5 audits' leaderboard impact, proposals and answers to the passes' inputs

reference_audit/2026-10-05-louisiana/README.md

@@ -90,7 +90,7 @@ If Max rules instead that the convention takes the amount published before the f
 ## Method
 
 1. **Sweep.** `scripts/sweep_la_standard_deduction.py` rebuilds every household with `Scenario.to_pe_household` and computes every output on policyengine-us 2.15.17 with `latest_final`. Its baseline reproduces all 1,928 scored references. It then reruns every household with the Louisiana 2026 amounts set to each candidate, joint at 200%. Outputs: `verification/sweep_la_standard_deduction.csv`, `verification/sweep_la_households.json` (each Louisiana household's AGI, deduction, taxable income and tax under each candidate) and the log.
-2. **Score.** `scripts/leaderboard_impact.py` copies the frozen run to scratch directories and never writes the snapshot. For each option it rewrites the two reference values (and the copy's reference digest) or appends the two outputs to the exclusion record. It scores each copy with `python -m policybench.cli analyze`, the command the freeze runs. The unchanged copy reproduces the published payload's scoring field for field. Outputs: `verification/leaderboard_impact.json`, `leaderboard_impact_models.csv` and the log.
+2. **Score.** `scripts/leaderboard_impact.py` copies the frozen run to scratch directories and never writes the snapshot. It reads the run and the sweep's values from git at the pass's commits, not from the working tree ([Pinned inputs](#pinned-inputs)). For each option it rewrites the two reference values (and the copy's reference digest) or appends the two outputs to the exclusion record. It scores each copy with `python -m policybench.cli analyze`, the command the freeze runs. The unchanged copy reproduces the published payload's scoring field for field. Outputs: `verification/leaderboard_impact.json`, `leaderboard_impact_models.csv` and the log, which is its standard output.
 3. **Module check.** `scripts/verify_convention_module.py` recomputes all 1,984 outputs with `fixes/latest_final_la.py` and compares each with the sweep's `ldr_published` column; they agree on every output. policyengine-us applies a reform set twice (before uprating and again with its structural reforms), so the module's guard accepts the engine's value or its own.
 4. **Tests.** `tests/test_la_standard_deduction_audit.py` checks these records without loading the engine:
    - the baseline matches every scored reference, and the candidates move only the two outputs;
@@ -103,3 +103,39 @@ If Max rules instead that the convention takes the amount published before the f
 Louisiana's withholding proxy (`la_withheld_income_tax`) subtracts the federal standard deduction, so the Louisiana amount does not reach the federal SALT deduction. The three other Louisiana households (038, 057, 074) have Louisiana AGI below the deduction under every candidate.
 
 Engine: `results/local/adds202609/triage/.venv-pe21517` (policyengine-us 2.15.17).
+
+## Pinned inputs
+
+This audit's passes read release dashboard-data-20260930's run, as #187 committed it at `8b4c0ca1`, and this directory's inputs as #202 merged them at `9ce4ade8`. Every committed file the scripts below write regenerates byte for byte from those inputs. Release dashboard-data-20261006 (#202) then excluded eight more outputs and rewrote the run's payload and exclusion record. On that working tree the unpinned impact script exited 0 after rewriting both evidence files. Every option scored eight fewer outputs than in the pass (1,920 instead of 1,928, and 1,918 instead of 1,926 for `exclude`), and the top-five lists and rank changes moved.
+
+Each script below now takes every file it reads from git with `git show <commit>:<path>`. It stops before scoring or writing anything unless each file's sha256 matches its pin, and it never reads the working tree. `<run>` is `paper/snapshot/20260501/runs/us_full_run_20260612_policyengine_4_16_1_populace`.
+
+### `scripts/leaderboard_impact.py`
+
+| Input | Commit | sha256 |
+|---|---|---|
+| `<run>/data.json.gz` | `8b4c0ca1` | `1e029aaa87d1dfbd2ceee88419599a919dd7c9d4aba78a308ec48d008d54ae18` |
+| `<run>/predictions.csv.gz` | `8b4c0ca1` | `ca2c4c48c7fd3e680c9c61a7380ecfcb60ce95f913c5c363762e023949d8ad12` |
+| `<run>/reference_outputs.csv` | `8b4c0ca1` | `e8bbba8fd3e90f78e7c0e83df06227bc1c94563e92f7405fe12be853a30b2466` |
+| `<run>/reference_outputs.csv.meta.json` | `8b4c0ca1` | `816fef53c452d8520a321bc12bc29b28da1e7956a06818e5ec13d7fc7b371a4b` |
+| `<run>/reference_exclusions.json` | `8b4c0ca1` | `bf4e6a249aeee01d0b71f5834ef7a35c4bab2266d2c59d0e81b12a0da44281c2` |
+| `<run>/scenarios.csv` | `8b4c0ca1` | `71b16212f0c0b3e5d13d8694ce57e362c23248665806c4d6dea7b23ef472858a` |
+| `<run>/scenarios.csv.meta.json` | `8b4c0ca1` | `03a66e90b86e9bd0cc77f27520784bd581777762f749675dc716e24c1b8eaebb` |
+| `verification/sweep_la_standard_deduction.csv` | `9ce4ade8` | `406bceb71581dfe676722740bd588896e2b72ef1baa74b4975d80b1ca851a8e0` |
+
+It stages each file under `<scratch>/pass_inputs/<path>` and reads only that copy. `verification/sweep_la_standard_deduction.csv` is the sweep's output (step 1); the score step reads the candidates' values from it.
+
+### Reproduce
+
+The scoring code is the checkout's. The impact script's check on its unchanged copy stops it if that code no longer scores the pinned run as the pinned payload records. The scripts run from any checkout that has both commits in its history; a shallow clone needs `git fetch --unshallow` first:
+
+```bash
+PYTHONPATH=$PWD uv run python reference_audit/2026-10-05-louisiana/scripts/leaderboard_impact.py --scratch <dir outside the repository> \
+  > reference_audit/2026-10-05-louisiana/verification/leaderboard_impact.log
+```
+
+`--out-dir <dir>` writes the files elsewhere. `tests/test_reference_audit_pins.py` checks the pins, that a changed byte in any input is refused, and that no script reads the working tree. Its slow test regenerates `leaderboard_impact.json` and `leaderboard_impact_models.csv` from this checkout, whose run #202 rewrote, and requires the committed files, and the log, byte for byte. The slow test takes one to two minutes per audit, so CI deselects it:
+
+```bash
+OPENBLAS_NUM_THREADS=1 uv run pytest -m slow tests/test_reference_audit_pins.py
+```

reference_audit/2026-10-05-louisiana/scripts/leaderboard_impact.py

@@ -1,6 +1,17 @@
 """Score the frozen run under each option for the two Louisiana state income tax outputs.
 
-Copies the frozen run (predictions, references, scenarios, exclusions) to scratch
+Reads the pass's inputs from git, never from the working tree: the frozen run
+(payload, predictions, references, scenarios, exclusions) as PASS_COMMIT (release
+dashboard-data-20260930, #187) holds it, and the sweep's candidate values as
+AUDIT_COMMIT (#202, which merged this audit) holds them. Later releases rewrite the
+working tree's run: dashboard-data-20261006 (#202) excluded eight more outputs, and
+scoring that run would silently regenerate different evidence (1,920 scored outputs
+instead of the pass's 1,928). Each input in INPUTS is staged under
+``<scratch>/pass_inputs`` and must match its pinned sha256, or the script stops before
+scoring anything. The scoring code is the checkout's; the ``keep`` check below stops
+the script if that code no longer scores the pinned run as the pinned payload records.
+
+Copies the staged run (predictions, references, scenarios, exclusions) to scratch
 directories and never writes the snapshot. Each option is one copy:
 
 ``keep``
@@ -19,6 +30,12 @@
 
   PYTHONPATH=<checkout> <policybench venv>/bin/python \\
     reference_audit/2026-10-05-louisiana/scripts/leaderboard_impact.py --scratch <dir>
+
+Its standard output is ``verification/leaderboard_impact.log``. The checkout needs
+PASS_COMMIT and AUDIT_COMMIT in its history (a shallow clone needs
+``git fetch --unshallow``). ``--out-dir`` writes the verification files elsewhere;
+tests/test_reference_audit_pins.py regenerates them that way and requires the
+committed ones, and the log, byte for byte.
 """
 
 from __future__ import annotations
@@ -37,19 +54,59 @@
 
 HERE = Path(__file__).resolve().parents[1]
 ROOT = HERE.parents[1]
-RUN = (
-    ROOT
-    / "paper/snapshot/20260501/runs/us_full_run_20260612_policyengine_4_16_1_populace"
-)
-SWEEP = HERE / "verification/sweep_la_standard_deduction.csv"
 OUT_DIR = HERE / "verification"
+# The pass's inputs, pinned by commit and sha256.
+PASS_COMMIT = "8b4c0ca146bb6f66deba6ce24009d49d70d92df2"
+AUDIT_COMMIT = "9ce4ade8382962a9134860c23f56d92509b5e57f"
+RUN_PATH = (
+    "paper/snapshot/20260501/runs/us_full_run_20260612_policyengine_4_16_1_populace"
+)
+AUDIT_PATH = "reference_audit/2026-10-05-louisiana"
+SWEEP_PATH = f"{AUDIT_PATH}/verification/sweep_la_standard_deduction.csv"
+# Every file the script reads from the repository: path -> (commit, sha256).
+# pass_inputs() stages each at the same path under its target.
+INPUTS = {
+    f"{RUN_PATH}/data.json.gz": (
+        PASS_COMMIT,
+        "1e029aaa87d1dfbd2ceee88419599a919dd7c9d4aba78a308ec48d008d54ae18",
+    ),
+    f"{RUN_PATH}/predictions.csv.gz": (
+        PASS_COMMIT,
+        "ca2c4c48c7fd3e680c9c61a7380ecfcb60ce95f913c5c363762e023949d8ad12",
+    ),
+    f"{RUN_PATH}/reference_outputs.csv": (
+        PASS_COMMIT,
+        "e8bbba8fd3e90f78e7c0e83df06227bc1c94563e92f7405fe12be853a30b2466",
+    ),
+    f"{RUN_PATH}/reference_outputs.csv.meta.json": (
+        PASS_COMMIT,
+        "816fef53c452d8520a321bc12bc29b28da1e7956a06818e5ec13d7fc7b371a4b",
+    ),
+    f"{RUN_PATH}/reference_exclusions.json": (
+        PASS_COMMIT,
+        "bf4e6a249aeee01d0b71f5834ef7a35c4bab2266d2c59d0e81b12a0da44281c2",
+    ),
+    f"{RUN_PATH}/scenarios.csv": (
+        PASS_COMMIT,
+        "71b16212f0c0b3e5d13d8694ce57e362c23248665806c4d6dea7b23ef472858a",
+    ),
+    f"{RUN_PATH}/scenarios.csv.meta.json": (
+        PASS_COMMIT,
+        "03a66e90b86e9bd0cc77f27520784bd581777762f749675dc716e24c1b8eaebb",
+    ),
+    SWEEP_PATH: (
+        AUDIT_COMMIT,
+        "406bceb71581dfe676722740bd588896e2b72ef1baa74b4975d80b1ca851a8e0",
+    ),
+}
 OUTPUTS = (
     ("scenario_051", "state_income_tax_before_refundable_credits"),
     ("scenario_077", "state_income_tax_before_refundable_credits"),
 )
 REGENERATE = ("ldr_published", "ldr_official", "hold_2025")
 OPTIONS = ("keep",) + REGENERATE + ("exclude",)
 TOLERANCE = 1.0
+# Copied into each copy and scored; data.json.gz is only compared against.
 RUN_FILES = (
     "predictions.csv.gz",
     "reference_outputs.csv",
@@ -71,12 +128,71 @@
 EXACT_PAYLOAD_KEYS = ("programStats", "heatmap", "globalWeights", "failureModes")
 
 
+def overlaps(scratch: Path, root: Path) -> bool:
+    """Whether ``scratch`` and ``root`` are one directory or one holds the other.
+
+    Judged by filesystem identity (device and inode), not spelling, so case on a
+    case-insensitive filesystem or a link cannot hide an overlap. ``scratch`` need
+    not exist yet; its nearest existing ancestors decide.
+    """
+
+    def identities(path: Path) -> set[tuple[int, int]]:
+        found = set()
+        for part in (path, *path.parents):
+            try:
+                status = part.stat()
+            except OSError:
+                continue
+            found.add((status.st_dev, status.st_ino))
+        return found
+
+    root_status = root.stat()
+    if (root_status.st_dev, root_status.st_ino) in identities(scratch):
+        return True
+    try:
+        scratch_status = scratch.stat()
+    except OSError:
+        return False
+    return (scratch_status.st_dev, scratch_status.st_ino) in identities(root)
+
+
 def sha256(path: Path) -> str:
     return hashlib.sha256(path.read_bytes()).hexdigest()
 
 
-def candidate_values() -> dict[str, dict[tuple[str, str], float]]:
-    sweep = pd.read_csv(SWEEP).set_index(["scenario_id", "variable"])
+def git_input(commit: str, path: str, pinned: str, target: Path) -> Path:
+    """Write ``path`` as ``commit`` holds it to ``target``, refusing any other bytes."""
+    result = subprocess.run(
+        ["git", "-C", str(ROOT), "show", f"{commit}:{path}"],
+        capture_output=True,
+    )
+    if result.returncode:
+        raise SystemExit(
+            f"cannot read {path} at {commit[:12]}; fetch full history "
+            f"(git fetch --unshallow): {result.stderr.decode().strip()}"
+        )
+    digest = hashlib.sha256(result.stdout).hexdigest()
+    if digest != pinned:
+        raise SystemExit(
+            f"{commit[:12]}:{path} has sha256 {digest}, not the pinned {pinned}"
+        )
+    target.parent.mkdir(parents=True, exist_ok=True)
+    target.write_bytes(result.stdout)
+    return target
+
+
+def pass_inputs(target: Path) -> Path:
+    """Stage every input in INPUTS at its repository path under ``target``."""
+    if target.exists():
+        shutil.rmtree(target)
+    for path, (commit, pinned) in INPUTS.items():
+        git_input(commit, path, pinned, target / path)
+    return target
+
+
+def candidate_values(path: Path) -> dict[str, dict[tuple[str, str], float]]:
+    """Each candidate's two reference values, from the sweep CSV at ``path``."""
+    sweep = pd.read_csv(path).set_index(["scenario_id", "variable"])
     values = {}
     for candidate in ("engine",) + REGENERATE:
         values[candidate] = {key: float(sweep.loc[key, candidate]) for key in OUTPUTS}
@@ -106,12 +222,13 @@ def exclusion_entries(values: dict) -> list[dict]:
     return entries
 
 
-def stage(target: Path, option: str, values: dict) -> Path:
+def stage(source: Path, target: Path, option: str, values: dict) -> Path:
+    """Copy the staged run at ``source`` to ``target`` and apply ``option``."""
     if target.exists():
         shutil.rmtree(target)
     target.mkdir(parents=True)
     for name in RUN_FILES:
-        shutil.copy2(RUN / name, target / name)
+        shutil.copy2(source / name, target / name)
     if option in REGENERATE:
         # Rewrite only the two rows' value field, so every other byte stays as frozen,
         # and record the new digest in the copy's sidecar, as a regeneration would.
@@ -169,8 +286,9 @@ def analyze(run_dir: Path) -> dict:
     return payload["countries"]["us"]
 
 
-def check_reproduces_published(base: dict) -> None:
-    published = json.loads(gzip.decompress((RUN / "data.json.gz").read_bytes()))
+def check_reproduces_published(base: dict, source: Path) -> None:
+    """Stop unless ``base`` scores as the staged ``source/data.json.gz`` does."""
+    published = json.loads(gzip.decompress((source / "data.json.gz").read_bytes()))
     for key in EXACT_PAYLOAD_KEYS:
         if published[key] != base[key]:
             raise SystemExit(f"unchanged copy does not reproduce published {key}")
@@ -207,9 +325,9 @@ def compare(base: dict, option: dict) -> pd.DataFrame:
     return rows.sort_values("exact_rank_published")
 
 
-def exact_models(values: dict, option: str) -> dict[str, list[str]]:
+def exact_models(values: dict, option: str, source: Path) -> dict[str, list[str]]:
     """Models within the $1 tolerance of each output's reference under an option."""
-    published = json.loads(gzip.decompress((RUN / "data.json.gz").read_bytes()))
+    published = json.loads(gzip.decompress((source / "data.json.gz").read_bytes()))
     predictions = published["scenarioPredictions"]
     result = {}
     for sid, variable in OUTPUTS:
@@ -230,17 +348,29 @@ def exact_models(values: dict, option: str) -> dict[str, list[str]]:
 def main() -> None:
     parser = argparse.ArgumentParser()
     parser.add_argument("--scratch", required=True)
+    parser.add_argument(
+        "--out-dir",
+        default=str(OUT_DIR),
+        help="where to write the verification files (default: %(default)s)",
+    )
     args = parser.parse_args()
-    scratch = Path(args.scratch)
-    values = candidate_values()
+    scratch = Path(args.scratch).resolve()
+    out_dir = Path(args.out_dir)
+    # stage() and pass_inputs() delete and rewrite scratch/<name>, so scratch and
+    # the repository must not overlap: neither may lie inside the other.
+    if overlaps(scratch, ROOT):
+        parser.error(f"--scratch must not overlap the repository ({ROOT})")
+    inputs = pass_inputs(scratch / "pass_inputs")
+    source = inputs / RUN_PATH
+    values = candidate_values(inputs / SWEEP_PATH)
 
     payloads = {}
     for option in OPTIONS:
-        payloads[option] = analyze(stage(scratch / option, option, values))
+        payloads[option] = analyze(stage(source, scratch / option, option, values))
         if option == "keep":
-            check_reproduces_published(payloads["keep"])
+            check_reproduces_published(payloads["keep"], source)
 
-    OUT_DIR.mkdir(parents=True, exist_ok=True)
+    out_dir.mkdir(parents=True, exist_ok=True)
     summary = {
         "outputs": [f"{sid}/{variable}" for sid, variable in OUTPUTS],
         "reference_values": {
@@ -267,7 +397,7 @@ def main() -> None:
         frames.append(frame.reset_index())
         summary["options"][option] = {
             "scored_outputs": int(payloads[option]["modelStats"][0]["n"]),
-            "exact_models": exact_models(values, option),
+            "exact_models": exact_models(values, option, source),
             "exact_delta_pp": {
                 "min": float(models["exact_delta"].min()),
                 "max": float(models["exact_delta"].max()),
@@ -295,8 +425,8 @@ def main() -> None:
             "top5_exact": models.sort_values("exact_rank_option").head(5).index.tolist(),
             "top5_score": models.sort_values("score_rank_option").head(5).index.tolist(),
         }
-    pd.concat(frames).to_csv(OUT_DIR / "leaderboard_impact_models.csv", index=False)
-    (OUT_DIR / "leaderboard_impact.json").write_text(
+    pd.concat(frames).to_csv(out_dir / "leaderboard_impact_models.csv", index=False)
+    (out_dir / "leaderboard_impact.json").write_text(
         json.dumps(summary, indent=2) + "\n"
     )
     print(json.dumps(summary, indent=2))