Diane Schanzenbach's 8/27 feedback (email thread "sampling error"): the QC system made several discrete shifts across our FY2012–24 panel, which is why Bauer–Schanzenbach stayed post-pandemic. Answer with numbers, not argument.
Extend analysis/persistence_backtest.py (merged in #87) with a regime_robustness block:
- Regime-crossing test: fit variance components on FY2012–19 only; score FY2022–24 targets with the same CRPS/pinball/coverage battery. If the persistence widths transfer across the pandemic break, the "shifts did not break prediction" claim gets its strongest form.
- Post-pandemic-targets summary: the existing per-cell rows already cover targets 2022–24 — add a filtered summary table alongside the full-panel one.
- Memo + locks per the existing conventions (generated-memo equality, live input hashes, exact regeneration; see
tests/test_persistence_backtest.py).
Context: the answers went to Diane in the held 8/27 reply ("models fit only on earlier years still score well on 2022-2024, and the persistence model wins 2024 outright") — this issue makes that claim a committed artifact. Existing per-year CRPS in persistence_backtest_results.json already shows 2022/2023/2024 results from expanding windows; the new piece is the frozen-2012–19 fit.
Gate: sol review before merge (both prior rounds' reports under ~/.cache/axiom-oracles/snap-fy27/backtest-review/).
Diane Schanzenbach's 8/27 feedback (email thread "sampling error"): the QC system made several discrete shifts across our FY2012–24 panel, which is why Bauer–Schanzenbach stayed post-pandemic. Answer with numbers, not argument.
Extend
analysis/persistence_backtest.py(merged in #87) with aregime_robustnessblock:tests/test_persistence_backtest.py).Context: the answers went to Diane in the held 8/27 reply ("models fit only on earlier years still score well on 2022-2024, and the persistence model wins 2024 outright") — this issue makes that claim a committed artifact. Existing per-year CRPS in
persistence_backtest_results.jsonalready shows 2022/2023/2024 results from expanding windows; the new piece is the frozen-2012–19 fit.Gate: sol review before merge (both prior rounds' reports under ~/.cache/axiom-oracles/snap-fy27/backtest-review/).