Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,8 +28,9 @@ noise alone materially affects tier assignment.
## What v0.1 finds (FY 2024 file, all 53 jurisdictions)

- **Tier assignment is noisy.** Several states' tiers are near coin flips
under QC sampling variation (Colorado: official rate 9.97%, 0.03 points
from the 15% boundary, with a ±0.9-point sampling SD).
under QC sampling variation (Colorado: official FY 2024 rate 9.97%, 0.03
points below the 10% rate where the 15% cost share begins, with a
±0.9-point sampling SD; its FY 2025 rate, 10.09%, is above it).
- **Simplification options carry large expected values** where they can move
a state across a boundary: about $609M/yr in combined expected state
cost-share reduction nationally at 50% category-suppression effectiveness
Expand Down
2 changes: 1 addition & 1 deletion analysis/ENGINE_COMPARISON.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ Stage concepts are pinned per state in the suite reports (Colorado exposes `snap

## Formula-benefit divergence catalog

`|RAWBEN - BENFIX|` equals the recorded error amount for every official-universe case in the seven states (asserted at build), and nationally for 99.997% of weighted FY2024 cases (n=44,800). The Minimodel's formula benefit taken as the deviation anchor — `|RAWBEN - FSBEN|` — matches the recorded error for only 83.64% weighted, because the formula recomputation conflates recorded allotment adjustments with error. The catalog partitions each state's `FSBEN != BENFIX` cases three ways:
`|RAWBEN - BENFIX|` equals the recorded error amount for every official-universe case in the seven states (asserted at build), and nationally for 99.997% of weighted FY2024 cases (n=44,800). The formula benefit (FSBEN) taken as the deviation anchor — `|RAWBEN - FSBEN|` — matches the recorded error for only 83.64% weighted, because the formula recomputation conflates recorded allotment adjustments with error. The catalog partitions each state's `FSBEN != BENFIX` cases three ways:

1. **Coded allotment adjustment** — `ALLADJ` records a prorated (2) or other (3) adjustment; the recorded allotment is not the full-month formula amount by design.
2. **Error-correction arithmetic** — no coded adjustment and `AMTERR > 0`: the reviewer's corrected allotment differs from the full recomputation (about two-thirds within $2, rounding-scale).
Expand Down
12 changes: 6 additions & 6 deletions analysis/engine_comparison.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,17 +9,17 @@
each state's FY2024 QC public-use reviews through the Axiom RuleSpec
SNAP composition and compare six recorded stages — gross income,
standard deduction, excess-shelter deduction, net income, maximum
allotment, and the benefit — at zero tolerance against the file's own
Minimodel-recomputed values (``FSBEN`` and companions). This module
allotment, and the benefit — at zero tolerance against the file's
computed values (``FSBEN`` and companions). This module
re-derives the per-state parity counts from those reports and asserts
the exactness the app displays (every compared case matches at every
stage; excluded cases are enumerated program-structure classes).

* **The formula-benefit divergence catalog** — the public file records
three benefit anchors per case: ``RAWBEN`` (the allotment the state
issued), ``BENFIX`` (the allotment corrected for the reviewer's
findings), and ``FSBEN`` (the Minimodel's full-formula recomputation
from the edited inputs). ``|RAWBEN - BENFIX|`` equals the recorded
findings), and ``FSBEN`` (the formula benefit Mathematica computes for
USDA from the edited inputs). ``|RAWBEN - BENFIX|`` equals the recorded
error amount ``AMTERR`` for every official-universe case in the seven
states; ``FSBEN`` differs from ``BENFIX`` for a catalogued minority,
so a formula recomputation alone conflates recorded allotment
Expand Down Expand Up @@ -495,8 +495,8 @@ def render_report(artifact: dict[str, Any]) -> str:
"official-universe case in the seven states (asserted at "
"build), and nationally for "
f"{100 * national['concordance_weighted']['benfix']:.3f}% of "
f"weighted FY2024 cases (n={national['n']:,}). The Minimodel's "
"formula benefit taken as the deviation anchor — "
f"weighted FY2024 cases (n={national['n']:,}). The formula "
"benefit (FSBEN) taken as the deviation anchor — "
"`|RAWBEN - FSBEN|` — matches the recorded error for only "
f"{100 * national['concordance_weighted']['fsben']:.2f}% "
"weighted, because the formula recomputation conflates "
Expand Down
8 changes: 4 additions & 4 deletions app/public/app.js
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ const TIERS = [[6, 0], [8, 5], [10, 10], [Infinity, 15]];
const TIER_LABELS = { 0: "0% share", 5: "5% share", 10: "10% share", 15: "15% share" };
const TIER_VARS = { 0: "--tier-0", 5: "--tier-5", 10: "--tier-10", 15: "--tier-15" };
const DRAWS = 4000;
const ASSET_V = "20261003a"; // bump with index.html's app.js?v= on every deploy that changes any asset
const ASSET_V = "20261004a"; // bump with index.html's app.js?v= on every deploy that changes any asset
const SCEN_SCHEMA = "snap_qc_sim.model_scenarios.v1";
const ENGINE_SCHEMA = "snap_qc_sim.engine_comparison.v1";
const INTERVENTIONS_SCHEMA = "snap_qc_sim.interventions.v1";
Expand Down Expand Up @@ -813,7 +813,7 @@ function renderFy2027(code) {
// ---- Axiom rules-engine comparison mode ----------------------------------
// The engine mode surfaces precomputed verification artifacts: the committed
// axiom-oracles suite reports (stage-by-stage parity of the Axiom RuleSpec
// computation against the QC file's Minimodel-recomputed benefit chain) and
// computation against the QC file's computed benefit chain) and
// the formula-benefit divergence catalog built by analysis/engine_comparison.py.
// It does not change the Monte Carlo simulation — engine runs are precomputed,
// never recomputed in the browser.
Expand Down Expand Up @@ -866,8 +866,8 @@ function enginePanelBody(code) {
const conflates =
`<h3>What the formula benefit conflates with error</h3>` +
`<p>The file records three benefit anchors per case: the allotment the state issued (RAWBEN), ` +
`the allotment corrected for the reviewer's findings (BENFIX), and the Minimodel's full-formula ` +
`recomputation (FSBEN). |RAWBEN − BENFIX| equals the recorded error amount for every ` +
`the allotment corrected for the reviewer's findings (BENFIX), and the formula benefit Mathematica ` +
`computes for USDA from the edited record with the QC Minimodel's benefit formula (FSBEN). |RAWBEN − BENFIX| equals the recorded error amount for every ` +
`official-universe case in the seven verified states — nationally ${pct(nat.concordance_weighted.benfix, 3)} of ` +
`weighted FY 2024 cases. Taking the formula benefit as the error anchor instead — |RAWBEN − FSBEN| — matches ` +
`the recorded error for only ${pct(nat.concordance_weighted.fsben, 1)} weighted: a formula recomputation alone ` +
Expand Down
8 changes: 4 additions & 4 deletions app/public/index.html
Original file line number Diff line number Diff line change
Expand Up @@ -118,7 +118,7 @@ <h2>Targeted review scenarios</h2>
<div class="control levers">
<span class="label">Benefit recomputation</span>
<label><input type="checkbox" id="engine-mode"> <span>Axiom rules engine — verification view</span></label>
<div class="hint">Shows how the open, certified Axiom RuleSpec computation compares against the QC file's FNA Minimodel benefit chain, stage by stage, with the formula-benefit divergence catalog. Precomputed artifacts; the simulation is unchanged.</div>
<div class="hint">Shows how the open, certified Axiom RuleSpec computation compares against the QC file's computed benefit chain, stage by stage, with the formula-benefit divergence catalog. Precomputed artifacts; the simulation is unchanged.</div>
</div>
<div class="control levers">
<span class="label">Uncertainty</span>
Expand All @@ -129,8 +129,8 @@ <h2>Targeted review scenarios</h2>
<p class="hint mechanism" id="mechanism">Sampling engine: resamples the state's observed FY 2024 QC error dollars, centered on the official rate.</p>

<section class="chart-card engine-panel" id="engine-panel" hidden>
<h2>Axiom rules engine vs the FNA QC Minimodel</h2>
<p class="sub">The QC file's formula benefit (FSBEN) is recomputed by the QC Minimodel used by the Food and Nutrition Administration (formerly FNS) — a closed model. The Axiom rules engine replays the same reviews through open, per-provision-cited RuleSpec encodings — compared stage by stage below against the file's own recorded values.</p>
<h2>Axiom rules engine vs the QC file's benefit chain</h2>
<p class="sub">Mathematica computes the QC file's formula benefit (FSBEN) for USDA from each edited case record with the benefit formula of the QC Minimodel, one of the SNAP microsimulation models of the Food and Nutrition Administration (formerly FNS). The Axiom rules engine replays the same reviews through open, per-provision-cited RuleSpec encodings — compared stage by stage below against the file's own recorded values.</p>
<div id="engine-body"></div>
</section>

Expand Down Expand Up @@ -236,6 +236,6 @@ <h2>All states at a glance</h2>
</nav>
</div></footer>

<script type="module" src="app.js?v=20261003a"></script>
<script type="module" src="app.js?v=20261004a"></script>
</body>
</html>
21 changes: 13 additions & 8 deletions app/public/paper/index.html
Original file line number Diff line number Diff line change
Expand Up @@ -76,23 +76,28 @@ <h1>Administrative quality control data as an oracle</h1>
failures, and the fiscal 2025 rates landing where the sampling-noise
mechanism says realized years should: 18 of 53 jurisdictions changed
cost-share tiers, only 7 of those changes beyond two-year sampling
noise. This page embeds the revision-7 manuscript (updated 2026-08-13), which
carries five rounds of adversarial review, archived unedited, and adds
the engine-adoption accounting bounds, the official-vs-file wedge
decomposition, and the pre-registered model-repair experiments.
noise. This page embeds revision 10 (2026-10-03), which corrects how
revision 9 described the QC file's computed benefit and the Colorado
error-case replay, adds Colorado's fiscal 2025 rate beside its fiscal
2024 margin, and withdraws a claim of errata in the federal technical
documentation that no source records; the fact catalog lists each
change. The manuscript carries six rounds of adversarial review,
archived unedited, and adds the engine-adoption accounting bounds, the
official-vs-file wedge decomposition, and the pre-registered
model-repair experiments.
</p>
<span class="paper-tag">Revision 9 · 2026-08-16 · Max Ghenis, PolicyEngine</span>
<span class="paper-tag">Revision 10 · 2026-10-03 · Max Ghenis, PolicyEngine</span>
</header>

<nav class="paper-actions" aria-label="Paper actions">
<a class="primary" href="web/index.html?v=r9-20260816">Open standalone HTML</a>
<a class="primary" href="web/index.html?v=r10-20261003">Open standalone HTML</a>
<a href="web/index.pdf">Download PDF</a>
<a href="../">Live simulator</a>
<a href="https://github.com/PolicyEngine/snap-qc-sim">Code and artifacts</a>
</nav>

<div class="paper-frame-card">
<iframe src="web/index.html?v=r9-20260816"
<iframe src="web/index.html?v=r10-20261003"
title="SNAP payment error working paper (embedded manuscript)"
loading="lazy"
sandbox="allow-same-origin allow-popups allow-popups-to-escape-sandbox"
Expand All @@ -101,7 +106,7 @@ <h1>Administrative quality control data as an oracle</h1>

<p class="paper-foot">
<a href="#paper-top">↑ Back to top</a> ·
<a href="web/index.html?v=r9-20260816">Open manuscript in a new page</a>
<a href="web/index.html?v=r10-20261003">Open manuscript in a new page</a>
</p>
</main>

Expand Down
Loading
Loading