diff --git a/paper/FACTS.md b/paper/FACTS.md index 9bac358..3fc5e56 100644 --- a/paper/FACTS.md +++ b/paper/FACTS.md @@ -130,7 +130,7 @@ Supersession notes (feature-round regeneration, 2026-08-09): E2's stage AUCs/mag | # | Fact | Source | |---|---|---| | K1 | Cause shares, any-presence over each case's nine `AGENCY` slots, share of official error dollars (sum HWGT × `AMTERR`; FY file pools 12 monthly samples so pooled dollars are annual): Colorado strict computation 7.4% ($6.9M/yr), broad rules-engine set 10.6% ($10.0M/yr), software-only 6.3% ($5.9M/yr); national strict 3.5% ($232M/yr), broad 19.3% ($1.27B/yr), software-only 2.5% ($163M/yr), of $6.59B/yr official error dollars. All 53 jurisdictions in the artifact; classes overlap by construction; accounting conventions, not causal estimates of engine-preventable error | analysis/cause_shares.json (deterministic; conventions and code-to-class map serialized in-file) | -| K2 | The lab snapshot's "10 cases, $3.28M, 3.3%" broad-residual line (ANALYSIS.md L62-67) does not reproduce in dollars: the 10-case count reproduces from the SAV cause slots, but HWGT × AMTERR gives $4.47M and HWGT × engine-gap gives $2.91M (4.5% of the 283-case replay dollars). The manuscript does not quote the $3.28M figure | analysis/cause_shares.json colorado_replay_reconciliation | +| K2 | The lab snapshot's "10 cases, $3.28M, 3.3%" broad-residual line (ANALYSIS.md L62-67 at 755a7f3; corrected in the lab on 2026-10-03, with every lab figure recomputed in paper/snapshot/labs/amterr/claims_audit.json) does not reproduce in dollars: the 10-case count reproduces from the SAV cause slots, but HWGT × AMTERR gives $4.47M (4.5% of the 283-case replay dollars) and HWGT × engine-gap gives $2.91M. The manuscript does not quote the $3.28M figure | analysis/cause_shares.json colorado_replay_reconciliation | | K3 | Layer-2 finding-nature tabulation, committed and test-locked: cases partition by whether populated findings are computational (NATURE in {36,42,43,54,64,65,75,79,80,98,123} or ELEMENT 520; deduction natures {52,53,56,57} only when the paired AGENCY cause is in layer 1's broad computing-apparatus set {10,17,19,20,21,22}). Colorado deviation convention reproduces the archived table exactly: 13/19/13/260 cases, $3.66M/$7.47M/$4.68M/$96.8M, 3.3/6.6/4.2/86.0% of $112.6M. National deviation: 4.6/11.1/6.5/77.7% of $8.133B. The artifact also serializes a wider agency-responsibility convention (system-side = all 14 agency-cause codes) with materially larger shares (CO system-caused 31.1% deviation) and both official-error-denominator variants | analysis/cause_shares.json (schema v2); analysis/NATURE_REPORT.md (techdoc line citations) | ## L. Engine-comparison artifact and the simulator's verification view (added after revision 5) diff --git a/paper/snapshot/labs/amterr/ANALYSIS.md b/paper/snapshot/labs/amterr/ANALYSIS.md index 1347d6c..4e23297 100644 --- a/paper/snapshot/labs/amterr/ANALYSIS.md +++ b/paper/snapshot/labs/amterr/ANALYSIS.md @@ -3,166 +3,367 @@ Colorado FY2024, from the USDA SNAP QC public-use file. Built 2026-07-11 on top of the 856/856 benefit-parity result (axiom-oracles#268) and the Giannella/Molin raw-variable reconstruction (github.com/giannella/snap_qc, -run for FY2024 in this lab: `reconstruct_co_fy2024.R`). +run for FY2024 in this lab: `reconstruct_co_fy2024.R`). Revised 2026-10-03: +layer 3, the software-cause split and the cost-share comparison were +recomputed and corrected, and the weight vintage is now stated (see +[revision history](#revision-history)). Every figure below that is computed +from the QC file is regenerated by `audit_claims.py` into +`claims_audit.json`; `README.md` gives the pins and commands to rerun the +whole lab. -## The three layers (increasing strictness) +## Data and weights -All dollars HWGT-weighted and annualized from the FY2024 file; Colorado has -856 sampled cases, 305 with payment errors (STATUS 2/3), $112.6M/yr weighted -error dollars on $1.268B issuance (8.88% file-derived; FNS's official -regression-adjusted FY2024 Colorado rate is **9.97%** — over 7.91 + under -2.06, snap-fy24QC-PER.pdf). +All dollars are HWGT-weighted from the FY2024 file. The file pools 12 +monthly samples, so a weighted sum of monthly amounts is an annual figure. +The lab ran on the May 2026 posting of the file. USDA re-posted it on +2026-08-18 to correct HWGT and FYWGT. A cell-by-cell comparison of the two +postings finds the same 44,891 rows and 1,177 columns, with changes only in +HWGT and FYWGT (16,948 rows, 287 in Colorado) and HWGT_OLD and FYWGT_OLD +(10,072 rows, 154 in Colorado). Every case-level result here (which cases the +solver moved, which the engine reproduces) is therefore the same under both +postings; only weighted dollars move. The text uses the May 2026 weights the +lab ran on; [August 2026 weights](#august-2026-weights) lists the headline +figures under the corrected weights. -### Layer 1 — QC's own cause coding (native, no model needed) +Colorado has 856 sampled cases, 305 with payment errors (STATUS 2/3), and +$112.6M/yr of weighted error dollars (sum HWGT × AMTERR) on $1.268B of +issuance (sum HWGT × RAWBEN): an 8.88% file-derived rate. FNS's official +regression-adjusted FY2024 Colorado rate is 9.97% (7.91 over + 2.06 under; +payment error rate table dated June 30, 2025). The FY2025 rate, published +June 24, 2026, is 10.09% (8.52 + 1.57). -Every error finding carries an AGENCY cause code (codebook: FY-2024 Tech Doc). -Case-attributed error dollars (a case counts if any finding has the code; -per-finding AMOUNTs are mostly zero-filled in Colorado, so case attribution -is the usable metric): +## Layer 1: QC cause coding -| class | cause codes | cases | $/yr | share of error $ | +Every error finding carries an AGENCY cause code (codebook: FY2024 technical +documentation, PDF p. 102). A case counts toward a class if any of its nine +AGENCY slots holds a code in the class, so classes overlap. Per-finding +AMOUNT fields sum to $4.7M/yr in Colorado against $112.6M of case error +dollars, so case attribution is the usable metric. + +| Class | Cause codes | Cases | $/yr | Share of error $ | |---|---|---|---|---| -| strict computation (programming 17, arithmetic 20, mass change 19) | 17/19/20 | — | $8.2M | 7.3% | -| + policy misapplied 10, budgeted wrong 22, computer user 21 | +10/21/22 | 35 | $11.9M | 10.5% | +| Strict computation (programming 17, mass change 19, arithmetic 20) | 17/19/20 | 23 | $8.2M | 7.3% | +| Broad: adds policy incorrectly applied 10, computer user 21, budgeted incorrect amount 22 | 10/17/19/20/21/22 | 35 | $11.9M | 10.5% | + +In Colorado no error case carries code 21 or 22; the broad class there is +codes 10 (12 cases), 17 (13), 19 (5) and 20 (5). 21 error cases ($2.3M, +2.1%) carry no cause code in any slot. -National, same coding: strict $320M/yr (3.9%); broad $1.5B/yr (18.4%) of -$8.1B/yr weighted error dollars on $88.8B issuance. +National, same coding: strict $320M/yr (3.9%) and broad $1.50B/yr (18.4%) of +$8.14B/yr weighted error dollars, on $91.4B of issuance (sum HWGT × RAWBEN). +The July text gave $88.8B as national issuance; that figure is sum HWGT × +FSBEN, the calculated benefit. -### Layer 2 — finding-nature classification (cause-disambiguated) +## Layer 2: finding-nature classification -Natures that inherently describe computation (rounding 36, conversion 42, -averaging 43, wrong standard 54/64/65, benefit incorrectly computed 75, -allotment tables 79, proration 80/123, transcription-or-computation 98, -element 520) plus deduction-amount natures 52/53/56/57 only when the cause -code is system-side: +A populated finding (ELEMENTi, NATUREi, AGENCYi) is computational if its +element is 520 (arithmetic computation), its nature is one that describes +computation (rounding 36, conversion 42, averaging 43, wrong standard 54/65, +amount after a move 64, benefit incorrectly computed 75, allotment tables 79, +initial-month proration 80, transcription or computation 98, proration 123), +or its nature is a deduction nature (52/53/56/57) and its own cause code is +in the broad layer-1 set. Cases partition by how many of their findings are +computational: -| class | cases | $/yr | share | +| Class | Cases | $/yr | Share | |---|---|---|---| -| pure_math (all findings computation) | 13 | $3.66M | 3.3% | -| input_system_caused | 19 | $7.47M | 6.6% | -| mixed | 13 | $4.68M | 4.2% | +| pure_math (all findings computational) | 13 | $3.66M | 3.3% | +| input_system_caused (none computational; a broad-set cause present) | 19 | $7.47M | 6.6% | +| mixed (some computational) | 13 | $4.68M | 4.2% | | input_other | 260 | $96.8M | 86.0% | -### Layer 3 — engine-verified (the demonstrable core) - -Method: reconstruct each error case's pre-edit ORIGINAL values with the -Giannella/Molin $3-shift solver (FY2024 adaptation, smoothing off), replay -them through the Axiom engine (the one proven 856/856 exact on corrected -inputs), compare to RAWBEN (what the agency actually issued), at the file's -own $5 editing tolerance. - -- 283 of 305 error cases survive the authors' consistency filters. -- **246/283 (86.9%): engine(original) ≈ RAWBEN** — the issuance is correct - arithmetic on wrong facts. Input/information errors, faithfully propagated. - (Perfect concordance: the R solver and the Rust engine partition the 283 - cases identically — 246 explained by both, 37 by neither.) -- **37/283: no single-variable original value + correct math reproduces the - issuance.** Upper bound on computation-side; includes solver limitations - (multi-element, household-composition interactions). -- **10 of those 37 also carry QC's own computation/policy cause coding**: - the engine-verified computation class. **$3.28M/yr = 3.3% of replayed - error dollars.** For each, on the facts the agency recorded, the verified - engine returns the reviewer-certified correct benefit and the agency - system did not — e.g. 202312-40441: issued $704, correct $973 = engine - (a $269/month underpayment from "benefit incorrectly computed"). - -Colorado's computation-class error patterns: initial-month proration bugs, -"benefit/allotment incorrectly computed," wrong SUA standard applied, -child-support deduction programming errors, Social Security COLA mass-change -failures, wage conversion/averaging misapplied, and one homeless-shelter -deduction wrongly omitted (element 362 — the same provision whose stale $143 -literal our own first run caught). - -## Why this is millions: the OBBBA cost-share tiers - -7 USC 2013(a)(2) (verified from the US Code, prelim): beginning FY2028 the -state share of benefit costs is 0% / 5% / 10% / 15% for payment error rates -<6 / 6–8 / 8–10 / ≥10, keyed for FY2028 to the state's FY2025 or FY2026 rate -(state's election), then to the third preceding year. - -- One tier ≈ 5% of Colorado's ~$1.27B issuance ≈ **$63M/yr**. -- Colorado's official FY2024 rate is **9.97% — 0.03 points from the 15% - tier**. Its QC-coded system/policy-application errors are ~10.5% of its - error dollars ≈ **~1.0 point of the rate** — 30× the margin to the worse - tier, and half the distance to the better (8%) one. -- Nationally: 10.93% official FY2024 rate; the broad system/policy class is - $1.5B/yr of weighted error dollars. - -The rate that determines the FY2028 share is FY2025/FY2026 — being measured -NOW. Computation-class errors are the share a state can eliminate by fixing -software, without changing verification practice or client behavior. - -## Distinguishing rules-engine causes from others - -Two axes separate "the software did it" from worker and client causes. - -**Axis 1 — QC cause codes.** Software-specific codes are 17 (computer -programming error) and 19 (computer-generated mass change): unambiguous -system attribution by the reviewer. Worker-computation codes: 20 -(arithmetic), 21 (computer user error). Human keying: 18 (data entry). -Ambiguous human-or-software: 10 (policy incorrectly applied), 22 (budgeted -wrong). Case-attributed error dollars: - -| class | Colorado | share | National | share | +The 26 pure_math and mixed cases carry 29 computational findings +(element/nature, with cause codes): initial-month proration (520/80) 4, all +cause 10; benefit incorrectly computed (520/75) 3, causes 15, 15, 20; +transcription or computation (520/98) 2, cause 20; a wrong utility standard +(364/54) 4, causes 2, 2, 12, 18; wage or self-employment conversion and +averaging (311 and 312 with 42/43) 6; child-support deduction amounts +(366/56) 2, cause 17; medical deduction (365/52, 365/53, 365/98) 3; and one +each for the utility allowance after a move (364/64), the utility allowance +omitted (364/52), the homeless shelter deduction omitted (362/52, cause 10), +a dependent care amount (323/57) and an RSDI transcription error (331/98). +Element 362 is the provision whose stale $143 literal the first parity run +caught in our own encoding. + +## Layer 3: engine replay + +### Method + +The Giannella/Molin solver (FY2024 adaptation, smoothing off) starts from +the file's edited inputs (household size, earned and unearned income, rent, +utilities and deductions) and moves only the input named by the case's first +finding (ELEMENT1): earned or unearned income, rent, the utility allowance, +medical, dependent care or child-support deduction in $3 steps, or household +size by one person for element 150 (natures 7, 12, 14, 16). Income steps +stop once the recomputed benefit passes RAWBEN (the benefit the agency +issued). Rent, utility and deduction steps also stop within $3 of it. Steps +that lower an input stop at zero; rent and utility steps also stop when the +shelter deduction reaches its cap; every step stops when the benefit reaches +$0 or after 1,000 steps. Rows the solver labels +`util_up` or `util_down` then have the utility allowance set to the nearest +value above (or below) the file's UTIL that more than 5 filtered cases in +that state and calendar year use; a `util_down` row with no such value is +set to 0. When ELEMENT1 is outside those lists the inputs stay as they are +(`correctednotes == "no_change"`). The Axiom +engine, which reproduces all 856 Colorado FSBEN values at zero tolerance on +the file's inputs (axiom-oracles#268), then computes the benefit on the +solver's inputs, and the result is compared with RAWBEN at the file's own $5 +editing tolerance. Below, "moved" means at least one of the eight replayed +inputs differs from the file value it started from. (The solver's +`correctedamount` column misses two utility rows, 202403-40765 and +202404-40803, because it is recorded before the utility snap.) + +FSBEN is a constructed variable: the final benefit Mathematica's model +calculates from the edited inputs (technical documentation: listed as +constructed on PDF p. 73, with the raw/constructed legend on p. 75 and the +formula on p. 91). RAWBEN is a raw variable: the benefit the unit was +certified to receive in the sample month (PDF p. 92). + +### Results + +- 283 of 305 error cases pass the solver's consistency filters + (|RAWBEN − FSBEN| within $5 of AMTERR, RENT and UTIL present, BENMAX equal + to the FY2024 table value). +- 246 of 283 (86.9% of cases, 72.0% of the $99.1M replayed error dollars) + reproduce the issued benefit within $5. In 230 the solver moved an input; + in 16 nothing moved and |RAWBEN − FSBEN| ≤ $5 already. +- 37 do not reproduce. In 20 the solver moved an input and stopped short. + In 17 nothing moved: 14 `no_change` rows and 3 rows where the solver's + element applied but it stopped before moving. For those 17 the replayed + input is the file's input, so the engine returns FSBEN. +- The solver's own recomputed benefit and the engine agree on the within-$5 + classification for all 283 cases (246 reproduced, 37 not). + +A reproduced case shows that one value of the first-listed input, run +through correct arithmetic, yields the issued benefit. That is consistent +with an input error on that element; it does not identify which input the +agency had wrong, and it does not rule out a computation error that a moved +input absorbed. A non-reproduced case does not by itself show a computation +error: the solver tries one element in one direction, so multi-element and +household-composition errors also land among the 37. + +The replay outcome does not separate the layer-2 computational findings. Of +the 26 cases that carry one, 13 reproduce ($3.6M/yr, 3.2% of Colorado error +dollars), 10 do not ($3.5M, 3.1%) and 3 were not replayed. Among the 13 are +202404-40794 and 202404-40823, which reproduce after the solver moved the +child-support deduction and whose finding is 366/56/17 (computer programming +error). + +### The 10 broad-coded cases that do not reproduce + +10 of the 37 carry a broad layer-1 code (10/17/19/20/21/22) in any slot. +They carry $4.47M/yr: 4.5% of replayed error dollars and 4.0% of Colorado +error dollars. The July text gave $3.28M (3.3%); that figure does not +reproduce. HWGT × AMTERR gives $4.47M, and HWGT × |engine − RAWBEN| gives +$2.91M. + +| Case | Issued (RAWBEN) | FSBEN | Engine on replayed input | Solver | Broad-coded finding (element/nature/cause) | Computation finding | +|---|---|---|---|---|---|---| +| 202310-40336 | $150 | $291 | $291 | no_change | 520/80/10 initial-month proration | yes | +| 202312-40456 | $146 | $75 | $102 | moved utility allowance (364) | 520/80/10 initial-month proration | yes | +| 202312-40466 | $281 | $291 | $291 | no_change | 520/75/20 benefit incorrectly computed | yes | +| 202312-40513 | $264 | $103 | $244 | moved utility allowance (364) | 331/44/17 RSDI, less income received than budgeted | no | +| 202401-40518 | $132 | $171 | $171 | no_change | 520/80/10 initial-month proration | yes | +| 202401-40548 | $594 | $886 | $746 | moved rent (363) | 350/44/17 child support received, less than budgeted | no | +| 202405-40910 | $95 | $149 | $149 | no_change | 362/52/10 homeless shelter deduction omitted | yes | +| 202406-40990 | $125 | $161 | $161 | no_change | 161/6/10 time-limited participation, eligible person excluded | no | +| 202407-41178 | $187 | $291 | $291 | no_change | 520/80/10 initial-month proration | yes | +| 202408-41271 | $240 | $227 | $227 | no_change | 520/98/20 transcription or computation | yes | + +What the replay shows for these cases: + +- In 7 (the `no_change` rows) the engine returns FSBEN because the solver + left the file's inputs in place. That restates the 856/856 parity result + for those cases. It does not show what facts the agency recorded or what + the agency's system computed from them. +- In the other 3 the solver moved a different finding's input, and the + engine's benefit matches neither RAWBEN nor FSBEN. +- 7 of the 10 broad-coded findings concern the benefit computation (520 in + six cases; 362/52 in one). These 7 computation candidates carry $2.0M/yr, + 1.8% of Colorado error dollars. The other 3 are income or eligibility + findings. +- The candidates' codes are 10 (policy incorrectly applied) in 5 and 20 + (arithmetic computation) in 2. None carries 17 or 19. + +The candidate count depends on the cause-code set. By the layer-2 rule alone, +the 10 non-reproduced cases with a computational finding are the 7 +candidates, two more 520/75 findings with cause 15 (agency failed to follow +up on impending changes; 202312-40441 and 202401-40566) and a wrong utility +standard with cause 18 (364/54/18; 202310-40297). + +The July text illustrated the class with 202312-40441 (issued $704, FSBEN +$973). That case's only cause code is 15, so it is outside the 10; it is a +layer-2 pure_math case (520/75/15) and a `no_change` row. + +## Software causes and the replay + +**Cause codes.** Codes 17 (computer programming error) and 19 (mass change) +name the system. This lab groups 20 (arithmetic computation error) and 21 +(computer user error) as worker computation; 18 is data entry or coding; 10 +and 22 (policy incorrectly applied; agency budgeted an incorrect amount) can +be either. Case-attributed error dollars: + +| Class | Colorado | Share | National | Share | |---|---|---|---|---| -| software (17+19) | $7.0M/yr (18 cases) | 6.2% | $240.6M/yr | 3.0% | -| worker computation (20+21) | $1.2M/yr | 1.1% | $107.1M/yr | 1.3% | -| data entry (18) | $18.5M/yr | 16.4% | $176.2M/yr | 2.2% | -| policy misapplied / budgeted wrong (10+22) | $3.7M+/yr | 3.3%+ | $1,210M/yr | 14.9% | - -Colorado's software-coded share is double the national rate (6.2% vs 3.0%), -and its data-entry share is 7x national — a CBMS-specific signature. -Worker arithmetic is nearly extinct (5 CO cases): the system does the math, -so when the math is wrong it is mostly the system. - -**Axis 2 — the engine replay splits HOW the software failed.** Of the 16 -replayable software-coded (17/19) Colorado cases: - -- **14 = automation fed itself the wrong input** ($4.3M/yr): COLA mass - changes writing wrong RSDI/SSI amounts, interfaces budgeting wrong child - support or rent — then computing correctly on the bad value. The engine - reproduces the issuance from the reconstructed wrong input. Fixed by - verified data integrations, not by rules logic. -- **2 = computation logic itself wrong** ($2.4M/yr): no input value under - correct rules reproduces what the system issued. - -The complementary class — the 10 engine-verified computation cases -($3.3M/yr) from layer 3 — includes those 2 plus worker-arithmetic and -policy-misapplication cases: everything where the benefit determination -was wrong GIVEN the recorded facts. That is the class a verified rules -engine eliminates regardless of whether human or machine did the math, -because it replaces both. Together with the automation-fed class, the -"modern verified eligibility stack" claim covers ~$7.6M/yr ≈ 6.7% of -Colorado's error dollars. - -Cause-coding caveat: reviewer cause assignment varies in quality across -states (the tech doc notes FY2024 coding-definition changes); the engine -replay is the independent check on it, which is exactly the role it plays -in axis 2. - -## Caveats (do not drop these when quoting) +| Software (17, 19) | $7.0M/yr (18 cases) | 6.2% | $240.6M/yr | 3.0% | +| Worker computation (20, 21) | $1.2M/yr (5 cases) | 1.1% | $106.9M/yr | 1.3% | +| Data entry (18) | $18.5M/yr (28 cases) | 16.4% | $176.2M/yr | 2.2% | +| Policy misapplied or budgeted wrong (10, 22) | $3.7M/yr (12 cases) | 3.3% | $1,188M/yr | 14.6% | + +Colorado's software-coded share is 2.1 times the national share, and its +data-entry share is 7.6 times. The July text gave national figures of $107.1M +for codes 20/21 and $1,210M (14.9%) for codes 10/22; neither reproduces. + +**What the replay adds.** 16 replayed Colorado cases carry code 17 or 19. + +- 14 reproduce the issued benefit ($4.3M/yr). In 10 of them ($1.8M) the + solver moved the input on the element that carries the 17/19 code: RSDI + (331), SSI (333), child support received (350) or the child-support + deduction (366). In 2 ($1.7M, almost all of it 202404-40805) it moved + total unearned income through a different unearned-income finding, so + the match does not separate that finding from the software-coded one. In + 2 ($0.8M) it moved rent. +- 2 do not reproduce ($2.4M/yr): 202312-40513 and 202401-40548. In both the + software-coded finding (331/44/17 and 350/44/17) is on an element the + solver never moved; it moved ELEMENT1 (364 and 363). The replay does not + show that the computation logic failed in either case. +- Mass-change (19) findings in these 16 are on RSDI (4 cases), SSI (2) and + the medical deduction (1); programming (17) findings are on RSDI, SSI, + child support received and the child-support deduction. None is on rent + or shelter. + +Cause-coding caveat: reviewer cause assignment varies across states, and the +technical documentation notes FY2024 coding-definition changes. + +## Cost-share stakes + +7 USC 2013(a)(2), as amended by P.L. 119-21: beginning FY2028 the state +share of benefit costs is 0% / 5% / 10% / 15% for payment error rates below +6 / 6 to 8 / 8 to 10 / 10 and above, keyed for FY2028 to the state's FY2025 +or FY2026 rate (the state's election), then to the third preceding year. + +- One tier is 5 percentage points of the state's share of benefit costs. + On FY2024 sample-weighted issuance ($1.268B, sum HWGT × RAWBEN) that is + $63.4M/yr. +- Colorado's FY2024 rate, 9.97%, sits 0.03 points below the 10% line of the + 15% tier. Its FY2025 rate, 10.09%, sits 0.09 points above it. FY2026 is + the remaining election year. + +Where the broad cause class lands in the replay, as a share of Colorado error +dollars and scaled to the official rate (share × 9.97), over all error +dollars and over error dollars above the $56 threshold ($94.1M): + +| Part of the broad class (10/17/19/20/21/22) | Cases (above $56) | Share of error $ | Points of 9.97 | Share above $56 | Points above $56 | +|---|---|---|---|---|---| +| All | 35 (16) | 10.5% | 1.05 | 10.6% | 1.06 | +| Replay reproduces the issued benefit | 22 (9) | 5.7% | 0.56 | 5.4% | 0.54 | +| Replay does not reproduce | 10 (5) | 4.0% | 0.40 | 4.1% | 0.41 | +| Not replayed (solver filters) | 3 (2) | 0.9% | 0.09 | 1.1% | 0.11 | +| The 7 computation candidates | 7 (3) | 1.8% | 0.18 | 1.6% | 0.16 | + +The July text set the whole broad class, about 1.0 point, against the +0.03-point margin. Of that point, 0.56 sits in cases the replay reproduces +from a single moved or unchanged input, and 0.18 (0.16 on above-threshold +dollars) sits in the 7 cases whose broad-coded finding concerns the +computation; 4 of those 7 are at or below the $56 threshold. The scaling is +approximate: the official rate is regression-adjusted and counts only errors +above the threshold. The 7 candidates are 7 sampled cases. + +## Caveats - Within-state tabulations from the QC sample are not designed to be - state-representative (tech doc warning); the official PER uses regression - adjustment. Shares (%) are more defensible than $ levels; both are - FY2024-sample-based. -- Cause codes mix software and caseworker action: 17/19/20/21 are - machine/system; 10/22 can be either. "System or policy-application" - is the honest label for the broad class; the engine-verified 10 are the - individually demonstrable core. -- The engine replay validates the BENEFIT COMPUTATION; eligibility-side - errors (person included/excluded) are only partially modeled (household - size shifts approximate member effects). -- The solver is $3-granular; $5 (the file's own edit-loop tolerance) is the - operating comparison tolerance. Exact-tolerance rates are not meaningful - at this layer. -- FY2024 evaluated via the compile-time COLA overlay at the nominal period - (rulespec-us#759 pending), same as the parity run. + state-representative (technical documentation warning); the official rate + uses regression adjustment. Shares (%) are more defensible than dollar + levels; both are FY2024-sample-based. +- Cause codes mix software and caseworker action: 17/19/20/21 name the + system or the worker's computation; 10/22 can be either. "System or + policy-application" is the label the broad class supports. +- The engine replay checks the benefit computation. Eligibility-side errors + (person included or excluded) enter only through the solver's household + size step. +- The solver moves one element in $3 steps; $5 is the comparison tolerance. + Exact-tolerance rates are not meaningful at this layer. +- FY2024 was evaluated through the compile-time COLA overlay at the nominal + period, as in the parity run; rulespec-us#759 (the dependency inversion + that would remove the overlay) was still open on 2026-10-03. +- Weights: May 2026 posting (see [August 2026 weights](#august-2026-weights)). + +## August 2026 weights + +Headline figures under the May 2026 weights (used above) and the August 2026 +corrected weights. Case counts are identical. + +| Figure | May 2026 posting | August 2026 posting | +|---|---|---| +| Colorado error dollars | $112.58M | $112.58M | +| Colorado issuance | $1.268B | $1.265B | +| One cost-share tier (5% of issuance) | $63.4M | $63.3M | +| Layer 1 strict, Colorado | $8.19M (7.3%) | $8.19M (7.3%) | +| Layer 1 broad, Colorado | $11.87M (10.5%) | $11.86M (10.5%) | +| Data entry (18), Colorado | $18.47M (16.4%) | $18.43M (16.4%) | +| Software (17, 19), Colorado | $6.98M (6.2%) | $6.98M (6.2%) | +| National error dollars | $8.136B | $8.108B | +| National broad | $1.499B (18.4%) | $1.492B (18.4%) | +| Replayed error dollars (283 cases) | $99.13M | $99.08M | +| 10 broad-coded non-reproduced cases | $4.47M (4.5% of replayed) | $4.47M (4.5%) | +| 7 computation candidates | $2.00M (1.77%) | $1.99M (1.77%) | +| Software-coded, replay reproduces | $4.27M | $4.26M | +| Software-coded, replay does not reproduce | $2.37M | $2.37M | ## Artifacts -- `native_decomposition.json`, `phase_a_classification.json` — layers 1–2 -- `reconstruct_co_fy2024.R` → `co_fy2024_reconstruction.csv` (283 CO rows), - `fy2024_reconstruction_national.csv` (15,902 rows, all states) -- `amterr_replay.py` → `amterr_replay_results.json` (engine vs RAWBEN per case) -- Official rate table: `../snap-qc/snap-fy24QC-PER.pdf`; tiers: 7 USC 2013(a)(2) +| File | Contents | Generator | +|---|---|---| +| `native_decomposition.json` | Layer-1 totals, Colorado and national | `audit_claims.py` (regenerates it from the May 2026 posting and checks the committed copy) | +| `../phase_a_classification.json` | Layer-2 classes and case lists | same | +| `co_fy2024_reconstruction.csv`, `fy2024_reconstruction_national.csv` | Solver output: 283 Colorado and 15,902 national error rows | `reconstruct_co_fy2024.R` | +| `amterr_replay_results.json` | Engine on reconstructed inputs vs RAWBEN, per case | `amterr_replay.py` | +| `claims_audit.json` | Every figure in this file, under both postings | `audit_claims.py` | + +Official rates: FNS payment error rate tables for FY2024, +https://fns-prod.azureedge.us/sites/default/files/resource-files/snap-fy24QC-PER.pdf +(dated June 30, 2025), and FY2025 (dated June 24, 2026; the table +`analysis/fy2025_movement.py` reads). Tiers: 7 USC 2013(a)(2), as amended by +P.L. 119-21 sec. 10105. Rerun instructions and pins: `README.md`. + +## Revision history + +- **2026-07-11.** First version. +- **2026-10-03.** Recomputed every figure (`audit_claims.py`). Changes: + - Layer 3: the 10 broad-coded non-reproduced cases carry $4.47M (4.5% of + replayed error dollars). The July text gave $3.28M (3.3%). + - The July text said the engine returns "the reviewer-certified correct + benefit" for those 10 "on the facts the agency recorded." The engine + equals FSBEN in 7 of the 10, all `no_change` rows where the replayed + input is the file's input. FSBEN is Mathematica's calculated benefit. + - Only 7 of the 10 broad-coded findings concern the computation. The July + example, 202312-40441, is outside the 10 (cause 15). + - "No single-variable original value + correct math reproduces the + issuance" now states what the solver does: it moves only the ELEMENT1 + input, and for 17 of the 37 it moved nothing (14 `no_change` rows). The + "upper bound on computation-side" reading of the 37 is withdrawn: of + the 26 cases with a layer-2 computational finding, 13 reproduce and 10 + do not. + - Software cases: of the 14 that reproduce, 10 matched by moving the + software-coded element's input, 2 by moving the same unearned-income + total through a different finding and 2 by moving rent. The 2 that do + not reproduce have their software finding on an element the solver did + not move, so the replay does not show computation-logic failure. No + software-coded finding is on rent. The combined "~$7.6M/yr ≈ 6.7%" + figure is withdrawn. + - Reproduced cases: 230 matched after the solver moved an input and 16 + without any move. + - Cost share: the broad class (about 1.05 points of 9.97) splits into + 0.56 points the replay reproduces from a single input, 0.40 not + reproduced and 0.09 not replayed; the 7 computation candidates are + 0.18 points (0.16 on above-threshold dollars), and none carries code + 17 or 19. + - National issuance is $91.4B (sum HWGT × RAWBEN); the July $88.8B was + sum HWGT × FSBEN. National codes 20/21 are $106.9M; codes 10/22 are + $1,188M (14.6%). + - The text now states the weight vintage (May 2026 posting) and adds the + August 2026 figures. + - FY2025's rate (10.09%, published June 24, 2026) replaces the statement + that FY2025 was still being measured. + - Paths in the scripts are configurable; `README.md` gives pins and + commands. `phase_a_classification.json` lives one directory up. diff --git a/paper/snapshot/labs/amterr/README.md b/paper/snapshot/labs/amterr/README.md new file mode 100644 index 0000000..2b5eb0b --- /dev/null +++ b/paper/snapshot/labs/amterr/README.md @@ -0,0 +1,140 @@ +# amterr lab: rerunning it + +This directory holds the Colorado FY2024 error-case lab described in +`ANALYSIS.md`. It has three stages: + +1. `reconstruct_co_fy2024.R` (R) reconstructs each error case's pre-correction + inputs and writes `co_fy2024_reconstruction.csv` and + `fy2024_reconstruction_national.csv`. +2. `amterr_replay.py` (Python, inside an axiom-oracles checkout) runs the Axiom + rules engine on those inputs and writes `amterr_replay_results.json`. +3. `audit_claims.py` (Python, this repository) recomputes every figure in + `ANALYSIS.md` into `claims_audit.json`. It also regenerates + `native_decomposition.json` and `../phase_a_classification.json` from the + May 2026 posting and checks them against the committed copies. + +Stage 3 needs only this repository and the QC files. Stages 1 and 2 need the +pinned external checkouts below. + +## Pins + +The lab ran on 2026-07-11; the replay output is stamped 23:36Z. The commits +below were the main-branch heads at that time. On 2026-10-03, reruns at these +pins reproduced `co_fy2024_reconstruction.csv`, +`fy2024_reconstruction_national.csv` and `amterr_replay_results.json` byte +for byte. + +| Component | Pin | +|---|---| +| giannella/snap_qc | `741e10bf75c36a9ce8ac20f922b5eec3d4539dc2` | +| snap_qc `qc_data/qc_pub_fy2024.sav` | sha256 `ab6420fa359ab9bcc280a21b9ba7b11172c79c6f7a661718bc6e318f97723fbb` (carries the May 2026 posting's weights) | +| TheAxiomFoundation/axiom-oracles | `d34aa6fa04287387e0dab6912d128ef746c7b6b2` (merge of #268) | +| TheAxiomFoundation/rulespec-us | `b53ce208771085030939db4b9691762506b6bca2` (#826) | +| TheAxiomFoundation/axiom-rules-engine | `de0efdc73b469132ee268e1c832e8f7148b91431` (#102) | +| Engine release binary | sha256 `bb8ec23689697a5417b74c38196c0488e002e4ee6fe3b33faabb39005e6e5eee`, the 2026-08-06 build of that commit recorded in `../../cert/CERT_REPORT.md` (built there with `cargo build --release --offline`) | +| FY2024 QC CSV, May 2026 posting | zip `https://snapqcdata.net/sites/default/files/2026-05/qcfy2024_csv.zip` (sha256 `0f3230a4318307d3088382546095eebfde03e781da6f65c9eac7f077bd4263f4`); member `qc_pub_fy2024.csv` sha256 `45193eb7370463ab3067d71da23a580fec34a5460341e4e750dda0be061e1aa9` | +| FY2024 QC CSV, August 2026 posting | zip `https://snapqcdata.net/sites/default/files/2026-08/qcfy2024_csv.zip` (sha256 `b8b29b8593f78aa51c48332c47d2d92fa5bbecf5346570acb45e26f2d9ebd2b5`); member sha256 `e871a8e9caca0be72e2003b09bdf71e1d020984b52289d2b74c4c6b88c4f793b` | +| R | 4.3.0 with haven 2.5.5, dplyr 1.1.2, tidyr 1.3.0 | + +The binary that ran on 2026-07-11 was not recorded. The 2026-10-03 rerun used +the 2026-08-06 build of the same commit, attested in `CERT_REPORT.md`. Its +output matched byte for byte under CPython 3.14 (free-threaded) and 3.13. + +The two postings differ only in HWGT, FYWGT, HWGT_OLD and FYWGT_OLD. The +replay reads weights only for its `weight` field. Run against the August +posting, it changes that field in 90 of 283 rows and nothing else. + +## Commands + +Set these once (example paths): + +```bash +export REPO=$PWD # this repository's root +export LAB=$REPO/paper/snapshot/labs/amterr +export WORK=$HOME/amterr-rerun # any scratch directory +mkdir -p $WORK +``` + +### Stage 3: audit (this repository only) + +Download the two zips named in the pins table into `$WORK/may2026/` and +`$WORK/aug2026/`. The axiom-oracles loader sends a `Referer` of +`https://snapqcdata.net/datafiles` with its request; the same page lists the +files for a browser download. USDA replaces postings, so the May 2026 zip may stop +being served; the hashes identify which posting a file is. Then unzip and +check the CSV hashes against the table: + +```bash +unzip -o -d $WORK/may2026 $WORK/may2026/qcfy2024_csv.zip +unzip -o -d $WORK/aug2026 $WORK/aug2026/qcfy2024_csv.zip +shasum -a 256 $WORK/may2026/qc_pub_fy2024.csv $WORK/aug2026/qc_pub_fy2024.csv +``` + +```bash +SNAP_QC_CSV_MAY2026=$WORK/may2026/qc_pub_fy2024.csv \ +SNAP_QC_CSV_AUG2026=$WORK/aug2026/qc_pub_fy2024.csv \ +uv run --frozen --extra analysis python $LAB/audit_claims.py --check +``` + +`--check` exits 1 if the regenerated audit differs from `claims_audit.json`; +drop it to rewrite the file. The script refuses a CSV whose hash does not +match its posting. `tests/test_amterr_lab.py` runs the same regeneration +when both CSVs are present at the default paths (`~/.cache/axiom-oracles/`) +or at the paths in those variables, and skips it otherwise. + +### Stage 1: reconstruction (R, about 70 s) + +```bash +git clone https://github.com/giannella/snap_qc $WORK/snap_qc +git -C $WORK/snap_qc checkout 741e10bf75c36a9ce8ac20f922b5eec3d4539dc2 +shasum -a 256 $WORK/snap_qc/qc_data/qc_pub_fy2024.sav +mkdir -p $WORK/recon +SNAP_QC_REPO=$WORK/snap_qc AMTERR_LAB_DIR=$WORK/recon Rscript $LAB/reconstruct_co_fy2024.R +cmp $WORK/recon/co_fy2024_reconstruction.csv $LAB/co_fy2024_reconstruction.csv +cmp $WORK/recon/fy2024_reconstruction_national.csv $LAB/fy2024_reconstruction_national.csv +``` + +The script prints `wrote 283 CO error rows; 15902 national error rows` and +`CO error cases within $5: 246 of 283`. Without `AMTERR_LAB_DIR` it writes +next to itself. + +### Stage 2: engine replay + +Build the engine at its pin, then check out the harness and the rules. The +2026-10-03 rerun used the attested binary in the table; a fresh build on +another machine may differ in sha256, so compare the replay output, which is +what the pins have to reproduce. + +```bash +git clone https://github.com/TheAxiomFoundation/axiom-rules-engine $WORK/engine +git -C $WORK/engine checkout de0efdc73b469132ee268e1c832e8f7148b91431 +(cd $WORK/engine && cargo build --release) +export BIN=$WORK/engine/target/release/axiom-rules-engine + +git clone https://github.com/TheAxiomFoundation/axiom-oracles $WORK/axiom-oracles +git -C $WORK/axiom-oracles checkout d34aa6fa04287387e0dab6912d128ef746c7b6b2 +git clone https://github.com/TheAxiomFoundation/rulespec-us $WORK/rulespec-us +git -C $WORK/rulespec-us checkout b53ce208771085030939db4b9691762506b6bca2 +mkdir -p $WORK/replay && cp $LAB/co_fy2024_reconstruction.csv $WORK/replay/ +``` + +The harness's QC loader hash-checks only a zip it downloads. A CSV it finds +in `AXIOM_SNAP_QC_DATA_DIR` or `~/.cache/axiom-oracles/snap-qc/` is used +unchecked, so point it at a directory holding the May 2026 posting's CSV and +check that file's hash first: + +```bash +shasum -a 256 $WORK/may2026/qc_pub_fy2024.csv +cd $WORK/axiom-oracles && uv sync --frozen +cd $WORK/axiom-oracles && \ + AXIOM_SNAP_QC_DATA_DIR=$WORK/may2026 \ + AXIOM_SNAP_QC_RULESPEC_ROOT=$WORK/rulespec-us \ + AXIOM_SNAP_QC_AXIOM_BINARY=$BIN \ + AMTERR_LAB_DIR=$WORK/replay \ + uv run --frozen python $LAB/amterr_replay.py +cmp $WORK/replay/amterr_replay_results.json $LAB/amterr_replay_results.json +``` + +The script prints `engine(original) vs RAWBEN <=$5: 246/283 (86.9%)` and +`solver within $5: 246/283; engine agrees (<=$5) on 246 of those`. Without +`AMTERR_LAB_DIR` it reads and writes next to itself. diff --git a/paper/snapshot/labs/amterr/amterr_replay.py b/paper/snapshot/labs/amterr/amterr_replay.py index 11c92a6..a117549 100644 --- a/paper/snapshot/labs/amterr/amterr_replay.py +++ b/paper/snapshot/labs/amterr/amterr_replay.py @@ -6,16 +6,17 @@ the ORIGINAL values — run the engine, and compare the allotment to RAWBEN (the benefit the agency actually issued). -Interpretation: - match -> the issuance is explained by correct arithmetic on the original - facts: an input/information error, faithfully propagated. - miss -> no plausible original value + correct math reproduces the - issuance: computation-side error (or reconstruction failure — - correctednotes/at_max separate those). - -Run from the axiom-oracles worktree: - uv run python /Users/maxghenis/.cache/axiom-oracles/amterr-lab/amterr_replay.py -with AXIOM_SNAP_QC_RULESPEC_ROOT / AXIOM_SNAP_QC_AXIOM_BINARY set. +Reading the result: + match -> the engine on the solver's inputs is within $5 of RAWBEN. + miss -> it is not. +ANALYSIS.md (layer 3) says what each does and does not show. + +Run from an axiom-oracles checkout at the commit pinned in README.md: + AXIOM_SNAP_QC_RULESPEC_ROOT= \ + AXIOM_SNAP_QC_AXIOM_BINARY= \ + uv run python /amterr_replay.py +AMTERR_LAB_DIR (default: this file's directory) holds the input +co_fy2024_reconstruction.csv and receives amterr_replay_results.json. """ from __future__ import annotations @@ -49,7 +50,7 @@ ) from axiom_oracles.populations.snap_qc import load_qc_units -LAB = Path("/Users/maxghenis/.cache/axiom-oracles/amterr-lab") +LAB = Path(os.environ.get("AMTERR_LAB_DIR", Path(__file__).resolve().parent)).expanduser() def _num(v): diff --git a/paper/snapshot/labs/amterr/audit_claims.py b/paper/snapshot/labs/amterr/audit_claims.py new file mode 100644 index 0000000..ef76733 --- /dev/null +++ b/paper/snapshot/labs/amterr/audit_claims.py @@ -0,0 +1,984 @@ +"""Recompute every quantitative claim in ANALYSIS.md from committed artifacts. + +Inputs: + +* ``amterr_replay_results.json`` and ``co_fy2024_reconstruction.csv`` (this + directory): the July 2026 replay and solver outputs, committed as-is; +* the FY2024 SNAP QC public-use CSV, in one or both of its two postings. + +Outputs ``claims_audit.json`` (this directory). The audit also regenerates +``native_decomposition.json`` and ``../phase_a_classification.json`` from the +May 2026 posting and records whether they match the committed copies. + +The two postings differ only in the weight columns (HWGT, FYWGT, HWGT_OLD, +FYWGT_OLD); the audit re-verifies that, so every case-level result (which +cases the solver moved, which the engine reproduces) is posting-invariant and +only weighted dollars move. + +Usage, from the repository root:: + + uv run --extra analysis python paper/snapshot/labs/amterr/audit_claims.py + uv run --extra analysis python paper/snapshot/labs/amterr/audit_claims.py --check + +Environment (defaults in ``POSTINGS``): + + SNAP_QC_CSV_MAY2026 qc_pub_fy2024.csv from the May 2026 posting + SNAP_QC_CSV_AUG2026 qc_pub_fy2024.csv from the August 2026 posting +""" + +from __future__ import annotations + +import argparse +import hashlib +import json +import os +import sys +from collections.abc import Iterable +from pathlib import Path +from typing import Any + +import numpy as np +import pandas as pd + +LAB = Path(__file__).resolve().parent +REPO = LAB.parents[3] +REPLAY_PATH = LAB / "amterr_replay_results.json" +RECONSTRUCTION_PATH = LAB / "co_fy2024_reconstruction.csv" +NATIVE_PATH = LAB / "native_decomposition.json" +PHASE_A_PATH = LAB.parent / "phase_a_classification.json" +OUTPUT_PATH = LAB / "claims_audit.json" + +# USDA replaces postings rather than editing them. Each entry pins the zip and +# the CSV member so a mislabeled local file fails loudly. +POSTINGS: dict[str, dict[str, str]] = { + "may2026": { + "env": "SNAP_QC_CSV_MAY2026", + "default": "~/.cache/axiom-oracles/snap-qc/qc_pub_fy2024.csv", + "zip_url": ( + "https://snapqcdata.net/sites/default/files/2026-05/qcfy2024_csv.zip" + ), + "zip_sha256": ( + "0f3230a4318307d3088382546095eebfde03e781da6f65c9eac7f077bd4263f4" + ), + "csv_sha256": ( + "45193eb7370463ab3067d71da23a580fec34a5460341e4e750dda0be061e1aa9" + ), + "last_modified": "2026-05-21", + "role": "weights used by the July 2026 lab run", + }, + "aug2026": { + "env": "SNAP_QC_CSV_AUG2026", + "default": "~/.cache/axiom-oracles/snap-qc/aug2026/qc_pub_fy2024.csv", + "zip_url": ( + "https://snapqcdata.net/sites/default/files/2026-08/qcfy2024_csv.zip" + ), + "zip_sha256": ( + "b8b29b8593f78aa51c48332c47d2d92fa5bbecf5346570acb45e26f2d9ebd2b5" + ), + "csv_sha256": ( + "e871a8e9caca0be72e2003b09bdf71e1d020984b52289d2b74c4c6b88c4f793b" + ), + "last_modified": "2026-08-18", + "role": "current posting; corrects HWGT and FYWGT", + }, +} +LEGACY_POSTING = "may2026" + +# FNS payment error rate tables (percent): FY2024 dated 2025-06-30, FY2025 +# dated 2026-06-24. +OFFICIAL_RATES = { + "fy2024": {"CO": 9.97, "US": 10.93, "dated": "2025-06-30"}, + "fy2025": {"CO": 10.09, "US": 10.62, "dated": "2026-06-24"}, +} +THRESHOLD_FY2024 = 56 +COLORADO_FIPS = 8 +REPLAY_TOLERANCE = 5 +COST_SHARE_STEP = 0.05 + +SLOTS = tuple(range(1, 10)) +STRICT_CODES = frozenset({17, 19, 20}) +BROAD_CODES = frozenset({10, 17, 19, 20, 21, 22}) +SOFTWARE_CODES = frozenset({17, 19}) +AXIS1_CLASSES = { + "software_17_19": frozenset({17, 19}), + "worker_computation_20_21": frozenset({20, 21}), + "data_entry_18": frozenset({18}), + "policy_or_budgeted_10_22": frozenset({10, 22}), +} +INHERENT_COMPUTATION_NATURES = frozenset({36, 42, 43, 54, 64, 65, 75, 79, 80, 98, 123}) +DEDUCTION_NATURES = frozenset({52, 53, 56, 57}) +# native_decomposition.json's "nc" field: the inherent natures plus 28 +# (incorrect income limit applied) and 127 (pass-through not considered or +# incorrectly applied). Neither occurs in a Colorado error case. +NC_NATURES = INHERENT_COMPUTATION_NATURES | {28, 127} +ARITHMETIC_ELEMENT = 520 +PHASE_A_CLASSES = ("input_other", "mixed", "pure_math", "input_system_caused") + +# The eight inputs amterr_replay.py feeds the engine, and the file fields the +# solver initializes them from (missing values become 0). +REPLAY_INPUTS = { + "rawusize": "FSUSIZE", + "rawearn": "FSEARN", + "rawunearn": "FSUNEARN", + "rawrent": "RENT", + "rawutil": "UTIL", + "rawmedded": "FSMEDDED", + "rawdepded": "FSDEPDED", + "rawcsded": "FSCSDED", +} +# The input the solver moves for each ELEMENT1 it handles +# (reconstruct_co_fy2024.R, adjust_* calls and the element-150 step). +SOLVER_ELEMENT_INPUTS = { + **dict.fromkeys( + (331, 332, 333, 334, 335, 336, 342, 343, 344, 345, 346, 350), "rawunearn" + ), + **dict.fromkeys((311, 312, 314, 321), "rawearn"), + 363: "rawrent", + 364: "rawutil", + 365: "rawmedded", + 323: "rawdepded", + 366: "rawcsded", + 150: "rawusize", +} + +FINDING_COLUMNS = tuple( + f"{root}{slot}" + for root in ("AGENCY", "ELEMENT", "NATURE", "AMOUNT") + for slot in SLOTS +) +QC_COLUMNS = ( + "STATE", + "YRMONTH", + "HHLDNO", + "STATUS", + "AMTERR", + "RAWBEN", + "FSBEN", + "HWGT", + *REPLAY_INPUTS.values(), + *FINDING_COLUMNS, +) +WEIGHT_COLUMNS = frozenset({"HWGT", "FYWGT", "HWGT_OLD", "FYWGT_OLD"}) + + +# -------------------------------------------------------------------------- +# small helpers + + +def sha256(path: Path) -> str: + digest = hashlib.sha256() + with path.open("rb") as handle: + for block in iter(lambda: handle.read(1 << 20), b""): + digest.update(block) + return digest.hexdigest() + + +def dollars(value: float) -> float: + return round(float(value), 2) + + +def share(numerator: float, denominator: float) -> float: + return round(float(numerator) / float(denominator), 12) if denominator else 0.0 + + +def points(value: float) -> float: + return round(float(value), 6) + + +def case_key(yrmonth: object, hhldno: object) -> str: + return f"{int(float(yrmonth))}-{int(float(hhldno))}" + + +def code(value: object) -> int | None: + if value is None or (isinstance(value, float) and np.isnan(value)): + return None + number = float(value) + if not number.is_integer(): + raise ValueError(f"fractional code {value!r}") + return int(number) + + +def posting_path(label: str) -> Path: + spec = POSTINGS[label] + return Path(os.environ.get(spec["env"], spec["default"])).expanduser() + + +def load_posting(label: str, *, verify: bool = True) -> pd.DataFrame: + path = posting_path(label) + if not path.exists(): + raise FileNotFoundError( + f"{label} posting not found at {path}; set {POSTINGS[label]['env']} " + f"(download {POSTINGS[label]['zip_url']})" + ) + if verify and sha256(path) != POSTINGS[label]["csv_sha256"]: + raise ValueError(f"{path} is not the {label} posting (sha256 mismatch)") + frame = pd.read_csv(path, usecols=list(QC_COLUMNS), low_memory=False) + frame["key"] = [case_key(y, h) for y, h in zip(frame["YRMONTH"], frame["HHLDNO"])] + return frame + + +def any_code(frame: pd.DataFrame, codes: Iterable[int]) -> pd.Series: + agencies = frame[[f"AGENCY{slot}" for slot in SLOTS]] + return agencies.isin(list(codes)).any(axis=1) + + +def findings(row: pd.Series) -> list[tuple[int, int | None, int | None]]: + """Populated (ELEMENT, NATURE, AGENCY) triples, in slot order.""" + out = [] + for slot in SLOTS: + element = code(row[f"ELEMENT{slot}"]) + if element is None: + continue + out.append((element, code(row[f"NATURE{slot}"]), code(row[f"AGENCY{slot}"]))) + return out + + +def is_computational( + finding: tuple[int, int | None, int | None], + system_codes: frozenset[int] = BROAD_CODES, +) -> bool: + """Layer-2 rule: inherent computation nature or the arithmetic element; + deduction natures only when the same slot's cause is system-side.""" + element, nature, agency = finding + return ( + element == ARITHMETIC_ELEMENT + or nature in INHERENT_COMPUTATION_NATURES + or (nature in DEDUCTION_NATURES and agency in system_codes) + ) + + +def error_cases(frame: pd.DataFrame) -> pd.DataFrame: + out = frame[frame["STATUS"].isin([2, 3])].copy() + out["error_dollars"] = out["HWGT"] * out["AMTERR"] + return out + + +def class_metric(cases: pd.DataFrame, mask: pd.Series, total: float) -> dict: + selected = cases.loc[mask.fillna(False).astype(bool)] + value = float(selected["error_dollars"].sum()) + return { + "n": len(selected), + "dollars": dollars(value), + "share": share(value, total), + } + + +# -------------------------------------------------------------------------- +# posting comparison + + +def posting_diff() -> dict[str, Any]: + """Columns that differ between the postings, over every column.""" + may_path, aug_path = posting_path("may2026"), posting_path("aug2026") + full_may = pd.read_csv(may_path, low_memory=False) + full_aug = pd.read_csv(aug_path, low_memory=False) + if list(full_may.columns) != list(full_aug.columns): + raise AssertionError("postings have different columns") + if len(full_may) != len(full_aug): + raise AssertionError("postings have different row counts") + colorado = full_may["STATE"].eq(COLORADO_FIPS) + changed = {} + for column in full_may.columns: + left, right = full_may[column], full_aug[column] + differs = ~((left == right) | (left.isna() & right.isna())) + if differs.any(): + changed[column] = { + "rows": int(differs.sum()), + "colorado_rows": int(differs[colorado].sum()), + } + if set(changed) - WEIGHT_COLUMNS: + raise AssertionError(f"non-weight columns changed: {sorted(changed)}") + return { + "rows": len(full_may), + "columns": len(full_may.columns), + "rows_in_same_order": bool( + full_may["YRMONTH"].equals(full_aug["YRMONTH"]) + and full_may["HHLDNO"].equals(full_aug["HHLDNO"]) + ), + "changed_columns": changed, + } + + +# -------------------------------------------------------------------------- +# layers 1 and 2: regenerate the archived artifacts + + +def native_decomposition(frame: pd.DataFrame) -> dict[str, dict[str, float]]: + """Rebuild native_decomposition.json (field meanings in ANALYSIS.md).""" + out = {} + for label, part in (("US", frame), ("CO", frame[frame["STATE"] == 8])): + errors = error_cases(part) + weights = errors["HWGT"].to_numpy()[:, None] + agencies = errors[[f"AGENCY{slot}" for slot in SLOTS]] + natures = errors[[f"NATURE{slot}" for slot in SLOTS]] + amounts = errors[[f"AMOUNT{slot}" for slot in SLOTS]].fillna(0).to_numpy() + row: dict[str, float] = { + "benefit_dollars": float((part["FSBEN"] * part["HWGT"]).sum()), + "cases": float(len(part)), + "err_cases": float(len(errors)), + "err_cases_w": float(errors["HWGT"].sum()), + "err_dollars": float(errors["error_dollars"].sum()), + "err_dollars_raw": float(errors["AMTERR"].sum()), + "under_dollars": float( + errors.loc[errors["STATUS"] == 3, "error_dollars"].sum() + ), + "finding_dollars": float((amounts * weights).sum()), + "over_dollars": float( + errors.loc[errors["STATUS"] == 2, "error_dollars"].sum() + ), + } + groups = ( + ("t1", agencies.isin(list(STRICT_CODES))), + ("t2", agencies.isin(list(BROAD_CODES))), + ("nc", natures.isin(list(NC_NATURES))), + ) + for prefix, slot_mask in groups: + case_mask = slot_mask.any(axis=1) + row[f"{prefix}_cases_w"] = float(errors.loc[case_mask, "HWGT"].sum()) + row[f"{prefix}_case_dollars"] = float( + errors.loc[case_mask, "error_dollars"].sum() + ) + row[f"{prefix}_finding_dollars"] = float( + (np.where(slot_mask.to_numpy(), amounts, 0.0) * weights).sum() + ) + out[label] = row + return out + + +def phase_a_classification(frame: pd.DataFrame) -> dict[str, Any]: + """Rebuild ../phase_a_classification.json (Colorado, layer 2).""" + errors = error_cases(frame[frame["STATE"] == COLORADO_FIPS]) + totals = {name: [0, 0.0, 0.0] for name in PHASE_A_CLASSES} + listed: dict[str, list[dict[str, Any]]] = { + "pure_math": [], + "input_system_caused": [], + } + for _, row in errors.sort_values(["YRMONTH", "HHLDNO"]).iterrows(): + found = findings(row) + flags = [is_computational(f) for f in found] + if flags and all(flags): + name = "pure_math" + elif any(flags): + name = "mixed" + elif any(agency in BROAD_CODES for _, _, agency in found): + name = "input_system_caused" + else: + name = "input_other" + totals[name][0] += 1 + totals[name][1] += float(row["HWGT"]) + totals[name][2] += float(row["error_dollars"]) + if name in listed: + listed[name].append( + { + "case": f"2024-{case_key(row['YRMONTH'], row['HHLDNO'])}", + "status": str(int(row["STATUS"])), + "amterr": float(row["AMTERR"]), + "w": float(row["HWGT"]), + "findings": [list(f) for f in found], + } + ) + return {"classes": totals, **listed} + + +def _max_relative_difference(actual: Any, expected: Any) -> float: + if isinstance(expected, dict): + if set(actual) != set(expected): + return float("inf") + return max( + (_max_relative_difference(actual[k], expected[k]) for k in expected), + default=0.0, + ) + if isinstance(expected, list): + if len(actual) != len(expected): + return float("inf") + return max( + (_max_relative_difference(a, e) for a, e in zip(actual, expected)), + default=0.0, + ) + if isinstance(expected, float) or isinstance(actual, float): + scale = max(abs(float(expected)), 1.0) + return abs(float(actual) - float(expected)) / scale + return 0.0 if actual == expected else float("inf") + + +def compare_to_committed(actual: Any, path: Path) -> dict[str, Any]: + expected = json.loads(path.read_text(encoding="utf-8")) + difference = _max_relative_difference(actual, expected) + return { + "path": path.relative_to(REPO).as_posix(), + "sha256": sha256(path), + "regenerated_from_posting": LEGACY_POSTING, + "matches_committed": bool(difference <= 1e-9), + "max_relative_difference": float(f"{difference:.3e}"), + } + + +# -------------------------------------------------------------------------- +# layer 3: the replay + + +def load_replay() -> pd.DataFrame: + replay = pd.DataFrame(json.loads(REPLAY_PATH.read_text(encoding="utf-8"))) + replay["key"] = [ + case_key(y, h) for y, h in zip(replay["yrmonth"], replay["hhldno"]) + ] + reconstruction = pd.read_csv(RECONSTRUCTION_PATH) + reconstruction["key"] = [ + case_key(y, h) + for y, h in zip(reconstruction["YRMONTH"], reconstruction["HHLDNO"]) + ] + keep = reconstruction[ + ["key", "correctedamount", "ELEMENT1", *REPLAY_INPUTS] + ].rename(columns={"ELEMENT1": "solver_element"}) + merged = replay.merge(keep, on="key", how="left", validate="1:1") + if merged["correctedamount"].isna().any(): + raise AssertionError("replay rows missing from the reconstruction") + merged["reproduced"] = merged["within5"].astype(bool) + # The solver moves only the input named by ELEMENT1; "no_change" means the + # element is outside its lists. + merged["solver_no_change"] = merged["correctednotes"].eq("no_change") + return merged + + +def moved_from_keys(replay: pd.DataFrame, unchanged: Iterable[str]) -> pd.Series: + """Rebuild the moved flag from the committed list of unchanged cases.""" + return ~replay["key"].isin(set(unchanged)) + + +def join_replay(replay: pd.DataFrame, posting: pd.DataFrame) -> pd.DataFrame: + colorado = posting[posting["STATE"] == COLORADO_FIPS] + if not colorado["key"].is_unique: + raise AssertionError("Colorado YRMONTH-HHLDNO keys are not unique") + joined = replay.merge(colorado, on="key", how="left", validate="1:1") + if joined["STATE"].isna().any(): + raise AssertionError("replay rows missing from the posting") + for left, right in (("amterr", "AMTERR"), ("rawben", "RAWBEN"), ("fsben", "FSBEN")): + if not np.array_equal(joined[left].astype(float), joined[right].astype(float)): + raise AssertionError(f"replay {left} disagrees with posting {right}") + joined["error_dollars"] = joined["HWGT"] * joined["AMTERR"] + joined["engine_equals_fsben"] = joined["engine_on_original"].eq(joined["FSBEN"]) + # An input moved when any replayed input differs from the file value it + # was initialized from. correctedamount understates this: it is recorded + # before the utility allowance is snapped to a common state value. + changed = pd.concat( + [ + joined[raw].astype(float).ne(joined[field].fillna(0).astype(float)) + for raw, field in REPLAY_INPUTS.items() + ], + axis=1, + ) + joined["solver_moved_input"] = changed.any(axis=1) + if (joined["solver_moved_input"] & joined["solver_no_change"]).any(): + raise AssertionError("a no_change row has a changed input") + return joined + + +def case_findings(joined: pd.DataFrame, mask: pd.Series) -> list[dict[str, Any]]: + rows = [] + for _, row in joined.loc[mask].sort_values("key").iterrows(): + found = findings(row) + broad = [f for f in found if f[2] in BROAD_CODES] + rows.append( + { + "key": row["key"], + "status": int(row["STATUS"]), + "rawben": float(row["RAWBEN"]), + "fsben": float(row["FSBEN"]), + "engine_on_original": float(row["engine_on_original"]), + "amterr": float(row["AMTERR"]), + "reproduced": bool(row["reproduced"]), + "correctednotes": row["correctednotes"], + "solver_moved_input": bool(row["solver_moved_input"]), + "solver_element": int(row["solver_element"]), + "findings": [list(f) for f in found], + "broad_coded_findings": [list(f) for f in broad], + "broad_coded_finding_is_computational": any( + is_computational(f) for f in broad + ), + } + ) + return rows + + +def replay_partition(replay: pd.DataFrame, moved: pd.Series) -> dict[str, Any]: + """Case counts from the replay, the solver output and a moved flag.""" + reproduced = replay["reproduced"] + miss = ~reproduced + trivial = (replay["rawben"] - replay["fsben"]).abs() <= REPLAY_TOLERANCE + engine_equals_fsben = replay["engine_on_original"].eq(replay["fsben"]) + misses = replay.loc[miss] + return { + "n": len(replay), + "reproduced_n": int(reproduced.sum()), + "not_reproduced_n": int(miss.sum()), + "solver_and_engine_agree_n": int( + (replay["within5"] == replay["solver_within5"]).sum() + ), + "reproduced_solver_moved_input_n": int((reproduced & moved).sum()), + "reproduced_without_move_n": int((reproduced & ~moved).sum()), + "reproduced_without_move_rawben_within_5_of_fsben_n": int( + (reproduced & ~moved & trivial).sum() + ), + "not_reproduced_solver_moved_input_n": int((miss & moved).sum()), + "not_reproduced_solver_no_change_n": int(misses["solver_no_change"].sum()), + "not_reproduced_eligible_but_unmoved_n": int( + (miss & ~moved & ~replay["solver_no_change"]).sum() + ), + "not_reproduced_engine_equals_fsben_n": int((miss & engine_equals_fsben).sum()), + "not_reproduced_unmoved_engine_equals_fsben_n": int( + (miss & ~moved & engine_equals_fsben).sum() + ), + "not_reproduced_by_correctednotes": { + note: int(count) + for note, count in sorted(misses["correctednotes"].value_counts().items()) + }, + } + + +def case_level(joined: pd.DataFrame) -> dict[str, Any]: + """Posting-invariant facts: which cases, never how many dollars.""" + miss = ~joined["reproduced"] + broad = any_code(joined, BROAD_CODES) + software = any_code(joined, SOFTWARE_CODES) + + broad_misses = case_findings(joined, miss & broad) + candidates = [ + r["key"] for r in broad_misses if r["broad_coded_finding_is_computational"] + ] + computational_misses = [ + r["key"] + for r in case_findings(joined, miss) + if any(is_computational(tuple(f)) for f in r["findings"]) + ] + computational_matches = [ + r["key"] + for r in case_findings(joined, ~miss) + if any(is_computational(tuple(f)) for f in r["findings"]) + ] + + software_rows = [] + for _, row in joined.loc[software].sort_values("key").iterrows(): + found = findings(row) + software_elements = sorted({f[0] for f in found if f[2] in SOFTWARE_CODES}) + software_inputs = {SOLVER_ELEMENT_INPUTS.get(e) for e in software_elements} + element = int(row["solver_element"]) + if not row["solver_moved_input"]: + what_moved = "nothing" + elif element in software_elements: + what_moved = "software_coded_element" + elif SOLVER_ELEMENT_INPUTS.get(element) in software_inputs: + what_moved = "same_input_other_element" + else: + what_moved = "other_input" + software_rows.append( + { + "key": row["key"], + "reproduced": bool(row["reproduced"]), + "amterr": float(row["AMTERR"]), + "correctednotes": row["correctednotes"], + "solver_element": element, + "solver_input": SOLVER_ELEMENT_INPUTS.get(element), + "software_coded_elements": software_elements, + "solver_moved": what_moved, + "findings": [list(f) for f in found], + } + ) + + example = joined.loc[joined["key"] == "202312-40441"].iloc[0] + example_codes = sorted({f[2] for f in findings(example) if f[2] is not None}) + unchanged = sorted(joined.loc[~joined["solver_moved_input"], "key"]) + snapped = joined["solver_moved_input"] & joined["correctedamount"].eq(0) + snapped &= ~joined["correctednotes"].str.startswith("hhsize") + return { + "replay": replay_partition(joined, joined["solver_moved_input"]), + "replay_inputs_unchanged_keys": unchanged, + "moved_with_zero_correctedamount_keys": sorted(joined.loc[snapped, "key"]), + "broad_coded_misses": { + "codes": sorted(BROAD_CODES), + "n": len(broad_misses), + "engine_equals_fsben_n": sum( + r["engine_on_original"] == r["fsben"] for r in broad_misses + ), + "solver_no_change_n": sum( + r["correctednotes"] == "no_change" for r in broad_misses + ), + "computation_candidate_keys": candidates, + "other_keys": [ + r["key"] for r in broad_misses if r["key"] not in candidates + ], + "cases": broad_misses, + }, + "software_coded_replayed": { + "codes": sorted(SOFTWARE_CODES), + "n": len(software_rows), + "reproduced_n": sum(r["reproduced"] for r in software_rows), + "reproduced_by_what_moved": { + kind: sum( + r["reproduced"] and r["solver_moved"] == kind for r in software_rows + ) + for kind in ( + "software_coded_element", + "same_input_other_element", + "other_input", + "nothing", + ) + }, + "cases": software_rows, + }, + "analysis_example_202312_40441": { + "agency_codes": example_codes, + "in_broad_coded_misses": bool(set(example_codes) & BROAD_CODES), + "correctednotes": example["correctednotes"], + "rawben": float(example["RAWBEN"]), + "fsben": float(example["FSBEN"]), + "engine_on_original": float(example["engine_on_original"]), + }, + "not_reproduced_with_computational_finding_keys": computational_misses, + "reproduced_with_computational_finding_keys": computational_matches, + "not_reproduced_cases": case_findings(joined, miss), + } + + +# -------------------------------------------------------------------------- +# weighted results for one posting + + +def colorado_and_national(frame: pd.DataFrame) -> dict[str, Any]: + out = {} + for label, part in (("CO", frame[frame["STATE"] == COLORADO_FIPS]), ("US", frame)): + errors = error_cases(part) + total = float(errors["error_dollars"].sum()) + issuance = float((part["RAWBEN"] * part["HWGT"]).sum()) + above = errors["AMTERR"] > THRESHOLD_FY2024 + amounts = errors[[f"AMOUNT{slot}" for slot in SLOTS]].fillna(0).to_numpy() + out[label] = { + "cases": len(part), + "error_cases": len(errors), + "finding_amount_dollars": dollars( + (amounts * errors["HWGT"].to_numpy()[:, None]).sum() + ), + "error_dollars": dollars(total), + "error_dollars_above_threshold": dollars( + errors.loc[above, "error_dollars"].sum() + ), + "issuance_rawben_dollars": dollars(issuance), + "calculated_fsben_dollars": dollars((part["FSBEN"] * part["HWGT"]).sum()), + "file_error_rate": share(total, issuance), + "cost_share_step_dollars": dollars(COST_SHARE_STEP * issuance), + "layer1": { + "strict_17_19_20": class_metric( + errors, any_code(errors, STRICT_CODES), total + ), + "broad_10_17_19_20_21_22": class_metric( + errors, any_code(errors, BROAD_CODES), total + ), + }, + "axis1": { + name: class_metric(errors, any_code(errors, codes), total) + for name, codes in AXIS1_CLASSES.items() + }, + "broad_code_case_counts": { + str(c): int(any_code(errors, {c}).sum()) for c in sorted(BROAD_CODES) + }, + "no_cause_code": class_metric( + errors, + errors[[f"AGENCY{slot}" for slot in SLOTS]].isna().all(axis=1), + total, + ), + } + out["CO"]["axis1_ratio_to_national"] = { + name: round( + out["CO"]["axis1"][name]["share"] / out["US"]["axis1"][name]["share"], 6 + ) + for name in AXIS1_CLASSES + } + return out + + +def computational_findings(frame: pd.DataFrame) -> dict[str, Any]: + """Tally the computational findings in layer-2 pure_math and mixed cases.""" + errors = error_cases(frame[frame["STATE"] == COLORADO_FIPS]) + tally: dict[tuple[int, int | None, int | None, str], int] = {} + cases = 0 + for _, row in errors.iterrows(): + found = findings(row) + flags = [is_computational(f) for f in found] + if not any(flags): + continue + cases += 1 + name = "pure_math" if all(flags) else "mixed" + for finding, flag in zip(found, flags): + if flag: + key = (*finding, name) + tally[key] = tally.get(key, 0) + 1 + rows = [ + {"element": e, "nature": n, "agency": a, "class": c, "findings": k} + for (e, n, a, c), k in sorted(tally.items(), key=lambda item: str(item[0])) + ] + return { + "cases": cases, + "findings": sum(r["findings"] for r in rows), + "by_code": rows, + } + + +def reconstruction_rows() -> dict[str, int]: + national = LAB / "fy2024_reconstruction_national.csv" + return { + "colorado_rows": len(pd.read_csv(RECONSTRUCTION_PATH)), + "national_rows": len(pd.read_csv(national, usecols=["STATE"])), + } + + +def phase_a_summary(frame: pd.DataFrame) -> dict[str, Any]: + regenerated = phase_a_classification(frame) + total = sum(v[2] for v in regenerated["classes"].values()) + return { + name: { + "n": values[0], + "dollars": dollars(values[2]), + "share": share(values[2], total), + } + for name, values in regenerated["classes"].items() + } + + +def replay_dollars( + joined: pd.DataFrame, colorado_errors: pd.DataFrame +) -> dict[str, Any]: + replayed_total = float(joined["error_dollars"].sum()) + colorado_total = float(colorado_errors["error_dollars"].sum()) + rate = OFFICIAL_RATES["fy2024"]["CO"] + broad = any_code(joined, BROAD_CODES) + miss = ~joined["reproduced"] + level = case_level(joined) + candidates = set(level["broad_coded_misses"]["computation_candidate_keys"]) + is_candidate = joined["key"].isin(candidates) + computational = joined["key"].isin( + set(level["not_reproduced_with_computational_finding_keys"]) + ) + + above_total = float( + colorado_errors.loc[ + colorado_errors["AMTERR"] > THRESHOLD_FY2024, "error_dollars" + ].sum() + ) + above = joined["AMTERR"] > THRESHOLD_FY2024 + + def metric(mask: pd.Series) -> dict[str, Any]: + value = float(joined.loc[mask, "error_dollars"].sum()) + above_value = float(joined.loc[mask & above, "error_dollars"].sum()) + colorado_share = share(value, colorado_total) + above_share = share(above_value, above_total) + return { + "n": int(mask.sum()), + "dollars": dollars(value), + "share_of_replayed": share(value, replayed_total), + "share_of_colorado_error_dollars": colorado_share, + "points_of_official_fy2024_rate": points(colorado_share * rate), + "above_threshold_n": int((mask & above).sum()), + "share_of_colorado_above_threshold_error_dollars": above_share, + "above_threshold_points_of_official_fy2024_rate": points( + above_share * rate + ), + } + + gap = (joined["engine_on_original"] - joined["RAWBEN"]).abs() * joined["HWGT"] + return { + "replayed": metric(pd.Series(True, index=joined.index)), + "reproduced": metric(joined["reproduced"]), + "not_reproduced": metric(miss), + "broad_coded_misses": { + **metric(miss & broad), + "engine_gap_dollars": dollars(gap[miss & broad].sum()), + }, + "computation_candidates": metric(is_candidate), + "not_reproduced_with_computational_finding": metric(computational), + "reproduced_with_computational_finding": metric( + joined["key"].isin(set(level["reproduced_with_computational_finding_keys"])) + ), + } + + +def broad_split(joined: pd.DataFrame, colorado_errors: pd.DataFrame) -> dict[str, Any]: + """Where the any-presence broad and software classes land in the replay.""" + rate = OFFICIAL_RATES["fy2024"]["CO"] + lookup = joined.set_index("key")["reproduced"] + out = {} + for name, codes in ( + ("broad_10_17_19_20_21_22", BROAD_CODES), + ("software_17_19", SOFTWARE_CODES), + ("other_10_20_21_22", frozenset({10, 20, 21, 22})), + ): + for convention, errors in ( + ("all_errors", colorado_errors), + ( + "above_threshold", + colorado_errors[colorado_errors["AMTERR"] > THRESHOLD_FY2024], + ), + ): + total = float(errors["error_dollars"].sum()) + in_class = any_code(errors, codes) + replay_status = errors["key"].map(lookup) + parts = { + "all": in_class, + "reproduced": in_class & replay_status.eq(True), + "not_reproduced": in_class & replay_status.eq(False), + "not_replayed": in_class & replay_status.isna(), + } + out.setdefault(name, {})[convention] = { + part: { + **class_metric(errors, mask, total), + "points_of_official_fy2024_rate": points( + share(errors.loc[mask, "error_dollars"].sum(), total) * rate + ), + } + for part, mask in parts.items() + } + return out + + +def software_dollars(joined: pd.DataFrame) -> dict[str, Any]: + software = any_code(joined, SOFTWARE_CODES) + cases = case_level(joined)["software_coded_replayed"]["cases"] + moved = joined["key"].map({r["key"]: r["solver_moved"] for r in cases}) + parts = [ + ("reproduced", software & joined["reproduced"]), + ("not_reproduced", software & ~joined["reproduced"]), + ] + for kind in ("software_coded_element", "same_input_other_element", "other_input"): + parts.append( + ( + f"reproduced_solver_moved_{kind}", + software & joined["reproduced"] & moved.eq(kind), + ) + ) + return { + part: { + "n": int(mask.sum()), + "dollars": dollars(joined.loc[mask, "error_dollars"].sum()), + } + for part, mask in parts + } + + +def weighted_results(frame: pd.DataFrame, replay: pd.DataFrame) -> dict[str, Any]: + joined = join_replay(replay, frame) + colorado_errors = error_cases(frame[frame["STATE"] == COLORADO_FIPS]) + return { + "totals": colorado_and_national(frame), + "layer2": phase_a_summary(frame), + "replay": replay_dollars(joined, colorado_errors), + "class_by_replay_outcome": broad_split(joined, colorado_errors), + "software_coded_replayed": software_dollars(joined), + } + + +# -------------------------------------------------------------------------- +# artifact + + +def build(postings: Iterable[str] = tuple(POSTINGS)) -> dict[str, Any]: + postings = tuple(postings) + frames = {label: load_posting(label) for label in postings} + replay = load_replay() + legacy = frames[LEGACY_POSTING] + artifact: dict[str, Any] = { + "schema": "amterr-lab-claims-audit/1", + "inputs": { + "committed": { + path.relative_to(REPO).as_posix(): sha256(path) + for path in (REPLAY_PATH, RECONSTRUCTION_PATH) + }, + "postings": { + label: { + k: v + for k, v in POSTINGS[label].items() + if k not in ("env", "default") + } + for label in postings + }, + "official_rates_percent": OFFICIAL_RATES, + }, + "conventions": { + "error_case": "STATUS in {2,3}", + "error_dollars": "sum(HWGT * AMTERR); the FY file pools 12 monthly samples, so the sum is annual", + "issuance": "sum(HWGT * RAWBEN) over all records", + "code_class": "a case counts if any of AGENCY1-AGENCY9 is in the class (classes overlap)", + "reproduced": f"abs(engine_on_original - RAWBEN) <= {REPLAY_TOLERANCE}", + "solver_moved_input": ( + "any replayed input (" + + ", ".join(REPLAY_INPUTS) + + ") differs from its file field (" + + ", ".join(REPLAY_INPUTS.values()) + + "; missing -> 0)" + ), + "computation_candidate": ( + "not reproduced, and a finding carrying a code in " + f"{sorted(BROAD_CODES)} is computational under the layer-2 rule" + ), + "points_of_official_fy2024_rate": ( + "share of file error dollars x 9.97; an approximation, since the " + "official rate is regression-adjusted and counts only errors above " + f"the ${THRESHOLD_FY2024} threshold" + ), + }, + "regenerated_artifacts": { + "native_decomposition": compare_to_committed( + native_decomposition(legacy), NATIVE_PATH + ), + "phase_a_classification": compare_to_committed( + phase_a_classification(legacy), PHASE_A_PATH + ), + }, + "case_level": { + **case_level(join_replay(replay, legacy)), + "layer2_computational_findings": computational_findings(legacy), + "reconstruction": reconstruction_rows(), + }, + "by_posting": { + label: weighted_results(frame, replay) for label, frame in frames.items() + }, + } + if {"may2026", "aug2026"} <= set(postings): + artifact["posting_diff"] = posting_diff() + for label in postings: + level = { + **case_level(join_replay(replay, frames[label])), + "layer2_computational_findings": computational_findings(frames[label]), + "reconstruction": artifact["case_level"]["reconstruction"], + } + if level != artifact["case_level"]: + raise AssertionError(f"case-level results differ under {label}") + return artifact + + +def render(artifact: dict[str, Any]) -> str: + return json.dumps(artifact, indent=1, sort_keys=False) + "\n" + + +def main(argv: Iterable[str] | None = None) -> int: + parser = argparse.ArgumentParser(description=__doc__.split("\n\n")[0]) + parser.add_argument( + "--check", + action="store_true", + help="exit 1 if the regenerated audit differs from claims_audit.json", + ) + args = parser.parse_args(list(argv) if argv is not None else None) + text = render(build()) + if args.check: + current = ( + OUTPUT_PATH.read_text(encoding="utf-8") if OUTPUT_PATH.exists() else "" + ) + if current != text: + print(f"{OUTPUT_PATH.relative_to(REPO)} is stale", file=sys.stderr) + return 1 + print(f"{OUTPUT_PATH.relative_to(REPO)} is current") + return 0 + OUTPUT_PATH.write_text(text, encoding="utf-8") + print(f"wrote {OUTPUT_PATH.relative_to(REPO)}") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/paper/snapshot/labs/amterr/claims_audit.json b/paper/snapshot/labs/amterr/claims_audit.json new file mode 100644 index 0000000..7be629d --- /dev/null +++ b/paper/snapshot/labs/amterr/claims_audit.json @@ -0,0 +1,2919 @@ +{ + "schema": "amterr-lab-claims-audit/1", + "inputs": { + "committed": { + "paper/snapshot/labs/amterr/amterr_replay_results.json": "3e214d2161269a8e5f1dad882aeec4dc0f7e35d957b6f836ea3170907b3368c6", + "paper/snapshot/labs/amterr/co_fy2024_reconstruction.csv": "c4ccbb4f06861132fefad304005b9c599df8375b9ab057988b674352951887df" + }, + "postings": { + "may2026": { + "zip_url": "https://snapqcdata.net/sites/default/files/2026-05/qcfy2024_csv.zip", + "zip_sha256": "0f3230a4318307d3088382546095eebfde03e781da6f65c9eac7f077bd4263f4", + "csv_sha256": "45193eb7370463ab3067d71da23a580fec34a5460341e4e750dda0be061e1aa9", + "last_modified": "2026-05-21", + "role": "weights used by the July 2026 lab run" + }, + "aug2026": { + "zip_url": "https://snapqcdata.net/sites/default/files/2026-08/qcfy2024_csv.zip", + "zip_sha256": "b8b29b8593f78aa51c48332c47d2d92fa5bbecf5346570acb45e26f2d9ebd2b5", + "csv_sha256": "e871a8e9caca0be72e2003b09bdf71e1d020984b52289d2b74c4c6b88c4f793b", + "last_modified": "2026-08-18", + "role": "current posting; corrects HWGT and FYWGT" + } + }, + "official_rates_percent": { + "fy2024": { + "CO": 9.97, + "US": 10.93, + "dated": "2025-06-30" + }, + "fy2025": { + "CO": 10.09, + "US": 10.62, + "dated": "2026-06-24" + } + } + }, + "conventions": { + "error_case": "STATUS in {2,3}", + "error_dollars": "sum(HWGT * AMTERR); the FY file pools 12 monthly samples, so the sum is annual", + "issuance": "sum(HWGT * RAWBEN) over all records", + "code_class": "a case counts if any of AGENCY1-AGENCY9 is in the class (classes overlap)", + "reproduced": "abs(engine_on_original - RAWBEN) <= 5", + "solver_moved_input": "any replayed input (rawusize, rawearn, rawunearn, rawrent, rawutil, rawmedded, rawdepded, rawcsded) differs from its file field (FSUSIZE, FSEARN, FSUNEARN, RENT, UTIL, FSMEDDED, FSDEPDED, FSCSDED; missing -> 0)", + "computation_candidate": "not reproduced, and a finding carrying a code in [10, 17, 19, 20, 21, 22] is computational under the layer-2 rule", + "points_of_official_fy2024_rate": "share of file error dollars x 9.97; an approximation, since the official rate is regression-adjusted and counts only errors above the $56 threshold" + }, + "regenerated_artifacts": { + "native_decomposition": { + "path": "paper/snapshot/labs/amterr/native_decomposition.json", + "sha256": "5c151c2946ed0b41fe033f1a9d5cb96871c07718f4d320504f21fda93f71285c", + "regenerated_from_posting": "may2026", + "matches_committed": true, + "max_relative_difference": 1.563e-14 + }, + "phase_a_classification": { + "path": "paper/snapshot/labs/phase_a_classification.json", + "sha256": "e46fe569ccf714f7f795b72ef8144b80e77d6090719dc932a7ae9ae6c26f5b79", + "regenerated_from_posting": "may2026", + "matches_committed": true, + "max_relative_difference": 0.0 + } + }, + "case_level": { + "replay": { + "n": 283, + "reproduced_n": 246, + "not_reproduced_n": 37, + "solver_and_engine_agree_n": 283, + "reproduced_solver_moved_input_n": 230, + "reproduced_without_move_n": 16, + "reproduced_without_move_rawben_within_5_of_fsben_n": 16, + "not_reproduced_solver_moved_input_n": 20, + "not_reproduced_solver_no_change_n": 14, + "not_reproduced_eligible_but_unmoved_n": 3, + "not_reproduced_engine_equals_fsben_n": 21, + "not_reproduced_unmoved_engine_equals_fsben_n": 17, + "not_reproduced_by_correctednotes": { + "earn_down": 3, + "earn_error": 3, + "hhsize_down": 2, + "hhsize_up": 3, + "no_change": 14, + "rent_down": 2, + "rent_error": 2, + "unearn_error": 1, + "util_down": 2, + "util_up": 5 + } + }, + "replay_inputs_unchanged_keys": [ + "202310-40314", + "202310-40336", + "202312-40441", + "202312-40466", + "202312-40481", + "202401-40518", + "202401-40549", + "202401-40566", + "202401-40592", + "202402-40607", + "202402-40633", + "202403-40689", + "202404-40798", + "202404-40799", + "202404-40800", + "202404-40818", + "202405-40877", + "202405-40910", + "202405-40916", + "202406-40986", + "202406-40990", + "202406-41064", + "202407-41082", + "202407-41087", + "202407-41100", + "202407-41125", + "202407-41127", + "202407-41141", + "202407-41159", + "202407-41178", + "202408-41196", + "202408-41271", + "202408-41272" + ], + "moved_with_zero_correctedamount_keys": [ + "202403-40765", + "202404-40803" + ], + "broad_coded_misses": { + "codes": [ + 10, + 17, + 19, + 20, + 21, + 22 + ], + "n": 10, + "engine_equals_fsben_n": 7, + "solver_no_change_n": 7, + "computation_candidate_keys": [ + "202310-40336", + "202312-40456", + "202312-40466", + "202401-40518", + "202405-40910", + "202407-41178", + "202408-41271" + ], + "other_keys": [ + "202312-40513", + "202401-40548", + "202406-40990" + ], + "cases": [ + { + "key": "202310-40336", + "status": 3, + "rawben": 150.0, + "fsben": 291.0, + "engine_on_original": 291.0, + "amterr": 141.0, + "reproduced": false, + "correctednotes": "no_change", + "solver_moved_input": false, + "solver_element": 520, + "findings": [ + [ + 520, + 80, + 10 + ] + ], + "broad_coded_findings": [ + [ + 520, + 80, + 10 + ] + ], + "broad_coded_finding_is_computational": true + }, + { + "key": "202312-40456", + "status": 2, + "rawben": 146.0, + "fsben": 75.0, + "engine_on_original": 102.0, + "amterr": 71.0, + "reproduced": false, + "correctednotes": "util_up", + "solver_moved_input": true, + "solver_element": 364, + "findings": [ + [ + 364, + 53, + 12 + ], + [ + 520, + 80, + 10 + ], + [ + 363, + 57, + 2 + ] + ], + "broad_coded_findings": [ + [ + 520, + 80, + 10 + ] + ], + "broad_coded_finding_is_computational": true + }, + { + "key": "202312-40466", + "status": 3, + "rawben": 281.0, + "fsben": 291.0, + "engine_on_original": 291.0, + "amterr": 10.0, + "reproduced": false, + "correctednotes": "no_change", + "solver_moved_input": false, + "solver_element": 520, + "findings": [ + [ + 520, + 75, + 20 + ] + ], + "broad_coded_findings": [ + [ + 520, + 75, + 20 + ] + ], + "broad_coded_finding_is_computational": true + }, + { + "key": "202312-40513", + "status": 2, + "rawben": 264.0, + "fsben": 103.0, + "engine_on_original": 244.0, + "amterr": 161.0, + "reproduced": false, + "correctednotes": "util_up", + "solver_moved_input": true, + "solver_element": 364, + "findings": [ + [ + 364, + 53, + 2 + ], + [ + 331, + 44, + 17 + ], + [ + 363, + 56, + 2 + ] + ], + "broad_coded_findings": [ + [ + 331, + 44, + 17 + ] + ], + "broad_coded_finding_is_computational": false + }, + { + "key": "202401-40518", + "status": 3, + "rawben": 132.0, + "fsben": 171.0, + "engine_on_original": 171.0, + "amterr": 39.0, + "reproduced": false, + "correctednotes": "no_change", + "solver_moved_input": false, + "solver_element": 520, + "findings": [ + [ + 520, + 80, + 10 + ] + ], + "broad_coded_findings": [ + [ + 520, + 80, + 10 + ] + ], + "broad_coded_finding_is_computational": true + }, + { + "key": "202401-40548", + "status": 3, + "rawben": 594.0, + "fsben": 886.0, + "engine_on_original": 746.0, + "amterr": 292.0, + "reproduced": false, + "correctednotes": "rent_down", + "solver_moved_input": true, + "solver_element": 363, + "findings": [ + [ + 363, + 53, + 14 + ], + [ + 366, + 52, + 12 + ], + [ + 350, + 44, + 17 + ] + ], + "broad_coded_findings": [ + [ + 350, + 44, + 17 + ] + ], + "broad_coded_finding_is_computational": false + }, + { + "key": "202405-40910", + "status": 3, + "rawben": 95.0, + "fsben": 149.0, + "engine_on_original": 149.0, + "amterr": 54.0, + "reproduced": false, + "correctednotes": "no_change", + "solver_moved_input": false, + "solver_element": 362, + "findings": [ + [ + 362, + 52, + 10 + ] + ], + "broad_coded_findings": [ + [ + 362, + 52, + 10 + ] + ], + "broad_coded_finding_is_computational": true + }, + { + "key": "202406-40990", + "status": 3, + "rawben": 125.0, + "fsben": 161.0, + "engine_on_original": 161.0, + "amterr": 36.0, + "reproduced": false, + "correctednotes": "no_change", + "solver_moved_input": false, + "solver_element": 161, + "findings": [ + [ + 161, + 6, + 10 + ], + [ + 311, + 97, + 26 + ] + ], + "broad_coded_findings": [ + [ + 161, + 6, + 10 + ] + ], + "broad_coded_finding_is_computational": false + }, + { + "key": "202407-41178", + "status": 3, + "rawben": 187.0, + "fsben": 291.0, + "engine_on_original": 291.0, + "amterr": 104.0, + "reproduced": false, + "correctednotes": "no_change", + "solver_moved_input": false, + "solver_element": 520, + "findings": [ + [ + 520, + 80, + 10 + ], + [ + 363, + 53, + 2 + ] + ], + "broad_coded_findings": [ + [ + 520, + 80, + 10 + ] + ], + "broad_coded_finding_is_computational": true + }, + { + "key": "202408-41271", + "status": 2, + "rawben": 240.0, + "fsben": 227.0, + "engine_on_original": 227.0, + "amterr": 13.0, + "reproduced": false, + "correctednotes": "no_change", + "solver_moved_input": false, + "solver_element": 520, + "findings": [ + [ + 520, + 98, + 20 + ] + ], + "broad_coded_findings": [ + [ + 520, + 98, + 20 + ] + ], + "broad_coded_finding_is_computational": true + } + ] + }, + "software_coded_replayed": { + "codes": [ + 17, + 19 + ], + "n": 16, + "reproduced_n": 14, + "reproduced_by_what_moved": { + "software_coded_element": 10, + "same_input_other_element": 2, + "other_input": 2, + "nothing": 0 + }, + "cases": [ + { + "key": "202311-40371", + "reproduced": true, + "amterr": 139.0, + "correctednotes": "unearn_up", + "solver_element": 331, + "solver_input": "rawunearn", + "software_coded_elements": [ + 331 + ], + "solver_moved": "software_coded_element", + "findings": [ + [ + 331, + 44, + 17 + ], + [ + 363, + 57, + 2 + ] + ] + }, + { + "key": "202312-40513", + "reproduced": false, + "amterr": 161.0, + "correctednotes": "util_up", + "solver_element": 364, + "solver_input": "rawutil", + "software_coded_elements": [ + 331 + ], + "solver_moved": "other_input", + "findings": [ + [ + 364, + 53, + 2 + ], + [ + 331, + 44, + 17 + ], + [ + 363, + 56, + 2 + ] + ] + }, + { + "key": "202401-40548", + "reproduced": false, + "amterr": 292.0, + "correctednotes": "rent_down", + "solver_element": 363, + "solver_input": "rawrent", + "software_coded_elements": [ + 350 + ], + "solver_moved": "other_input", + "findings": [ + [ + 363, + 53, + 14 + ], + [ + 366, + 52, + 12 + ], + [ + 350, + 44, + 17 + ] + ] + }, + { + "key": "202401-40551", + "reproduced": true, + "amterr": 116.0, + "correctednotes": "rent_down", + "solver_element": 363, + "solver_input": "rawrent", + "software_coded_elements": [ + 333 + ], + "solver_moved": "other_input", + "findings": [ + [ + 363, + 52, + 1 + ], + [ + 333, + 37, + 19 + ] + ] + }, + { + "key": "202401-40577", + "reproduced": true, + "amterr": 9.0, + "correctednotes": "unearn_down", + "solver_element": 333, + "solver_input": "rawunearn", + "software_coded_elements": [ + 333 + ], + "solver_moved": "software_coded_element", + "findings": [ + [ + 333, + 38, + 17 + ] + ] + }, + { + "key": "202403-40730", + "reproduced": true, + "amterr": 13.0, + "correctednotes": "unearn_down", + "solver_element": 331, + "solver_input": "rawunearn", + "software_coded_elements": [ + 331 + ], + "solver_moved": "software_coded_element", + "findings": [ + [ + 331, + 38, + 19 + ], + [ + 363, + 56, + 2 + ] + ] + }, + { + "key": "202403-40768", + "reproduced": true, + "amterr": 7.0, + "correctednotes": "unearn_down", + "solver_element": 346, + "solver_input": "rawunearn", + "software_coded_elements": [ + 331 + ], + "solver_moved": "same_input_other_element", + "findings": [ + [ + 346, + 97, + 26 + ], + [ + 364, + 97, + 26 + ], + [ + 331, + 44, + 17 + ], + [ + 365, + 57, + 15 + ] + ] + }, + { + "key": "202404-40794", + "reproduced": true, + "amterr": 128.0, + "correctednotes": "cs_up", + "solver_element": 366, + "solver_input": "rawcsded", + "software_coded_elements": [ + 366 + ], + "solver_moved": "software_coded_element", + "findings": [ + [ + 366, + 56, + 17 + ] + ] + }, + { + "key": "202404-40805", + "reproduced": true, + "amterr": 453.0, + "correctednotes": "unearn_down", + "solver_element": 331, + "solver_input": "rawunearn", + "software_coded_elements": [ + 350 + ], + "solver_moved": "same_input_other_element", + "findings": [ + [ + 331, + 35, + 1 + ], + [ + 350, + 112, + 17 + ], + [ + 363, + 57, + 2 + ] + ] + }, + { + "key": "202404-40823", + "reproduced": true, + "amterr": 41.0, + "correctednotes": "cs_up", + "solver_element": 366, + "solver_input": "rawcsded", + "software_coded_elements": [ + 366 + ], + "solver_moved": "software_coded_element", + "findings": [ + [ + 366, + 56, + 17 + ], + [ + 363, + 56, + 2 + ] + ] + }, + { + "key": "202404-40846", + "reproduced": true, + "amterr": 35.0, + "correctednotes": "unearn_up", + "solver_element": 331, + "solver_input": "rawunearn", + "software_coded_elements": [ + 331, + 365 + ], + "solver_moved": "software_coded_element", + "findings": [ + [ + 331, + 38, + 19 + ], + [ + 365, + 52, + 19 + ], + [ + 363, + 56, + 2 + ] + ] + }, + { + "key": "202405-40904", + "reproduced": true, + "amterr": 16.0, + "correctednotes": "unearn_down", + "solver_element": 331, + "solver_input": "rawunearn", + "software_coded_elements": [ + 331 + ], + "solver_moved": "software_coded_element", + "findings": [ + [ + 331, + 38, + 19 + ], + [ + 363, + 97, + 26 + ] + ] + }, + { + "key": "202406-40991", + "reproduced": true, + "amterr": 3.0, + "correctednotes": "unearn_up", + "solver_element": 350, + "solver_input": "rawunearn", + "software_coded_elements": [ + 350 + ], + "solver_moved": "software_coded_element", + "findings": [ + [ + 350, + 44, + 17 + ] + ] + }, + { + "key": "202407-41112", + "reproduced": true, + "amterr": 54.0, + "correctednotes": "rent_up", + "solver_element": 363, + "solver_input": "rawrent", + "software_coded_elements": [ + 350 + ], + "solver_moved": "other_input", + "findings": [ + [ + 363, + 53, + 2 + ], + [ + 350, + 38, + 17 + ] + ] + }, + { + "key": "202408-41201", + "reproduced": true, + "amterr": 49.0, + "correctednotes": "unearn_down", + "solver_element": 331, + "solver_input": "rawunearn", + "software_coded_elements": [ + 331, + 333 + ], + "solver_moved": "software_coded_element", + "findings": [ + [ + 331, + 38, + 19 + ], + [ + 333, + 38, + 19 + ] + ] + }, + { + "key": "202408-41206", + "reproduced": true, + "amterr": 30.0, + "correctednotes": "unearn_up", + "solver_element": 350, + "solver_input": "rawunearn", + "software_coded_elements": [ + 350 + ], + "solver_moved": "software_coded_element", + "findings": [ + [ + 350, + 44, + 17 + ] + ] + } + ] + }, + "analysis_example_202312_40441": { + "agency_codes": [ + 15 + ], + "in_broad_coded_misses": false, + "correctednotes": "no_change", + "rawben": 704.0, + "fsben": 973.0, + "engine_on_original": 973.0 + }, + "not_reproduced_with_computational_finding_keys": [ + "202310-40297", + "202310-40336", + "202312-40441", + "202312-40456", + "202312-40466", + "202401-40518", + "202401-40566", + "202405-40910", + "202407-41178", + "202408-41271" + ], + "reproduced_with_computational_finding_keys": [ + "202310-40290", + "202312-40490", + "202312-40509", + "202402-40643", + "202403-40716", + "202404-40794", + "202404-40823", + "202404-40846", + "202406-40984", + "202407-41132", + "202407-41153", + "202409-41363", + "202409-41371" + ], + "not_reproduced_cases": [ + { + "key": "202310-40265", + "status": 2, + "rawben": 108.0, + "fsben": 42.0, + "engine_on_original": 0.0, + "amterr": 66.0, + "reproduced": false, + "correctednotes": "hhsize_up", + "solver_moved_input": true, + "solver_element": 150, + "findings": [ + [ + 150, + 12, + 2 + ], + [ + 363, + 53, + 2 + ] + ], + "broad_coded_findings": [], + "broad_coded_finding_is_computational": false + }, + { + "key": "202310-40297", + "status": 2, + "rawben": 619.0, + "fsben": 532.0, + "engine_on_original": 658.0, + "amterr": 87.0, + "reproduced": false, + "correctednotes": "util_up", + "solver_moved_input": true, + "solver_element": 364, + "findings": [ + [ + 364, + 54, + 18 + ] + ], + "broad_coded_findings": [], + "broad_coded_finding_is_computational": false + }, + { + "key": "202310-40304", + "status": 2, + "rawben": 101.0, + "fsben": 23.0, + "engine_on_original": 23.0, + "amterr": 78.0, + "reproduced": false, + "correctednotes": "earn_down", + "solver_moved_input": true, + "solver_element": 311, + "findings": [ + [ + 311, + 35, + 1 + ] + ], + "broad_coded_findings": [], + "broad_coded_finding_is_computational": false + }, + { + "key": "202310-40336", + "status": 3, + "rawben": 150.0, + "fsben": 291.0, + "engine_on_original": 291.0, + "amterr": 141.0, + "reproduced": false, + "correctednotes": "no_change", + "solver_moved_input": false, + "solver_element": 520, + "findings": [ + [ + 520, + 80, + 10 + ] + ], + "broad_coded_findings": [ + [ + 520, + 80, + 10 + ] + ], + "broad_coded_finding_is_computational": true + }, + { + "key": "202311-40369", + "status": 2, + "rawben": 897.0, + "fsben": 452.0, + "engine_on_original": 766.0, + "amterr": 445.0, + "reproduced": false, + "correctednotes": "earn_error", + "solver_moved_input": true, + "solver_element": 312, + "findings": [ + [ + 312, + 38, + 12 + ], + [ + 111, + 7, + 18 + ], + [ + 363, + 53, + 2 + ], + [ + 346, + 35, + 2 + ] + ], + "broad_coded_findings": [], + "broad_coded_finding_is_computational": false + }, + { + "key": "202311-40407", + "status": 3, + "rawben": 23.0, + "fsben": 973.0, + "engine_on_original": 766.0, + "amterr": 950.0, + "reproduced": false, + "correctednotes": "hhsize_up", + "solver_moved_input": true, + "solver_element": 150, + "findings": [ + [ + 150, + 12, + 18 + ], + [ + 331, + 44, + 14 + ], + [ + 342, + 35, + 14 + ], + [ + 363, + 53, + 14 + ] + ], + "broad_coded_findings": [], + "broad_coded_finding_is_computational": false + }, + { + "key": "202312-40441", + "status": 3, + "rawben": 704.0, + "fsben": 973.0, + "engine_on_original": 973.0, + "amterr": 269.0, + "reproduced": false, + "correctednotes": "no_change", + "solver_moved_input": false, + "solver_element": 520, + "findings": [ + [ + 520, + 75, + 15 + ] + ], + "broad_coded_findings": [], + "broad_coded_finding_is_computational": false + }, + { + "key": "202312-40456", + "status": 2, + "rawben": 146.0, + "fsben": 75.0, + "engine_on_original": 102.0, + "amterr": 71.0, + "reproduced": false, + "correctednotes": "util_up", + "solver_moved_input": true, + "solver_element": 364, + "findings": [ + [ + 364, + 53, + 12 + ], + [ + 520, + 80, + 10 + ], + [ + 363, + 57, + 2 + ] + ], + "broad_coded_findings": [ + [ + 520, + 80, + 10 + ] + ], + "broad_coded_finding_is_computational": true + }, + { + "key": "202312-40466", + "status": 3, + "rawben": 281.0, + "fsben": 291.0, + "engine_on_original": 291.0, + "amterr": 10.0, + "reproduced": false, + "correctednotes": "no_change", + "solver_moved_input": false, + "solver_element": 520, + "findings": [ + [ + 520, + 75, + 20 + ] + ], + "broad_coded_findings": [ + [ + 520, + 75, + 20 + ] + ], + "broad_coded_finding_is_computational": true + }, + { + "key": "202312-40481", + "status": 2, + "rawben": 738.0, + "fsben": 685.0, + "engine_on_original": 685.0, + "amterr": 54.0, + "reproduced": false, + "correctednotes": "no_change", + "solver_moved_input": false, + "solver_element": 362, + "findings": [ + [ + 362, + 53, + 2 + ] + ], + "broad_coded_findings": [], + "broad_coded_finding_is_computational": false + }, + { + "key": "202312-40513", + "status": 2, + "rawben": 264.0, + "fsben": 103.0, + "engine_on_original": 244.0, + "amterr": 161.0, + "reproduced": false, + "correctednotes": "util_up", + "solver_moved_input": true, + "solver_element": 364, + "findings": [ + [ + 364, + 53, + 2 + ], + [ + 331, + 44, + 17 + ], + [ + 363, + 56, + 2 + ] + ], + "broad_coded_findings": [ + [ + 331, + 44, + 17 + ] + ], + "broad_coded_finding_is_computational": false + }, + { + "key": "202401-40518", + "status": 3, + "rawben": 132.0, + "fsben": 171.0, + "engine_on_original": 171.0, + "amterr": 39.0, + "reproduced": false, + "correctednotes": "no_change", + "solver_moved_input": false, + "solver_element": 520, + "findings": [ + [ + 520, + 80, + 10 + ] + ], + "broad_coded_findings": [ + [ + 520, + 80, + 10 + ] + ], + "broad_coded_finding_is_computational": true + }, + { + "key": "202401-40546", + "status": 3, + "rawben": 241.0, + "fsben": 291.0, + "engine_on_original": 291.0, + "amterr": 50.0, + "reproduced": false, + "correctednotes": "util_down", + "solver_moved_input": true, + "solver_element": 364, + "findings": [ + [ + 364, + 97, + null + ] + ], + "broad_coded_findings": [], + "broad_coded_finding_is_computational": false + }, + { + "key": "202401-40548", + "status": 3, + "rawben": 594.0, + "fsben": 886.0, + "engine_on_original": 746.0, + "amterr": 292.0, + "reproduced": false, + "correctednotes": "rent_down", + "solver_moved_input": true, + "solver_element": 363, + "findings": [ + [ + 363, + 53, + 14 + ], + [ + 366, + 52, + 12 + ], + [ + 350, + 44, + 17 + ] + ], + "broad_coded_findings": [ + [ + 350, + 44, + 17 + ] + ], + "broad_coded_finding_is_computational": false + }, + { + "key": "202401-40549", + "status": 3, + "rawben": 651.0, + "fsben": 706.0, + "engine_on_original": 706.0, + "amterr": 55.0, + "reproduced": false, + "correctednotes": "rent_down", + "solver_moved_input": false, + "solver_element": 363, + "findings": [ + [ + 363, + 56, + 2 + ], + [ + 311, + 97, + 26 + ] + ], + "broad_coded_findings": [], + "broad_coded_finding_is_computational": false + }, + { + "key": "202401-40566", + "status": 3, + "rawben": 213.0, + "fsben": 228.0, + "engine_on_original": 228.0, + "amterr": 15.0, + "reproduced": false, + "correctednotes": "no_change", + "solver_moved_input": false, + "solver_element": 520, + "findings": [ + [ + 520, + 75, + 15 + ] + ], + "broad_coded_findings": [], + "broad_coded_finding_is_computational": false + }, + { + "key": "202401-40592", + "status": 3, + "rawben": 189.0, + "fsben": 535.0, + "engine_on_original": 535.0, + "amterr": 346.0, + "reproduced": false, + "correctednotes": "no_change", + "solver_moved_input": false, + "solver_element": 150, + "findings": [ + [ + 150, + 13, + 1 + ], + [ + 342, + 44, + 12 + ] + ], + "broad_coded_findings": [], + "broad_coded_finding_is_computational": false + }, + { + "key": "202402-40603", + "status": 2, + "rawben": 187.0, + "fsben": 67.0, + "engine_on_original": 205.0, + "amterr": 120.0, + "reproduced": false, + "correctednotes": "util_up", + "solver_moved_input": true, + "solver_element": 364, + "findings": [ + [ + 364, + 53, + 12 + ] + ], + "broad_coded_findings": [], + "broad_coded_finding_is_computational": false + }, + { + "key": "202402-40618", + "status": 3, + "rawben": 421.0, + "fsben": 766.0, + "engine_on_original": 973.0, + "amterr": 345.0, + "reproduced": false, + "correctednotes": "hhsize_down", + "solver_moved_input": true, + "solver_element": 150, + "findings": [ + [ + 150, + 7, + 12 + ] + ], + "broad_coded_findings": [], + "broad_coded_finding_is_computational": false + }, + { + "key": "202403-40689", + "status": 3, + "rawben": 276.0, + "fsben": 291.0, + "engine_on_original": 291.0, + "amterr": 15.0, + "reproduced": false, + "correctednotes": "no_change", + "solver_moved_input": false, + "solver_element": 362, + "findings": [ + [ + 362, + 52, + 12 + ] + ], + "broad_coded_findings": [], + "broad_coded_finding_is_computational": false + }, + { + "key": "202403-40738", + "status": 3, + "rawben": 525.0, + "fsben": 819.0, + "engine_on_original": 673.0, + "amterr": 294.0, + "reproduced": false, + "correctednotes": "hhsize_up", + "solver_moved_input": true, + "solver_element": 150, + "findings": [ + [ + 150, + 12, + 14 + ], + [ + 160, + 6, + 14 + ], + [ + 311, + 44, + 14 + ] + ], + "broad_coded_findings": [], + "broad_coded_finding_is_computational": false + }, + { + "key": "202404-40818", + "status": 2, + "rawben": 885.0, + "fsben": 831.0, + "engine_on_original": 831.0, + "amterr": 54.0, + "reproduced": false, + "correctednotes": "no_change", + "solver_moved_input": false, + "solver_element": 362, + "findings": [ + [ + 362, + 53, + 18 + ] + ], + "broad_coded_findings": [], + "broad_coded_finding_is_computational": false + }, + { + "key": "202405-40908", + "status": 3, + "rawben": 184.0, + "fsben": 192.0, + "engine_on_original": 130.0, + "amterr": 7.0, + "reproduced": false, + "correctednotes": "util_down", + "solver_moved_input": true, + "solver_element": 364, + "findings": [ + [ + 364, + 97, + 26 + ] + ], + "broad_coded_findings": [], + "broad_coded_finding_is_computational": false + }, + { + "key": "202405-40910", + "status": 3, + "rawben": 95.0, + "fsben": 149.0, + "engine_on_original": 149.0, + "amterr": 54.0, + "reproduced": false, + "correctednotes": "no_change", + "solver_moved_input": false, + "solver_element": 362, + "findings": [ + [ + 362, + 52, + 10 + ] + ], + "broad_coded_findings": [ + [ + 362, + 52, + 10 + ] + ], + "broad_coded_finding_is_computational": true + }, + { + "key": "202405-40916", + "status": 2, + "rawben": 139.0, + "fsben": 124.0, + "engine_on_original": 124.0, + "amterr": 15.0, + "reproduced": false, + "correctednotes": "unearn_error", + "solver_moved_input": false, + "solver_element": 350, + "findings": [ + [ + 350, + 44, + 16 + ], + [ + 311, + 97, + 26 + ] + ], + "broad_coded_findings": [], + "broad_coded_finding_is_computational": false + }, + { + "key": "202405-40957", + "status": 2, + "rawben": 1154.0, + "fsben": 56.0, + "engine_on_original": 896.0, + "amterr": 1098.0, + "reproduced": false, + "correctednotes": "earn_down", + "solver_moved_input": true, + "solver_element": 311, + "findings": [ + [ + 311, + 35, + 15 + ] + ], + "broad_coded_findings": [], + "broad_coded_finding_is_computational": false + }, + { + "key": "202406-40990", + "status": 3, + "rawben": 125.0, + "fsben": 161.0, + "engine_on_original": 161.0, + "amterr": 36.0, + "reproduced": false, + "correctednotes": "no_change", + "solver_moved_input": false, + "solver_element": 161, + "findings": [ + [ + 161, + 6, + 10 + ], + [ + 311, + 97, + 26 + ] + ], + "broad_coded_findings": [ + [ + 161, + 6, + 10 + ] + ], + "broad_coded_finding_is_computational": false + }, + { + "key": "202406-40998", + "status": 2, + "rawben": 766.0, + "fsben": 291.0, + "engine_on_original": 535.0, + "amterr": 475.0, + "reproduced": false, + "correctednotes": "hhsize_down", + "solver_moved_input": true, + "solver_element": 150, + "findings": [ + [ + 150, + 7, + 4 + ] + ], + "broad_coded_findings": [], + "broad_coded_finding_is_computational": false + }, + { + "key": "202406-41065", + "status": 2, + "rawben": 415.0, + "fsben": 309.0, + "engine_on_original": 343.0, + "amterr": 106.0, + "reproduced": false, + "correctednotes": "earn_error", + "solver_moved_input": true, + "solver_element": 312, + "findings": [ + [ + 312, + 45, + 14 + ], + [ + 363, + 56, + 14 + ] + ], + "broad_coded_findings": [], + "broad_coded_finding_is_computational": false + }, + { + "key": "202407-41087", + "status": 2, + "rawben": 409.0, + "fsben": 364.0, + "engine_on_original": 364.0, + "amterr": 45.0, + "reproduced": false, + "correctednotes": "rent_error", + "solver_moved_input": false, + "solver_element": 363, + "findings": [ + [ + 363, + 56, + 2 + ], + [ + 342, + 97, + 26 + ] + ], + "broad_coded_findings": [], + "broad_coded_finding_is_computational": false + }, + { + "key": "202407-41159", + "status": 2, + "rawben": 766.0, + "fsben": 583.0, + "engine_on_original": 583.0, + "amterr": 183.0, + "reproduced": false, + "correctednotes": "no_change", + "solver_moved_input": false, + "solver_element": 150, + "findings": [ + [ + 150, + 13, + 1 + ], + [ + 363, + 52, + 1 + ], + [ + 364, + 52, + 1 + ] + ], + "broad_coded_findings": [], + "broad_coded_finding_is_computational": false + }, + { + "key": "202407-41178", + "status": 3, + "rawben": 187.0, + "fsben": 291.0, + "engine_on_original": 291.0, + "amterr": 104.0, + "reproduced": false, + "correctednotes": "no_change", + "solver_moved_input": false, + "solver_element": 520, + "findings": [ + [ + 520, + 80, + 10 + ], + [ + 363, + 53, + 2 + ] + ], + "broad_coded_findings": [ + [ + 520, + 80, + 10 + ] + ], + "broad_coded_finding_is_computational": true + }, + { + "key": "202408-41268", + "status": 2, + "rawben": 240.0, + "fsben": 208.0, + "engine_on_original": 232.0, + "amterr": 32.0, + "reproduced": false, + "correctednotes": "rent_error", + "solver_moved_input": true, + "solver_element": 363, + "findings": [ + [ + 363, + 56, + 2 + ], + [ + 311, + 38, + 12 + ] + ], + "broad_coded_findings": [], + "broad_coded_finding_is_computational": false + }, + { + "key": "202408-41271", + "status": 2, + "rawben": 240.0, + "fsben": 227.0, + "engine_on_original": 227.0, + "amterr": 13.0, + "reproduced": false, + "correctednotes": "no_change", + "solver_moved_input": false, + "solver_element": 520, + "findings": [ + [ + 520, + 98, + 20 + ] + ], + "broad_coded_findings": [ + [ + 520, + 98, + 20 + ] + ], + "broad_coded_finding_is_computational": true + }, + { + "key": "202409-41302", + "status": 2, + "rawben": 148.0, + "fsben": 50.0, + "engine_on_original": 95.0, + "amterr": 98.0, + "reproduced": false, + "correctednotes": "earn_error", + "solver_moved_input": true, + "solver_element": 314, + "findings": [ + [ + 314, + 33, + 12 + ], + [ + 363, + 53, + 2 + ] + ], + "broad_coded_findings": [], + "broad_coded_finding_is_computational": false + }, + { + "key": "202409-41321", + "status": 2, + "rawben": 55.0, + "fsben": 23.0, + "engine_on_original": 23.0, + "amterr": 32.0, + "reproduced": false, + "correctednotes": "util_up", + "solver_moved_input": true, + "solver_element": 364, + "findings": [ + [ + 364, + 53, + 1 + ], + [ + 363, + 97, + 26 + ] + ], + "broad_coded_findings": [], + "broad_coded_finding_is_computational": false + }, + { + "key": "202409-41362", + "status": 2, + "rawben": 291.0, + "fsben": 23.0, + "engine_on_original": 23.0, + "amterr": 268.0, + "reproduced": false, + "correctednotes": "earn_down", + "solver_moved_input": true, + "solver_element": 311, + "findings": [ + [ + 311, + 35, + 1 + ], + [ + 334, + 35, + 1 + ], + [ + 363, + 57, + 2 + ] + ], + "broad_coded_findings": [], + "broad_coded_finding_is_computational": false + } + ], + "layer2_computational_findings": { + "cases": 26, + "findings": 29, + "by_code": [ + { + "element": 311, + "nature": 42, + "agency": 18, + "class": "mixed", + "findings": 1 + }, + { + "element": 311, + "nature": 42, + "agency": 20, + "class": "mixed", + "findings": 1 + }, + { + "element": 311, + "nature": 43, + "agency": 10, + "class": "mixed", + "findings": 2 + }, + { + "element": 312, + "nature": 42, + "agency": 3, + "class": "pure_math", + "findings": 1 + }, + { + "element": 312, + "nature": 43, + "agency": 18, + "class": "mixed", + "findings": 1 + }, + { + "element": 323, + "nature": 57, + "agency": 20, + "class": "pure_math", + "findings": 1 + }, + { + "element": 331, + "nature": 98, + "agency": 18, + "class": "pure_math", + "findings": 1 + }, + { + "element": 362, + "nature": 52, + "agency": 10, + "class": "pure_math", + "findings": 1 + }, + { + "element": 364, + "nature": 52, + "agency": 10, + "class": "mixed", + "findings": 1 + }, + { + "element": 364, + "nature": 54, + "agency": 12, + "class": "mixed", + "findings": 1 + }, + { + "element": 364, + "nature": 54, + "agency": 18, + "class": "pure_math", + "findings": 1 + }, + { + "element": 364, + "nature": 54, + "agency": 2, + "class": "mixed", + "findings": 1 + }, + { + "element": 364, + "nature": 54, + "agency": 2, + "class": "pure_math", + "findings": 1 + }, + { + "element": 364, + "nature": 64, + "agency": 1, + "class": "mixed", + "findings": 1 + }, + { + "element": 365, + "nature": 52, + "agency": 19, + "class": "mixed", + "findings": 1 + }, + { + "element": 365, + "nature": 53, + "agency": 17, + "class": "mixed", + "findings": 1 + }, + { + "element": 365, + "nature": 98, + "agency": 14, + "class": "mixed", + "findings": 1 + }, + { + "element": 366, + "nature": 56, + "agency": 17, + "class": "mixed", + "findings": 1 + }, + { + "element": 366, + "nature": 56, + "agency": 17, + "class": "pure_math", + "findings": 1 + }, + { + "element": 520, + "nature": 75, + "agency": 15, + "class": "pure_math", + "findings": 2 + }, + { + "element": 520, + "nature": 75, + "agency": 20, + "class": "pure_math", + "findings": 1 + }, + { + "element": 520, + "nature": 80, + "agency": 10, + "class": "mixed", + "findings": 2 + }, + { + "element": 520, + "nature": 80, + "agency": 10, + "class": "pure_math", + "findings": 2 + }, + { + "element": 520, + "nature": 98, + "agency": 20, + "class": "pure_math", + "findings": 2 + } + ] + }, + "reconstruction": { + "colorado_rows": 283, + "national_rows": 15902 + } + }, + "by_posting": { + "may2026": { + "totals": { + "CO": { + "cases": 856, + "error_cases": 305, + "finding_amount_dollars": 4740779.39, + "error_dollars": 112575220.49, + "error_dollars_above_threshold": 94092792.02, + "issuance_rawben_dollars": 1267963387.75, + "calculated_fsben_dollars": 1255069797.92, + "file_error_rate": 0.088784283186, + "cost_share_step_dollars": 63398169.39, + "layer1": { + "strict_17_19_20": { + "n": 23, + "dollars": 8189989.7, + "share": 0.072751264987 + }, + "broad_10_17_19_20_21_22": { + "n": 35, + "dollars": 11870971.25, + "share": 0.105449238258 + } + }, + "axis1": { + "software_17_19": { + "n": 18, + "dollars": 6984182.17, + "share": 0.062040137634 + }, + "worker_computation_20_21": { + "n": 5, + "dollars": 1205807.52, + "share": 0.010711127353 + }, + "data_entry_18": { + "n": 28, + "dollars": 18466974.77, + "share": 0.164041204548 + }, + "policy_or_budgeted_10_22": { + "n": 12, + "dollars": 3680981.55, + "share": 0.03269797327 + } + }, + "broad_code_case_counts": { + "10": 12, + "17": 13, + "19": 5, + "20": 5, + "21": 0, + "22": 0 + }, + "no_cause_code": { + "n": 21, + "dollars": 2312762.2, + "share": 0.020544149868 + }, + "axis1_ratio_to_national": { + "software_17_19": 2.097496, + "worker_computation_20_21": 0.815154, + "data_entry_18": 7.575577, + "policy_or_budgeted_10_22": 0.223851 + } + }, + "US": { + "cases": 44891, + "error_cases": 17836, + "finding_amount_dollars": 6607691015.15, + "error_dollars": 8135965305.97, + "error_dollars_above_threshold": 6593382671.8, + "issuance_rawben_dollars": 91385805108.85, + "calculated_fsben_dollars": 88823856912.58, + "file_error_rate": 0.089028764328, + "cost_share_step_dollars": 4569290255.44, + "layer1": { + "strict_17_19_20": { + "n": 1145, + "dollars": 320088850.45, + "share": 0.039342455187 + }, + "broad_10_17_19_20_21_22": { + "n": 3095, + "dollars": 1498501938.9, + "share": 0.184182439643 + } + }, + "axis1": { + "software_17_19": { + "n": 855, + "dollars": 240647188.9, + "share": 0.029578197528 + }, + "worker_computation_20_21": { + "n": 322, + "dollars": 106906574.61, + "share": 0.013139998831 + }, + "data_entry_18": { + "n": 338, + "dollars": 176175825.73, + "share": 0.021653954892 + }, + "policy_or_budgeted_10_22": { + "n": 1992, + "dollars": 1188420695.28, + "share": 0.146070029872 + } + }, + "broad_code_case_counts": { + "10": 1288, + "17": 211, + "19": 644, + "20": 297, + "21": 26, + "22": 730 + }, + "no_cause_code": { + "n": 2308, + "dollars": 270293004.43, + "share": 0.033221995702 + } + } + }, + "layer2": { + "input_other": { + "n": 260, + "dollars": 96760825.27, + "share": 0.85952152574 + }, + "mixed": { + "n": 13, + "dollars": 4683116.8, + "share": 0.041599890114 + }, + "pure_math": { + "n": 13, + "dollars": 3662840.51, + "share": 0.032536827296 + }, + "input_system_caused": { + "n": 19, + "dollars": 7468437.9, + "share": 0.06634175685 + } + }, + "replay": { + "replayed": { + "n": 283, + "dollars": 99128122.62, + "share_of_replayed": 1.0, + "share_of_colorado_error_dollars": 0.880550108579, + "points_of_official_fy2024_rate": 8.779085, + "above_threshold_n": 97, + "share_of_colorado_above_threshold_error_dollars": 0.864632484315, + "above_threshold_points_of_official_fy2024_rate": 8.620386 + }, + "reproduced": { + "n": 246, + "dollars": 71388303.72, + "share_of_replayed": 0.720161966447, + "share_of_colorado_error_dollars": 0.634138697749, + "points_of_official_fy2024_rate": 6.322363, + "above_threshold_n": 76, + "share_of_colorado_above_threshold_error_dollars": 0.594870585264, + "above_threshold_points_of_official_fy2024_rate": 5.93086 + }, + "not_reproduced": { + "n": 37, + "dollars": 27739818.9, + "share_of_replayed": 0.279838033553, + "share_of_colorado_error_dollars": 0.24641141083, + "points_of_official_fy2024_rate": 2.456722, + "above_threshold_n": 21, + "share_of_colorado_above_threshold_error_dollars": 0.269761899051, + "above_threshold_points_of_official_fy2024_rate": 2.689526 + }, + "broad_coded_misses": { + "n": 10, + "dollars": 4469987.48, + "share_of_replayed": 0.0450930307, + "share_of_colorado_error_dollars": 0.039706673079, + "points_of_official_fy2024_rate": 0.395876, + "above_threshold_n": 5, + "share_of_colorado_above_threshold_error_dollars": 0.040923618775, + "above_threshold_points_of_official_fy2024_rate": 0.408008, + "engine_gap_dollars": 2913739.34 + }, + "computation_candidates": { + "n": 7, + "dollars": 1995070.46, + "share_of_replayed": 0.020126180163, + "share_of_colorado_error_dollars": 0.017722110128, + "points_of_official_fy2024_rate": 0.176689, + "above_threshold_n": 3, + "share_of_colorado_above_threshold_error_dollars": 0.015786488796, + "above_threshold_points_of_official_fy2024_rate": 0.157391 + }, + "not_reproduced_with_computational_finding": { + "n": 10, + "dollars": 3509155.62, + "share_of_replayed": 0.035400202545, + "share_of_colorado_error_dollars": 0.031171652194, + "points_of_official_fy2024_rate": 0.310781, + "above_threshold_n": 5, + "share_of_colorado_above_threshold_error_dollars": 0.031087423957, + "above_threshold_points_of_official_fy2024_rate": 0.309942 + }, + "reproduced_with_computational_finding": { + "n": 13, + "dollars": 3624188.18, + "share_of_replayed": 0.036560645823, + "share_of_colorado_error_dollars": 0.032193480649, + "points_of_official_fy2024_rate": 0.320969, + "above_threshold_n": 6, + "share_of_colorado_above_threshold_error_dollars": 0.033435772289, + "above_threshold_points_of_official_fy2024_rate": 0.333355 + } + }, + "class_by_replay_outcome": { + "broad_10_17_19_20_21_22": { + "all_errors": { + "all": { + "n": 35, + "dollars": 11870971.25, + "share": 0.105449238258, + "points_of_official_fy2024_rate": 1.051329 + }, + "reproduced": { + "n": 22, + "dollars": 6363473.6, + "share": 0.056526414699, + "points_of_official_fy2024_rate": 0.563568 + }, + "not_reproduced": { + "n": 10, + "dollars": 4469987.48, + "share": 0.039706673079, + "points_of_official_fy2024_rate": 0.395876 + }, + "not_replayed": { + "n": 3, + "dollars": 1037510.17, + "share": 0.00921615048, + "points_of_official_fy2024_rate": 0.091885 + } + }, + "above_threshold": { + "all": { + "n": 16, + "dollars": 9987978.46, + "share": 0.106150303781, + "points_of_official_fy2024_rate": 1.058319 + }, + "reproduced": { + "n": 9, + "dollars": 5109701.53, + "share": 0.054304919816, + "points_of_official_fy2024_rate": 0.54142 + }, + "not_reproduced": { + "n": 5, + "dollars": 3850617.55, + "share": 0.040923618775, + "points_of_official_fy2024_rate": 0.408008 + }, + "not_replayed": { + "n": 2, + "dollars": 1027659.38, + "share": 0.01092176519, + "points_of_official_fy2024_rate": 0.10889 + } + } + }, + "software_17_19": { + "all_errors": { + "all": { + "n": 18, + "dollars": 6984182.17, + "share": 0.062040137634, + "points_of_official_fy2024_rate": 0.61854 + }, + "reproduced": { + "n": 14, + "dollars": 4265365.91, + "share": 0.03788903006, + "points_of_official_fy2024_rate": 0.377754 + }, + "not_reproduced": { + "n": 2, + "dollars": 2365222.74, + "share": 0.021010154215, + "points_of_official_fy2024_rate": 0.209471 + }, + "not_replayed": { + "n": 2, + "dollars": 353593.52, + "share": 0.00314095336, + "points_of_official_fy2024_rate": 0.031315 + } + }, + "above_threshold": { + "all": { + "n": 7, + "dollars": 5914070.74, + "share": 0.062853600309, + "points_of_official_fy2024_rate": 0.62665 + }, + "reproduced": { + "n": 4, + "dollars": 3205105.27, + "share": 0.034063239111, + "points_of_official_fy2024_rate": 0.33961 + }, + "not_reproduced": { + "n": 2, + "dollars": 2365222.74, + "share": 0.025137129979, + "points_of_official_fy2024_rate": 0.250617 + }, + "not_replayed": { + "n": 1, + "dollars": 343742.73, + "share": 0.003653231218, + "points_of_official_fy2024_rate": 0.036423 + } + } + }, + "other_10_20_21_22": { + "all_errors": { + "all": { + "n": 17, + "dollars": 4886789.07, + "share": 0.043409100623, + "points_of_official_fy2024_rate": 0.432789 + }, + "reproduced": { + "n": 8, + "dollars": 2098107.69, + "share": 0.018637384639, + "points_of_official_fy2024_rate": 0.185815 + }, + "not_reproduced": { + "n": 8, + "dollars": 2104764.73, + "share": 0.018696518864, + "points_of_official_fy2024_rate": 0.186404 + }, + "not_replayed": { + "n": 1, + "dollars": 683916.66, + "share": 0.00607519712, + "points_of_official_fy2024_rate": 0.06057 + } + }, + "above_threshold": { + "all": { + "n": 9, + "dollars": 4073907.72, + "share": 0.043296703472, + "points_of_official_fy2024_rate": 0.431668 + }, + "reproduced": { + "n": 5, + "dollars": 1904596.25, + "share": 0.020241680705, + "points_of_official_fy2024_rate": 0.20181 + }, + "not_reproduced": { + "n": 3, + "dollars": 1485394.81, + "share": 0.015786488796, + "points_of_official_fy2024_rate": 0.157391 + }, + "not_replayed": { + "n": 1, + "dollars": 683916.66, + "share": 0.007268533971, + "points_of_official_fy2024_rate": 0.072467 + } + } + } + }, + "software_coded_replayed": { + "reproduced": { + "n": 14, + "dollars": 4265365.91 + }, + "not_reproduced": { + "n": 2, + "dollars": 2365222.74 + }, + "reproduced_solver_moved_software_coded_element": { + "n": 10, + "dollars": 1767510.92 + }, + "reproduced_solver_moved_same_input_other_element": { + "n": 2, + "dollars": 1693841.88 + }, + "reproduced_solver_moved_other_input": { + "n": 2, + "dollars": 804013.11 + } + } + }, + "aug2026": { + "totals": { + "CO": { + "cases": 856, + "error_cases": 305, + "finding_amount_dollars": 4741045.5, + "error_dollars": 112576406.09, + "error_dollars_above_threshold": 94149414.89, + "issuance_rawben_dollars": 1265236193.59, + "calculated_fsben_dollars": 1252193779.35, + "file_error_rate": 0.088976593193, + "cost_share_step_dollars": 63261809.68, + "layer1": { + "strict_17_19_20": { + "n": 23, + "dollars": 8185101.86, + "share": 0.072707080812 + }, + "broad_10_17_19_20_21_22": { + "n": 35, + "dollars": 11862583.77, + "share": 0.105373622984 + } + }, + "axis1": { + "software_17_19": { + "n": 18, + "dollars": 6981968.09, + "share": 0.062019816866 + }, + "worker_computation_20_21": { + "n": 5, + "dollars": 1203133.77, + "share": 0.010687263946 + }, + "data_entry_18": { + "n": 28, + "dollars": 18425329.7, + "share": 0.163669549784 + }, + "policy_or_budgeted_10_22": { + "n": 12, + "dollars": 3677481.92, + "share": 0.032666542172 + } + }, + "broad_code_case_counts": { + "10": 12, + "17": 13, + "19": 5, + "20": 5, + "21": 0, + "22": 0 + }, + "no_cause_code": { + "n": 21, + "dollars": 2313703.86, + "share": 0.020552298135 + }, + "axis1_ratio_to_national": { + "software_17_19": 2.097495, + "worker_computation_20_21": 0.814726, + "data_entry_18": 7.55532, + "policy_or_budgeted_10_22": 0.223749 + } + }, + "US": { + "cases": 44891, + "error_cases": 17836, + "finding_amount_dollars": 6586710752.37, + "error_dollars": 8108035016.39, + "error_dollars_above_threshold": 6572048865.22, + "issuance_rawben_dollars": 90963473968.81, + "calculated_fsben_dollars": 88398413140.47, + "file_error_rate": 0.089135063368, + "cost_share_step_dollars": 4548173698.44, + "layer1": { + "strict_17_19_20": { + "n": 1145, + "dollars": 318587584.1, + "share": 0.039292822916 + }, + "broad_10_17_19_20_21_22": { + "n": 3095, + "dollars": 1492473803.46, + "share": 0.18407342845 + } + }, + "axis1": { + "software_17_19": { + "n": 855, + "dollars": 239742609.43, + "share": 0.029568521712 + }, + "worker_computation_20_21": { + "n": 322, + "dollars": 106358141.64, + "share": 0.013117622386 + }, + "data_entry_18": { + "n": 338, + "dollars": 175642926.26, + "share": 0.021662822855 + }, + "policy_or_budgeted_10_22": { + "n": 1992, + "dollars": 1183743628.07, + "share": 0.145996363568 + } + }, + "broad_code_case_counts": { + "10": 1288, + "17": 211, + "19": 644, + "20": 297, + "21": 26, + "22": 730 + }, + "no_cause_code": { + "n": 2308, + "dollars": 269515996.53, + "share": 0.033240605891 + } + } + }, + "layer2": { + "input_other": { + "n": 260, + "dollars": 96769184.36, + "share": 0.859586726169 + }, + "mixed": { + "n": 13, + "dollars": 4685262.58, + "share": 0.041618512635 + }, + "pure_math": { + "n": 13, + "dollars": 3660895.45, + "share": 0.032519206951 + }, + "input_system_caused": { + "n": 19, + "dollars": 7461063.71, + "share": 0.066275554245 + } + }, + "replay": { + "replayed": { + "n": 283, + "dollars": 99078929.05, + "share_of_replayed": 1.0, + "share_of_colorado_error_dollars": 0.880103855643, + "points_of_official_fy2024_rate": 8.774635, + "above_threshold_n": 97, + "share_of_colorado_above_threshold_error_dollars": 0.864163422904, + "above_threshold_points_of_official_fy2024_rate": 8.615709 + }, + "reproduced": { + "n": 246, + "dollars": 71205745.95, + "share_of_replayed": 0.718676984393, + "share_of_colorado_error_dollars": 0.632510384926, + "points_of_official_fy2024_rate": 6.306129, + "above_threshold_n": 76, + "share_of_colorado_above_threshold_error_dollars": 0.593124241242, + "above_threshold_points_of_official_fy2024_rate": 5.913449 + }, + "not_reproduced": { + "n": 37, + "dollars": 27873183.1, + "share_of_replayed": 0.281323015607, + "share_of_colorado_error_dollars": 0.247593470717, + "points_of_official_fy2024_rate": 2.468507, + "above_threshold_n": 21, + "share_of_colorado_above_threshold_error_dollars": 0.271039181662, + "above_threshold_points_of_official_fy2024_rate": 2.702261 + }, + "broad_coded_misses": { + "n": 10, + "dollars": 4468608.19, + "share_of_replayed": 0.045101498724, + "share_of_colorado_error_dollars": 0.039694002922, + "points_of_official_fy2024_rate": 0.395749, + "above_threshold_n": 5, + "share_of_colorado_above_threshold_error_dollars": 0.040898782922, + "above_threshold_points_of_official_fy2024_rate": 0.407761, + "engine_gap_dollars": 2912067.81 + }, + "computation_candidates": { + "n": 7, + "dollars": 1993431.11, + "share_of_replayed": 0.020119627165, + "share_of_colorado_error_dollars": 0.017707361442, + "points_of_official_fy2024_rate": 0.176542, + "above_threshold_n": 3, + "share_of_colorado_above_threshold_error_dollars": 0.015774008601, + "above_threshold_points_of_official_fy2024_rate": 0.157267 + }, + "not_reproduced_with_computational_finding": { + "n": 10, + "dollars": 3507380.14, + "share_of_replayed": 0.035399859254, + "share_of_colorado_error_dollars": 0.031155552619, + "points_of_official_fy2024_rate": 0.310621, + "above_threshold_n": 5, + "share_of_colorado_above_threshold_error_dollars": 0.031064295632, + "above_threshold_points_of_official_fy2024_rate": 0.309711 + }, + "reproduced_with_computational_finding": { + "n": 13, + "dollars": 3626164.38, + "share_of_replayed": 0.036598744154, + "share_of_colorado_error_dollars": 0.032210695842, + "points_of_official_fy2024_rate": 0.321141, + "above_threshold_n": 6, + "share_of_colorado_above_threshold_error_dollars": 0.033436716173, + "above_threshold_points_of_official_fy2024_rate": 0.333364 + } + }, + "class_by_replay_outcome": { + "broad_10_17_19_20_21_22": { + "all_errors": { + "all": { + "n": 35, + "dollars": 11862583.77, + "share": 0.105373622984, + "points_of_official_fy2024_rate": 1.050575 + }, + "reproduced": { + "n": 22, + "dollars": 6356465.41, + "share": 0.056463566643, + "points_of_official_fy2024_rate": 0.562942 + }, + "not_reproduced": { + "n": 10, + "dollars": 4468608.19, + "share": 0.039694002922, + "points_of_official_fy2024_rate": 0.395749 + }, + "not_replayed": { + "n": 3, + "dollars": 1037510.17, + "share": 0.009216053419, + "points_of_official_fy2024_rate": 0.091884 + } + }, + "above_threshold": { + "all": { + "n": 16, + "dollars": 9985369.56, + "share": 0.106058753245, + "points_of_official_fy2024_rate": 1.057406 + }, + "reproduced": { + "n": 9, + "dollars": 5107113.7, + "share": 0.054244773647, + "points_of_official_fy2024_rate": 0.54082 + }, + "not_reproduced": { + "n": 5, + "dollars": 3850596.48, + "share": 0.040898782922, + "points_of_official_fy2024_rate": 0.407761 + }, + "not_replayed": { + "n": 2, + "dollars": 1027659.38, + "share": 0.010915196677, + "points_of_official_fy2024_rate": 0.108825 + } + } + }, + "software_17_19": { + "all_errors": { + "all": { + "n": 18, + "dollars": 6981968.09, + "share": 0.062019816866, + "points_of_official_fy2024_rate": 0.618338 + }, + "reproduced": { + "n": 14, + "dollars": 4262891.77, + "share": 0.03786665358, + "points_of_official_fy2024_rate": 0.377531 + }, + "not_reproduced": { + "n": 2, + "dollars": 2365482.8, + "share": 0.021012243006, + "points_of_official_fy2024_rate": 0.209492 + }, + "not_replayed": { + "n": 2, + "dollars": 353593.52, + "share": 0.003140920281, + "points_of_official_fy2024_rate": 0.031315 + } + }, + "above_threshold": { + "all": { + "n": 7, + "dollars": 5914330.8, + "share": 0.0628185614, + "points_of_official_fy2024_rate": 0.626301 + }, + "reproduced": { + "n": 4, + "dollars": 3205105.27, + "share": 0.034042752969, + "points_of_official_fy2024_rate": 0.339406 + }, + "not_reproduced": { + "n": 2, + "dollars": 2365482.8, + "share": 0.025124774321, + "points_of_official_fy2024_rate": 0.250494 + }, + "not_replayed": { + "n": 1, + "dollars": 343742.73, + "share": 0.00365103411, + "points_of_official_fy2024_rate": 0.036401 + } + } + }, + "other_10_20_21_22": { + "all_errors": { + "all": { + "n": 17, + "dollars": 4880615.68, + "share": 0.043353806118, + "points_of_official_fy2024_rate": 0.432237 + }, + "reproduced": { + "n": 8, + "dollars": 2093573.64, + "share": 0.018596913063, + "points_of_official_fy2024_rate": 0.185411 + }, + "not_reproduced": { + "n": 8, + "dollars": 2103125.39, + "share": 0.018681759916, + "points_of_official_fy2024_rate": 0.186257 + }, + "not_replayed": { + "n": 1, + "dollars": 683916.66, + "share": 0.006075133139, + "points_of_official_fy2024_rate": 0.060569 + } + }, + "above_threshold": { + "all": { + "n": 9, + "dollars": 4071038.76, + "share": 0.043240191845, + "points_of_official_fy2024_rate": 0.431105 + }, + "reproduced": { + "n": 5, + "dollars": 1902008.43, + "share": 0.020202020677, + "points_of_official_fy2024_rate": 0.201414 + }, + "not_reproduced": { + "n": 3, + "dollars": 1485113.68, + "share": 0.015774008601, + "points_of_official_fy2024_rate": 0.157267 + }, + "not_replayed": { + "n": 1, + "dollars": 683916.66, + "share": 0.007264162566, + "points_of_official_fy2024_rate": 0.072424 + } + } + } + }, + "software_coded_replayed": { + "reproduced": { + "n": 14, + "dollars": 4262891.77 + }, + "not_reproduced": { + "n": 2, + "dollars": 2365482.8 + }, + "reproduced_solver_moved_software_coded_element": { + "n": 10, + "dollars": 1765036.78 + }, + "reproduced_solver_moved_same_input_other_element": { + "n": 2, + "dollars": 1693841.88 + }, + "reproduced_solver_moved_other_input": { + "n": 2, + "dollars": 804013.11 + } + } + } + }, + "posting_diff": { + "rows": 44891, + "columns": 1177, + "rows_in_same_order": true, + "changed_columns": { + "HWGT": { + "rows": 16948, + "colorado_rows": 287 + }, + "HWGT_OLD": { + "rows": 10072, + "colorado_rows": 154 + }, + "FYWGT": { + "rows": 16948, + "colorado_rows": 287 + }, + "FYWGT_OLD": { + "rows": 10072, + "colorado_rows": 154 + } + } + } +} diff --git a/paper/snapshot/labs/amterr/reconstruct_co_fy2024.R b/paper/snapshot/labs/amterr/reconstruct_co_fy2024.R index cbbdd40..0b3e83c 100644 --- a/paper/snapshot/labs/amterr/reconstruct_co_fy2024.R +++ b/paper/snapshot/labs/amterr/reconstruct_co_fy2024.R @@ -9,8 +9,14 @@ # - keeps ALL solver/filter logic verbatim otherwise. suppressPackageStartupMessages({library(haven); library(dplyr); library(tidyr)}) -repo <- Sys.getenv("SNAP_QC_REPO", "~/snap_qc/") # clone of github.com/giannella/snap_qc -out_dir <- "/Users/maxghenis/.cache/axiom-oracles/amterr-lab/" +# SNAP_QC_REPO: clone of github.com/giannella/snap_qc at the commit pinned in +# README.md. AMTERR_LAB_DIR: output directory (default: this script's directory). +script_dir <- local({ + arg <- grep("^--file=", commandArgs(trailingOnly = FALSE), value = TRUE) + if (length(arg)) dirname(normalizePath(sub("^--file=", "", arg[1]))) else getwd() +}) +repo <- paste0(normalizePath(Sys.getenv("SNAP_QC_REPO", "~/snap_qc"), mustWork = TRUE), "/") +out_dir <- paste0(normalizePath(Sys.getenv("AMTERR_LAB_DIR", script_dir), mustWork = TRUE), "/") correct_variables <- TRUE mydata <- read_sav(paste0(repo, "qc_data/qc_pub_fy2024.sav")) diff --git a/pyproject.toml b/pyproject.toml index 5c75a36..902641d 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -15,7 +15,11 @@ analysis = [ "scikit-learn>=1.8,<1.9", "scipy>=1.17,<1.18", ] -dev = ["pytest>=8,<10", "ruff>=0.4,<1"] +dev = [ + "hypothesis>=6.100,<7", + "pytest>=8,<10", + "ruff>=0.4,<1", +] [build-system] requires = ["hatchling"] diff --git a/tests/test_amterr_lab.py b/tests/test_amterr_lab.py new file mode 100644 index 0000000..b128459 --- /dev/null +++ b/tests/test_amterr_lab.py @@ -0,0 +1,472 @@ +"""Guards for the amterr lab's claims audit (paper/snapshot/labs/amterr). + +The committed ``claims_audit.json`` recomputes every figure in the lab's +ANALYSIS.md. Most tests here need only committed files. The regeneration test +also needs both FY2024 QC postings and skips without them. +""" + +from __future__ import annotations + +import importlib.util +import json +import sys +from pathlib import Path + +import numpy as np +import pandas as pd +import pytest +from hypothesis import given, settings +from hypothesis import strategies as st + +LAB = Path(__file__).parents[1] / "paper/snapshot/labs/amterr" + + +def _load_module(): + spec = importlib.util.spec_from_file_location( + "amterr_audit_claims", LAB / "audit_claims.py" + ) + module = importlib.util.module_from_spec(spec) + sys.modules[spec.name] = module + spec.loader.exec_module(module) + return module + + +audit = _load_module() +AUDIT = json.loads((LAB / "claims_audit.json").read_text(encoding="utf-8")) +POSTINGS = tuple(AUDIT["by_posting"]) +CASES = AUDIT["case_level"] + + +# --------------------------------------------------------------------------- +# committed-file checks (no QC data needed) + + +def test_replay_partition_recomputes_from_committed_replay(): + replay = audit.load_replay() + moved = audit.moved_from_keys(replay, CASES["replay_inputs_unchanged_keys"]) + assert audit.replay_partition(replay, moved) == CASES["replay"] + + +def test_unchanged_inputs_cover_every_no_change_row(): + replay = audit.load_replay() + no_change = set(replay.loc[replay["solver_no_change"], "key"]) + unchanged = set(CASES["replay_inputs_unchanged_keys"]) + assert no_change <= unchanged + assert unchanged.isdisjoint(CASES["moved_with_zero_correctedamount_keys"]) + + +def test_replay_partition_is_exhaustive(): + replay = CASES["replay"] + assert replay["reproduced_n"] + replay["not_reproduced_n"] == replay["n"] + assert ( + replay["reproduced_solver_moved_input_n"] + replay["reproduced_without_move_n"] + == replay["reproduced_n"] + ) + unmoved = ( + replay["not_reproduced_solver_no_change_n"] + + replay["not_reproduced_eligible_but_unmoved_n"] + ) + assert ( + replay["not_reproduced_solver_moved_input_n"] + unmoved + == (replay["not_reproduced_n"]) + ) + # An unmoved row replays the file's own inputs, so the engine returns FSBEN. + assert replay["not_reproduced_unmoved_engine_equals_fsben_n"] == unmoved + assert ( + sum(replay["not_reproduced_by_correctednotes"].values()) + == (replay["not_reproduced_n"]) + ) + + +def test_case_lists_nest(): + missed = {row["key"] for row in CASES["not_reproduced_cases"]} + broad = CASES["broad_coded_misses"] + broad_keys = {row["key"] for row in broad["cases"]} + candidates = set(broad["computation_candidate_keys"]) + assert candidates <= broad_keys <= missed + assert candidates.isdisjoint(broad["other_keys"]) + assert candidates | set(broad["other_keys"]) == broad_keys + assert len(missed) == CASES["replay"]["not_reproduced_n"] + for row in broad["cases"]: + assert row["broad_coded_findings"], row["key"] + assert all(f[2] in audit.BROAD_CODES for f in row["broad_coded_findings"]) + + +def test_audit_case_rows_agree_with_committed_replay(): + replay = audit.load_replay().set_index("key") + for row in CASES["not_reproduced_cases"]: + source = replay.loc[row["key"]] + assert not source["reproduced"] + assert row["rawben"] == source["rawben"] + assert row["fsben"] == source["fsben"] + assert row["engine_on_original"] == source["engine_on_original"] + assert row["correctednotes"] == source["correctednotes"] + + +def test_analysis_example_is_outside_the_broad_class(): + example = CASES["analysis_example_202312_40441"] + assert example["agency_codes"] == [15] + assert not example["in_broad_coded_misses"] + assert example["engine_on_original"] == example["fsben"] + + +def test_regenerated_layer_one_and_two_artifacts_match_committed_copies(): + for name, record in AUDIT["regenerated_artifacts"].items(): + assert record["matches_committed"], name + # The audit compared against this exact file; an edit to it must fail. + assert record["sha256"] == audit.sha256(audit.REPO / record["path"]), name + + +def test_audit_was_built_from_the_committed_replay_and_solver_output(): + for path, digest in AUDIT["inputs"]["committed"].items(): + assert audit.sha256(audit.REPO / path) == digest, path + + +def test_postings_differ_only_in_weight_columns(): + assert set(AUDIT["posting_diff"]["changed_columns"]) <= audit.WEIGHT_COLUMNS + assert AUDIT["posting_diff"]["rows_in_same_order"] + + +@pytest.mark.parametrize("posting", POSTINGS) +def test_weighted_partitions_conserve_dollars(posting): + result = AUDIT["by_posting"][posting] + replay = result["replay"] + assert replay["reproduced"]["n"] + replay["not_reproduced"]["n"] == 283 + assert replay["reproduced"]["dollars"] + replay["not_reproduced"]["dollars"] == ( + pytest.approx(replay["replayed"]["dollars"], abs=0.02) + ) + assert ( + replay["computation_candidates"]["dollars"] + <= replay["broad_coded_misses"]["dollars"] + <= replay["not_reproduced"]["dollars"] + ) + rate = audit.OFFICIAL_RATES["fy2024"]["CO"] + for metric in replay.values(): + assert 0 <= metric["share_of_colorado_error_dollars"] <= 1 + assert metric["points_of_official_fy2024_rate"] == pytest.approx( + metric["share_of_colorado_error_dollars"] * rate, abs=1e-6 + ) + assert metric["above_threshold_n"] <= metric["n"] + candidates = set(CASES["broad_coded_misses"]["computation_candidate_keys"]) + assert candidates <= set(CASES["not_reproduced_with_computational_finding_keys"]) + # 26 cases carry a computational finding: 13 reproduce, 10 do not, 3 were + # not replayed. + assert ( + CASES["layer2_computational_findings"]["cases"] + - len(CASES["reproduced_with_computational_finding_keys"]) + - len(CASES["not_reproduced_with_computational_finding_keys"]) + == 3 + ) + + for classes in result["class_by_replay_outcome"].values(): + for parts in classes.values(): + pieces = ("reproduced", "not_reproduced", "not_replayed") + assert sum(parts[p]["n"] for p in pieces) == parts["all"]["n"] + assert sum(parts[p]["dollars"] for p in pieces) == pytest.approx( + parts["all"]["dollars"], abs=0.03 + ) + + software = result["software_coded_replayed"] + kinds = ("software_coded_element", "same_input_other_element", "other_input") + parts = [software[f"reproduced_solver_moved_{kind}"] for kind in kinds] + assert sum(p["n"] for p in parts) == software["reproduced"]["n"] + assert sum(p["dollars"] for p in parts) == pytest.approx( + software["reproduced"]["dollars"], abs=0.02 + ) + + colorado = result["totals"]["CO"] + layer2 = result["layer2"] + assert sum(v["n"] for v in layer2.values()) == colorado["error_cases"] + assert sum(v["dollars"] for v in layer2.values()) == pytest.approx( + colorado["error_dollars"], abs=0.05 + ) + + +def test_case_counts_do_not_depend_on_posting(): + first, *rest = (AUDIT["by_posting"][p] for p in POSTINGS) + for other in rest: + for state in ("CO", "US"): + for block in ("layer1", "axis1"): + for name, metric in first["totals"][state][block].items(): + assert other["totals"][state][block][name]["n"] == metric["n"] + for name, metric in first["layer2"].items(): + assert other["layer2"][name]["n"] == metric["n"] + + +def _millions(value: float, digits: int) -> str: + return f"${value / 1e6:.{digits}f}M" + + +def _percent(value: float, digits: int = 1) -> str: + return f"{100 * value:.{digits}f}%" + + +def test_analysis_quotes_the_audit(): + """Whole sentences and table rows, so a figure cannot match elsewhere.""" + text = " ".join((LAB / "ANALYSIS.md").read_text(encoding="utf-8").split()) + may = AUDIT["by_posting"]["may2026"] + aug = AUDIT["by_posting"]["aug2026"] + counts = CASES["replay"] + replay = may["replay"] + misses = replay["broad_coded_misses"] + quoted = [ + ( + f"{counts['reproduced_n']} of {counts['n']} " + f"({_percent(counts['reproduced_n'] / counts['n'])} of cases, " + f"{_percent(replay['reproduced']['share_of_replayed'])} of the " + f"{_millions(replay['replayed']['dollars'], 1)} replayed error dollars)" + ), + ( + f"In {counts['reproduced_solver_moved_input_n']} the solver moved an input; " + f"in {counts['reproduced_without_move_n']} nothing moved" + ), + ( + f"In {counts['not_reproduced_solver_moved_input_n']} the solver moved an " + "input and stopped short" + ), + ( + f"They carry {_millions(misses['dollars'], 2)}/yr: " + f"{_percent(misses['share_of_replayed'])} of replayed error dollars and " + f"{_percent(misses['share_of_colorado_error_dollars'])} of Colorado" + ), + f"gives {_millions(misses['engine_gap_dollars'], 2)}.", + ( + f"{replay['reproduced_with_computational_finding']['n']} reproduce " + f"({_millions(replay['reproduced_with_computational_finding']['dollars'], 1)}" + f"/yr, " + f"{_percent(replay['reproduced_with_computational_finding']['share_of_colorado_error_dollars'])}" + " of Colorado error dollars), " + f"{replay['not_reproduced_with_computational_finding']['n']} do not " + f"({_millions(replay['not_reproduced_with_computational_finding']['dollars'], 1)}" + f", " + f"{_percent(replay['not_reproduced_with_computational_finding']['share_of_colorado_error_dollars'])})" + ), + ( + f"The {CASES['layer2_computational_findings']['cases']} pure_math and mixed " + f"cases carry {CASES['layer2_computational_findings']['findings']} " + "computational findings" + ), + f"on ${may['totals']['US']['issuance_rawben_dollars'] / 1e9:.1f}B of issuance", + f"{CASES['reconstruction']['national_rows']:,} national error rows", + ] + + # Software-cause table rows (Colorado and national). + labels = { + "software_17_19": "Software (17, 19)", + "worker_computation_20_21": "Worker computation (20, 21)", + "data_entry_18": "Data entry (18)", + "policy_or_budgeted_10_22": "Policy misapplied or budgeted wrong (10, 22)", + } + for name, label in labels.items(): + colorado = may["totals"]["CO"]["axis1"][name] + national = may["totals"]["US"]["axis1"][name] + national_dollars = national["dollars"] / 1e6 + shown = ( + f"${national_dollars:,.0f}M" + if national_dollars >= 1000 + else f"${national_dollars:.1f}M" + ) + quoted.append( + f"| {label} | {_millions(colorado['dollars'], 1)}/yr " + f"({colorado['n']} cases) | {_percent(colorado['share'])} | " + f"{shown}/yr | {_percent(national['share'])} |" + ) + + # Cost-share table rows: all error dollars, then above the $56 threshold. + split = may["class_by_replay_outcome"]["broad_10_17_19_20_21_22"] + rows = { + "All": "all", + "Replay reproduces the issued benefit": "reproduced", + "Replay does not reproduce": "not_reproduced", + "Not replayed (solver filters)": "not_replayed", + } + for label, part in rows.items(): + every, above = split["all_errors"][part], split["above_threshold"][part] + quoted.append( + f"| {label} | {every['n']} ({above['n']}) | {_percent(every['share'])} | " + f"{every['points_of_official_fy2024_rate']:.2f} | " + f"{_percent(above['share'])} | " + f"{above['points_of_official_fy2024_rate']:.2f} |" + ) + candidates = replay["computation_candidates"] + quoted.append( + f"| The 7 computation candidates | {candidates['n']} " + f"({candidates['above_threshold_n']}) | " + f"{_percent(candidates['share_of_colorado_error_dollars'])} | " + f"{candidates['points_of_official_fy2024_rate']:.2f} | " + f"{_percent(candidates['share_of_colorado_above_threshold_error_dollars'])} | " + f"{candidates['above_threshold_points_of_official_fy2024_rate']:.2f} |" + ) + + # August-weights table rows that carry both postings. + def both(label, path, digits=2): + left, right = may, aug + for key in path: + left, right = left[key], right[key] + return ( + f"| {label} | {_millions(left['dollars'], digits)} " + f"({_percent(left['share'])}) | {_millions(right['dollars'], digits)} " + f"({_percent(right['share'])}) |" + ) + + quoted += [ + both("Layer 1 strict, Colorado", ("totals", "CO", "layer1", "strict_17_19_20")), + both( + "Layer 1 broad, Colorado", + ("totals", "CO", "layer1", "broad_10_17_19_20_21_22"), + ), + both("Data entry (18), Colorado", ("totals", "CO", "axis1", "data_entry_18")), + both( + "Software (17, 19), Colorado", ("totals", "CO", "axis1", "software_17_19") + ), + ( + f"| One cost-share tier (5% of issuance) | " + f"{_millions(may['totals']['CO']['cost_share_step_dollars'], 1)} | " + f"{_millions(aug['totals']['CO']['cost_share_step_dollars'], 1)} |" + ), + ] + missing = [q for q in quoted if q not in text] + assert not missing, missing + + +# --------------------------------------------------------------------------- +# regeneration from the raw postings (skips without the data) + + +def _postings_available() -> bool: + for label, spec in audit.POSTINGS.items(): + path = audit.posting_path(label) + if not path.exists() or audit.sha256(path) != spec["csv_sha256"]: + return False + return True + + +@pytest.mark.skipif(not _postings_available(), reason="FY2024 QC postings absent") +def test_audit_regenerates_from_postings(assert_artifact_values_match): + assert_artifact_values_match(json.loads(audit.render(audit.build())), AUDIT) + + +# --------------------------------------------------------------------------- +# property tests on the classification and partition logic + +ELEMENTS = (150, 311, 331, 350, 362, 364, 366, 520) +NATURES = (6, 37, 38, 44, 52, 53, 54, 56, 57, 75, 80, 97, 98, 123) +AGENCIES = (1, 2, 10, 12, 15, 17, 18, 19, 20, 21, 22, 26) + + +@st.composite +def colorado_errors(draw): + n = draw(st.integers(min_value=1, max_value=25)) + rows = [] + for index in range(n): + row = { + "STATE": 8, + "YRMONTH": 202310 + index // 10, + "HHLDNO": 40000 + index, + "STATUS": draw(st.sampled_from((2, 3))), + "AMTERR": float(draw(st.integers(min_value=1, max_value=900))), + "HWGT": draw(st.floats(min_value=1.0, max_value=6000.0)), + } + populated = draw(st.integers(min_value=0, max_value=4)) + for slot in audit.SLOTS: + if slot <= populated: + row[f"ELEMENT{slot}"] = float(draw(st.sampled_from(ELEMENTS))) + row[f"NATURE{slot}"] = float(draw(st.sampled_from(NATURES))) + agency = draw(st.one_of(st.none(), st.sampled_from(AGENCIES))) + row[f"AGENCY{slot}"] = np.nan if agency is None else float(agency) + else: + row[f"ELEMENT{slot}"] = np.nan + row[f"NATURE{slot}"] = np.nan + row[f"AGENCY{slot}"] = np.nan + rows.append(row) + frame = pd.DataFrame(rows) + frame["key"] = [audit.case_key(y, h) for y, h in zip(frame.YRMONTH, frame.HHLDNO)] + return frame + + +@given( + st.sampled_from(ELEMENTS), + st.sampled_from(NATURES), + st.one_of(st.none(), st.sampled_from(AGENCIES)), +) +def test_is_computational_rule(element, nature, agency): + result = audit.is_computational((element, nature, agency)) + if ( + element == audit.ARITHMETIC_ELEMENT + or nature in audit.INHERENT_COMPUTATION_NATURES + ): + assert result + elif nature in audit.DEDUCTION_NATURES: + assert result == (agency in audit.BROAD_CODES) + else: + assert not result + + +@settings(max_examples=150, deadline=None) +@given(colorado_errors()) +def test_layer2_classes_partition_cases_and_dollars(frame): + result = audit.phase_a_classification(frame) + totals = result["classes"] + assert sum(v[0] for v in totals.values()) == len(frame) + dollars = float((frame.HWGT * frame.AMTERR).sum()) + assert sum(v[2] for v in totals.values()) == pytest.approx(dollars, rel=1e-12) + assert len(result["pure_math"]) == totals["pure_math"][0] + assert len(result["input_system_caused"]) == totals["input_system_caused"][0] + for case in result["pure_math"]: + assert case["findings"] + assert all(audit.is_computational(tuple(f)) for f in case["findings"]) + for case in result["input_system_caused"]: + assert not any(audit.is_computational(tuple(f)) for f in case["findings"]) + assert any(f[2] in audit.BROAD_CODES for f in case["findings"]) + + +@settings(max_examples=150, deadline=None) +@given(colorado_errors(), st.data()) +def test_replay_split_partitions_each_class(frame, data): + errors = audit.error_cases(frame) + replayed = data.draw( + st.lists(st.booleans(), min_size=len(frame), max_size=len(frame)) + ) + reproduced = data.draw( + st.lists(st.booleans(), min_size=len(frame), max_size=len(frame)) + ) + joined = pd.DataFrame( + { + "key": frame.key[np.array(replayed)], + "reproduced": np.array(reproduced)[np.array(replayed)], + } + ) + split = audit.broad_split(joined, errors) + for classes in split.values(): + for parts in classes.values(): + pieces = ("reproduced", "not_reproduced", "not_replayed") + assert sum(parts[p]["n"] for p in pieces) == parts["all"]["n"] + assert sum(parts[p]["share"] for p in pieces) == pytest.approx( + parts["all"]["share"], abs=1e-9 + ) + + +@settings(max_examples=100, deadline=None) +@given(colorado_errors(), st.floats(min_value=0.01, max_value=100.0)) +def test_shares_are_invariant_to_rescaling_weights(frame, factor): + scaled = frame.assign(HWGT=frame.HWGT * factor) + for codes in (audit.STRICT_CODES, audit.BROAD_CODES, audit.SOFTWARE_CODES): + base, other = audit.error_cases(frame), audit.error_cases(scaled) + total, total_scaled = base.error_dollars.sum(), other.error_dollars.sum() + left = audit.class_metric(base, audit.any_code(base, codes), total) + right = audit.class_metric(other, audit.any_code(other, codes), total_scaled) + assert left["n"] == right["n"] + assert left["share"] == pytest.approx(right["share"], abs=1e-9) + + +@settings(max_examples=100, deadline=None) +@given(colorado_errors()) +def test_any_presence_is_monotone_in_the_code_set(frame): + errors = audit.error_cases(frame) + total = errors.error_dollars.sum() + strict = audit.class_metric( + errors, audit.any_code(errors, audit.STRICT_CODES), total + ) + broad = audit.class_metric(errors, audit.any_code(errors, audit.BROAD_CODES), total) + assert strict["n"] <= broad["n"] + assert strict["dollars"] <= broad["dollars"] + 0.01 diff --git a/uv.lock b/uv.lock index e11a2f3..41ee9f1 100644 --- a/uv.lock +++ b/uv.lock @@ -15,6 +15,94 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/d1/d6/3965ed04c63042e047cb6a3e6ed1a63a35087b6a609aa3a15ed8ac56c221/colorama-0.4.6-py2.py3-none-any.whl", hash = "sha256:4f1d9991f5acc0ca119f9d443620b77f9d6b33703e51011c16baf57afb285fc6", size = 25335, upload-time = "2022-10-25T02:36:20.889Z" }, ] +[[package]] +name = "hypothesis" +version = "6.168.3" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "sortedcontainers" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/09/b7/13118bbc45d6d8b9d04e2de779e2a4ff23145ea39efa692b994fb874ca72/hypothesis-6.168.3.tar.gz", hash = "sha256:a43388f9067678fef6e13bdff325b6cfa6961a590498bb37f7ff31589c83bc75", size = 511022, upload-time = "2026-09-28T05:20:58.499Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/2b/cd/746a2fc1e5e5ba34f07ea6ee1ad07525a3dde5ff4836db3a7980af49a891/hypothesis-6.168.3-cp310-abi3-macosx_10_12_x86_64.whl", hash = "sha256:d20972ca134e652e9928ecd200966a8a2857adc2812c1d74b12a872e5cd503eb", size = 791536, upload-time = "2026-09-28T05:18:57.162Z" }, + { url = "https://files.pythonhosted.org/packages/c3/0f/7a158e377b69556c8e12c25c8fd811d0103e54c24a12dab1f3b9819f202e/hypothesis-6.168.3-cp310-abi3-macosx_11_0_arm64.whl", hash = "sha256:eafbec09d3e87d13d8242411d1f5868b5e879f1bd6d95e233e9ff80c527bea1c", size = 787313, upload-time = "2026-09-28T05:20:18.122Z" }, + { url = "https://files.pythonhosted.org/packages/45/c6/7df9104ee359e8fcd77791781c47e70794964ca69673665e0de2a6590f4c/hypothesis-6.168.3-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:73c5627497968cc62d140e9fde1ea21e12cac7b64dc419fea8286cd34ff1ad4e", size = 1124036, upload-time = "2026-09-28T05:20:04.372Z" }, + { url = "https://files.pythonhosted.org/packages/92/0f/8a61715404a73b9a82cb1f976803428d603cdea976a9c30266c72ad6a7ce/hypothesis-6.168.3-cp310-abi3-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:29dc56e6dc6eeb0aeb08ac463f847279bde2336c4786faf7ee0af4b93cd031a7", size = 1147875, upload-time = "2026-09-28T05:19:15.495Z" }, + { url = "https://files.pythonhosted.org/packages/e4/3f/2b16e95cc3b9a069afa23830a012aefe377ce4a457a227e5a5026eb827ff/hypothesis-6.168.3-cp310-abi3-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:753bb501f8560d2e321ed3b62596b4496c56f15668b0a0669231ccc3e6c4e80d", size = 1149489, upload-time = "2026-09-28T05:19:18.886Z" }, + { url = "https://files.pythonhosted.org/packages/f6/e7/d7cf6dc068bb02732b2a6e38a35a0dcb7f2137608198fc79d740f69d197c/hypothesis-6.168.3-cp310-abi3-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:71ab606c472449cb872ef2a7acaec679ee4f651b7f0a477a04ebc85d8933ef8c", size = 1191992, upload-time = "2026-09-28T05:19:21.83Z" }, + { url = "https://files.pythonhosted.org/packages/5f/26/f1e5b25dec15998e8375d722825ec3dae1075cbe7cb1f74221b2312a2e33/hypothesis-6.168.3-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:26928956c54748e4dfa587333741ab750246123b93484955646ff316cca27eef", size = 1169911, upload-time = "2026-09-28T05:19:27.996Z" }, + { url = "https://files.pythonhosted.org/packages/03/36/901234a49147e6bf8a5b9e54b775706a223c2b52befea1106ea2aaf15d48/hypothesis-6.168.3-cp310-abi3-manylinux_2_31_riscv64.whl", hash = "sha256:0bdfc53c041b61c854fa3761735736b991bc2bddafee45a361cfe4d43027c1ef", size = 1129406, upload-time = "2026-09-28T05:18:42.632Z" }, + { url = "https://files.pythonhosted.org/packages/55/c1/5fc9ff91ec9fba6bb527833386c66812524b5cc75fe7d4a4f9122ee1c420/hypothesis-6.168.3-cp310-abi3-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:650528e2b1e2a45e95c2df624d4b9364b8ac5a027ba84e1eea2bc5398dd01dcd", size = 1160399, upload-time = "2026-09-28T05:20:23.911Z" }, + { url = "https://files.pythonhosted.org/packages/9b/d2/49ad1ef5c55547c8abfd0fc571b3493806dda530fd5cb3d8263dbad86a72/hypothesis-6.168.3-cp310-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:209dc54cdb1b4d6d7020ad8d09e44c49756361b7183adacea4d6f5705085b595", size = 1299906, upload-time = "2026-09-28T05:18:58.417Z" }, + { url = "https://files.pythonhosted.org/packages/0d/5f/f98a26094c4f462c4215807bed0f9a6b5508b27a3c329bcba1bc8adc7ac7/hypothesis-6.168.3-cp310-abi3-musllinux_1_2_armv7l.whl", hash = "sha256:af8ca98cd11dc7f9427bb90483abcd9688d4df2e8e63893a30a2423b027ebb11", size = 1425538, upload-time = "2026-09-28T05:20:49.074Z" }, + { url = "https://files.pythonhosted.org/packages/d0/b8/c5b4406e3a6591e51a42f85469351d1d824e0a54b84167aec9b2353759fe/hypothesis-6.168.3-cp310-abi3-musllinux_1_2_i686.whl", hash = "sha256:f8122cfdd0bc0ba843063effa22ac310bdebdb3a1319bf1e63f14baa119c72ab", size = 1377084, upload-time = "2026-09-28T05:20:06.618Z" }, + { url = "https://files.pythonhosted.org/packages/e0/01/0d84a8ea469d024c15f602799c0b8410fbe03d99fdb0370449b33c3c916f/hypothesis-6.168.3-cp310-abi3-musllinux_1_2_ppc64le.whl", hash = "sha256:4bedbb379eab34f792af7ee9a05aae04e9c08bbb52e0f90f5a2110d4fe4b2fbd", size = 1281162, upload-time = "2026-09-28T05:19:42.768Z" }, + { url = "https://files.pythonhosted.org/packages/22/b5/ea6435038de1a795f005cb603531e8ccc053febc50180f3ab55583c9c60e/hypothesis-6.168.3-cp310-abi3-musllinux_1_2_riscv64.whl", hash = "sha256:5a953115b9f5c95133ab2d04efffeec96e5658c3207923dca7285f7db3e6bbef", size = 1300433, upload-time = "2026-09-28T05:18:55.634Z" }, + { url = "https://files.pythonhosted.org/packages/6c/78/fa6c77d9f64b6d592e69e582d83cbce7d7e56897faa4762885a2123b0166/hypothesis-6.168.3-cp310-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:6173558e676ad25ed1e20507fa4024a0d816dd90f715f77006e5a28f19109026", size = 1336273, upload-time = "2026-09-28T05:19:08.273Z" }, + { url = "https://files.pythonhosted.org/packages/f1/56/4acd0d778bab818c9ea336fcba768bf43bc70db7f84cc373579298df4641/hypothesis-6.168.3-cp310-abi3-win32.whl", hash = "sha256:dc66390fb12d80585aa9222bf538ce8b7aa22cf5d118250647355a1c9e8f62f4", size = 678211, upload-time = "2026-09-28T05:19:23.355Z" }, + { url = "https://files.pythonhosted.org/packages/b0/db/0a02b146ad1c30f9716362f2e1a379dfd6b0f7be03ac1e22c9da24c9e2a9/hypothesis-6.168.3-cp310-abi3-win_amd64.whl", hash = "sha256:92325b276360fe86c5bf71a568c0d53a6d140b0de36dcf029f9164a17803bb24", size = 684906, upload-time = "2026-09-28T05:18:35.099Z" }, + { url = "https://files.pythonhosted.org/packages/a3/c0/958deaf726848f96f52250740bf39f13f476b068e22f73e2b358eaa07532/hypothesis-6.168.3-cp310-abi3-win_arm64.whl", hash = "sha256:3cf6f1eeaf41cd8d60cf1f88fde905ca1dd77c906929a507d6ac7f66f2ccba2a", size = 683292, upload-time = "2026-09-28T05:19:57.404Z" }, + { url = "https://files.pythonhosted.org/packages/f4/ab/443208c2988a24eaafbfbd6791b897ad9a8a1dd7f870d5b455d7d15a28ad/hypothesis-6.168.3-cp311-cp311-macosx_10_12_x86_64.whl", hash = "sha256:9029775dc2e25e3e8e596c4306926f0079158353631c7c52c285bbc5c8073c2b", size = 792239, upload-time = "2026-09-28T05:19:59.144Z" }, + { url = "https://files.pythonhosted.org/packages/cf/7b/619c1a76c52de2d3cfd68a2f9876c09464d63097db5bf86a28b62dc37bb1/hypothesis-6.168.3-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:0e1d91530cefdb9e0b6467c203eb509c61a11d70480a37db59cb53a2f9913102", size = 788113, upload-time = "2026-09-28T05:19:17.145Z" }, + { url = "https://files.pythonhosted.org/packages/ea/dd/33841e8833105f850214a486d83740ba52041892ca94e687a05829c0ce83/hypothesis-6.168.3-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:0df00cbe8133aa220308d11fa28e3add996cf5102a39b279c1d711015fd9114b", size = 1124241, upload-time = "2026-09-28T05:18:37.589Z" }, + { url = "https://files.pythonhosted.org/packages/4b/43/c84a7900feced6a420e824ab61d316e4f59a6887d991bc143ecc294a09ed/hypothesis-6.168.3-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:fd7f75a2e23288ee82ee965a952473d9c5447c2cbc1afc94be09d2402201774e", size = 1170406, upload-time = "2026-09-28T05:19:46.365Z" }, + { url = "https://files.pythonhosted.org/packages/47/5d/6e2a32b9c3020af7a06b00fe446a2c5155ae76ac468c0b68d7188e524002/hypothesis-6.168.3-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:bc2b1c37631f2231dd0d077225aa5908fc2bd34a4c3faf475053b0bc2eb24c95", size = 1300339, upload-time = "2026-09-28T05:20:31.968Z" }, + { url = "https://files.pythonhosted.org/packages/f9/78/19f3c9eb9a5c215b9fbde57be18ef7c1c78afa509b460d4e8f86026d741a/hypothesis-6.168.3-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:73960a58f6efc8cbc57d8ad0647955f7c5b25f3d562e3eaa57afa1c69894f7aa", size = 1336484, upload-time = "2026-09-28T05:19:02.841Z" }, + { url = "https://files.pythonhosted.org/packages/88/bb/d46c4519e989e022667705d8a56e83a09ad4ea50eae07ca4bb0f4708dd23/hypothesis-6.168.3-cp311-cp311-win_amd64.whl", hash = "sha256:3f1122223759acc0fe5301c1ae505d762ee61af91db7b191a791ccdbcd965462", size = 684646, upload-time = "2026-09-28T05:20:00.842Z" }, + { url = "https://files.pythonhosted.org/packages/7a/56/44a17266bcf1add5fba7c3df5c3920396a5e924d917fb0534a6f9e2b9b9a/hypothesis-6.168.3-cp312-cp312-macosx_10_12_x86_64.whl", hash = "sha256:bff12e036b67abc5ded682f724f4d7afaff5223190270a0763695f218c079068", size = 793313, upload-time = "2026-09-28T05:20:22.045Z" }, + { url = "https://files.pythonhosted.org/packages/d8/18/1a25b11e9133cb54c02ee6cf99a61296ad461aabc7aae3399e1d0ff1f095/hypothesis-6.168.3-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:c73c6188056e6dc110260e39d141c69c3d02f23e3ba1d8610c7a8b6f2510ec96", size = 784855, upload-time = "2026-09-28T05:19:34.496Z" }, + { url = "https://files.pythonhosted.org/packages/8b/da/728d3210f0b0089337f37206a100d31ef546d1f99839e709f0a9ca443f8e/hypothesis-6.168.3-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:bb1063cb794765097b8c0add598bd34a904f44dc15db81275dd1f0ed1ac567ed", size = 1123043, upload-time = "2026-09-28T05:19:05.468Z" }, + { url = "https://files.pythonhosted.org/packages/53/86/bfb9ec344a8b3ef2fa6f59bb3d30c5b3a7d86d594107670539c34ee53d1b/hypothesis-6.168.3-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:9b2d47f5d9060b049039bef0b267e88480f6468f95ee394e514e84a81ccdc5f5", size = 1169121, upload-time = "2026-09-28T05:20:40.512Z" }, + { url = "https://files.pythonhosted.org/packages/3b/e0/5a76dbafbcaf8fff075f09377cddefa17289ef05dbf51878dde88385b966/hypothesis-6.168.3-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:8367f623a98cbf90f33fd2a43b100671b152b5975bc343a578290945f52f7018", size = 1298749, upload-time = "2026-09-28T05:19:32.883Z" }, + { url = "https://files.pythonhosted.org/packages/28/15/0e65af96b353f2c4bdc283c53ed0980e5ac5009afe97ef3b23c77b364664/hypothesis-6.168.3-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:fd81241f4cd76fb18d481e87b9688b30a0cb5c30841730513323ba1c7aba1837", size = 1335271, upload-time = "2026-09-28T05:20:54.005Z" }, + { url = "https://files.pythonhosted.org/packages/f5/c5/37eab2155664aa544df321f6d3940b89c6702d79a72fdc98d7b5a2286f8c/hypothesis-6.168.3-cp312-cp312-win_amd64.whl", hash = "sha256:377438de53afb94347d9845b7d90db5905c6d2a8d612108e9938e0f80503bb90", size = 682211, upload-time = "2026-09-28T05:18:51.206Z" }, + { url = "https://files.pythonhosted.org/packages/0c/a2/6787da846d929e52fc3344d803299c45782dbeac287528af08380e984bf8/hypothesis-6.168.3-cp313-cp313-macosx_10_12_x86_64.whl", hash = "sha256:1b230a850de63334c16654a34a2d547e0179d36b9071d4439b3e7237f6d077e7", size = 793254, upload-time = "2026-09-28T05:19:36.294Z" }, + { url = "https://files.pythonhosted.org/packages/89/ba/2893f5ca42501f4d562cba3229c8994c3a3e5fac66ef60c1e0e0b5220a5d/hypothesis-6.168.3-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:d63b0226cd3e0d8bdd97c3384b22a21934ed4d53246c1c93575dada616672499", size = 784699, upload-time = "2026-09-28T05:18:54.194Z" }, + { url = "https://files.pythonhosted.org/packages/b7/aa/7d7349daf75b71f6f35876f8de115e974c5c04d600e86df1b2779b0883ab/hypothesis-6.168.3-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:0369f5df055f96e117ab12e5f249668ff144731ab5280bc7a205fdf81f990b89", size = 1123016, upload-time = "2026-09-28T05:20:36.372Z" }, + { url = "https://files.pythonhosted.org/packages/00/f0/7774e1ea072708ea46cb25c4aeee5f9978b6c271764809069e8a6789858e/hypothesis-6.168.3-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:f076bcd0f77fdcdb7797826c879099573d03a02e228ebe649ea917081b962ac0", size = 1168989, upload-time = "2026-09-28T05:20:08.44Z" }, + { url = "https://files.pythonhosted.org/packages/a7/7d/113992abed9efbd7944e3a496a382da58152dce126fddf6419ff31a0a57d/hypothesis-6.168.3-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:b823ba1fcec8da730f29316d010b06d3f7e0c3828dcf91020e24c55e7d24652a", size = 1298626, upload-time = "2026-09-28T05:19:40.986Z" }, + { url = "https://files.pythonhosted.org/packages/d5/65/3659fa5e733027e5b37a27486e57f4053535e40853d23501c422bb275043/hypothesis-6.168.3-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:dd2849c269d674e4618590f3b48d443bd4c06b5aef3d8d869086c2e6d213d248", size = 1335184, upload-time = "2026-09-28T05:20:20.087Z" }, + { url = "https://files.pythonhosted.org/packages/b4/6c/35aab2221b5ea65125340f236e1a765a9e9f3b28d89a389ea0033fed29f7/hypothesis-6.168.3-cp313-cp313-win_amd64.whl", hash = "sha256:3ef7d26f5789e691401d5f87eafed9bd2763f0dbe47af6d6b66012a509404766", size = 682173, upload-time = "2026-09-28T05:19:12.762Z" }, + { url = "https://files.pythonhosted.org/packages/6e/79/27b0cb56ff5d2bc92458bf6fcebdb1f0dc01f57f043a178e9f9d74498169/hypothesis-6.168.3-cp314-cp314-macosx_10_12_x86_64.whl", hash = "sha256:e2d4c68729a13df9af4998d2652cfb5d541c5609b88c306880a5dfb284938ec2", size = 793287, upload-time = "2026-09-28T05:20:38.522Z" }, + { url = "https://files.pythonhosted.org/packages/e8/bd/5342f95c3bc36586777ef8cb8eba87ac7acf0184210327fb0e6d8654e295/hypothesis-6.168.3-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:b345f818083ec99966a43ca4f7b38feb62c6920ce28572bc2bde4948a8b7eaba", size = 784857, upload-time = "2026-09-28T05:20:42.609Z" }, + { url = "https://files.pythonhosted.org/packages/4a/5f/fc774241518a0588d362680a3084275058aab86d6bc02a2c61e1073142d0/hypothesis-6.168.3-cp314-cp314-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:0608c610fc002978fc5de0f471770d8817e9455cb8649a47983c9f8bccfc1897", size = 1123330, upload-time = "2026-09-28T05:19:29.735Z" }, + { url = "https://files.pythonhosted.org/packages/b0/e6/131f16775a3dca5f4fe27f0d6ad9a9851600098a54a872d163f6ebf0b664/hypothesis-6.168.3-cp314-cp314-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:9dcf6448b1ecc37f2b23f2d1b3ddfc9ff6b6910614c15a642dc82f419b96037b", size = 1169178, upload-time = "2026-09-28T05:19:53.436Z" }, + { url = "https://files.pythonhosted.org/packages/06/61/c9f5bff8b73321c9fd12f2d69666c6e4861b97b60f5f24fdd6ac293007fd/hypothesis-6.168.3-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:ae4f9f094041dcce5119ebd7bab71062743ab02b6e654d056b370beda78c19e2", size = 1299127, upload-time = "2026-09-28T05:20:14.486Z" }, + { url = "https://files.pythonhosted.org/packages/e4/6d/d4618f8ab12dd76c4d58ea052252759aa2bf0c4e71f4b502ba5d577e0b9f/hypothesis-6.168.3-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:d479985fe73af97badfb72dc6d20c6a353e486a36f6d065f600026e0cf954a86", size = 1335390, upload-time = "2026-09-28T05:20:10.326Z" }, + { url = "https://files.pythonhosted.org/packages/95/b6/727161e17cc297df1aeea945de05c532033b66751625bec192bc5b5ff707/hypothesis-6.168.3-cp314-cp314-pyemscripten_2026_0_wasm32.whl", hash = "sha256:785e2c45f8c08e274b4bf1ccae97f1a4e09407e9a68e80790cfab1d17a4a45fa", size = 624291, upload-time = "2026-09-28T05:18:36.376Z" }, + { url = "https://files.pythonhosted.org/packages/77/23/2f9b506392a0b8712440bcb781abc5713be9ffac1303b6d10f3669050882/hypothesis-6.168.3-cp314-cp314-win_amd64.whl", hash = "sha256:320920b1e3dae8611eee8a03d063cf2187446f2a17c38cfb8a7fc1466f71eee2", size = 682064, upload-time = "2026-09-28T05:20:16.336Z" }, + { url = "https://files.pythonhosted.org/packages/50/93/efabfd95eb2b69c0c9fa1e1c83c8290aa2715d46baeb5a7764faf09c133a/hypothesis-6.168.3-cp314-cp314t-macosx_10_12_x86_64.whl", hash = "sha256:35380baa981108a7f60c4eab71e46acd8d8f58440520346a1a6aba06dca7e074", size = 791875, upload-time = "2026-09-28T05:19:37.789Z" }, + { url = "https://files.pythonhosted.org/packages/1b/fc/2a0ada1623a9a33048bbf9f00a7c92513398c6b8865cba5efb854262ce30/hypothesis-6.168.3-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:f0aaaed00438fa6673d856aaee12a4a8afbde82a9f3c5ff487cacfb064239afe", size = 783412, upload-time = "2026-09-28T05:18:48.266Z" }, + { url = "https://files.pythonhosted.org/packages/41/31/73c615d37eeb12da0209555612c32ced628d860267c23cfd2691c6207aea/hypothesis-6.168.3-cp314-cp314t-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:f27e6df1576bf838e7f484d4ae5cba92114497ca30afb917056bf1ea33e95675", size = 1121662, upload-time = "2026-09-28T05:19:26.225Z" }, + { url = "https://files.pythonhosted.org/packages/92/54/1893344a3b8bbf83fa7c9bb7e96f6836580213854f46671cb749f7b4706c/hypothesis-6.168.3-cp314-cp314t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:3bc85014577982ec6d2e266edc5cd6e7a7d3674c648791ca983c67a52f89a0c4", size = 1167761, upload-time = "2026-09-28T05:18:49.85Z" }, + { url = "https://files.pythonhosted.org/packages/26/97/a61d82968febd25a0f08ffbbfabfaeead66f68fc5fa73f3c9ece1c36fc8f/hypothesis-6.168.3-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:ccfc29505aa1cdcc254cb9cd701fd0811fef5480addd97d4127a00df023410f7", size = 1297309, upload-time = "2026-09-28T05:19:51.711Z" }, + { url = "https://files.pythonhosted.org/packages/bc/01/fe8e4cf6d39d02f4efadaa351427fa13722086cfc54a2da3579741c5177a/hypothesis-6.168.3-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:32d0699566aaa93f9e97a44705d7164386f1e91de78d277f53bada35e78bcc96", size = 1334260, upload-time = "2026-09-28T05:19:00.121Z" }, + { url = "https://files.pythonhosted.org/packages/39/6a/09177ebc62f4778f94379ecade6a51299d3d8c4f4b75ba636bf3a0202364/hypothesis-6.168.3-cp314-cp314t-win_amd64.whl", hash = "sha256:28d88fa174ecbd4ecbd7bb290f06d0db084a3971c3f511ae2830b65b5e25500f", size = 681992, upload-time = "2026-09-28T05:19:48.043Z" }, + { url = "https://files.pythonhosted.org/packages/f9/f6/890bf33d608cd63348d3146a3ca83362fbf09e589c5e5230a00c403070f4/hypothesis-6.168.3-cp315-abi3.abi3t-macosx_10_12_x86_64.whl", hash = "sha256:61f5782d807b1e6aa5c9beef1054cb2037e7d1add48a64972cd6ee447778781f", size = 791231, upload-time = "2026-09-28T05:18:43.998Z" }, + { url = "https://files.pythonhosted.org/packages/03/dd/fc36b204f8aa7437c676576d9437f0422de46aa8a729e6cd5bbb0224e759/hypothesis-6.168.3-cp315-abi3.abi3t-macosx_11_0_arm64.whl", hash = "sha256:7aacf3cf40c7ce8f9e4924d5b57bc0b068beafdfed2347bcf160a28b4cfbba7b", size = 783171, upload-time = "2026-09-28T05:19:20.353Z" }, + { url = "https://files.pythonhosted.org/packages/5c/d2/3cd087d577db67c404991d7723e64d7cd75570cdc82a0f7329aed6f2086c/hypothesis-6.168.3-cp315-abi3.abi3t-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:01768a03a30dc54df7fe457c0b34016c84d598fa00d5965ededab93ba4eb3408", size = 1121152, upload-time = "2026-09-28T05:18:39.095Z" }, + { url = "https://files.pythonhosted.org/packages/42/6f/a05a68cc66e7a22701c50bf94fb9ddeb36f4f5d0de3cc130066709eda621/hypothesis-6.168.3-cp315-abi3.abi3t-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:1a8a4ffc6c6e6e577f2bfbcfebf7cffbb310283ba2532c729570a0a752cebaaa", size = 1144069, upload-time = "2026-09-28T05:20:44.705Z" }, + { url = "https://files.pythonhosted.org/packages/4a/f4/fdd7a093fb2fb3a0ce24256c92ad5bf67936f4669bd06d5ceae4298150ec/hypothesis-6.168.3-cp315-abi3.abi3t-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:b987d73eba95183a7e59cca6d1925c588aa4307922d9852cc2b4d282a2ae4128", size = 1146692, upload-time = "2026-09-28T05:20:25.83Z" }, + { url = "https://files.pythonhosted.org/packages/f1/c8/6dbd4377e935505ae4fc8ee4c7b18c69994c4bbaca015447c470218bdbee/hypothesis-6.168.3-cp315-abi3.abi3t-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:d4569c39bd97d9573e946429ed676f3b55a7c8ed80920addd67d384a47d59b38", size = 1189480, upload-time = "2026-09-28T05:20:46.734Z" }, + { url = "https://files.pythonhosted.org/packages/ba/bf/ff288b496b690000d2686dcfa7f67855e4c5a6dddb464ed8e4880be83bb2/hypothesis-6.168.3-cp315-abi3.abi3t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:6042b8707a4b25bbbfe20b110258fb7549a68951e5525608a8ce06091039e9fc", size = 1167109, upload-time = "2026-09-28T05:19:24.758Z" }, + { url = "https://files.pythonhosted.org/packages/d9/d1/3fee2bc29fc747fabbb27e174515592e390809ca465d3bbbabbcfa3235a8/hypothesis-6.168.3-cp315-abi3.abi3t-manylinux_2_31_riscv64.whl", hash = "sha256:f413629de94d38a7a2ad259697a6752526143cb49c2e2d7bdba19381c80693f6", size = 1126823, upload-time = "2026-09-28T05:18:52.636Z" }, + { url = "https://files.pythonhosted.org/packages/58/6a/585294392fa6a6d9719446a042a777ba98ecca265d9e48cbedc10c88df0f/hypothesis-6.168.3-cp315-abi3.abi3t-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:f071737e4e775bebba07e319eb645a880d1e0186b4d24bad23255e849d86d483", size = 1155773, upload-time = "2026-09-28T05:20:51.268Z" }, + { url = "https://files.pythonhosted.org/packages/cd/4d/fa283ff79996debf1ff08593e01f8b773eb8211b19b3d6d5d491317a65bf/hypothesis-6.168.3-cp315-abi3.abi3t-musllinux_1_2_aarch64.whl", hash = "sha256:f59a3912858d0609c26054aa1c474847c797937f5e0e1c77f0d3f08032e50f09", size = 1296671, upload-time = "2026-09-28T05:18:46.937Z" }, + { url = "https://files.pythonhosted.org/packages/42/05/98f9c2f628da5afabb6c3e8df5d2d299d2d50bf306b14a6e26d3954c4951/hypothesis-6.168.3-cp315-abi3.abi3t-musllinux_1_2_armv7l.whl", hash = "sha256:f09c05a23a8025dd5cad07c2ed299a46778e72d49ff0dd5bf353bcc0f7dc1580", size = 1422001, upload-time = "2026-09-28T05:19:04.188Z" }, + { url = "https://files.pythonhosted.org/packages/b8/c9/f2109ade29a7ec284d55bca328d784743e8b13e321576fd34ee7e65f99af/hypothesis-6.168.3-cp315-abi3.abi3t-musllinux_1_2_i686.whl", hash = "sha256:44ace770bda3a0301739fc5a413c764de790c739df1f1a7218dd049f8594d9f5", size = 1374182, upload-time = "2026-09-28T05:20:27.764Z" }, + { url = "https://files.pythonhosted.org/packages/ce/6d/e05d5f72441564a3bebc71fa155deafd0fd3b015d6014ec8a00edf6e42bb/hypothesis-6.168.3-cp315-abi3.abi3t-musllinux_1_2_ppc64le.whl", hash = "sha256:03b131043608f94a2578a2a896a1a72092acb7079815eef5b8513b08447e0e62", size = 1278402, upload-time = "2026-09-28T05:20:56.332Z" }, + { url = "https://files.pythonhosted.org/packages/96/d7/6988a7f1f69c5c530687dce03a32c157e2d878e3a2e32af9269ce016b78a/hypothesis-6.168.3-cp315-abi3.abi3t-musllinux_1_2_riscv64.whl", hash = "sha256:e9784aca26eddfe99b03a0292320db742cc8a74200ef864fdce949f526f973cd", size = 1297770, upload-time = "2026-09-28T05:19:11.387Z" }, + { url = "https://files.pythonhosted.org/packages/a2/28/5bb82b60b836bd2329e1fe01ad94efc2b14dfa806f8e50ed14b795d78770/hypothesis-6.168.3-cp315-abi3.abi3t-musllinux_1_2_x86_64.whl", hash = "sha256:2b52ac363096232bebc2add117e9178d91f4248f4cdb919fd1026b5f86a4bb16", size = 1333977, upload-time = "2026-09-28T05:20:02.483Z" }, + { url = "https://files.pythonhosted.org/packages/8e/7d/d419841b8f65481ea1a50c4ba36f2670d48b8c7a51e4e8586089564cf7f6/hypothesis-6.168.3-cp315-abi3.abi3t-win32.whl", hash = "sha256:d28e3a6b511a74ce37df5274b51f36c2b274fea7365e00b16f0c81e22acd5957", size = 675394, upload-time = "2026-09-28T05:19:55.237Z" }, + { url = "https://files.pythonhosted.org/packages/d8/d2/1de6a2ad100e44817f2e9a8e8ba3eeaf4769621f4f87f2d4966c4971fcc4/hypothesis-6.168.3-cp315-abi3.abi3t-win_amd64.whl", hash = "sha256:7b9638789548361a57d984f56619ac694a914c328181d911271618409261ff4a", size = 681699, upload-time = "2026-09-28T05:19:09.957Z" }, + { url = "https://files.pythonhosted.org/packages/c0/79/be3370fd02734d6b1d950183ae9224580d19fa8d356b95348d60f1e3eeb7/hypothesis-6.168.3-cp315-abi3.abi3t-win_arm64.whl", hash = "sha256:65d78e4357ec48ed2c67825f06740ee3599be4cfe770a6092bed07108d679ac5", size = 679849, upload-time = "2026-09-28T05:19:06.884Z" }, + { url = "https://files.pythonhosted.org/packages/3f/0e/27960db1d45a94497e1681fb9d7b9140b44da4e655ba20f0d82f95d10230/hypothesis-6.168.3-pp311-pypy311_pp73-macosx_10_12_x86_64.whl", hash = "sha256:101531b4ccf7fa12965a6b295d10b64c2f8a8fab61e1f9b18c55335d8fb02575", size = 793135, upload-time = "2026-09-28T05:19:14.056Z" }, + { url = "https://files.pythonhosted.org/packages/9a/cb/be08379286c1925fcd651b92ed835f470a83b859b5c484b3431da9d851f5/hypothesis-6.168.3-pp311-pypy311_pp73-macosx_11_0_arm64.whl", hash = "sha256:41e0120814de5c3a6b58d8cb27cb11b8873ab33fba130cd09855a27f5a12a1ca", size = 788995, upload-time = "2026-09-28T05:20:34.164Z" }, + { url = "https://files.pythonhosted.org/packages/38/7b/6d885d139b2ed76e2f4908cf10eae56acc73bf05cbef286357c4559c9a04/hypothesis-6.168.3-pp311-pypy311_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:9d120009d145697909f2f0c532f2d568558ddd6990707f8cba6576bfb9770696", size = 1124974, upload-time = "2026-09-28T05:19:49.903Z" }, + { url = "https://files.pythonhosted.org/packages/05/cc/b302ae809b48254830cae57801d0af0b66783df515b5dd56b9f21c8dfa86/hypothesis-6.168.3-pp311-pypy311_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:818b3d09bba60ef90463e47a4bc87275dba1e8aeb184d3102888440ca7c7837d", size = 1171967, upload-time = "2026-09-28T05:20:12.295Z" }, + { url = "https://files.pythonhosted.org/packages/2c/ab/51031080a6600f5b07adf5f9dfa117345c2adda07a5c24bda523e70bd1e9/hypothesis-6.168.3-pp311-pypy311_pp73-win_amd64.whl", hash = "sha256:9e3c36eba80e089ecfa28ec08049723223cb41d3cfe4d9988169cf98440e402b", size = 685641, upload-time = "2026-09-28T05:19:44.609Z" }, +] + [[package]] name = "iniconfig" version = "2.3.0" @@ -293,18 +381,6 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/1d/5f/8528479ef76dfd4c0f276a5d5518f79c4728048751d661b7b16d2edf6ecc/pyreadstat-1.3.5-cp314-cp314-win_amd64.whl", hash = "sha256:12d6697735d57ca59415493e1177e5977969b1d6804c251e55f4641db4ace7f5", size = 2044671, upload-time = "2026-05-19T12:22:08.178Z" }, ] -[[package]] -name = "quantile-forest" -version = "1.4.1" -source = { registry = "https://pypi.org/simple" } -dependencies = [ - { name = "numpy", version = "2.4.6", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.12'" }, - { name = "numpy", version = "2.5.1", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.12'" }, - { name = "scikit-learn" }, - { name = "scipy" }, -] -sdist = { url = "https://files.pythonhosted.org/packages/62/6e/3f1493d4abcce71fdc82ed575475d3e02da7b03375129e84be2622e1532f/quantile_forest-1.4.1.tar.gz", hash = "sha256:713a23c69562b7551ba4a05c22ce9d0e90db6a73d043e760b29c331cb19dc552", size = 486249, upload-time = "2025-09-10T12:48:04.578Z" } - [[package]] name = "pytest" version = "9.1.1" @@ -342,6 +418,35 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/ec/dd/96da98f892250475bdf2328112d7468abdd4acc7b902b6af23f4ed958ea0/pytz-2026.2-py2.py3-none-any.whl", hash = "sha256:04156e608bee23d3792fd45c94ae47fae1036688e75032eea2e3bf0323d1f126", size = 510141, upload-time = "2026-05-04T01:35:27.408Z" }, ] +[[package]] +name = "quantile-forest" +version = "1.4.1" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "numpy", version = "2.4.6", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.12'" }, + { name = "numpy", version = "2.5.1", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.12'" }, + { name = "scikit-learn" }, + { name = "scipy" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/62/6e/3f1493d4abcce71fdc82ed575475d3e02da7b03375129e84be2622e1532f/quantile_forest-1.4.1.tar.gz", hash = "sha256:713a23c69562b7551ba4a05c22ce9d0e90db6a73d043e760b29c331cb19dc552", size = 486249, upload-time = "2025-09-10T12:48:04.578Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/75/cc/dc1d8d7a3bf1bf8eaff4d810f56970237458482f0a8e892a4d20a27d2386/quantile_forest-1.4.1-cp311-cp311-macosx_10_9_universal2.whl", hash = "sha256:f4d1866c694defc077ee01190d1c69c9ef4092b31c0f86e5ae7ae3098ef7b9be", size = 954993, upload-time = "2025-09-10T12:47:38.784Z" }, + { url = "https://files.pythonhosted.org/packages/d4/eb/b9931f40427665a8bbfbbc00dfe26ecb0d8f9df08be8df6c5f20e4ae43c3/quantile_forest-1.4.1-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:da3e40acf24b60aeb1bf24f7648aeb40f984d6b9a722513e8f9bb13d7a75e1f9", size = 717871, upload-time = "2025-09-10T12:47:39.957Z" }, + { url = "https://files.pythonhosted.org/packages/ef/9a/47e0d2f81115ea4112f41239a669b7440bf71ad50dce92dad86be14aad86/quantile_forest-1.4.1-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:591e12ae0356206668e2ae8f2808749600da7c587ce7819b39b97d0a7c4053d2", size = 709737, upload-time = "2025-09-10T12:47:41.351Z" }, + { url = "https://files.pythonhosted.org/packages/02/2b/dfca97f4b6a8c63cdc839f119719a0f68455c3b1a013711a72f63b3dd90d/quantile_forest-1.4.1-cp311-cp311-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:443341c9047160f36464d72871da7babae04cb8092b9fd19eca86682277ee810", size = 2436079, upload-time = "2025-09-10T12:47:42.936Z" }, + { url = "https://files.pythonhosted.org/packages/a8/f0/9e375572814f44bb93caf942c0de36c483e22a0488241042536c0dc39fb6/quantile_forest-1.4.1-cp311-cp311-win_amd64.whl", hash = "sha256:69d39db8c434fa2aaa48716eb05774491b22d1087f2f24bfcd853b52869d01bc", size = 685513, upload-time = "2025-09-10T12:47:44.045Z" }, + { url = "https://files.pythonhosted.org/packages/93/53/63c400659404b45221405f7dbdb42fb0cea4b9cae0877a567d56d760a995/quantile_forest-1.4.1-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:f7d4eae276928f07c13e4784842768569e92c50e93f66c1feadf85c4967b3be4", size = 959038, upload-time = "2025-09-10T12:47:45.193Z" }, + { url = "https://files.pythonhosted.org/packages/e3/d7/694d428f94b5aec95bd9bb3805b119c1845bb63e215deeeab64e60812037/quantile_forest-1.4.1-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:c0526c117be0df98e79e1ce378968f1e1faa9ca23e08da449baa0651a52a81d1", size = 720471, upload-time = "2025-09-10T12:47:46.873Z" }, + { url = "https://files.pythonhosted.org/packages/8d/fb/747bf715bfba7570f88c7c601ef3f3350eceb4ce4bf72a1d36fb9845fdd2/quantile_forest-1.4.1-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:b67fc17c82ea85f575617f7a093f3ad8ef0dc5a159f886a9948224b98483ad8c", size = 710769, upload-time = "2025-09-10T12:47:47.88Z" }, + { url = "https://files.pythonhosted.org/packages/99/05/86bbce5503c007cfeeb74068edf608c4216e570ad13c9500513f5473740c/quantile_forest-1.4.1-cp312-cp312-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:d402c4af3f72d21c3ca3e9dda25a68207d29ae4d34b8126bcf19fc3680ce23e0", size = 2406284, upload-time = "2025-09-10T12:47:49.42Z" }, + { url = "https://files.pythonhosted.org/packages/8b/93/1ae45144ab80bdd8cf8e7bf983137440b1c3430516a7db340caee9b6d77d/quantile_forest-1.4.1-cp312-cp312-win_amd64.whl", hash = "sha256:b1513b039f7ea5b9467201807b41594d25ecaf088868221e2f1ddea4edeb13b8", size = 685743, upload-time = "2025-09-10T12:47:50.525Z" }, + { url = "https://files.pythonhosted.org/packages/33/61/f8ff4e348dc2d265ea97287f921b92bca265229c48be64b94756ecff4078/quantile_forest-1.4.1-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:37c2da2ab54aceacdf5292065147f40a073b13cc3844262f0f3cbd5b8a8d928e", size = 955098, upload-time = "2025-09-10T12:47:52.137Z" }, + { url = "https://files.pythonhosted.org/packages/4f/95/75f3eea1c7cc3786c1ffdf4685e79c4979a4ae6ccedfed80362c9162f0d4/quantile_forest-1.4.1-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:3f0436ac7622442c2995cf121e0960332e769791f3f3c7ea62363e8480803bb3", size = 718470, upload-time = "2025-09-10T12:47:53.566Z" }, + { url = "https://files.pythonhosted.org/packages/fe/f1/0f26386bf164ede156099d18e3e4493dd21dc48e329e1be68232e5cf8b52/quantile_forest-1.4.1-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:a594bd3552507beffa6ca6002143601be5defd5cc7329154f41317110f895f7a", size = 709245, upload-time = "2025-09-10T12:47:54.54Z" }, + { url = "https://files.pythonhosted.org/packages/4f/cd/6501c8c200f34a87e1e94d7ea4f1a9dc842154fbfaa0fe65f072817fbc41/quantile_forest-1.4.1-cp313-cp313-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:697c48faf52a04e7e47f97187650d16cecc9c971fe2f83d56854b4a454289f60", size = 2403543, upload-time = "2025-09-10T12:47:55.956Z" }, + { url = "https://files.pythonhosted.org/packages/f2/be/f77c6705e974b23353c43da1cd93e11fe0afc7e859c2d14f748d25cc0376/quantile_forest-1.4.1-cp313-cp313-win_amd64.whl", hash = "sha256:fe33f6a8b63b3617568cc1254e1802a70ce3ac23897790f3be10f8db5257fe83", size = 685417, upload-time = "2025-09-10T12:47:57.346Z" }, +] + [[package]] name = "ruff" version = "0.16.1" @@ -517,12 +622,14 @@ analysis = [ { name = "scipy" }, ] dev = [ + { name = "hypothesis" }, { name = "pytest" }, { name = "ruff" }, ] [package.metadata] requires-dist = [ + { name = "hypothesis", marker = "extra == 'dev'", specifier = ">=6.100,<7" }, { name = "numpy", specifier = ">=1.26,<3" }, { name = "pandas", marker = "extra == 'analysis'", specifier = ">=2.2,<3" }, { name = "pyreadstat", marker = "extra == 'analysis'", specifier = ">=1.2.8,<2" }, @@ -534,6 +641,15 @@ requires-dist = [ ] provides-extras = ["analysis", "dev"] +[[package]] +name = "sortedcontainers" +version = "2.4.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/e8/c4/ba2f8066cceb6f23394729afe52f3bf7adec04bf9ed2c820b39e19299111/sortedcontainers-2.4.0.tar.gz", hash = "sha256:25caa5a06cc30b6b83d11423433f65d1f9d76c4c6a0c90e3379eaa43b9bfdb88", size = 30594, upload-time = "2021-05-16T22:03:42.897Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/32/46/9cb0e58b2deb7f82b84065f37f3bffeb12413f947f9388e4cac22c4621ce/sortedcontainers-2.4.0-py2.py3-none-any.whl", hash = "sha256:a163dcaede0f1c021485e957a39245190e74249897e2ae4b2aa38595db237ee0", size = 29575, upload-time = "2021-05-16T22:03:41.177Z" }, +] + [[package]] name = "threadpoolctl" version = "3.6.0"