Repository navigation
Conversation
…clusions policyengine-us 2.15.17 fills the state income tax part of the federal SALT deduction with state_withheld_income_tax, an AGI-formula estimate of withholding, and no prompt states state income tax withheld or paid. This adds the audit (reference_audit/2026-10-05): a sweep of all 1,984 outputs under the reference, liability, net and zero readings (baseline reproduces all 1,928 scored references), the side readings the records quote, three proposed reference_depends_on_unlisted_input records (022 CA, 081 MA, 114 VA federal income tax), and the leaderboard impact scored through the analyze CLI on scratch copies. No reference, exclusion record or published number changes here. The exclusions change published scores and wait for Max's ruling. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
Companion audit: #193 (the Medicare Part B premium the engine assumes for every Medicare-eligible person, the "Related engine issues" follow-up in this PR's README).
Two rounding slips in this PR's README, which a release adopting the records should correct:
🤖 Generated with Claude Code |
…ct summary and independent reviews - State the prompt's "assume program take-up" sentence in both records and the README, and why it does not settle Medicare enrollment (the benchmark's DEFAULT_TAKEUP_INPUTS has no Medicare entry; the request asks eligibility only). - Name both ACA coverage tests in the mechanism section. - leaderboard_impact.py recomputes the freeze's legacy impact_summary_by_model.csv in every case (the unchanged copy reproduces the frozen file). The stale-weight check now reports that file, which reorders 10 of 46 models with 2.15.17's weights, so the finding is narrowed to "no change in the analyze payload". - Read #191's records from a sha256-checked copy at its head 8af912a, so this directory does not depend on #191's branch. - Add the four workflow reviewers' reports (verification/independent_reviews.json). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…last answer (release dashboard-data-20261006) (#202) * Date the model response window from the last answer, not the release date The release drivers set the manifest's model_response_date to "2026-06-12 to <release date>". Release dashboard-data-20260930 recorded an end of 2026-09-30, but the last answer on that board (GPT-6.1 Sol) completed at 2026-09-29 22:36:32 UTC and no model answered on the 30th. freeze_snapshot.model_response_window now derives the end as the UTC date of the latest request_completed_at in the predictions the freeze gzips into the snapshot. main() derives it before its first write and passes it to build_manifest. The start stays configured (MODEL_RESPONSE_START), because the first waves' rows carry no timestamps and the run label gives their date. A driver that still sets MODEL_RESPONSE_DATE must name the derived window, or the freeze refuses. Both drivers stop setting it, and freeze_gpt61sol derives the window in its preflight so --dry-run reports it. The live manifest is not changed here: correcting it is a published-claim change queued for Max (cos decision d831). A strict xfail test records the live mismatch and fails once a freeze corrects the manifest, so that freeze must remove the mark. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Audit the state income tax withheld in federal SALT; propose three exclusions policyengine-us 2.15.17 fills the state income tax part of the federal SALT deduction with state_withheld_income_tax, an AGI-formula estimate of withholding, and no prompt states state income tax withheld or paid. This adds the audit (reference_audit/2026-10-05): a sweep of all 1,984 outputs under the reference, liability, net and zero readings (baseline reproduces all 1,928 scored references), the side readings the records quote, three proposed reference_depends_on_unlisted_input records (022 CA, 081 MA, 114 VA federal income tax), and the leaderboard impact scored through the analyze CLI on scratch copies. No reference, exclusion record or published number changes here. The exclusions change published scores and wait for Max's ruling. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Audit Louisiana's 2026 standard deduction behind two scored references scenario_051 and scenario_077 state income tax rest on policyengine-us's explicit 2026 Louisiana standard deduction of $12,835 (PR #8411), which no Louisiana publication states. The audit sources what Louisiana published, sweeps all 1,984 outputs under six candidate deductions on policyengine-us 2.15.17 with latest_final, and scores each option through the analyze CLI. It changes no reference. Before the 2026-07-03 freeze the Department of Revenue published only a provisional $12,875 (withholding tables and estimated-tax worksheet), saying the return amount would differ; RIB 26-019 (2026-09-28) set it at $12,838, which moves the references by $0.09 and no exact match. The audit recommends keeping both references and drafts the documentation; the ruling is Max's. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Keep the sweep, scoring and check logs the audit README cites Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Audit optional employer pass-through in the payroll tax references; draft both records Four scored payroll_tax references (scenario_032 MN, 043 CO, 081 MA, 082 NY) count an employee share of a state paid-leave premium that the law lets the employer deduct but does not require. policyengine-us 2.15.17 counts the largest share the employer may deduct. The output asks for "mandatory employee state payroll taxes". reference_audit/2026-10-05-payroll classifies every state program in the payroll references from primary law (two adversarial reviews per program, none refuted), sweeps all 1,984 outputs with an output-scope adapter that drops the optional shares (four outputs move, nothing else), and drafts both candidate records: proposed_exclusions.json and proposed_regenerations.json. It scores each through the analyze CLI; the unchanged copy reproduces the published payload. Changes no reference or published number. Draft README; review pending. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Audit the Medicare Part B premium in medical expenses; propose excluding scenario_114's Virginia income tax policyengine-us 2.15.17 treats every Medicare-eligible person as enrolled (takes_up_medicare_if_eligible defaults to true) and adds the modeled Part B premium, $2,434.80 for 2026, to medical expenses. No prompt states Medicare enrollment or a premium. A sweep of all 1,984 outputs on the reference system with the premium at 0, and with enrollment off, moves exactly two: scenario_114 federal (10,729.61 -> 11,265.27, already proposed in #191) and Virginia (3,514.15 -> 3,654.15). No SNAP or eligibility output moves by any amount. Adds reference_audit/2026-10-05-medicare-part-b: the sweep, a per-household explanation, the drafted records (one for the Virginia output, and a fallback for the federal output if #191's record is not adopted) and the leaderboard impact under each combination with #191. Changes no reference or published number; the exclusion waits for Max's ruling. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Rest the payroll recommendation on the reference rules; record the independent review An eight-agent review of 8714770 (differential impact and sweep, mechanism, advocates for keep and regenerate with a judge, release checklist, claim audit) agreed on every number: 2,438 impact values, all 1,984 swept outputs bit for bit, and the 36 hand-computed payroll parts. The judge upheld exclusion but struck the draft's board-effect and model-behaviour arguments, which no rule uses. - README: the recommendation now rests on 2026-09-22/09-28 rule 4 and the r24, r25 and r14 precedents; regeneration is the strict reading of "mandatory". Adds both options' board effects side by side, the counter-evidence the reviewers found, the Acts 2026 c. 101 line mapping, the review summary and the release checklist. Fixes the claim audit's five errors and thirteen imprecisions. - proposed_exclusions.json: one shared unlisted input; the basis cites the rules and precedents (no engine input exists for the employer's choice). - payroll_mandatory_scope.py: docstring states its scope (DE, ME, VT and WA left for upstream). Sweep, proposals and impact rerun on the new hash; values unchanged. - verification/independent_reviews.json: the eight reports. Changes no reference or published number. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Name the payroll decision (cos d972) in the audit README Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Answer the take-up reading in the Part B records; add the legacy impact summary and independent reviews - State the prompt's "assume program take-up" sentence in both records and the README, and why it does not settle Medicare enrollment (the benchmark's DEFAULT_TAKEUP_INPUTS has no Medicare entry; the request asks eligibility only). - Name both ACA coverage tests in the mechanism section. - leaderboard_impact.py recomputes the freeze's legacy impact_summary_by_model.csv in every case (the unchanged copy reproduces the frozen file). The stale-weight check now reports that file, which reorders 10 of 46 models with 2.15.17's weights, so the finding is narrowed to "no change in the analyze payload". - Read #191's records from a sha256-checked copy at its head 8af912a, so this directory does not depend on #191's branch. - Add the four workflow reviewers' reports (verification/independent_reviews.json). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Correct scenario_031's Medicaid annotations to the engine's California disregard The case note, reference explanation and 43 row annotations for scenario_031 head_medicaid_eligible explained the reference (eligible) by subtracting a Medicare Part B premium from countable income. policyengine-us 2.15.17 subtracts no premium. It counts $23,853.47 of SSI unearned income and applies California's $230 monthly disregard in place of SSI's $20 exclusion. That gives $21,093.47 against $22,024.80 (138% of the 2026 guideline). Wording-only: every row stays llm_error with its subtype. The ledger (reference_audit/2026-10-05-medicaid-031-annotations/rewrites.json) is applied by scripts/apply_rewrites.py, which re-pins the manifest hashes, and tests/test_annotation_rewrites.py keeps it in force. engine_values.py asserts every figure on the reference system. payload_diff.py shows a re-export changes only this cell's text fields. The frozen run dir is untouched. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Point the Part B audit's annotation findings to #197 and the scenario_114 row check Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Apply the independent review's wording fixes to scenario_031's annotations Narrows 'holds only if the disregard is skipped' to the engine's limit, names gemini-3.5-flash's source for $23,853, drops 'about' before exact figures, rewords claude-fable-5, the case note's category clause and the explanation's limit phrase. Records immigration_status in engine_values.json and tightens payload_diff's reproduction check. Adds the review as verification/independent_review.md. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Keep CSV timestamps exact and test the real manifest builder's window Answers the hard-tier review of #188 (b6b385c): - model_response_window parses with float_precision=round_trip; pandas' default parser rounded the last double before 2027-02-01 UTC up to midnight and dated a January 31 answer February 1. New end-to-end CSV regression. - A test now calls the real build_manifest with a window no release has used and requires it in model_response_date and both reproducibility notes; a builder hard-coded to September 30 fails it (mutation f). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Add the release 20261006 driver, its spec and its tests scripts/release_20261006.py stages, exports and freezes release dashboard-data-20261006 from release 20260930 (read from git at 8b4c0ca): the eight ruled exclusion records and their adjudications, the record edits the three audits wrote, the annotation rewrites, and the derived response window. docs/release_20261006/spec.json holds its inputs. Gates: the base stage is 20260930's; the annotations differ from 20260930's only on the excluded and reworded outputs; two exports agree byte for byte; the payload differs from 20260930's only where the release allows; rebuilt with 20260930's exclusion record, the staged bundle reproduces 20260930's statistics exactly; the freeze leaves references, predictions and serving configuration as they were and changes the manifest only where the release allows. tests/test_release_20261006.py rebuilds each committed record from git and the spec, rescores the payload's rows independently, and checks the gates with Hypothesis properties. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Date the 2026-10-05 wave's decisions in UTC, document the driver, widen the in-force test - spec: the new wave's adjudications were written on 2026-10-06 UTC; - DATE_CONVENTIONS_NEW names Max's rulings of 2026-10-05 US Eastern time; - docs/release_20261006/design.md documents the driver's inputs, gates and order; - the in-force test strips any adjudication sentence before re-appending it; - rescore_sensitivity_summaries knows the word for 64. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * WIP: partial content edits written against the trial freeze Test pins, paper and docs prose, and a draft release note from the stopped content pass. Some are reviewed and some are not; the content pass after the fresh freeze finishes them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Add the release's annotation rewrite ledger and its evidence 306 whole-field rewrites in four groups, each drafted from its audit's evidence, checked by an adversarial verifier and revised: - salt (81): scenario_022 and scenario_081 federal income tax; the SALT state income tax is the engine's withholding estimate, not the household's; - s114 (67): scenario_114 federal and Virginia income tax; the withholding estimate and the imputed Medicare Part B premium; - payroll (151): the four paid-leave payroll outputs; optional employer pass-through shares, and scenario_032's and 043's note errors; - louisiana (7): scenario_051 and 077 case notes and rows; sourced $12,835 chain, no change to scored references. Each group's evidence file quotes the model explanations and engine values behind every changed number. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Record the d994 hold in the release spec * Freeze dashboard-data-20261006 and rescore sensitivity evidence * Pin the release note to the draft v2 contract and document the d994 hold * Render the paper, pin it, record d994's ruling and the validation logs Renders the paper outside the sandbox with the installed Quarto 1.9.36 and re-pins it with freeze_snapshot.py --rendered-only (pdf sha256 9403564b...). The Louisiana audit README now records d994 (ruled 2026-10-06, after this release was built): the convention is kept as written, and both Louisiana references are excluded in the release after this one. Validation logs: render exit 0; app lint, 170 bun tests and the build exit 0; full pytest 1,651 passed, 9 skipped, 1 failed. The failure is test_local_claude_models_resolve_without_remote_cost_map[claude-sonnet-5-5], which fails on main too once litellm loads its remote cost map. PR #201 fixes it, and this branch takes the fix from main before it merges. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Record the reviewed head Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Name the published rates behind Massachusetts's maximum; restore the pre-freeze tree The independent review of head 8b831f2 found that the release note and 32 annotation rewrites called $805.01 (0.46%) the most a Massachusetts employer may deduct. The payroll audit records a competing reading: a literal application of Acts 2026 c. 101 s. 45 would let a large employer deduct up to 0.772% ($1,351.02) for 2026, while the Department's rates page kept 0.46%. The texts now say "the most Massachusetts's published 2026 rates let the employer deduct": the ledger's scenario_081 payroll texts, its adjudication reasoning, a fifth record edit for the exclusion record, and the note. The exclusion and every score are unchanged. Also from the review: verify_payload now requires the staged payload's households and outputs to be the base's before it compares cells, with a test that a dropped cell or household is refused; the design note records d994's ruling instead of calling it pending. The freeze's outputs (5435bab) are restored to their pre-freeze state so prepare, export and freeze can run again on the corrected ledger. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Freeze dashboard-data-20261006 again on the corrected Massachusetts wording Payload f03441f64316997de843e82c27272442c794148a121c25e1343fbddbe38574a0, 127,224,818 bytes. Against the reviewed head 8b831f2 the freeze changes 31 row annotations, one case note, one adjudication's reasoning and one exclusion record's alternative reading, all for scenario_081's payroll output, and the hashes that follow. The analysis files and the sensitivity evidence are byte-identical, so no score moved. The paper is rendered and pinned again (pdf sha256 f3ccebc8...). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Point the effects record at the corrected payload; drop the session handoff drafts The summary CSV the effects record binds is byte-identical after the re-freeze, so every expected effect still holds; only the payload hash it names changes. The handoff drafts were one session's working notes, not repo documentation. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Record the validation logs of the corrected freeze Full pytest on the merged head: 1,662 passed, 9 skipped, exit 0. App lint, 170 bun tests and the build: exit 0. Paper render: exit 0. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Record the reviewed head Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Qualify Massachusetts's caps in the two remaining exclusion-record sentences The round-2 review found two record sentences that still stated Massachusetts's limits without the published-rates qualification: scenario_081's federal-tax note ('the largest share the employer may deduct') and its payroll record, which attributed the 100% family and 40% medical caps directly to s. 6(c). Acts 2026 c. 101 ss. 25-26 and 45 can be read to swap those caps for 2026. The note now names the published 2026 rates, and a sixth record edit qualifies the caps the same way. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Restore the pre-freeze tree for the round-3 freeze Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Freeze dashboard-data-20261006 on the round-3 record edits Payload aa34e5c9ea926848dc7af460a98f17021bc42dae54a8015c9988a85e79a5d462, 127,225,036 bytes. Against round 2 only scenario_081's two exclusion-record sentences change; the analysis files and sensitivity evidence remain byte-identical to the first freeze, so no score moved. The release asset is replaced and verified against the pointer. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Record the round-3 validation logs Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Record the reviewed head Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
Shipped in release |
What this adds
reference_audit/2026-10-05/: an audit of how the US references fill the state income tax part of the federal SALT deduction. It proposes threereference_depends_on_unlisted_inputrecords and changes no reference, exclusion record or published number. Adopting the records changes published scores, so they wait for Max's ruling (cos decision).Finding
In policyengine-us 2.15.17 (verified in the installed source):
salt_deduction→salt→state_and_local_sales_or_income_tax, which is the larger of (state_withheld_income_tax+local_income_tax) and sales tax.state_withheld_income_taxadds per-state*_withheld_income_taxformulas on the person's whole federal AGI. For example, MA = 5% × (AGI − $4,400), which is $8,484.41 for scenario_081. They are not the household's state liability.A sweep of all 1,984 outputs on the reference system (2.15.17 +
latest_final) reproduces all 1,928 scored references. It then reruns each output under three readings: liability, net and zero. Only five outputs move, all federal income tax before refundable credits:No other output moves by even a cent. No model is within $1 of any of the three frozen references. On 081, four models land within $1 of the liability reading once the corrected MA liability is used ($26,992.64).
Leaderboard impact (if adopted)
Computed with
policybench.cli analyzeon scratch copies. The unchanged copy reproduces the published payload's scoring exactly.Verification
Five independent reviewers ran in a Claude Code workflow (structured reports in
verification/independent_reviews.json):scripts/variants.pyrecomputes every side number the records quote.Related findings (follow-up tasks)
Invariants (load-bearing logic)
sweep_salt_withholding.csvbyte for byte.These are scripts run once over frozen data, not library code. They are excluded from ruff like the rest of
reference_audit/, and they add no tests.🤖 Generated with Claude Code