Skip to content

Audit the state income tax withheld in federal SALT; propose three exclusions - #191

Closed
MaxGhenis wants to merge 1 commit into
mainfrom
salt-withholding-20261005
Closed

MaxGhenis wants to merge 1 commit into
mainfrom
salt-withholding-20261005

Conversation

@MaxGhenis

Copy link
Copy Markdown
Contributor

What this adds

reference_audit/2026-10-05/: an audit of how the US references fill the state income tax part of the federal SALT deduction. It proposes three reference_depends_on_unlisted_input records and changes no reference, exclusion record or published number. Adopting the records changes published scores, so they wait for Max's ruling (cos decision).

Finding

In policyengine-us 2.15.17 (verified in the installed source):

  • salt_deduction → salt → state_and_local_sales_or_income_tax, which is the larger of (state_withheld_income_tax + local_income_tax) and sales tax.
  • state_withheld_income_tax adds per-state *_withheld_income_tax formulas on the person's whole federal AGI. For example, MA = 5% × (AGI − $4,400), which is $8,484.41 for scenario_081. They are not the household's state liability.
  • No prompt states state income tax withheld or paid, and the prompt says unlisted numeric inputs are 0.

A sweep of all 1,984 outputs on the reference system (2.15.17 + latest_final) reproduces all 1,928 scored references. It then reruns each output under three readings: liability, net and zero. Only five outputs move, all federal income tax before refundable credits:

Output Reference Liability reading Zero reading Status
scenario_022 (CA) 11,113.57 11,895.23 12,192.29 scored → proposed
scenario_081 (MA) 26,932.27 26,991.31 28,763.81 scored → proposed
scenario_114 (VA) 10,729.61 11,137.31 11,758.09 scored → proposed
scenario_078 (MD) 24,164.46 24,799.16 27,290.70 already excluded; note text added
scenario_120 (CT) 40,416.67 (2.15.17) 40,438.95 43,619.93 already excluded; note text added

No other output moves by even a cent. No model is within $1 of any of the three frozen references. On 081, four models land within $1 of the liability reading once the corrected MA liability is used ($26,992.64).

Leaderboard impact (if adopted)

Computed with policybench.cli analyze on scratch copies. The unchanged copy reproduces the published payload's scoring exactly.

Verification

Five independent reviewers ran in a Claude Code workflow (structured reports in verification/independent_reviews.json):

  • engine-free recomputation of every withholding estimate and federal value, to the cent;
  • a differential re-implementation, bit-identical on all 1,984 outputs;
  • an adversarial review, which could not refute the proposal and found the payroll-contribution defect;
  • an impact reproduction, byte-identical;
  • a completeness critic, which found the Part B premium and the county dependence.

scripts/variants.py recomputes every side number the records quote.

Related findings (follow-up tasks)

  • policyengine-us leaves mandatory employee state payroll contributions out of SALT (CA SDI, MA/CT paid leave; Rev. Rul. 2025-4). This moves only outputs that are excluded or proposed.
  • A modeled Medicare Part B premium, which no prompt states, also moves scenario_114's scored VA state output, $3,514.15 → $3,654.15. Audited separately.
  • The README lists what a release adopting the records must also do: adjudications, annotation relabels, pinned counts and paper text.

Invariants (load-bearing logic)

  • Baseline reproduction: the sweep's baseline equals every scored published reference (asserted; 0 mismatches). The impact script stops unless the unchanged copy reproduces the published scoring.
  • Determinism: re-running the sweep with the committed script reproduced sweep_salt_withholding.csv byte for byte.
  • Differential: two independent mechanisms (tax-unit vs per-person withholding input) agree on all 1,984 outputs.
  • Fixed point: every household reaches a fixed point; all converged on the first pass.
  • Exclusion symmetry: no model had an exact hit on the three outputs, so exclusion only removes misses. Each model's exact rate change equals (1/100)·Σ S_h·w_h/(1−w_h), checked by the impact reviewer to 1.7e-14.

These are scripts run once over frozen data, not library code. They are excluded from ruff like the rest of reference_audit/, and they add no tests.

🤖 Generated with Claude Code

…clusions

policyengine-us 2.15.17 fills the state income tax part of the federal SALT
deduction with state_withheld_income_tax, an AGI-formula estimate of
withholding, and no prompt states state income tax withheld or paid. This adds
the audit (reference_audit/2026-10-05): a sweep of all 1,984 outputs under the
reference, liability, net and zero readings (baseline reproduces all 1,928
scored references), the side readings the records quote, three proposed
reference_depends_on_unlisted_input records (022 CA, 081 MA, 114 VA federal
income tax), and the leaderboard impact scored through the analyze CLI on
scratch copies.

No reference, exclusion record or published number changes here. The
exclusions change published scores and wait for Max's ruling.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@vercel

vercel Bot commented Oct 5, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
policybench-site Ready Ready Preview Oct 5, 2026 1:27pm UTC

Request Review

@MaxGhenis

Copy link
Copy Markdown
Contributor Author

Companion audit: #193 (the Medicare Part B premium the engine assumes for every Medicare-eligible person, the "Related engine issues" follow-up in this PR's README).

Two rounding slips in this PR's README, which a release adopting the records should correct:

  • GPT-6 Sol with the three records is 95.6146, which rounds to 95.61%, not 95.62%.
  • The always-zero baseline is 70.3946, which rounds to 70.39%, not 70.40%.

🤖 Generated with Claude Code

MaxGhenis added a commit that referenced this pull request Oct 5, 2026
…ct summary and independent reviews

- State the prompt's "assume program take-up" sentence in both records and the
  README, and why it does not settle Medicare enrollment (the benchmark's
  DEFAULT_TAKEUP_INPUTS has no Medicare entry; the request asks eligibility only).
- Name both ACA coverage tests in the mechanism section.
- leaderboard_impact.py recomputes the freeze's legacy impact_summary_by_model.csv
  in every case (the unchanged copy reproduces the frozen file). The stale-weight
  check now reports that file, which reorders 10 of 46 models with 2.15.17's
  weights, so the finding is narrowed to "no change in the analyze payload".
- Read #191's records from a sha256-checked copy at its head 8af912a, so this
  directory does not depend on #191's branch.
- Add the four workflow reviewers' reports (verification/independent_reviews.json).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MaxGhenis added a commit that referenced this pull request Oct 9, 2026
…last answer (release dashboard-data-20261006) (#202)

* Date the model response window from the last answer, not the release date

The release drivers set the manifest's model_response_date to
"2026-06-12 to <release date>". Release dashboard-data-20260930 recorded
an end of 2026-09-30, but the last answer on that board (GPT-6.1 Sol)
completed at 2026-09-29 22:36:32 UTC and no model answered on the 30th.

freeze_snapshot.model_response_window now derives the end as the UTC date
of the latest request_completed_at in the predictions the freeze gzips
into the snapshot. main() derives it before its first write and passes it
to build_manifest. The start stays configured (MODEL_RESPONSE_START),
because the first waves' rows carry no timestamps and the run label gives
their date. A driver that still sets MODEL_RESPONSE_DATE must name the
derived window, or the freeze refuses. Both drivers stop setting it, and
freeze_gpt61sol derives the window in its preflight so --dry-run reports
it.

The live manifest is not changed here: correcting it is a published-claim
change queued for Max (cos decision d831). A strict xfail test records the
live mismatch and fails once a freeze corrects the manifest, so that
freeze must remove the mark.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Audit the state income tax withheld in federal SALT; propose three exclusions

policyengine-us 2.15.17 fills the state income tax part of the federal SALT
deduction with state_withheld_income_tax, an AGI-formula estimate of
withholding, and no prompt states state income tax withheld or paid. This adds
the audit (reference_audit/2026-10-05): a sweep of all 1,984 outputs under the
reference, liability, net and zero readings (baseline reproduces all 1,928
scored references), the side readings the records quote, three proposed
reference_depends_on_unlisted_input records (022 CA, 081 MA, 114 VA federal
income tax), and the leaderboard impact scored through the analyze CLI on
scratch copies.

No reference, exclusion record or published number changes here. The
exclusions change published scores and wait for Max's ruling.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Audit Louisiana's 2026 standard deduction behind two scored references

scenario_051 and scenario_077 state income tax rest on policyengine-us's
explicit 2026 Louisiana standard deduction of $12,835 (PR #8411), which
no Louisiana publication states. The audit sources what Louisiana
published, sweeps all 1,984 outputs under six candidate deductions on
policyengine-us 2.15.17 with latest_final, and scores each option
through the analyze CLI. It changes no reference.

Before the 2026-07-03 freeze the Department of Revenue published only a
provisional $12,875 (withholding tables and estimated-tax worksheet),
saying the return amount would differ; RIB 26-019 (2026-09-28) set it
at $12,838, which moves the references by $0.09 and no exact match.
The audit recommends keeping both references and drafts the
documentation; the ruling is Max's.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Keep the sweep, scoring and check logs the audit README cites

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Audit optional employer pass-through in the payroll tax references; draft both records

Four scored payroll_tax references (scenario_032 MN, 043 CO, 081 MA, 082 NY) count
an employee share of a state paid-leave premium that the law lets the employer
deduct but does not require. policyengine-us 2.15.17 counts the largest share the
employer may deduct. The output asks for "mandatory employee state payroll taxes".

reference_audit/2026-10-05-payroll classifies every state program in the payroll
references from primary law (two adversarial reviews per program, none refuted),
sweeps all 1,984 outputs with an output-scope adapter that drops the optional
shares (four outputs move, nothing else), and drafts both candidate records:
proposed_exclusions.json and proposed_regenerations.json. It scores each through
the analyze CLI; the unchanged copy reproduces the published payload.

Changes no reference or published number. Draft README; review pending.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Audit the Medicare Part B premium in medical expenses; propose excluding scenario_114's Virginia income tax

policyengine-us 2.15.17 treats every Medicare-eligible person as enrolled
(takes_up_medicare_if_eligible defaults to true) and adds the modeled Part B
premium, $2,434.80 for 2026, to medical expenses. No prompt states Medicare
enrollment or a premium. A sweep of all 1,984 outputs on the reference system
with the premium at 0, and with enrollment off, moves exactly two: scenario_114
federal (10,729.61 -> 11,265.27, already proposed in #191) and Virginia
(3,514.15 -> 3,654.15). No SNAP or eligibility output moves by any amount.

Adds reference_audit/2026-10-05-medicare-part-b: the sweep, a per-household
explanation, the drafted records (one for the Virginia output, and a fallback
for the federal output if #191's record is not adopted) and the leaderboard
impact under each combination with #191. Changes no reference or published
number; the exclusion waits for Max's ruling.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Rest the payroll recommendation on the reference rules; record the independent review

An eight-agent review of 8714770 (differential impact and sweep, mechanism,
advocates for keep and regenerate with a judge, release checklist, claim audit)
agreed on every number: 2,438 impact values, all 1,984 swept outputs bit for bit,
and the 36 hand-computed payroll parts. The judge upheld exclusion but struck the
draft's board-effect and model-behaviour arguments, which no rule uses.

- README: the recommendation now rests on 2026-09-22/09-28 rule 4 and the r24,
  r25 and r14 precedents; regeneration is the strict reading of "mandatory".
  Adds both options' board effects side by side, the counter-evidence the
  reviewers found, the Acts 2026 c. 101 line mapping, the review summary and
  the release checklist. Fixes the claim audit's five errors and thirteen
  imprecisions.
- proposed_exclusions.json: one shared unlisted input; the basis cites the
  rules and precedents (no engine input exists for the employer's choice).
- payroll_mandatory_scope.py: docstring states its scope (DE, ME, VT and WA left
  for upstream). Sweep, proposals and impact rerun on the new hash; values
  unchanged.
- verification/independent_reviews.json: the eight reports.

Changes no reference or published number.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Name the payroll decision (cos d972) in the audit README

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Answer the take-up reading in the Part B records; add the legacy impact summary and independent reviews

- State the prompt's "assume program take-up" sentence in both records and the
  README, and why it does not settle Medicare enrollment (the benchmark's
  DEFAULT_TAKEUP_INPUTS has no Medicare entry; the request asks eligibility only).
- Name both ACA coverage tests in the mechanism section.
- leaderboard_impact.py recomputes the freeze's legacy impact_summary_by_model.csv
  in every case (the unchanged copy reproduces the frozen file). The stale-weight
  check now reports that file, which reorders 10 of 46 models with 2.15.17's
  weights, so the finding is narrowed to "no change in the analyze payload".
- Read #191's records from a sha256-checked copy at its head 8af912a, so this
  directory does not depend on #191's branch.
- Add the four workflow reviewers' reports (verification/independent_reviews.json).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Correct scenario_031's Medicaid annotations to the engine's California disregard

The case note, reference explanation and 43 row annotations for
scenario_031 head_medicaid_eligible explained the reference (eligible) by
subtracting a Medicare Part B premium from countable income. policyengine-us
2.15.17 subtracts no premium. It counts $23,853.47 of SSI unearned income and
applies California's $230 monthly disregard in place of SSI's $20 exclusion.
That gives $21,093.47 against $22,024.80 (138% of the 2026 guideline).

Wording-only: every row stays llm_error with its subtype. The ledger
(reference_audit/2026-10-05-medicaid-031-annotations/rewrites.json) is
applied by scripts/apply_rewrites.py, which re-pins the manifest hashes, and
tests/test_annotation_rewrites.py keeps it in force. engine_values.py
asserts every figure on the reference system. payload_diff.py shows a
re-export changes only this cell's text fields. The frozen run dir is
untouched.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Point the Part B audit's annotation findings to #197 and the scenario_114 row check

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Apply the independent review's wording fixes to scenario_031's annotations

Narrows 'holds only if the disregard is skipped' to the engine's limit, names
gemini-3.5-flash's source for $23,853, drops 'about' before exact figures,
rewords claude-fable-5, the case note's category clause and the explanation's
limit phrase. Records immigration_status in engine_values.json and tightens
payload_diff's reproduction check. Adds the review as
verification/independent_review.md.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Keep CSV timestamps exact and test the real manifest builder's window

Answers the hard-tier review of #188 (b6b385c):
- model_response_window parses with float_precision=round_trip; pandas' default
  parser rounded the last double before 2027-02-01 UTC up to midnight and dated
  a January 31 answer February 1. New end-to-end CSV regression.
- A test now calls the real build_manifest with a window no release has used
  and requires it in model_response_date and both reproducibility notes; a
  builder hard-coded to September 30 fails it (mutation f).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Add the release 20261006 driver, its spec and its tests

scripts/release_20261006.py stages, exports and freezes release
dashboard-data-20261006 from release 20260930 (read from git at 8b4c0ca):
the eight ruled exclusion records and their adjudications, the record edits
the three audits wrote, the annotation rewrites, and the derived response
window. docs/release_20261006/spec.json holds its inputs.

Gates: the base stage is 20260930's; the annotations differ from 20260930's
only on the excluded and reworded outputs; two exports agree byte for byte;
the payload differs from 20260930's only where the release allows; rebuilt
with 20260930's exclusion record, the staged bundle reproduces 20260930's
statistics exactly; the freeze leaves references, predictions and serving
configuration as they were and changes the manifest only where the release
allows.

tests/test_release_20261006.py rebuilds each committed record from git and
the spec, rescores the payload's rows independently, and checks the gates
with Hypothesis properties.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Date the 2026-10-05 wave's decisions in UTC, document the driver, widen the in-force test

- spec: the new wave's adjudications were written on 2026-10-06 UTC;
- DATE_CONVENTIONS_NEW names Max's rulings of 2026-10-05 US Eastern time;
- docs/release_20261006/design.md documents the driver's inputs, gates and order;
- the in-force test strips any adjudication sentence before re-appending it;
- rescore_sensitivity_summaries knows the word for 64.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* WIP: partial content edits written against the trial freeze

Test pins, paper and docs prose, and a draft release note from the stopped
content pass. Some are reviewed and some are not; the content pass after the
fresh freeze finishes them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Add the release's annotation rewrite ledger and its evidence

306 whole-field rewrites in four groups, each drafted from its audit's
evidence, checked by an adversarial verifier and revised:
- salt (81): scenario_022 and scenario_081 federal income tax; the SALT
  state income tax is the engine's withholding estimate, not the household's;
- s114 (67): scenario_114 federal and Virginia income tax; the withholding
  estimate and the imputed Medicare Part B premium;
- payroll (151): the four paid-leave payroll outputs; optional employer
  pass-through shares, and scenario_032's and 043's note errors;
- louisiana (7): scenario_051 and 077 case notes and rows; sourced
  $12,835 chain, no change to scored references.

Each group's evidence file quotes the model explanations and engine
values behind every changed number.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Record the d994 hold in the release spec

* Freeze dashboard-data-20261006 and rescore sensitivity evidence

* Pin the release note to the draft v2 contract and document the d994 hold

* Render the paper, pin it, record d994's ruling and the validation logs

Renders the paper outside the sandbox with the installed Quarto 1.9.36 and
re-pins it with freeze_snapshot.py --rendered-only (pdf sha256 9403564b...).
The Louisiana audit README now records d994 (ruled 2026-10-06, after this
release was built): the convention is kept as written, and both Louisiana
references are excluded in the release after this one.

Validation logs: render exit 0; app lint, 170 bun tests and the build exit
0; full pytest 1,651 passed, 9 skipped, 1 failed. The failure is
test_local_claude_models_resolve_without_remote_cost_map[claude-sonnet-5-5],
which fails on main too once litellm loads its remote cost map. PR #201
fixes it, and this branch takes the fix from main before it merges.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Record the reviewed head

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Name the published rates behind Massachusetts's maximum; restore the pre-freeze tree

The independent review of head 8b831f2 found that the release note and 32
annotation rewrites called $805.01 (0.46%) the most a Massachusetts employer
may deduct. The payroll audit records a competing reading: a literal
application of Acts 2026 c. 101 s. 45 would let a large employer deduct up to
0.772% ($1,351.02) for 2026, while the Department's rates page kept 0.46%.
The texts now say "the most Massachusetts's published 2026 rates let the
employer deduct": the ledger's scenario_081 payroll texts, its adjudication
reasoning, a fifth record edit for the exclusion record, and the note. The
exclusion and every score are unchanged.

Also from the review: verify_payload now requires the staged payload's
households and outputs to be the base's before it compares cells, with a
test that a dropped cell or household is refused; the design note records
d994's ruling instead of calling it pending.

The freeze's outputs (5435bab) are restored to their pre-freeze state so
prepare, export and freeze can run again on the corrected ledger.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Freeze dashboard-data-20261006 again on the corrected Massachusetts wording

Payload f03441f64316997de843e82c27272442c794148a121c25e1343fbddbe38574a0,
127,224,818 bytes. Against the reviewed head 8b831f2 the freeze changes 31
row annotations, one case note, one adjudication's reasoning and one
exclusion record's alternative reading, all for scenario_081's payroll
output, and the hashes that follow. The analysis files and the sensitivity
evidence are byte-identical, so no score moved. The paper is rendered and
pinned again (pdf sha256 f3ccebc8...).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Point the effects record at the corrected payload; drop the session handoff drafts

The summary CSV the effects record binds is byte-identical after the
re-freeze, so every expected effect still holds; only the payload hash it
names changes. The handoff drafts were one session's working notes, not
repo documentation.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Record the validation logs of the corrected freeze

Full pytest on the merged head: 1,662 passed, 9 skipped, exit 0. App lint,
170 bun tests and the build: exit 0. Paper render: exit 0.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Record the reviewed head

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Qualify Massachusetts's caps in the two remaining exclusion-record sentences

The round-2 review found two record sentences that still stated
Massachusetts's limits without the published-rates qualification:
scenario_081's federal-tax note ('the largest share the employer may
deduct') and its payroll record, which attributed the 100% family and 40%
medical caps directly to s. 6(c). Acts 2026 c. 101 ss. 25-26 and 45 can be
read to swap those caps for 2026. The note now names the published 2026
rates, and a sixth record edit qualifies the caps the same way.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Restore the pre-freeze tree for the round-3 freeze

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Freeze dashboard-data-20261006 on the round-3 record edits

Payload aa34e5c9ea926848dc7af460a98f17021bc42dae54a8015c9988a85e79a5d462,
127,225,036 bytes. Against round 2 only scenario_081's two exclusion-record
sentences change; the analysis files and sensitivity evidence remain
byte-identical to the first freeze, so no score moved. The release asset is
replaced and verified against the pointer.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Record the round-3 validation logs

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Record the reviewed head

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
@MaxGhenis

Copy link
Copy Markdown
Contributor Author

Shipped in release dashboard-data-20261006 (PR #202, squash-merged as 9ce4ade8, now the latest data release and live on policybench.org). This PR's commits were folded into that release as a merge commit, so its content is on main; closing it rather than merging it a second time. Ruling d963: federal income tax before refundable credits for scenario_022, 081 and 114 is excluded (SALT rests on the engine's formula estimate of state income tax withheld).

This branch was successfully deployed

1 active deployment
Preview — 8af912a0 Deployed Oct 5, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant