Skip to content

v2 prompt and data: program-specific disability facts, integer inputs, and the take-up boundary #165

Open

Description

@MaxGhenis

v2 prompt and data: make disability facts program-specific, expose integer inputs, state the take-up boundary

Problem (found on the 39-model board, 2026-09-05)

The prompt lists a single undefined fact, - is disabled, and tells models that unlisted status facts are false and to assume program take-up. The engine's disability-linked gates are program-specific and read other inputs:

Program Engine gate Present in the certified June build?
SSI meets_ssi_disability_criteria and the SGA screen (42 USC 1382c) false for all 160,858 persons; never appears in any prompt
Medicare under 65 social_security_disability > 0 and months_receiving_social_security_disability >= 24 (42 USC 426) months absent in every build, and integer-typed, which the prompt builder excludes
SNAP elderly/disabled receipt-based is_usda_disabled works (SSI-disabled, SSDI income, veteran statuses)
IRC 152 / 22 is_permanently_and_totally_disabled never populated in any build
IRC 21 CDCC is_incapable_of_self_care absent until the 2026-07-23 builds (2,067 from PEDISDRS)

is_disabled itself (any of six CPS difficulty items) feeds 128 variables: 113 state child-care, TANF and state-tax provisions, HUD, Medicaid work-requirement exemptions, VITA, and the SSI working-student exclusion (which should read is_ssi_disabled).

Consequences on the 39-model board: 34 SSI model-rows across five households paid SSI on a disabled under-65 member against a $0 reference fixed by the data; 18 of them are pure disability-gate disagreements (within 10% of the clean alternative amount), 16 carry independent errors (resource limit exceeded, one-third reduction from inferred shelter, wrong rates). One Medicare row (scenario_074) rests on the unlisted SSDI duration. See annotations//us_adjudications.json and the paper's audit section.

The preamble also does not state the take-up boundary: the engine deems the TANF non-cash benefit received when eligible (SNAP BBCE) while models are told not to infer unlisted benefit receipt.

Proposed preamble (Astra draft, 2026-09-05, to edit)

Estimate the requested outputs under the PolicyEngine-US version and calculation conventions identified with this question. All listed people live together in the stated household group. Unless dated explicitly, facts apply throughout the tax-benefit year without changes in status or income volatility; wage amounts are annual totals including overtime, and hourly rates are straight-time rates. Unlisted numeric inputs, including integer inputs, are zero, and unlisted boolean inputs are false, subject to the filing and take-up assumptions below. "General disability indicator" is a survey characteristic and does not establish SSI disability, tax-specific disability, SSDI entitlement, or Medicare eligibility. SSI disability uses the separately stated "meets SSI disability criteria before the substantial-gainful-activity test"; apply the earnings test separately. Tax-specific disability and self-care conditions are also stated separately. Medicare eligibility is evaluated on January 1; qualifying disability-benefit months are measured as of that date. Assume filing and full take-up of eligible requested benefits, plus eligible TANF/MOE noncash benefits used for SNAP categorical eligibility, and apply those computed benefits in downstream calculations. This exception permits computed receipt; it does not create substantive eligibility or historical entitlement. Social Security, SSDI, and veterans payments and histories are supplied inputs only. Infer no other income, expenses, assets, receipt, rent, or coverage. The SSI output includes federal SSI only; state supplements require a separate requested output.

Data and builder requirements

  • Draw v2 households from a microcosm build that carries the SIPP-imputed meets_ssi_disability_criteria (builds from 2026-07-17) and is_incapable_of_self_care (from 2026-07-23).
  • Populate is_permanently_and_totally_disabled (or state that IRC 152/22 disability pathways are out of scope).
  • Carry SSDI entitlement duration (or start date) with a defined reference date, or state the assumption; allow integer inputs to be promptable.
  • Label employer_sponsored_insurance_premiums as employer-paid (never in the stated wages).
  • Record provenance per disability fact: observed, imputed, or unknown. Unknown must not print as an observed negative.
  • Engine nit: is_ssi_blind_or_disabled_working_student_exclusion_eligible reads is_disabled; it should read is_ssi_disabled.

Related: #139 (v2 board condition), the prompt-ambiguity adjudication record introduced in #164.

Additions from the 2026-09-05 work (PR #164)

Activity

  1. MaxGhenis commented on Oct 10, 2026

    @MaxGhenis
    ContributorAuthor

    Two more prompt labels for v2, from a check of release dashboard-data-20261010's scenario_039 (VA) federal income tax. Six strong models agree against that reference; the reference holds, and these two labels are what misled the models or left the question open. Evidence: #213 (reference_audit/2026-10-10-held-references/). Neither label is in this issue yet.

    1. "is a surviving spouse" is the Code's name for a filing status; the data flag is only "widowed"

    • is_surviving_spouse prints as "is a surviving spouse" (policybench/prompts.py). 26 U.S.C. 2(a) uses that term for the status that gets the joint rate table and standard deduction. It requires a dependent child living in the home and a death in one of the two preceding tax years.
    • The flag carries neither fact. It is the survey's marital status:
      • policyengine-us-data sets it as person.A_MARITL == 4 (policyengine_us_data/datasets/cps/cps.py line 1212 at main 42ed5d45);
      • Microcosm sets it as marital_status == 4 (packages/microcosm-build/src/microcosm/build/us_runtime/relationship_inputs.py line 181 at main aca40a69).
    • The engine applies the child test itself (surviving_spouse_eligible), so the reference files a lone widow as single.
    • On the board: scenario_039 is one person, aged 61. Six models (claude-opus-5.5, claude-sonnet-5.5, gpt-6-astra, claude-fable-5.1, gpt-6.1-sol, gpt-5.5) file her as a qualifying surviving spouse and answer $5,145.11 against a reference of $8,596.03. None of the six mentions a child. gpt-5.6-sol and gpt-5.6-luna make the same status error.
    • The label is in 12 of the 100 prompts (scenarios 000, 021, 022, 039, 042, 048, 062, 084, 095, 108, 111, 118), and none of those households lists a child.
    • Add an unactivated v2 household prompt contract for issue 165 #173's draft contract already treats the label as loaded: it renames it "Surviving spouse indicator" and adds deciding facts (the death year, with a synthetic one when the data has none, and whether a dependent child lives in the home). This cell points to a simpler fix: say what the data says, "is widowed", and let the filing status follow from the listed household. Either way the label should stop borrowing the statute's term.

    2. "estate income" states no character

    • estate_income prints as "estate income". The engine's input is ordinary income: Schedule E Part III income carried to Schedule 1 line 5, with an estate's interest, dividends and capital gains in separate inputs (the variable's documentation in policyengine-us 2.38.6).
    • The prompt says none of that. Read as qualified dividends or long-term gain, scenario_039's $25,950 is taxed at preferential rates and the answer is $6,259.99, not $8,596.03.
    • No model answers $6,259.99. The independent reviewer of this cell (GPT-6.1 Sol) would still exclude the output on this ground; whether to is queued for Max as d1252.
    • v2: label it by character, for example "ordinary income from an estate or trust (Schedule E)", or state in the preamble that estate income is ordinary and excludes dividends and gains. Two prompts carry it: scenario_039 and scenario_110 (OH).

    "employer sponsored insurance premiums", the label behind the other cell in #213 (OH scenario_025), is already in this issue's data requirements.

  2. MaxGhenis commented on Oct 14, 2026

    @MaxGhenis
    ContributorAuthor

    Two rulings from Max on 2026-10-13 add items for the v2 prompt:

    • d1263 (Tax Table vs rate schedule): keep the section 1 rate schedule. The v2 prompt's federal income tax output definition should say that the answer is the tax computed from the section 1 rate schedule on taxable income, to the cent, not the IRS Tax Table amount. Evidence: reference_audit/2026-10-10-tax-table/README.md ("What each option would take"). The benchmark card and paper state the convention from release dashboard-data-20261014 (PR Add Gemini 4 Argon and Muse Spark 1.3 (release, 50 models) [WIP] #221).
    • d1252 (held references): VA scenario_039 federal and OH scenario_025 state income tax stay scored. Fix the labels the reviewers found ambiguous: "estate income" states no character (ordinary vs preferential), and OH 025's employer premiums don't say who paid. Evidence: reference_audit/2026-10-10-held-references/README.md ("Prompt labels" and the reviewers' dissents).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions