Validation: baseline comparisons for the programs the reform changes

[PavelMakarchuk]

Summary

The "Baseline comparisons" card in the v3 report validation listed comparisons the reform does not change, every row read "Period not recorded separately", and no row linked to its own place on the scorecard.

Report 4332 shows the problem. It is a nationwide 2026 report with one change: gov.irs.credits.cdcc.eligibility.max (the CDCC's cap on qualifying people) goes from 2 to 4. The card listed six rows: the refundable CTC, two CDCC rows, the Maryland EITC, the saver's credit and the South Carolina EITC. All six matched only because their program variable is somewhere among the 216 variables the reform reaches within six formula steps. For example, sc_eitc is reached through sc_cdcc and md_eitc through md_agi. With this PR, the card shows only the two national CDCC rows (US Treasury and IRS SOI).

What changed

• Only the programs the reform changes (libs/flagship/comparisonRelevance.ts, a pure function relevantComparisons). A comparison counts when its program is named in a changed parameter's path or in a variable that reads that parameter directly. The direct readers come from the existing parameterDependencies map. The last path segment is ignored because it names a setting, not a program (…min_children_for_ss_taxes_minus_eitc is a CTC rule). Qualifiers like refundable/non are dropped, so refundable_ctc and non_refundable_ctc both count as the CTC. A short alias table covers scorecard ids that are not model names (co_fac, wa_wftc, wi_eic, …). City and territory parameters match no federal or state program.

• Jurisdiction. A nationwide report keeps national comparisons, plus a state's comparisons only when the reform changes that state's own rules (for example a DC SNAP minimum shows DC SNAP rows). A state or congressional-district report keeps that state's comparisons and the national comparisons of the federal programs the reform changes. Other states never show.

• Empty case. When nothing applies, the card shows one line: "No scorecard comparison measures the programs this reform changes." Loose matches no longer appear as rows.

• Per-row scorecard links. Each row now has a "View in scorecard" link to the scorecard view that lists the entry:

  • Published claims (the populations feed, where baseline claims live) go to https://www.policyengine.org/scorecard?view=validation. Outside the US, the link adds &country=xx.
  • Comparison-table rows (the track record) go to ?view=scorecard.

  The scorecard's URL state carries only view and country (see app/src/urlState.ts in policyengine-scorecard). It has no address for a single entry, so this is the closest link it supports.

• Periods. A period shows only when the scorecard records it. "Period not recorded separately" and "time basis not recorded" are gone.

• ReportValidation takes the report's region, which ReportView already has.

• Data calibration uses the same rule. A calibration target counts when the model variables it measures (policyengine_variables) are a program the reform changes, in the report's geography. A group with no target left drops out. An empty card shows one line: "No calibration target measures the programs this reform changes." For report 4332, the CTC and refundable CTC targets drop, so the card shows that line.

• Calibration targets read by name. The Target column showed raw dashboard ids, such as irs_soi.ty2022.historic_table_2.us.all.ctc_amount@2024.

  • Each target now shows its measure and table slice, for example "Amount", "Claims" or "Returns · 18k to 19k · no qualifying children".
  • The raw id is still in the button tooltip and in a "Target id" row in the details.
  • Group headings use our copy first, then the model's variable label, then the name in plain words with acronyms. For example, "Ctc limiting tax liability" now reads "CTC-limiting tax liability".
  • /api/calibration-targets now also passes the dashboard's source_measure_id and breakdown.

• Name matching. Both sides of a match now drop words that only qualify a program (refundable, enrolled, eligible, the tax_unit entity). So medicaid_enrolled counts as Medicaid. income_tax matches only income tax, not any path that contains "income". Aliases cover the ACA, education credit and LIHEAP targets.

• Validation pin and run stages. The scorecard track record now applies the rule at its source. The pin (trackRecordPrograms), the "scorecard programs found" stage and the baseline card now use the same programs.

v2 impact

None. Every changed file is under components/flagship, libs/flagship or their tests, and only the v3 report and bill-report pages import them. The only API change adds two fields to the v3 /api/calibration-targets response.

Tests

• comparisonRelevance.test.ts covers these cases:
  • Report 4332 keeps only the CDCC rows.
  • A federal CTC reform nationwide keeps only the national CTC rows.
  • A Maryland EITC reform keeps only the Maryland EITC row (nationwide, Maryland and a Maryland district).
  • A reform with no match keeps nothing.
  • A state that reads a federal rule shows in that state only.
  • A federal program measured in a state shows in that state's report only.
  • A state option of a federal program shows that state and the nation.
  • A setting named after another program does not match it.
  • A city credit matches nothing.
  • A scorecard abbreviation matches through the alias table.
  • A claim with no recorded program needs a receipt that names the changed parameter.
• ValidationPanel.test.tsx and ReportValidation.test.tsx render a 4332-like fixture (the six real claims and the traced readers of the CDCC path). They check that only the two CDCC rows show, that each links to ?view=validation, that no "not recorded" text shows, and that unrelated claims give the one-line empty state.
• ScorecardComparisons.test.tsx checks the per-row links (US, UK, comparison table) and that periods show only when recorded. modelValidation.test.ts checks scorecardEntryUrl.
• calibrationMatching.test.ts covers these cases:
  • Report 4332's CTC targets drop, and no group is left.
  • A CTC reform keeps its CTC targets with their fit.
  • A group with another program's targets keeps only the program the reform changes.
  • An EITC path keeps only the EITC targets.
  • The target and group labels.
• CalibrationMatches.test.tsx checks the one-line empty state, the readable target names with the raw id as the tooltip and in the details, and the group heading for a raw variable name.
• modelValidation.test.ts checks that an EITC reform's track record no longer includes the refundable CTC.
• Checks from app/: bun run typecheck passes. eslint --max-warnings 0 and prettier --check pass on the changed files. bun run vitest passes: 424 files, 4863 tests, 10 skipped.

Base branch

This PR targets fix/v3-overlaps (#1273), the top of the open stack of validation PRs. Retarget it to main after #1273 merges.

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
policyengine-calculator-next	Ready	Preview	Oct 15, 2026 12:37pm UTC
policyengine-website	Ready	Preview	Oct 15, 2026 12:37pm UTC