diff --git a/docs/architecture.md b/docs/architecture.md index 3f9a24f8..72438fbf 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -141,8 +141,9 @@ Consumer-facing source vocabulary is a compatibility contract. Keep concept has not changed; put publication-specific wording in `source_table` and labels instead. Within a `source_name`, `measure_id` must distinguish different statistical measures even when their publisher columns share labels such as -`band_a` or `total`. A deliberate rename requires a migration note and a -regression test for the affected consumer selector. +`band_a` or `total`. A deliberate rename requires a migration note in +[concept migrations](concept-migrations.md) and a regression test for the +affected consumer selector. The `policyengine_chronicle.normalization` package owns low-assumption representation helpers: diff --git a/docs/concept-migrations.md b/docs/concept-migrations.md new file mode 100644 index 00000000..ed4b43c5 --- /dev/null +++ b/docs/concept-migrations.md @@ -0,0 +1,76 @@ +# Concept migrations + +Consumer-facing source vocabulary is a compatibility contract +(see [architecture](architecture.md#source-facts-and-microcosm-targets)). A +deliberate concept rename is recorded here, with the regression test that pins +the affected consumer selector. Changing a concept changes each affected +consumer row's `observed_measure.source_concept`, `observed_measure_key`, +`aggregate_fact_key`, `semantic_fact_key`, `legacy_fact_key` and generated +`label`; a label change also moves `layout.measure_label`. `measure_id`, +`source_record_id`, values and lineage do not change. + +## 2026-09-27: IRS SOI Form 1040 line 7 capital gain or (loss) + +| Packages | Columns | Old concept | New concept | +|---|---|---|---| +| `soi-historic-table-2`, `soi-historic-table-2-state-broad-2022` | `N01000` | `irs_soi.returns_with_taxable_net_capital_gains` | `irs_soi.returns_with_form_1040_capital_gain_or_loss` | +| same | `A01000` | `irs_soi.taxable_net_capital_gains` | `irs_soi.form_1040_capital_gain_or_loss` | +| `soi-congressional-district-2022` | `N01000` | `irs_soi.returns_with_net_capital_gains` | `irs_soi.returns_with_form_1040_capital_gain_or_loss` | +| same | `A01000` | `irs_soi.net_capital_gains` | `irs_soi.form_1040_capital_gain_or_loss` | + +Labels become the IRS wording, "Returns with net capital gain (less loss)" and +"Net capital gain (less loss)". `measure_id` stays `net_capital_gains_returns` / +`net_capital_gains_amount`. + +**Why.** The IRS documentation guides for Historic Table 2 (TY2020–TY2023) and +for the TY2022 congressional-district, ZIP and county files define `N01000` as +"Number of returns with net capital gain (less loss)" and `A01000` as "Net +capital gain (less loss) amount", both from Form 1040 line 7, "Capital gain or +(loss)". That line holds a Schedule D gain, a Schedule D loss limited to $3,000 +($1,500 married filing separately), or capital gain distributions reported +without a Schedule D. + +- The Historic Table 2 packages had borrowed the concept of Table 1.4 columns + 37/38 (25/26 before TY2022), "Sales of capital assets reported on Form 1040, Schedule D: Taxable + net gain", which counts only Schedule D returns with a gain. For TY2022 the + two counts are 30,465,850 and 12,915,122 returns. Historic Table 2's US count + equals Table 1.4's capital-gain-distribution, taxable-net-gain and + taxable-net-loss returns combined to within 0.25% in every year TY2020–TY2023 + (the TY2022 identity is checked in CI against Chronicle's registered files). +- The congressional-district ids read as a gain-only amount. IRC section + 1222(11) and the Publication 1304 Explanation of Terms both use "net capital + gain" for a positive amount only. +- The ids name the Form 1040 line rather than repeat the IRS phrase "net + capital gain (less loss)". Publication 1304 Table A uses that phrase for + Schedule D gain and loss returns without the distribution-only returns + (TY2022: 26,480,998), so the phrase alone does not identify the population. + +**Semantic keys.** Historic Table 2 and the congressional-district file now +share one concept for the same IRS variable. Before this change that held for 26 +of the 38 IRS variables the national package shares with the +congressional-district file (24 of 36 for the state package); the capital-gains +pair was 2 of the same 12 exceptions in both. Their TY2022 state and US rows therefore share semantic +keys, which adds 104 semantic-duplicate keys to the default bundle (51 states +and the US, returns and amount). The values differ: the IRS guide for the +congressional-district file says its state totals "may not be comparable to +State totals published elsewhere by SOI because of disclosure protection +procedures or the exclusion of returns that did not match based on the ZIP +code." Historic Table 2 and Table 1.4 capital-gain rows for the same year can +no longer share a semantic key. In the default bundle they did not collide, +because Historic Table 2 builds at TY2022 and Table 1.4 at TY2023, but a TY2022 +build of both did. + +**Unchanged.** Table 1.4 keeps `irs_soi.returns_with_taxable_net_capital_gains` +/ `irs_soi.taxable_net_capital_gains`. `measure_id` stays +`net_capital_gains_returns` / `net_capital_gains_amount` in all four packages +because Microcosm selects on it, so one `measure_id` still names two different +measures; chronicle#307 tracks giving them distinct ids together with the +Microcosm selector change, and the 10 other IRS columns that Historic Table 2 +and the congressional-district file still label with different concepts. + +**Pinned by** `tests/test_chronicle_soi_capital_gain_concepts.py`: every +`irs_soi` measure (any vintage) that names `N01000`/`A01000` in its +`source_column_id` or `expected_column_header` guard must declare the new +concept, only Table 1.4 may declare the Schedule D gain concept, the retired +ids may not reappear, and the consumer selector (`source_measure_id`) and +published values are unchanged. diff --git a/packages/irs_soi/congressional_district_2022/source_package.yaml b/packages/irs_soi/congressional_district_2022/source_package.yaml index ef37a997..b53fed9b 100644 --- a/packages/irs_soi/congressional_district_2022/source_package.yaml +++ b/packages/irs_soi/congressional_district_2022/source_package.yaml @@ -14124,24 +14124,24 @@ record_sets: value_scale: 1000 expected_cell_type: number - measure_id: net_capital_gains_returns - label: Returns with net capital gains + label: Returns with net capital gain (less loss) ordinal: 13 column: AL source_column_id: N01000 expected_column_header_row: 1 expected_column_header: N01000 - concept: irs_soi.returns_with_net_capital_gains + concept: irs_soi.returns_with_form_1040_capital_gain_or_loss unit: count aggregation: sum expected_cell_type: number - measure_id: net_capital_gains_amount - label: Net capital gains + label: Net capital gain (less loss) ordinal: 14 column: AM source_column_id: A01000 expected_column_header_row: 1 expected_column_header: A01000 - concept: irs_soi.net_capital_gains + concept: irs_soi.form_1040_capital_gain_or_loss unit: usd aggregation: sum value_scale: 1000 diff --git a/packages/irs_soi/historic_table_2/source_package.yaml b/packages/irs_soi/historic_table_2/source_package.yaml index 723ae241..fee51ec4 100644 --- a/packages/irs_soi/historic_table_2/source_package.yaml +++ b/packages/irs_soi/historic_table_2/source_package.yaml @@ -559,11 +559,11 @@ record_sets: value_scale: 1000 - measure_id: net_capital_gains_returns - label: Returns with taxable net capital gains + label: Returns with net capital gain (less loss) ordinal: 27 column: AK source_column_id: N01000 - concept: irs_soi.returns_with_taxable_net_capital_gains + concept: irs_soi.returns_with_form_1040_capital_gain_or_loss unit: count aggregation: sum expected_cell_type: number @@ -571,11 +571,11 @@ record_sets: expected_column_header: N01000 - measure_id: net_capital_gains_amount - label: Taxable net capital gains + label: Net capital gain (less loss) ordinal: 28 column: AL source_column_id: A01000 - concept: irs_soi.taxable_net_capital_gains + concept: irs_soi.form_1040_capital_gain_or_loss unit: usd aggregation: sum expected_cell_type: number diff --git a/packages/irs_soi/historic_table_2_state_broad_2022/source_package.yaml b/packages/irs_soi/historic_table_2_state_broad_2022/source_package.yaml index 297a6a01..dea8999e 100644 --- a/packages/irs_soi/historic_table_2_state_broad_2022/source_package.yaml +++ b/packages/irs_soi/historic_table_2_state_broad_2022/source_package.yaml @@ -368,22 +368,22 @@ record_sets: expected_column_header: A00900 value_scale: 1000 - measure_id: net_capital_gains_returns - label: Returns with taxable net capital gains + label: Returns with net capital gain (less loss) ordinal: 17 column: AK source_column_id: N01000 - concept: irs_soi.returns_with_taxable_net_capital_gains + concept: irs_soi.returns_with_form_1040_capital_gain_or_loss unit: count aggregation: sum expected_cell_type: number expected_column_header_row: 1 expected_column_header: N01000 - measure_id: net_capital_gains_amount - label: Taxable net capital gains + label: Net capital gain (less loss) ordinal: 18 column: AL source_column_id: A01000 - concept: irs_soi.taxable_net_capital_gains + concept: irs_soi.form_1040_capital_gain_or_loss unit: usd aggregation: sum expected_cell_type: number diff --git a/tests/test_chronicle_bundle.py b/tests/test_chronicle_bundle.py index 9b7cb177..62cb504e 100644 --- a/tests/test_chronicle_bundle.py +++ b/tests/test_chronicle_bundle.py @@ -144,8 +144,11 @@ def test_build_bundle_writes_merged_consumer_contract(tmp_path): # file's state-total and US rows share semantic keys with the Historic # Table 2 rows for the same TY2022 cells (two IRS publications of one # cell): 1,560 new duplicate keys among the changed packages' own - # builds, for a bundle-wide net of +1,555. - "semantic_duplicate_key_count": 2022, + # builds, for a bundle-wide net of +1,555. The HT2 and CD N01000/A01000 + # rows then took one Form 1040 line 7 concept + # (docs/concept-migrations.md), adding 104 more: 51 states and the US + # for returns and amount. + "semantic_duplicate_key_count": 2126, "skipped_source_count": 10, "source_count": 50, "source_package_count": 227, @@ -1231,7 +1234,7 @@ def test_build_bundle_writes_merged_consumer_contract(tmp_path): "tax_unit": 41368, } assert not coverage["duplicates"]["aggregate_fact_keys"] - assert len(coverage["duplicates"]["semantic_fact_keys"]) == 2022 + assert len(coverage["duplicates"]["semantic_fact_keys"]) == 2126 assert Counter(warning["code"] for warning in summary["warnings"]) == { "conflicting_geography_name_across_packages": 50, "conflicting_groupby_value_label": 16, diff --git a/tests/test_chronicle_soi_capital_gain_concepts.py b/tests/test_chronicle_soi_capital_gain_concepts.py new file mode 100644 index 00000000..04cf9db2 --- /dev/null +++ b/tests/test_chronicle_soi_capital_gain_concepts.py @@ -0,0 +1,275 @@ +"""IRS SOI capital-gain columns carry the concept their publisher documents. + +Two different IRS capital-gain counts share the ``net_capital_gains_*`` +measure ids: + +- Historic Table 2 and the congressional-district file read ``N01000`` / + ``A01000``. Every IRS documentation guide for those files (TY2020-TY2023) + defines them as "Number of returns with net capital gain (less loss)" and + "Net capital gain (less loss) amount", sourced from Form 1040 line 7, + "Capital gain or (loss)". That line holds a Schedule D gain, a loss limited + to $3,000 ($1,500 married filing separately), or capital gain distributions + reported without a Schedule D. +- Table 1.4 columns 37/38 (TY2022-TY2023; 25/26 before) are "Sales of capital + assets reported on Form 1040, Schedule D: Taxable net gain", a Schedule D + gain-only count. For TY2022 it is 12,915,122 returns against Historic Table + 2's 30,465,850. + +The concept ids keep those apart. ``measure_id`` stays shared because it is +the selector consumers key on; this module pins both properties so a cloned +vintage (for example a TY2023 package generated from its TY2022 twin) cannot +reintroduce the Table 1.4 concept on a line-7 column. See +``docs/concept-migrations.md`` for the rename record. +""" + +from __future__ import annotations + +from pathlib import Path +from typing import Any, Iterator + +import pytest +import yaml + +from chronicle.consumer_contract import consumer_fact_row +from chronicle.source_package import load_source_package + +REPO_ROOT = Path(__file__).resolve().parents[1] +IRS_SOI_PACKAGES = REPO_ROOT / "packages" / "irs_soi" + +LINE_7_RETURNS_CONCEPT = "irs_soi.returns_with_form_1040_capital_gain_or_loss" +LINE_7_AMOUNT_CONCEPT = "irs_soi.form_1040_capital_gain_or_loss" +LINE_7_DECLARATIONS = { + "N01000": ( + "net_capital_gains_returns", + "Returns with net capital gain (less loss)", + LINE_7_RETURNS_CONCEPT, + ), + "A01000": ( + "net_capital_gains_amount", + "Net capital gain (less loss)", + LINE_7_AMOUNT_CONCEPT, + ), +} +SCHEDULE_D_GAIN_CONCEPTS = frozenset( + { + "irs_soi.returns_with_taxable_net_capital_gains", + "irs_soi.taxable_net_capital_gains", + } +) +# Ids retired by the rename recorded in docs/concept-migrations.md. The +# congressional-district package used these for the same line-7 columns. +RETIRED_LINE_7_CONCEPTS = frozenset( + { + "irs_soi.returns_with_net_capital_gains", + "irs_soi.net_capital_gains", + } +) +# Packages on main that read the line-7 columns. Later vintages join +# automatically because the scan below keys on the IRS variable name. +KNOWN_LINE_7_PACKAGES = frozenset( + { + "historic_table_2", + "historic_table_2_state_broad_2022", + "congressional_district_2022", + } +) + + +def _measures(node: Any) -> Iterator[dict[str, Any]]: + # Every measure declaration carries a measure_id; a missing concept is + # yielded too, so the guards below fail on it instead of skipping it. + if isinstance(node, dict): + if "measure_id" in node: + yield node + for value in node.values(): + yield from _measures(value) + elif isinstance(node, list): + for value in node: + yield from _measures(value) + + +def _irs_soi_measures() -> Iterator[tuple[str, dict[str, Any]]]: + for path in sorted(IRS_SOI_PACKAGES.glob("*/source_package.yaml")): + package = yaml.safe_load(path.read_text()) + for measure in _measures(package): + yield path.parent.name, measure + + +def _line_7_variable(measure: dict[str, Any]) -> str | None: + for key in ("source_column_id", "expected_column_header"): + if measure.get(key) in LINE_7_DECLARATIONS: + return measure[key] + return None + + +def test_every_line_7_column_carries_the_line_7_concept(): + seen: set[str] = set() + for package_dir, measure in _irs_soi_measures(): + variable = _line_7_variable(measure) + if variable is None: + continue + seen.add(package_dir) + measure_id, label, concept = LINE_7_DECLARATIONS[variable] + where = f"{package_dir}:{measure['measure_id']}" + assert measure["measure_id"] == measure_id, where + assert measure["label"] == label, where + assert measure.get("concept") == concept, where + # Both guards name the same IRS variable, so a column shift cannot + # move the concept onto a neighbour. + assert measure.get("source_column_id") == variable, where + assert measure.get("expected_column_header") == variable, where + + assert KNOWN_LINE_7_PACKAGES <= seen + + +def test_schedule_d_gain_concepts_stay_on_table_1_4(): + # Table 1.4 vintage packages (table_1_4, table_1_4_, ...) may carry + # the Schedule D taxable-net-gain concept; nothing else may. + declaring = { + (package_dir, measure["measure_id"]) + for package_dir, measure in _irs_soi_measures() + if measure.get("concept") in SCHEDULE_D_GAIN_CONCEPTS + } + + assert ("table_1_4", "net_capital_gains_returns") in declaring + assert ("table_1_4", "net_capital_gains_amount") in declaring + assert { + (package_dir, measure_id) + for package_dir, measure_id in declaring + if not package_dir.startswith("table_1_4") + or measure_id not in {"net_capital_gains_returns", "net_capital_gains_amount"} + } == set() + + +def test_retired_line_7_concepts_are_not_declared(): + declaring = sorted( + f"{package_dir}:{measure['measure_id']}" + for package_dir, measure in _irs_soi_measures() + if measure.get("concept") in RETIRED_LINE_7_CONCEPTS + ) + + assert declaring == [] + + +def _facts_by_record(package_id: str, year: int) -> dict[str, Any]: + package = load_source_package(package_id) + return {fact.source_record_id: fact for fact in package.build_facts(year)} + + +@pytest.fixture(scope="module") +def capital_gain_rows() -> dict[str, dict[str, Any]]: + """Consumer rows for the US all-returns capital-gain facts, TY2022.""" + ht2 = _facts_by_record("soi-historic-table-2", 2022) + cd = _facts_by_record("soi-congressional-district-2022", 2022) + table_1_4 = _facts_by_record("soi-table-1-4", 2022) + facts = { + "ht2_returns": ht2[ + "irs_soi.ty2022.historic_table_2.us.all.net_capital_gains_returns" + ], + "ht2_amount": ht2[ + "irs_soi.ty2022.historic_table_2.us.all.net_capital_gains_amount" + ], + "cd_returns": cd[ + "irs_soi.ty2022.congressional_district_2022.all_returns.us." + "net_capital_gains_returns" + ], + "cd_amount": cd[ + "irs_soi.ty2022.congressional_district_2022.all_returns.us." + "net_capital_gains_amount" + ], + "t14_returns": table_1_4[ + "irs_soi.ty2022.table_1_4.all.net_capital_gains_returns" + ], + "t14_amount": table_1_4[ + "irs_soi.ty2022.table_1_4.all.net_capital_gains_amount" + ], + } + return {name: consumer_fact_row(fact) for name, fact in facts.items()} + + +def test_consumer_selector_survives_the_concept_rename(capital_gain_rows): + # Consumers select these facts by source_measure_id; the rename must not + # move it, and the values stay the published cells. + expected = { + "ht2_returns": ( + "net_capital_gains_returns", + LINE_7_RETURNS_CONCEPT, + 30_465_850, + ), + "ht2_amount": ( + "net_capital_gains_amount", + LINE_7_AMOUNT_CONCEPT, + 1_251_675_034_000, + ), + "cd_returns": ("net_capital_gains_returns", LINE_7_RETURNS_CONCEPT, 29_845_710), + "cd_amount": ( + "net_capital_gains_amount", + LINE_7_AMOUNT_CONCEPT, + 1_157_234_600_000, + ), + "t14_returns": ( + "net_capital_gains_returns", + "irs_soi.returns_with_taxable_net_capital_gains", + 12_915_122, + ), + "t14_amount": ( + "net_capital_gains_amount", + "irs_soi.taxable_net_capital_gains", + 1_269_785_083_000, + ), + } + + for name, (measure_id, concept, value) in expected.items(): + row = capital_gain_rows[name] + assert row["observed_measure"]["source_measure_id"] == measure_id, name + assert row["observed_measure"]["source_concept"] == concept, name + assert row["value"] == value, name + + +def _cells_by_address(package_id: str, year: int) -> dict[str, Any]: + cells = load_source_package(package_id).build_source_cells(year) + assert len({cell.sheet_name for cell in cells}) == 1 + return {cell.address: cell for cell in cells} + + +def test_publisher_cells_show_n01000_is_line_7_not_schedule_d_gain(): + """Differential check of the concept choice against the registered bytes. + + Form 1040 line 7 holds a Schedule D gain, a limited Schedule D loss, or + capital gain distributions filed without a Schedule D. Table 1.4 reports + those three groups in separate columns. Historic Table 2's N01000 (a + population count rounded to tens) matches their sum, not the gain column. + """ + ht2 = _cells_by_address("soi-historic-table-2", 2022) + table_1_4 = _cells_by_address("soi-table-1-4", 2022) + + assert (ht2["A2"].raw_value, ht2["B2"].raw_value) == ("US", 0) + assert ht2["AK1"].raw_value == "N01000" + n01000 = ht2["AK2"].raw_value + + assert table_1_4["A9"].raw_value == "All returns, total" + assert "Capital gain distributions" in table_1_4["AJ3"].raw_value + assert "Schedule D" in table_1_4["AL3"].raw_value + assert table_1_4["AL4"].raw_value == "Taxable\nnet gain" + assert table_1_4["AN4"].raw_value == "Taxable\nnet loss" + distributions_only = table_1_4["AJ9"].raw_value + schedule_d_gain = table_1_4["AL9"].raw_value + schedule_d_loss = table_1_4["AN9"].raw_value + + line_7_components = distributions_only + schedule_d_gain + schedule_d_loss + assert (n01000, line_7_components) == (30_465_850, 30_461_045) + assert abs(n01000 - line_7_components) / line_7_components < 0.0025 + assert n01000 / schedule_d_gain > 2.3 + + +def test_line_7_and_schedule_d_gain_facts_are_different_semantic_facts( + capital_gain_rows, +): + rows = capital_gain_rows + for kind in ("returns", "amount"): + ht2, table_1_4 = rows[f"ht2_{kind}"], rows[f"t14_{kind}"] + # Same period, geography, entity and measure id: only the concept + # separates Form 1040 line 7 from Schedule D taxable net gain. + assert ht2["period"] == table_1_4["period"] + assert ht2["geography"]["id"] == table_1_4["geography"]["id"] + assert ht2["semantic_fact_key"] != table_1_4["semantic_fact_key"]