From 0f6b73c105f21f9ed5c3588e73c2029aeb41bf53 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Tue, 29 Sep 2026 05:26:11 -0400 Subject: [PATCH 1/3] Label IRS SOI N01000/A01000 as Form 1040 line 7 capital gain or loss Historic Table 2 declared N01000/A01000 with Table 1.4's Schedule D taxable-net-gain concept (irs_soi.returns_with_taxable_net_capital_gains / irs_soi.taxable_net_capital_gains). Every IRS guide for Historic Table 2 (TY2020-TY2023) and for the TY2022 congressional-district, ZIP and county files defines them as "net capital gain (less loss)" from Form 1040 line 7: a Schedule D gain, a limited Schedule D loss, or capital gain distributions filed without Schedule D. For TY2022 the two counts are 30,465,850 and 12,915,122 returns, yet they shared one semantic fact key. Historic Table 2 and the congressional-district file now declare irs_soi.returns_with_form_1040_capital_gain_or_loss / irs_soi.form_1040_capital_gain_or_loss with the IRS labels. measure_id, values and lineage are unchanged; Table 1.4 keeps its concept. - docs/concept-migrations.md records the rename, as docs/architecture.md requires, and architecture.md links it. - tests/test_chronicle_soi_capital_gain_concepts.py requires every irs_soi package reading N01000/A01000 (any vintage) to carry the line 7 concept, keeps the Schedule D concept on Table 1.4 only, pins the consumer selector and values, and checks the concept against the registered publisher cells (N01000 = Table 1.4 distributions-only + Schedule D gain + Schedule D loss returns within 0.25%). - The default-bundle snapshot gains 104 semantic-duplicate keys (2,022 -> 2,126): Historic Table 2 and congressional-district state and US rows now share a concept, as 19 of the 31 IRS columns they share already do. Co-Authored-By: Claude Opus 5.5 --- docs/architecture.md | 5 +- docs/concept-migrations.md | 64 +++++ .../source_package.yaml | 8 +- .../historic_table_2/source_package.yaml | 8 +- .../source_package.yaml | 8 +- tests/test_chronicle_bundle.py | 9 +- ...est_chronicle_soi_capital_gain_concepts.py | 267 ++++++++++++++++++ 7 files changed, 352 insertions(+), 17 deletions(-) create mode 100644 docs/concept-migrations.md create mode 100644 tests/test_chronicle_soi_capital_gain_concepts.py diff --git a/docs/architecture.md b/docs/architecture.md index 3f9a24f8..72438fbf 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -141,8 +141,9 @@ Consumer-facing source vocabulary is a compatibility contract. Keep concept has not changed; put publication-specific wording in `source_table` and labels instead. Within a `source_name`, `measure_id` must distinguish different statistical measures even when their publisher columns share labels such as -`band_a` or `total`. A deliberate rename requires a migration note and a -regression test for the affected consumer selector. +`band_a` or `total`. A deliberate rename requires a migration note in +[concept migrations](concept-migrations.md) and a regression test for the +affected consumer selector. The `policyengine_chronicle.normalization` package owns low-assumption representation helpers: diff --git a/docs/concept-migrations.md b/docs/concept-migrations.md new file mode 100644 index 00000000..945d81bb --- /dev/null +++ b/docs/concept-migrations.md @@ -0,0 +1,64 @@ +# Concept migrations + +Consumer-facing source vocabulary is a compatibility contract +(see [architecture](architecture.md#source-facts-and-microcosm-targets)). A +deliberate concept rename is recorded here, with the regression test that pins +the affected consumer selector. Changing a concept changes each affected +consumer row's `observed_measure.source_concept`, `observed_measure_key`, +`aggregate_fact_key`, `semantic_fact_key`, `legacy_fact_key` and generated +`label`; a label change also moves `layout.measure_label`. `measure_id`, +`source_record_id`, values and lineage do not change. + +## 2026-09-27: IRS SOI Form 1040 line 7 capital gain or (loss) + +| Packages | Columns | Old concept | New concept | +|---|---|---|---| +| `soi-historic-table-2`, `soi-historic-table-2-state-broad-2022` | `N01000` | `irs_soi.returns_with_taxable_net_capital_gains` | `irs_soi.returns_with_form_1040_capital_gain_or_loss` | +| same | `A01000` | `irs_soi.taxable_net_capital_gains` | `irs_soi.form_1040_capital_gain_or_loss` | +| `soi-congressional-district-2022` | `N01000` | `irs_soi.returns_with_net_capital_gains` | `irs_soi.returns_with_form_1040_capital_gain_or_loss` | +| same | `A01000` | `irs_soi.net_capital_gains` | `irs_soi.form_1040_capital_gain_or_loss` | + +Labels become the IRS wording, "Returns with net capital gain (less loss)" and +"Net capital gain (less loss)". `measure_id` stays `net_capital_gains_returns` / +`net_capital_gains_amount`. + +**Why.** The IRS documentation guides for Historic Table 2 (TY2020–TY2023) and +for the TY2022 congressional-district, ZIP and county files define `N01000` as +"Number of returns with net capital gain (less loss)" and `A01000` as "Net +capital gain (less loss) amount", both from Form 1040 line 7, "Capital gain or +(loss)". That line holds a Schedule D gain, a Schedule D loss limited to $3,000 +($1,500 married filing separately), or capital gain distributions reported +without a Schedule D. + +- The Historic Table 2 packages had borrowed the concept of Table 1.4 columns + 37/38, "Sales of capital assets reported on Form 1040, Schedule D: Taxable + net gain", which counts only Schedule D returns with a gain. For TY2022 the + two counts are 30,465,850 and 12,915,122 returns. Historic Table 2's US count + equals Table 1.4's capital-gain-distribution, taxable-net-gain and + taxable-net-loss returns combined to within 0.25% in every year TY2020–TY2023. +- The congressional-district ids read as a gain-only amount. IRC section + 1222(11) and the Publication 1304 Explanation of Terms both use "net capital + gain" for a positive amount only. +- The ids name the Form 1040 line rather than repeat the IRS phrase "net + capital gain (less loss)". Publication 1304 Table A uses that phrase for + Schedule D gain and loss returns without the distribution-only returns + (TY2022: 26,480,998), so the phrase alone does not identify the population. + +**Semantic keys.** Historic Table 2 and the congressional-district file now +share one concept for the same IRS variable, as they already did for 19 of the +31 IRS columns they share. Their TY2022 state and US rows therefore share +semantic keys, which adds 104 semantic-duplicate keys to the default bundle (51 +states and the US, returns and amount). The values differ: the IRS guide for the +congressional-district file says its state totals "may not be comparable to +State totals published elsewhere by SOI because of disclosure protection +procedures or the exclusion of returns that did not match based on the ZIP +code." The Historic Table 2 and Table 1.4 rows no longer share a semantic key. + +**Unchanged.** Table 1.4 keeps `irs_soi.returns_with_taxable_net_capital_gains` +/ `irs_soi.taxable_net_capital_gains`. + +**Pinned by** `tests/test_chronicle_soi_capital_gain_concepts.py`: every +`irs_soi` package that reads `N01000`/`A01000` (any vintage) must declare the +new concept, only Table 1.4 may declare the Schedule D gain concept, the retired +ids may not reappear, and the consumer selector (`source_measure_id`) and +published values are unchanged. diff --git a/packages/irs_soi/congressional_district_2022/source_package.yaml b/packages/irs_soi/congressional_district_2022/source_package.yaml index ef37a997..b53fed9b 100644 --- a/packages/irs_soi/congressional_district_2022/source_package.yaml +++ b/packages/irs_soi/congressional_district_2022/source_package.yaml @@ -14124,24 +14124,24 @@ record_sets: value_scale: 1000 expected_cell_type: number - measure_id: net_capital_gains_returns - label: Returns with net capital gains + label: Returns with net capital gain (less loss) ordinal: 13 column: AL source_column_id: N01000 expected_column_header_row: 1 expected_column_header: N01000 - concept: irs_soi.returns_with_net_capital_gains + concept: irs_soi.returns_with_form_1040_capital_gain_or_loss unit: count aggregation: sum expected_cell_type: number - measure_id: net_capital_gains_amount - label: Net capital gains + label: Net capital gain (less loss) ordinal: 14 column: AM source_column_id: A01000 expected_column_header_row: 1 expected_column_header: A01000 - concept: irs_soi.net_capital_gains + concept: irs_soi.form_1040_capital_gain_or_loss unit: usd aggregation: sum value_scale: 1000 diff --git a/packages/irs_soi/historic_table_2/source_package.yaml b/packages/irs_soi/historic_table_2/source_package.yaml index 723ae241..fee51ec4 100644 --- a/packages/irs_soi/historic_table_2/source_package.yaml +++ b/packages/irs_soi/historic_table_2/source_package.yaml @@ -559,11 +559,11 @@ record_sets: value_scale: 1000 - measure_id: net_capital_gains_returns - label: Returns with taxable net capital gains + label: Returns with net capital gain (less loss) ordinal: 27 column: AK source_column_id: N01000 - concept: irs_soi.returns_with_taxable_net_capital_gains + concept: irs_soi.returns_with_form_1040_capital_gain_or_loss unit: count aggregation: sum expected_cell_type: number @@ -571,11 +571,11 @@ record_sets: expected_column_header: N01000 - measure_id: net_capital_gains_amount - label: Taxable net capital gains + label: Net capital gain (less loss) ordinal: 28 column: AL source_column_id: A01000 - concept: irs_soi.taxable_net_capital_gains + concept: irs_soi.form_1040_capital_gain_or_loss unit: usd aggregation: sum expected_cell_type: number diff --git a/packages/irs_soi/historic_table_2_state_broad_2022/source_package.yaml b/packages/irs_soi/historic_table_2_state_broad_2022/source_package.yaml index 297a6a01..dea8999e 100644 --- a/packages/irs_soi/historic_table_2_state_broad_2022/source_package.yaml +++ b/packages/irs_soi/historic_table_2_state_broad_2022/source_package.yaml @@ -368,22 +368,22 @@ record_sets: expected_column_header: A00900 value_scale: 1000 - measure_id: net_capital_gains_returns - label: Returns with taxable net capital gains + label: Returns with net capital gain (less loss) ordinal: 17 column: AK source_column_id: N01000 - concept: irs_soi.returns_with_taxable_net_capital_gains + concept: irs_soi.returns_with_form_1040_capital_gain_or_loss unit: count aggregation: sum expected_cell_type: number expected_column_header_row: 1 expected_column_header: N01000 - measure_id: net_capital_gains_amount - label: Taxable net capital gains + label: Net capital gain (less loss) ordinal: 18 column: AL source_column_id: A01000 - concept: irs_soi.taxable_net_capital_gains + concept: irs_soi.form_1040_capital_gain_or_loss unit: usd aggregation: sum expected_cell_type: number diff --git a/tests/test_chronicle_bundle.py b/tests/test_chronicle_bundle.py index 9b7cb177..62cb504e 100644 --- a/tests/test_chronicle_bundle.py +++ b/tests/test_chronicle_bundle.py @@ -144,8 +144,11 @@ def test_build_bundle_writes_merged_consumer_contract(tmp_path): # file's state-total and US rows share semantic keys with the Historic # Table 2 rows for the same TY2022 cells (two IRS publications of one # cell): 1,560 new duplicate keys among the changed packages' own - # builds, for a bundle-wide net of +1,555. - "semantic_duplicate_key_count": 2022, + # builds, for a bundle-wide net of +1,555. The HT2 and CD N01000/A01000 + # rows then took one Form 1040 line 7 concept + # (docs/concept-migrations.md), adding 104 more: 51 states and the US + # for returns and amount. + "semantic_duplicate_key_count": 2126, "skipped_source_count": 10, "source_count": 50, "source_package_count": 227, @@ -1231,7 +1234,7 @@ def test_build_bundle_writes_merged_consumer_contract(tmp_path): "tax_unit": 41368, } assert not coverage["duplicates"]["aggregate_fact_keys"] - assert len(coverage["duplicates"]["semantic_fact_keys"]) == 2022 + assert len(coverage["duplicates"]["semantic_fact_keys"]) == 2126 assert Counter(warning["code"] for warning in summary["warnings"]) == { "conflicting_geography_name_across_packages": 50, "conflicting_groupby_value_label": 16, diff --git a/tests/test_chronicle_soi_capital_gain_concepts.py b/tests/test_chronicle_soi_capital_gain_concepts.py new file mode 100644 index 00000000..35260fda --- /dev/null +++ b/tests/test_chronicle_soi_capital_gain_concepts.py @@ -0,0 +1,267 @@ +"""IRS SOI capital-gain columns carry the concept their publisher documents. + +Two different IRS capital-gain counts share the ``net_capital_gains_*`` +measure ids: + +- Historic Table 2 and the congressional-district file read ``N01000`` / + ``A01000``. Every IRS documentation guide for those files (TY2020-TY2023) + defines them as "Number of returns with net capital gain (less loss)" and + "Net capital gain (less loss) amount", sourced from Form 1040 line 7, + "Capital gain or (loss)". That line holds a Schedule D gain, a loss limited + to $3,000 ($1,500 married filing separately), or capital gain distributions + reported without a Schedule D. +- Table 1.4 columns 37/38 (TY2022-TY2023; 25/26 before) are "Sales of capital + assets reported on Form 1040, Schedule D: Taxable net gain", a Schedule D + gain-only count. For TY2022 it is 12,915,122 returns against Historic Table + 2's 30,465,850. + +The concept ids keep those apart. ``measure_id`` stays shared because it is +the selector consumers key on; this module pins both properties so a cloned +vintage (for example a TY2023 package generated from its TY2022 twin) cannot +reintroduce the Table 1.4 concept on a line-7 column. See +``docs/concept-migrations.md`` for the rename record. +""" + +from __future__ import annotations + +from pathlib import Path +from typing import Any, Iterator + +import pytest +import yaml + +from chronicle.consumer_contract import consumer_fact_row +from chronicle.source_package import load_source_package + +REPO_ROOT = Path(__file__).resolve().parents[1] +IRS_SOI_PACKAGES = REPO_ROOT / "packages" / "irs_soi" + +LINE_7_RETURNS_CONCEPT = "irs_soi.returns_with_form_1040_capital_gain_or_loss" +LINE_7_AMOUNT_CONCEPT = "irs_soi.form_1040_capital_gain_or_loss" +LINE_7_DECLARATIONS = { + "N01000": ( + "net_capital_gains_returns", + "Returns with net capital gain (less loss)", + LINE_7_RETURNS_CONCEPT, + ), + "A01000": ( + "net_capital_gains_amount", + "Net capital gain (less loss)", + LINE_7_AMOUNT_CONCEPT, + ), +} +SCHEDULE_D_GAIN_CONCEPTS = frozenset( + { + "irs_soi.returns_with_taxable_net_capital_gains", + "irs_soi.taxable_net_capital_gains", + } +) +# Ids retired by the rename recorded in docs/concept-migrations.md. The +# congressional-district package used these for the same line-7 columns. +RETIRED_LINE_7_CONCEPTS = frozenset( + { + "irs_soi.returns_with_net_capital_gains", + "irs_soi.net_capital_gains", + } +) +# Packages on main that read the line-7 columns. Later vintages join +# automatically because the scan below keys on the IRS variable name. +KNOWN_LINE_7_PACKAGES = frozenset( + { + "historic_table_2", + "historic_table_2_state_broad_2022", + "congressional_district_2022", + } +) + + +def _measures(node: Any) -> Iterator[dict[str, Any]]: + if isinstance(node, dict): + if "measure_id" in node and "concept" in node: + yield node + for value in node.values(): + yield from _measures(value) + elif isinstance(node, list): + for value in node: + yield from _measures(value) + + +def _irs_soi_measures() -> Iterator[tuple[str, dict[str, Any]]]: + for path in sorted(IRS_SOI_PACKAGES.glob("*/source_package.yaml")): + package = yaml.safe_load(path.read_text()) + for measure in _measures(package): + yield path.parent.name, measure + + +def _line_7_variable(measure: dict[str, Any]) -> str | None: + for key in ("source_column_id", "expected_column_header"): + if measure.get(key) in LINE_7_DECLARATIONS: + return measure[key] + return None + + +def test_every_line_7_column_carries_the_line_7_concept(): + seen: set[str] = set() + for package_dir, measure in _irs_soi_measures(): + variable = _line_7_variable(measure) + if variable is None: + continue + seen.add(package_dir) + measure_id, label, concept = LINE_7_DECLARATIONS[variable] + where = f"{package_dir}:{measure['measure_id']}" + assert measure["measure_id"] == measure_id, where + assert measure["label"] == label, where + assert measure["concept"] == concept, where + # Both guards name the same IRS variable, so a column shift cannot + # move the concept onto a neighbour. + assert measure.get("source_column_id") == variable, where + assert measure.get("expected_column_header") == variable, where + + assert KNOWN_LINE_7_PACKAGES <= seen + + +def test_schedule_d_gain_concepts_stay_on_table_1_4(): + declaring = { + (package_dir, measure["measure_id"]) + for package_dir, measure in _irs_soi_measures() + if measure["concept"] in SCHEDULE_D_GAIN_CONCEPTS + } + + assert declaring == { + ("table_1_4", "net_capital_gains_returns"), + ("table_1_4", "net_capital_gains_amount"), + } + + +def test_retired_line_7_concepts_are_not_declared(): + declaring = sorted( + f"{package_dir}:{measure['measure_id']}" + for package_dir, measure in _irs_soi_measures() + if measure["concept"] in RETIRED_LINE_7_CONCEPTS + ) + + assert declaring == [] + + +def _facts_by_record(package_id: str, year: int) -> dict[str, Any]: + package = load_source_package(package_id) + return {fact.source_record_id: fact for fact in package.build_facts(year)} + + +@pytest.fixture(scope="module") +def capital_gain_rows() -> dict[str, dict[str, Any]]: + """Consumer rows for the US all-returns capital-gain facts, TY2022.""" + ht2 = _facts_by_record("soi-historic-table-2", 2022) + cd = _facts_by_record("soi-congressional-district-2022", 2022) + table_1_4 = _facts_by_record("soi-table-1-4", 2022) + facts = { + "ht2_returns": ht2[ + "irs_soi.ty2022.historic_table_2.us.all.net_capital_gains_returns" + ], + "ht2_amount": ht2[ + "irs_soi.ty2022.historic_table_2.us.all.net_capital_gains_amount" + ], + "cd_returns": cd[ + "irs_soi.ty2022.congressional_district_2022.all_returns.us." + "net_capital_gains_returns" + ], + "cd_amount": cd[ + "irs_soi.ty2022.congressional_district_2022.all_returns.us." + "net_capital_gains_amount" + ], + "t14_returns": table_1_4[ + "irs_soi.ty2022.table_1_4.all.net_capital_gains_returns" + ], + "t14_amount": table_1_4[ + "irs_soi.ty2022.table_1_4.all.net_capital_gains_amount" + ], + } + return {name: consumer_fact_row(fact) for name, fact in facts.items()} + + +def test_consumer_selector_survives_the_concept_rename(capital_gain_rows): + # Consumers select these facts by source_measure_id; the rename must not + # move it, and the values stay the published cells. + expected = { + "ht2_returns": ( + "net_capital_gains_returns", + LINE_7_RETURNS_CONCEPT, + 30_465_850, + ), + "ht2_amount": ( + "net_capital_gains_amount", + LINE_7_AMOUNT_CONCEPT, + 1_251_675_034_000, + ), + "cd_returns": ("net_capital_gains_returns", LINE_7_RETURNS_CONCEPT, 29_845_710), + "cd_amount": ( + "net_capital_gains_amount", + LINE_7_AMOUNT_CONCEPT, + 1_157_234_600_000, + ), + "t14_returns": ( + "net_capital_gains_returns", + "irs_soi.returns_with_taxable_net_capital_gains", + 12_915_122, + ), + "t14_amount": ( + "net_capital_gains_amount", + "irs_soi.taxable_net_capital_gains", + 1_269_785_083_000, + ), + } + + for name, (measure_id, concept, value) in expected.items(): + row = capital_gain_rows[name] + assert row["observed_measure"]["source_measure_id"] == measure_id, name + assert row["observed_measure"]["source_concept"] == concept, name + assert row["value"] == value, name + + +def _cells_by_address(package_id: str, year: int) -> dict[str, Any]: + cells = load_source_package(package_id).build_source_cells(year) + assert len({cell.sheet_name for cell in cells}) == 1 + return {cell.address: cell for cell in cells} + + +def test_publisher_cells_show_n01000_is_line_7_not_schedule_d_gain(): + """Differential check of the concept choice against the registered bytes. + + Form 1040 line 7 holds a Schedule D gain, a limited Schedule D loss, or + capital gain distributions filed without a Schedule D. Table 1.4 reports + those three groups in separate columns. Historic Table 2's N01000 (a + population count rounded to tens) matches their sum, not the gain column. + """ + ht2 = _cells_by_address("soi-historic-table-2", 2022) + table_1_4 = _cells_by_address("soi-table-1-4", 2022) + + assert (ht2["A2"].raw_value, ht2["B2"].raw_value) == ("US", 0) + assert ht2["AK1"].raw_value == "N01000" + n01000 = ht2["AK2"].raw_value + + assert table_1_4["A9"].raw_value == "All returns, total" + assert "Capital gain distributions" in table_1_4["AJ3"].raw_value + assert "Schedule D" in table_1_4["AL3"].raw_value + assert table_1_4["AL4"].raw_value == "Taxable\nnet gain" + assert table_1_4["AN4"].raw_value == "Taxable\nnet loss" + distributions_only = table_1_4["AJ9"].raw_value + schedule_d_gain = table_1_4["AL9"].raw_value + schedule_d_loss = table_1_4["AN9"].raw_value + + line_7_components = distributions_only + schedule_d_gain + schedule_d_loss + assert (n01000, line_7_components) == (30_465_850, 30_461_045) + assert abs(n01000 - line_7_components) / line_7_components < 0.0025 + assert n01000 / schedule_d_gain > 2.3 + + +def test_line_7_and_schedule_d_gain_facts_are_different_semantic_facts( + capital_gain_rows, +): + rows = capital_gain_rows + for kind in ("returns", "amount"): + ht2, table_1_4 = rows[f"ht2_{kind}"], rows[f"t14_{kind}"] + # Same period, geography, entity and measure id: only the concept + # separates Form 1040 line 7 from Schedule D taxable net gain. + assert ht2["period"] == table_1_4["period"] + assert ht2["geography"]["id"] == table_1_4["geography"]["id"] + assert ht2["semantic_fact_key"] != table_1_4["semantic_fact_key"] From 7c83e891dd8ee6f7df629654570289ba3869fd5f Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Tue, 29 Sep 2026 16:57:41 -0400 Subject: [PATCH 2/3] Address review: harden capital-gain guards, link chronicle#307 - The guard scan now yields every measure declaration, so a measure missing its concept fails the line 7 check instead of being skipped. - The Schedule D gain concept may live on any Table 1.4 vintage package (table_1_4*), still only on the net_capital_gains_* measures. - docs/concept-migrations.md: count shared columns per package (19 of 31 national, 24 of 36 state), say the HT2/Table 1.4 collision was latent in the default bundle, and point the shared measure_id and the 10 other HT2/CD concept mismatches at chronicle#307. Co-Authored-By: Claude Opus 5.5 --- docs/concept-migrations.md | 21 +++++++++++----- ...est_chronicle_soi_capital_gain_concepts.py | 24 ++++++++++++------- 2 files changed, 31 insertions(+), 14 deletions(-) diff --git a/docs/concept-migrations.md b/docs/concept-migrations.md index 945d81bb..e00ea5e1 100644 --- a/docs/concept-migrations.md +++ b/docs/concept-migrations.md @@ -45,17 +45,26 @@ without a Schedule D. (TY2022: 26,480,998), so the phrase alone does not identify the population. **Semantic keys.** Historic Table 2 and the congressional-district file now -share one concept for the same IRS variable, as they already did for 19 of the -31 IRS columns they share. Their TY2022 state and US rows therefore share -semantic keys, which adds 104 semantic-duplicate keys to the default bundle (51 -states and the US, returns and amount). The values differ: the IRS guide for the +share one concept for the same IRS variable. That was already the case for 19 of +the 31 IRS columns the national package shares with the congressional-district +file (24 of 36 for the state package); the capital-gains pair was 2 of the same +12 exceptions in both. Their TY2022 state and US rows therefore share semantic +keys, which adds 104 semantic-duplicate keys to the default bundle (51 states +and the US, returns and amount). The values differ: the IRS guide for the congressional-district file says its state totals "may not be comparable to State totals published elsewhere by SOI because of disclosure protection procedures or the exclusion of returns that did not match based on the ZIP -code." The Historic Table 2 and Table 1.4 rows no longer share a semantic key. +code." Historic Table 2 and Table 1.4 rows for the same year can no longer share +a semantic key. In the default bundle they did not collide, because Historic +Table 2 builds at TY2022 and Table 1.4 at TY2023, but a same-year build did. **Unchanged.** Table 1.4 keeps `irs_soi.returns_with_taxable_net_capital_gains` -/ `irs_soi.taxable_net_capital_gains`. +/ `irs_soi.taxable_net_capital_gains`. `measure_id` stays +`net_capital_gains_returns` / `net_capital_gains_amount` in all four packages +because Microcosm selects on it, so one `measure_id` still names two different +measures; chronicle#307 tracks giving them distinct ids together with the +Microcosm selector change, and the 10 other columns the two files label with +different concepts. **Pinned by** `tests/test_chronicle_soi_capital_gain_concepts.py`: every `irs_soi` package that reads `N01000`/`A01000` (any vintage) must declare the diff --git a/tests/test_chronicle_soi_capital_gain_concepts.py b/tests/test_chronicle_soi_capital_gain_concepts.py index 35260fda..04cf9db2 100644 --- a/tests/test_chronicle_soi_capital_gain_concepts.py +++ b/tests/test_chronicle_soi_capital_gain_concepts.py @@ -76,8 +76,10 @@ def _measures(node: Any) -> Iterator[dict[str, Any]]: + # Every measure declaration carries a measure_id; a missing concept is + # yielded too, so the guards below fail on it instead of skipping it. if isinstance(node, dict): - if "measure_id" in node and "concept" in node: + if "measure_id" in node: yield node for value in node.values(): yield from _measures(value) @@ -111,7 +113,7 @@ def test_every_line_7_column_carries_the_line_7_concept(): where = f"{package_dir}:{measure['measure_id']}" assert measure["measure_id"] == measure_id, where assert measure["label"] == label, where - assert measure["concept"] == concept, where + assert measure.get("concept") == concept, where # Both guards name the same IRS variable, so a column shift cannot # move the concept onto a neighbour. assert measure.get("source_column_id") == variable, where @@ -121,23 +123,29 @@ def test_every_line_7_column_carries_the_line_7_concept(): def test_schedule_d_gain_concepts_stay_on_table_1_4(): + # Table 1.4 vintage packages (table_1_4, table_1_4_, ...) may carry + # the Schedule D taxable-net-gain concept; nothing else may. declaring = { (package_dir, measure["measure_id"]) for package_dir, measure in _irs_soi_measures() - if measure["concept"] in SCHEDULE_D_GAIN_CONCEPTS + if measure.get("concept") in SCHEDULE_D_GAIN_CONCEPTS } - assert declaring == { - ("table_1_4", "net_capital_gains_returns"), - ("table_1_4", "net_capital_gains_amount"), - } + assert ("table_1_4", "net_capital_gains_returns") in declaring + assert ("table_1_4", "net_capital_gains_amount") in declaring + assert { + (package_dir, measure_id) + for package_dir, measure_id in declaring + if not package_dir.startswith("table_1_4") + or measure_id not in {"net_capital_gains_returns", "net_capital_gains_amount"} + } == set() def test_retired_line_7_concepts_are_not_declared(): declaring = sorted( f"{package_dir}:{measure['measure_id']}" for package_dir, measure in _irs_soi_measures() - if measure["concept"] in RETIRED_LINE_7_CONCEPTS + if measure.get("concept") in RETIRED_LINE_7_CONCEPTS ) assert declaring == [] From 8e4c3b7268999ec05e31c089b7a4552e24c96711 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Tue, 29 Sep 2026 18:40:14 -0400 Subject: [PATCH 3/3] Correct the concept-migration note's counts and scope From the second review of chronicle#304: - Count shared IRS variables by the header each measure reads, not only by source_column_id: before this change 26 of 38 (national) and 24 of 36 (state) carried one concept in Historic Table 2 and the congressional-district file, with the same 12 exceptions. The first commit's message said 19 of 31, counting source_column_id only. - Only the capital-gain rows stop sharing a semantic key with Table 1.4. - State what the guard enforces: measures that name N01000/A01000 in a source_column_id or expected_column_header guard. - Give Table 1.4's pre-TY2022 column numbers and say which year CI checks. Co-Authored-By: Claude Opus 5.5 --- docs/concept-migrations.md | 29 ++++++++++++++++------------- 1 file changed, 16 insertions(+), 13 deletions(-) diff --git a/docs/concept-migrations.md b/docs/concept-migrations.md index e00ea5e1..ed4b43c5 100644 --- a/docs/concept-migrations.md +++ b/docs/concept-migrations.md @@ -31,11 +31,12 @@ capital gain (less loss) amount", both from Form 1040 line 7, "Capital gain or without a Schedule D. - The Historic Table 2 packages had borrowed the concept of Table 1.4 columns - 37/38, "Sales of capital assets reported on Form 1040, Schedule D: Taxable + 37/38 (25/26 before TY2022), "Sales of capital assets reported on Form 1040, Schedule D: Taxable net gain", which counts only Schedule D returns with a gain. For TY2022 the two counts are 30,465,850 and 12,915,122 returns. Historic Table 2's US count equals Table 1.4's capital-gain-distribution, taxable-net-gain and - taxable-net-loss returns combined to within 0.25% in every year TY2020–TY2023. + taxable-net-loss returns combined to within 0.25% in every year TY2020–TY2023 + (the TY2022 identity is checked in CI against Chronicle's registered files). - The congressional-district ids read as a gain-only amount. IRC section 1222(11) and the Publication 1304 Explanation of Terms both use "net capital gain" for a positive amount only. @@ -45,29 +46,31 @@ without a Schedule D. (TY2022: 26,480,998), so the phrase alone does not identify the population. **Semantic keys.** Historic Table 2 and the congressional-district file now -share one concept for the same IRS variable. That was already the case for 19 of -the 31 IRS columns the national package shares with the congressional-district -file (24 of 36 for the state package); the capital-gains pair was 2 of the same -12 exceptions in both. Their TY2022 state and US rows therefore share semantic +share one concept for the same IRS variable. Before this change that held for 26 +of the 38 IRS variables the national package shares with the +congressional-district file (24 of 36 for the state package); the capital-gains +pair was 2 of the same 12 exceptions in both. Their TY2022 state and US rows therefore share semantic keys, which adds 104 semantic-duplicate keys to the default bundle (51 states and the US, returns and amount). The values differ: the IRS guide for the congressional-district file says its state totals "may not be comparable to State totals published elsewhere by SOI because of disclosure protection procedures or the exclusion of returns that did not match based on the ZIP -code." Historic Table 2 and Table 1.4 rows for the same year can no longer share -a semantic key. In the default bundle they did not collide, because Historic -Table 2 builds at TY2022 and Table 1.4 at TY2023, but a same-year build did. +code." Historic Table 2 and Table 1.4 capital-gain rows for the same year can +no longer share a semantic key. In the default bundle they did not collide, +because Historic Table 2 builds at TY2022 and Table 1.4 at TY2023, but a TY2022 +build of both did. **Unchanged.** Table 1.4 keeps `irs_soi.returns_with_taxable_net_capital_gains` / `irs_soi.taxable_net_capital_gains`. `measure_id` stays `net_capital_gains_returns` / `net_capital_gains_amount` in all four packages because Microcosm selects on it, so one `measure_id` still names two different measures; chronicle#307 tracks giving them distinct ids together with the -Microcosm selector change, and the 10 other columns the two files label with -different concepts. +Microcosm selector change, and the 10 other IRS columns that Historic Table 2 +and the congressional-district file still label with different concepts. **Pinned by** `tests/test_chronicle_soi_capital_gain_concepts.py`: every -`irs_soi` package that reads `N01000`/`A01000` (any vintage) must declare the -new concept, only Table 1.4 may declare the Schedule D gain concept, the retired +`irs_soi` measure (any vintage) that names `N01000`/`A01000` in its +`source_column_id` or `expected_column_header` guard must declare the new +concept, only Table 1.4 may declare the Schedule D gain concept, the retired ids may not reappear, and the consumer selector (`source_measure_id`) and published values are unchanged.