Choose the capital-gains rebase control by concept and period, not feed order

changelog.d/1035-capital-gains-control-concept.fixed.md

@@ -0,0 +1 @@
+Rebase Historic Table 2 capital-gains rows only onto the Table 1.4 Schedule D taxable-net-gain control, the concept `capital_gains_gross` measures. Feed order no longer picks the control: congressional-district and Historic Table 2 rows (Form 1040 line 7, gain or loss) never qualify, a tie between matched records refuses the compile, and a line-7 row with no usable control is dropped instead of shipping as a level. On the pinned feed the 51 `national_state` state return-count targets move from 29,599,604 (2.39x the national target) to 12,289,836 (microcosm#1035).

docs/us-chronicle-feed-repin.md

@@ -171,6 +171,16 @@ compiled register goes from 32,867 to 32,843 targets, and the
 count, the only change: 32,842 compiled targets and 5,694 in
 `national_state` (`d315c75804ef`).
 
+**Erratum (27 September 2026, microcosm#1035).** Row order changed too. The
+new feed is sorted by the re-derived `aggregate_fact_key`s, and the
+capital-gains rebase kept the first of two equal-period controls. The
+congressional-district US row (TY2022 data stamped ty2023) now sorted ahead of
+Table 1.4 ty2023. It took over the returns control and raised the 51 Historic
+Table 2 state return counts from 12,289,836 to 29,599,604. The control is now
+chosen by concept and period alone
+([us-soi-capital-gains-concepts.md](us-soi-capital-gains-concepts.md)). The
+counts are unchanged and `national_state` is `65e4dde11c83`.
+
 The labelled feed exposed one latent defect in microcosm, fixed in this
 change: `apply_us_medicaid_enrollment_substitutions` built Rhode Island's
 substituted spec by cloning a neighbouring state's spec and kept that state's

docs/us-soi-capital-gains-concepts.md

@@ -0,0 +1,120 @@
+# SOI capital-gains concepts and the rebase control
+
+`capital_gains_gross` targets come from three IRS SOI products that count
+different populations under the same measure ids (`net_capital_gains_returns`,
+`net_capital_gains_amount`). This page records what each one counts, which
+one the model measures, and how `fiscal_targets.py` combines them
+(microcosm#1035).
+
+## What each table counts
+
+| Source | Columns | What it counts |
+|---|---|---|
+| Table 1.4 (`23in14ar.xls`) | col 37 / 38, "Sales of capital assets reported on Form 1040, Schedule D: Taxable net gain" | Schedule D returns that net to a gain, and that gain |
+| Table 1.4 | col 39 / 40, "Taxable net loss" | Schedule D returns that net to a loss, and the loss after the $3,000 limit |
+| Table 1.4 | col 35 / 36, "Capital gain distributions reported on Form 1040" | Returns that report only distributions, without Schedule D |
+| Historic Table 2 (`22in55cmcsv.csv`), congressional-district file (`22incd.csv`) | N01000 / A01000 | "Number of returns with net capital gain (less loss)" and its amount, at Form 1040 line 7 (both documentation guides) |
+
+Line 7 carries a Schedule D gain, a loss-limited Schedule D loss, or
+distributions reported without Schedule D (2022 Form 1040 instructions, line
+7, Exception 1). Columns 35, 37 and 39 are disjoint: Table 1.3 row 18, "Sales
+of capital assets net gain", equals col 35 + col 37 exactly in TY2022 and
+TY2023.
+
+N01000 is the union of the three Table 1.4 populations:
+
+| Tax year | HT2 US N01000 | Table 1.4 col 35 + 37 + 39 | Difference | HT2 A01000 vs col 36 + 38 − 40 |
+|---|---:|---:|---:|---:|
+| 2020 | 29,008,620 | 29,003,885 | +0.016% | −0.63% |
+| 2021 | 32,996,180 | 33,076,998 | −0.244% | −0.29% |
+| 2022 | 30,465,850 | 30,461,045 | +0.016% | −0.17% |
+| 2023 | 29,481,840 | 29,434,188 | +0.162% | −0.28% |
+
+In TY2022, col 37 (12,915,122) is 42.4% of N01000. The amounts are close
+(A01000 is 1.4% below col 38), because distributions and capped losses are
+small in dollars. The counts differ by 2.36x.
+
+The congressional-district file covers a subset of HT2: it leaves out
+territories, APO/FPO and foreign addresses and returns without a matched ZIP
+code. Its US N01000 is 98.0% of HT2's.
+
+## What the model measures
+
+`capital_gains_gross` maps to PE-US `capital_gains`, which is short-term plus
+long-term gains, i.e. Schedule D before the loss limit. Distributions reported
+without Schedule D are the separate `non_sch_d_capital_gains`, with its own
+Table 1.4 col 35 target. The SOI materializer counts a tax unit when the
+unit's summed `capital_gains` is positive and sums its positive part, so the
+count is col 37 and the sum is col 38. The Build P release met col 37 at
+12,391,446 against 12,392,020.
+
+No released record has negative capital gains, so the model has no Schedule D
+loss returns. An N01000 count therefore has no model counterpart at any
+level.
+
+## How the targets combine
+
+- **Controls.** Only a family registered with the model's concept
+  (`_SOI_CAPITAL_GAINS_FAMILY_CONCEPTS`, today Table 1.4 alone) can supply the
+  national level, and the latest national all-AGI fact not after the build
+  period wins. HT2 and congressional-district rows never do, whatever their
+  period stamp. Two different records at the winning period raise
+  `AmbiguousSoiCapitalGainsControlError`, so feed order never decides.
+- **Historic Table 2 rows** enter only as shares. Each state row is its share
+  of the HT2 US total, scaled to the control, landed at the control's period,
+  and tagged `soi_source_concept = form_1040_line_7_net_gain_or_loss` and
+  `soi_control_concept = schedule_d_taxable_net_gain`. A row with no control
+  at or after its own period is dropped, because a line-7 level would be a
+  2.36x mismatch. The HT2 national all-AGI rows retire because the control's
+  own row owns the national concept.
+- **Congressional-district rows** are not controls. Their own targets, which
+  are on the `full` surface only, are not reconciled here.
+
+The share step assumes that each state's share of line-7 returns equals its
+share of Schedule D gain returns. No IRS state product publishes a gain-only
+count, so this cannot be checked directly. AGI composition is one measurable
+source of difference: weighting each state's HT2 N01000 by AGI class with
+Table 1.4's gain share per class would move TY2022 state targets by −2.9% (WV)
+to +4.3% (DC), with a median absolute change of 1.6% and no state beyond 5%.
+
+## How the returns control drifted
+
+The rebase used to keep the first fact among equal-period candidates. The
+July pin listed Table 1.4 ty2023 first, so Build P rebased the 51 state
+return counts onto 12,392,020 (factor 0.406751165649407; states summing to
+12,289,836). The September re-pin (`consumer_facts_us_c5e5bf8`) sorts rows by
+`aggregate_fact_key`, a content hash. There, the congressional-district US row
+sorts first; it carries TY2022 data stamped ty2023 (PolicyEngine/chronicle#117).
+It became the control (factor 0.97964), and the states summed to 29,599,604,
+2.39x the national target of the same model quantity on the same surface. The
+amount control happened to stay on Table 1.4, but reversing the feed would
+have moved it to the congressional-district row (factor 0.92455 instead of
+0.77190).
+
+## Effect on the pinned feed
+
+Compiled at 2024 with target aging and the packaged congressional-district
+crosswalk, then narrowed to `national_state`:
+
+| Targets | Before | After |
+|---|---:|---:|
+| 51 HT2 state return counts, sum | 29,599,604 | 12,289,836 |
+| California | 3,774,052 | 1,566,997 |
+| Texas | 2,108,705 | 875,540 |
+| Florida | 2,093,099 | 869,060 |
+| New York | 1,974,435 | 819,791 |
+| National Table 1.4 return count (unchanged) | 12,392,020 | 12,392,020 |
+| `national_state` registry | `d315c75804ef` | `65e4dde11c83` |
+
+- All 60 HT2 return-count specs move by the same factor, 0.415202720927. That
+  is the 51 state rows plus 9 congressional-district proxies, which are on
+  `full` only.
+- The 60 matching amount specs keep their values and gain only the two
+  concept keys.
+- No spec enters or leaves either surface, which keeps 32,842 compiled and
+  5,694 on `national_state`.
+- The state return counts are back at the Build P values; California's
+  1,566,996.66 equals the July scorecard's.
+
+`test_pinned_feed_no_soi_state_family_sums_past_its_national_target` holds
+the invariant across all 63 SOI state families on the surface.

packages/microcosm-build/src/microcosm/build/us_runtime/fiscal_targets.py

@@ -45,6 +45,7 @@
 from microcosm.calibrate.geography_constants import US_STATE_FIPS_TO_POSTAL
 
 __all__ = [
+    "AmbiguousSoiCapitalGainsControlError",
     "US_FISCAL_MACRO_REALISM_BANDS",
     "US_FISCAL_TARGET_REGISTRY",
     "US_FISCAL_TARGET_SPECS",
@@ -1471,13 +1472,64 @@ class _SoiTotalControl:
     period_key: tuple[int, int, str]
 
 
+#: What an SOI record-set family's ``net_capital_gains_*`` columns count, read
+#: from the published IRS workbooks and documentation guides, TY2020-TY2023
+#: (microcosm#1035):
+#:
+#: - Table 1.4 cols 37/38, "Sales of capital assets reported on Form 1040,
+#:   Schedule D: Taxable net gain", cover Schedule D returns that net to a
+#:   gain. Returns reporting only capital gain distributions on Form 1040
+#:   (col 35) and Schedule D loss returns (col 39) are outside them: Table 1.3
+#:   row 18 "Sales of capital assets net gain" equals col 35 + col 37 exactly.
+#: - Historic Table 2 and the congressional-district file carry N01000/A01000,
+#:   which both documentation guides define as "Number of returns with net
+#:   capital gain (less loss)" / "Net capital gain (less loss) amount" at Form
+#:   1040 line 7: a Schedule D gain, a loss-limited Schedule D loss, or
+#:   distributions only. HT2 US N01000 equals Table 1.4 col 35 + col 37 +
+#:   col 39 within 0.25% in each of TY2020-TY2023, and is 2.36x col 37 in
+#:   TY2022. Its amount, net of losses, is 1.4% below col 38 in TY2022.
+#:
+#: A family absent from this register has no reviewed concept and is never a
+#: capital-gains control.
+_SOI_SCHEDULE_D_TAXABLE_NET_GAIN = "schedule_d_taxable_net_gain"
+_SOI_FORM_1040_LINE_7_NET_GAIN_OR_LOSS = "form_1040_line_7_net_gain_or_loss"
+_SOI_CAPITAL_GAINS_FAMILY_CONCEPTS: dict[str, str] = {
+    "table_1_4": _SOI_SCHEDULE_D_TAXABLE_NET_GAIN,
+    "historic_table_2": _SOI_FORM_1040_LINE_7_NET_GAIN_OR_LOSS,
+    "congressional_district": _SOI_FORM_1040_LINE_7_NET_GAIN_OR_LOSS,
+}
+#: The concept ``capital_gains_gross`` measures. It is the positive part of
+#: PE-US ``capital_gains`` (short-term plus long-term, i.e. Schedule D, before
+#: the loss limit); distributions reported without Schedule D are the separate
+#: ``non_sch_d_capital_gains``. Its indicator therefore counts Table 1.4 col 37
+#: returns and its sum is col 38; a line-7 count, which also counts losses and
+#: distributions-only returns, is a different population. (PE-US allows
+#: negative gains, but the Build P release data carries no tax unit with a
+#: net loss, so it could not reach a line-7 count at any level.)
+_SOI_CAPITAL_GAINS_MODEL_CONCEPT = _SOI_SCHEDULE_D_TAXABLE_NET_GAIN
+
+
+class AmbiguousSoiCapitalGainsControlError(ValueError):
+    """Two different concept-matched records tie for one capital-gains control."""
+
+
 def _rebase_stale_soi_capital_gains_distributions(
     registry: TargetRegistry,
     facts: tuple[object, ...],
     *,
     target_period: int | str,
 ) -> TargetRegistry:
-    """Use stale SOI capital-gains rows as shares, not hard old-year totals."""
+    """Use Historic Table 2 capital-gains rows as shares of a Table 1.4 level.
+
+    HT2 rows count Form 1040 line 7 (gain or loss), a population
+    ``capital_gains_gross`` cannot measure, so they never ship as levels. Each
+    one ships as its share of the HT2 national total, scaled to the
+    concept-matched control of ``_soi_capital_gains_active_totals`` at or after
+    its own period, and declares the bridge in ``soi_source_concept`` /
+    ``soi_control_concept``. Without such a control, the row is dropped. The
+    national all-AGI HT2 rows retire because the control's own row owns the
+    national concept.
+    """
 
     controls = _soi_capital_gains_active_totals(
         facts,
@@ -1494,15 +1546,11 @@ def _rebase_stale_soi_capital_gains_distributions(
         key = _soi_capital_gains_control_key_from_spec(spec)
         control = controls.get(key)
         source_total = stale_national_totals.get((*key, spec.metadata["source_period"]))
-        if control is None:
-            specs.append(spec)
-            continue
-        if source_total in (None, 0):
-            continue
-        if not _period_not_before(
+        if control is None or not _period_not_before(
             control.period_key, _period_key_from_value(spec.metadata["source_period"])
         ):
-            specs.append(spec)
+            continue
+        if source_total in (None, 0):
             continue
 
         if _is_national_all_agi_spec(spec):
@@ -1525,6 +1573,8 @@ def _rebase_stale_soi_capital_gains_distributions(
                     "uprating_index_source_record_id": control.source_record_id,
                     "uprating_factor": _format_float(factor),
                     "stale_distribution_rebased_to_active_total": "true",
+                    "soi_source_concept": _SOI_FORM_1040_LINE_7_NET_GAIN_OR_LOSS,
+                    "soi_control_concept": _SOI_CAPITAL_GAINS_MODEL_CONCEPT,
                 },
             )
         )
@@ -1536,13 +1586,26 @@ def _soi_capital_gains_active_totals(
     *,
     target_period: int | str,
 ) -> dict[tuple[str, str, str], _SoiTotalControl]:
+    """The capital-gains control per (measure, filing status, universe).
+
+    Only a national full-AGI fact whose family counts what
+    ``capital_gains_gross`` measures qualifies, so HT2 and
+    congressional-district rows never do, whatever their period stamp. The
+    latest qualifying period not after the build period wins, and the choice
+    depends on the fact set alone: two different candidates at the winning
+    period (different records, or one record with different values or period
+    labels) refuse the compile. Feed order used to break that tie, and the
+    September re-pin, which re-sorted the feed by content hash, handed the
+    returns control to the congressional-district US row (microcosm#1035).
+    """
     controls: dict[tuple[str, str, str], _SoiTotalControl] = {}
+    tied: dict[tuple[str, str, str], set[tuple[str, float, str]]] = {}
     target_period_key = _period_key_from_value(target_period)
     for fact in facts:
         key = _soi_capital_gains_control_key_from_fact(fact)
         if key is None:
             continue
-        if _is_stale_soi_historic_capital_gains_fact(fact):
+        if _soi_capital_gains_concept(fact) != _SOI_CAPITAL_GAINS_MODEL_CONCEPT:
             continue
         period_key = _period_key(fact)
         if not _not_after_target_period(period_key, target_period_key):
@@ -1556,16 +1619,55 @@ def _soi_capital_gains_active_totals(
             source_record_id=source_record_id,
             period_key=period_key,
         )
+        identity = (source_record_id, candidate.value, candidate.source_period)
         current = controls.get(key)
-        if current is None or _prefer_candidate(
-            candidate.period_key,
-            current.period_key,
-            target_period_key=target_period_key,
-        ):
+        if current is not None and period_key[:2] == current.period_key[:2]:
+            tied[key].add(identity)
+            continue
+        if current is None or period_key[:2] > current.period_key[:2]:
             controls[key] = candidate
+            tied[key] = {identity}
+    ambiguous = {
+        key: sorted(candidates)
+        for key, candidates in tied.items()
+        if len(candidates) > 1
+    }
+    if ambiguous:
+        raise AmbiguousSoiCapitalGainsControlError(
+            "SOI capital-gains control is ambiguous: different candidates tie at "
+            f"the latest period for {ambiguous}. Deduplicate them upstream; "
+            "feed order must not pick a rebase control (microcosm#1035)."
+        )
     return controls
 
 
+def _soi_record_set_family(record_set_id: str) -> str:
+    """An SOI record set's table family, without period or data vintage.
+
+    ``irs_soi.ty2023.table_1_4`` is ``table_1_4``,
+    ``irs_soi.ty2022.historic_table_2.state_broad`` is ``historic_table_2`` and
+    ``irs_soi.ty2023.congressional_district_2022.all_returns`` is
+    ``congressional_district``.
+    """
+    parts = record_set_id.split(".")
+    if parts[0] != "irs_soi":
+        return ""
+    rest = parts[2:] if len(parts) > 1 and _is_period_token(parts[1]) else parts[1:]
+    if not rest:
+        return ""
+    family = rest[0]
+    if family.startswith("congressional_district"):
+        return "congressional_district"
+    return family
+
+
+def _soi_capital_gains_concept(fact: object) -> str | None:
+    """The reviewed IRS concept of a capital-gains fact's family, if any."""
+    return _SOI_CAPITAL_GAINS_FAMILY_CONCEPTS.get(
+        _soi_record_set_family(_str_at(fact, "layout", "record_set_id"))
+    )
+
+
 def _soi_capital_gains_stale_national_totals(
     facts: tuple[object, ...],
 ) -> dict[tuple[str, str, str, str], float]: