Chi-square design-weight penalty and softmax mass parametrization for calibration; ACS local ESS frontier

[MaxGhenis]

Why

On 2026-09-28 David Trimmer measured concentrated weights in the ACS local release:

• national Kish ESS about 14k of 1.59M;
• Massachusetts 477, with half its weight on 152 records;
• district ESS 41–64.

He asked whether to anchor calibration to the design weights. This PR adds that anchor and tests it at full scale. The full write-up is experiments/us-acs-local-l2-basis-20260928/README.md; the method is in docs/calibration-l2-basis.md.

What the evidence shows:

1. Most of the concentration is in the starting weights. The staging's --acs-share 0.5 gives the 57,240 donor rows half the mass. Before calibration, national ESS is 36,288 and Massachusetts 1,020. Calibration takes ESS to 13,646.
2. The old l2_lambda knob backfires. Its record-weighted form pulls toward w ∝ d², and at λ = 0.1 it lowers ESS to 8,689.
3. The new chi-square basis works. It is GREG's distance to the design weights, which it pulls toward directly. At λ = 0.03 with the default projection parametrization (on today's equal-weight loss), against the reproduced release:
   • national ESS goes from 13,646 to 20,690;
   • Massachusetts from 448 to 589;
   • districts below ESS 50 from 352 to 278, and the smallest district from 12 to 22;
   • training targets within 10% from 97.6% to 97.3%;
   • on two rotated holdout folds, held-out fit is ahead on both measures: capped error 0.1247/0.1191 against 0.1296/0.1236, and share within 10% 64.2%/67.0% against 63.3%/65.8%. Held-out run-to-run variation was not measured and λ was picked on those folds, so read this as "no worse, probably slightly better".
4. The ACS share is the bigger lever, at an out-of-sample cost. At share 0.9: ESS 93,524, no district below 50, and about 40% more held-out error, mostly on SOI targets. That and the default λ are queued for Max, not changed here.
5. A release gate. No district below a quarter of its starting ESS separates cleanly at the current seeding: the release fails it in 15-24 districts across its three solves, and every penalized solve passes. A state floor near 200 sits inside the solves' spread across folds (177-216), so none is proposed.
6. The softmax mass parametrization is opt-in, and projection stays recommended. Under mass="conserve", the historical projection solve can stall on small problems where every target pulls the same way: it is up to 0.34 off the exact optimum, against softmax's 0.0034. At the release's scale, gradients are near Adam's eps with mixed signs, so the stall does not occur. The two parametrizations agree for λ ≤ 0.03, and projection falls inside softmax's training frontier at λ ≥ 0.1. Softmax's per-step cap loop runs out of its 32 rounds on most epochs at that scale (counted in the receipt; the closing projection makes the returned weights exact). The doc states this as a known limitation.

What changes

microcosm-calibrate gets two opt-in options. The defaults are byte-identical to bda72cb.

• l2_basis:
  • "record" (default) is the historical mean((w/d)**2).
  • "chi_square" is sum(d*(w/d-1)**2)/sum(d).
  • Threaded through calibrate, the L0 budget search, refit_l0_selection, calibrate_l0_refit (refit_l2_basis) and static_aging.
• mass_parametrization:
  • "projection" (default) is the historical scheme.
  • "softmax" is w = total*softmax(log_w), for Adam with mass="conserve" and no L0 gates. calibrate_l0_refit takes refit_mass_parametrization.
  • Softmax epochs whose cap rounds run out are counted in the iterate-selection receipt.
• CalibrationResult.chi_square_distance (also on L0RefitResult) and chi_square_distance(weights, anchor).

tools/build_us_acs_local_release.py. The changes are flags on top of #1053; other open PRs edit this tool too.

• New flags --l2-basis and --mass-parametrization, with the defaults unchanged.
• Both join _solver_settings, so resume and the already-complete shortcut cover them. A stamp written before they existed reads as the historical solve.
• The summary and the build manifest record them with the realized chi-square distance; a legacy summary reads as the historical solve.
• The ESS limitation keeps the "certified default" text only for the historical solve, and fills the historical basis for a legacy summary.
• The manifest's refresh recipe always names --l2-lambda, --l2-basis and --mass-parametrization at the recorded values, so a later default change cannot move it.
• The summary and manifest record softmax_cap_rounds_exhausted_epochs.

Evidence lives in experiments/us-acs-local-l2-basis-20260928/:

• the harness, and the CSR checkpoint verified against the dense one;
• the Modal grid of 57 solves, gated on reproducing the release (loss 0.015485 vs 0.015491, ESS 13,646 vs 13,631), run on three kernel heads whose solve.py differs only in docstrings, a receipt counter and guards;
• two reruns measuring run-to-run variation (training metrics only). They are not byte-identical to their originals: trajectories differ from epoch 0 or 1, though solve.py's module hash is the same at 9ef71ed, bc763cb and d82ff85 (it differs from 35ad665, where the originals ran, only by docstrings and the counter). Over 800 epochs the release's final loss moved 2.7% (0.3% at λ = 0.03), while every concentration measure moved under 1%;
• a receipt for the published weights' own concentration (results/published_weights.json);
• the seeding options, the gradient scale, and the CLARABEL references;
• the frontier tables and chart.

CI-pinned identities this PR moves

The US spec-engine seed protocol attests the source bytes of microcosm.calibrate.solve, so editing solve.py re-pins EXPECTED_HASHES["seed_protocol"] and ["seed_map"], the US spec_sha256 in test_us_multispine_pool_tool.py, and docs/evidence/spec-engine/us-f0-coverage.json (regenerated; --check passes 41/41). Each new value was computed on this tree (final values: seed_protocol be39eb6b…, seed_map 085d8d39…, US spec_sha256 e0b757ce…).

Invariants

These are property-tested in packages/microcosm-calibrate/tests/engine_free/shared/test_l2_basis.py.

1. The chi-square penalty is exactly 0 at w = anchor and nonnegative everywhere (Hypothesis).
2. Differential: the float32 torch penalty equals the float64 chi_square_distance, and the record penalty equals mean((w/d)**2) (Hypothesis).
3. When sum(w) = sum(d), the distance equals sum(w**2/d)/sum(d) - 1. With uniform d it equals n/ESS - 1 (Hypothesis).
4. Under a mass constraint, the reduced gradient vanishes at w = d for chi-square and at w ∝ d**2 for the record basis (Hypothesis).
5. With no targets and mass="conserve", the chi-square solve returns the design weights within 1% from any warm start, under both parametrizations (Hypothesis).
6. Softmax conserves the total to 1e-12 relative, respects the cap and keeps weights positive (Hypothesis).
7. Differential against an exact solver, on the tests' own problems (fixture plus generator):
   • each softmax-conserve and free-mass solve lands within 0.006 of CLARABEL's optimum;
   • the bound is twice the measured maximum of 0.0026;
   • the regularization-path identity holds with the slack 2·eps/Δλ derived from that bound, on informative λ pairs only;
   • with uniform design weights, ESS = n/(1+P) holds exactly.
8. Defaults reproduce the pre-change optimizer byte for byte across 7 configurations, with the defaults implicit and spelled out. This was mutation-tested: perturbing the record penalty by 1e-6, or reusing the loss path's exp node, fails it.

Intended violation, pinned as such: under "projection", invariant 7 does not hold in general. test_projection_parametrization_stalls_under_uniform_mass_pressure shows the stall when every target sits above its design total.

Review

Independent Opus 5.5 reviews, each through subfleet run --task review:

• Round 1 requested changes (one high, four medium). Fixed: truthful limitation provenance for an unpenalized softmax solve; the missing README; the stall claim qualified by measured gradient scale; test bounds derived from an exact reference on the tests' own problems; the cap-loop docs corrected and exhausted rounds counted.
• Round 2 died on a lane error after partial findings; those were fixed (no dominance claim, the prior as a limit not a bound, the λ = 1 district count, the resume text).
• Round 5 approved the round-4 fixes (delta review; six cosmetic nits, fixed).
• Round 4 approved (no blocker, high or medium). Its lows are fixed: the recipe names every penalty flag; the held-out claim no longer leans on training-loss noise; noise-sized differences are no longer used as arguments; a receipt for the published Massachusetts figure; floor ranges over every solve; the softmax limitation surfaced in calibrate()'s docstring, the CLI help and the summary.
• Round 3 requested changes (four medium, seven low), all in the write-up or the tool's evidence. Fixed: the cap-loop exhaustion stated as a known limitation, with projection recommended; the frontier claim limited to λ ≤ 0.03; both held-out metrics reported, with fold selection disclosed; the gate led by the relative floor, the state floor dropped; the kernel heads listed and checked; the legacy basis in the limitation text; the refresh recipe's penalty flags; a receipt for the Massachusetts split; labels and nits.

axiom: n/a: calibration kernel and ACS local tool; no policy rules change.

🤖 Generated with Claude Code

[MaxGhenis]

Merge audit (head b7b4c29):

• gh pr checks 1078 exits 0 on run 37001587847 (all 10 jobs pass); mergeable MERGEABLE / CLEAN; no CHANGES_REQUESTED.
• Independent review: Opus 5.5 rounds 4 and 5 via subfleet run --task review --tier standard approved (reports in _build_artifacts/acs-local-l2-basis-20260928/review/review-r4.md, review-r5.md). Every finding and nit from rounds 1-5 is fixed in the branch.
• Changes after round 5's approval are mechanical re-pins only: the H1 calibrate parity pin (b115745, tools/graph_parity_repin.py calibrate, which refuses if outputs move) and the spec-engine loader golden vector (b7b4c29). Both move because solve.py's source bytes are attested; numerics are byte-identical to bda72cb under the defaults (pinned by the frozen-oracle test).
• Main has moved since the merge base, but not in any seed-kernel module or pinned spec-engine file, so the pins hold on the merge ref.
• No default changes. Adopting the chi-square penalty (λ 0.03, projection) and the 25%-of-starting-ESS district gate is queued for Max as d797 (supersedes d792); the seeding question is d793.