Merge develop into the tomography branch

[cailmdaley]

This merges develop into the tomography branch, so that the later merge of the tomography branch into develop (step 4 of #375) becomes a review rather than a rebuild.
The result keeps the tomography branch's API and takes develop's I/O.
Every per-version result stays keyed by bin pair, with non-tomographic results as the ("all","all") pair.
Catalogues are read through develop's readers, and the ("all","all") pair is written through develop's SACC writers.

Closes #374. Part of #375.

How to review

GitHub's diff against the tomography branch is about 30k lines, and almost all of it is develop's own PRs, each reviewed when it merged. Two smaller surfaces matter:

• What this PR adds to either side (about 5.7k lines): the conflict resolutions, git show --remerge-diff 0d5f2ad1 (2.1k lines) and git show --remerge-diff 8631f608 (0.7k), and the 30 commits after the first merge (git log -p --no-merges 0d5f2ad1..).
• What develop takes on when the tomography branch merges into it: git diff develop...merge/develop-into-tomo. That is 60 files, +9.0k/−2.2k, of which 1k is n(z) tables. It is the tomographic science, and the part where review moves the science most.

Structure

The first commit, 0d5f2ad, is a true merge of develop (8db2aa5) and only resolves conflicts.
Each later change is its own commit:

Commits	What
7781c87 … b9a9e9b	Fixes for breakages that the merge left without conflict markers: the workflow scripts on the new API, pol_factor: -1, the B-mode summary and SACC n(z) on the bin-pair shapes, the tests, and this branch's own callers of reshaped helpers
8631f60	Second merge of this branch, after #307 (tomographic leakage). uv.lock re-resolved: shear-psf-leakage develop f1a2c071 (includes CosmoStat/shear_psf_leakage#48), glass 2026.2
45c51e5	Port of #366: pw² only when cell_method == "map", in both the SACC bandpower window and the iNKA fiducial
e6406e4	Port of #367: seeded jackknife patch centres, computed once per version and shared by every bin pair and by both catalogues of a cross pair
f7f2abb, 551f5c9, 465ae28, e3bd484	Review fixes: jackknife ρ/τ draws read under the name the library writes; the catalogue pseudo-Cℓ wrapper defaults to ("all","all"); the object-wise leakage reader accepts #48's row selection; Hartlap debiasing of simulation ρ/τ covariances restored
88fa858, d6356c9	ρ/τ paths in the workflow rules built from one helper; covariance rule robust to NFS cleanup
454e802, a769e10	τ plot: no cov_type=None in the covariance file name; error bars from the configured τ covariance
03bd3ad … 114486c	The audit's merge checklist (#374 comment): its regression tests brought in as passing tests, and the fixes they needed
a6e0765, 5c25e3e, 5d3b5d2, 7b34411	From a review of the merge surface: each ρ/τ jackknife draw (cov_estimate_method='jk') gets its own seeded layout instead of all repeating seed 0; the (all,all) SACC readback provides BE (= EB); tomographic COSEBIs refuse a single covariance file for every bin pair

Choices worth a look

• Catalogue-path covariance without pw². The workflow measures pseudo-Cℓ from catalogues (NmtFieldCatalog), which have no pixels. This branch already dropped pw² from the iNKA fiducial, and develop applied it. pw² now applies only on the map path.
• Noise below the first band. The analytic iNKA noise below ℓ_min continues at the lowest band's value, as develop had it, because white shape noise is flat (3367b2b, agreed with Sacha in review).
• The iNKA fiducial stays on CAMB. CAMB differs from develop's CCL by about −4% at ℓ = 1000, mostly through the non-linear model. cs_util's CCL→CAMB conversion dropped m_nu; fixed in Preserve CCL neutrino masses and species in CAMB predictions cs_util#97.
• File names. The non-tomographic ρ/τ products and the ξ± dump carry _tomo_bin_all, as basename() builds them. Existing products under the old names are recomputed on the next run.
• No tomographic SACC writers. Tomographic pairs keep this branch's FITS outputs; multi-tracer SACC is Tomographic SACC writers: one tracer and n(z) per bin #376.

Verification

• The audit's merge checklist (Merge develop into the tomography branch #374, tests from assay/decision-record, epic Scientific audit of sp_validation: decision record, regression tests, defects #377) passes: 19 tests pass, and 2 are strict xfail for per-bin τ centering, whose fix belongs upstream (Centre galaxy ellipticities within the selected tomographic bin shear_psf_leakage#52). Two checklist items needed code changes:

  • the iNKA covariance merge filled its lower cross-polarisation blocks from the wrong HDU (0250493; the merged 36×36 fixture is symmetric and positive definite);
  • plot_cosebis ignored compute_tomography (a71e5fa).

  The other items were already right on the merged branch, and their tests now hold them there.

• Full container test suite, slow tests included: 516 pass and 1 fails. The failure is a missing configured data path, and it also fails on a769e10. The workflow DAG tests pass, and CI builds the image and runs the unit tests in it.

• Tomographic ρ/τ rerun on the GLASS mock (the one catalogue with bins): each bin's galaxy mask selects exactly its TOM_BIN_ID rows, and the τ theory covariance is computed per bin from the masked catalogue.

Parity with develop

Both branches ran the full papers/cosmo_val suite on SP_v1.4.6.3 and its _leak_corr variant in the same image: develop 8db2aa5, and this branch d6356c9. To compare, I recomputed and diffed every product.

Identical to float precision:

• pseudo-Cℓ EE/EB/BB (≤1e-14 σ);
• ξ± on the npatch=1 grid (≤1e-10);
• COSEBIs (≤3e-9);
• pure-E/B means and covariance;
• the additive bias;
• the object-wise leakage fit.

ρ/τ differ by up to 0.8σ. This comes from #295's star mask, which drops the 947 of 6,554,897 stars with nonzero HSM flags. Switching the mask off reproduces develop to 1e-12. The flagged stars look like failed fits (e ≈ 0, T ≈ 0.07 against a median of 0.18), so the mask stays. Downstream, the SP_v1.4.6.3 fit moves α 0.029→0.033 (σ 0.011) and η 0.93→1.52 (σ 1.36).

Deliberate changes:

• Seeded patches (Seed the jackknife k-means and drop the patch-centre file #367). The ξ± jackknife covariance diagonal changes by 0.72–1.35×. The npatch=100 means also move, by up to 0.84σ: TreeCorr's tree approximation depends on the patch layout. With develop's patch centres this branch reproduces develop exactly, so the shift comes from the layout, not from this merge. A fix that makes the published means independent of the layout is proposed separately.
  
• iNKA covariance (pw² on the catalogue path, ℓ↔index alignment, CAMB fiducial). Each step switched on separately, against the earlier e2e reference (the curves multiply to the total). The leftover difference, up to 7% in EE near ℓ≈70, means the reference also differs elsewhere; its provenance is unknown. Develop has no covariance of its own to compare (Fix one-multipole shift in the pseudo-Cℓ Gaussian covariance theory #372).
  
• α from τ₀₊/ρ₀₊ (Update scale-dependent and objectwise leakage computations to tomography #307). For SP_v1.4.6.3, 0.020→0.021. For leak_corr, 0.004→0.009. Develop's ξ^gp/ξ^pp uses the PSF model at galaxy positions, against which the leakage correction was fitted. τ/ρ use the stars. My reading is that this explains the leak_corr gap; I haven't tested it.
  

Tomographic ρ/τ (GLASS mock): τ₀₊ per bin, with and without the bin mask.

Develop's own suite fails in three places, independent of this merge: the pseudo-Cℓ covariance rule (the dict bug of closed #372), COSEBIs and pure-E/B refusing the unblinded ξ± part the same workflow writes, and the τ plot (CosmoStat/shear_psf_leakage#46).

— Claude Opus 5.5 on behalf of Cail

🤖 Generated with Claude Code

[LisaGoh]

Thank you for this! The table in #375 makes things very clear now. I will fix #306 and #256 once everything upstream is merged.

[sachaguer]

Looks good to me. I do not remember why I traded low ell noise bias equal to l_min for zeroes. I think there is no good reason. It does not have much impact I believe but we should go back to the constant noise bias