Skip to content

Separate correlation means from the jackknife patch layout - #399

Draft
cailmdaley wants to merge 9 commits into
feature/sp_validation-extend-to-tomographyfrom
fix/layout-independent-means
Draft

cailmdaley wants to merge 9 commits into
feature/sp_validation-extend-to-tomographyfrom
fix/layout-independent-means

Conversation

@cailmdaley

@cailmdaley cailmdaley commented Oct 5, 2026 •

Copy link
Copy Markdown
Collaborator

Full-sample TreeCorr correlation means depend on the jackknife patch layout; seeded patches make a layout reproducible but do not remove that dependence.

  • correlation.measure_with_patches runs the published-means pass without patches using bin_slop=min(1, b_target/bin_size). The patched pass omits bin_slop, so TreeCorr uses its standard default for covariance and resampling.
  • ξ±, M_ap² and measured ρ/τ share this mechanism, including cross-catalogue and tomographic pairs; no shear_psf_leakage changes. In Derive reporting xi means from an exactly nested fine grid #400, the fine-grid means pass retains bin_slop=1; angle_slop stays at its default.
  • Cache sidecars record both configurations, so products from the former shared-tolerance policy do not match. Plot consumers reuse full-precision columns without repeating covariance jobs.
  • Synthetic layout tests are bit-exact for published means on one thread while covariance estimates differ. Reverting to patched means fails the ξ± and M_ap² regressions. The SACC producer retains dense jackknife covariance when its text cache exists.
  • Full container suite at the stacked Derive reporting xi means from an exactly nested fine grid #400 head, slow tests included: 537 passed; three environment failures reproduce on T (seven missing configured paths and two NFS FITS-cleanup errors). Host workflow suite: 11 passed, 1 skipped.
  • SP_v1.4.6.3, 1–250′/20 bins/100 patches: layout-dependent mean difference falls from 0.322σ rms / 0.809σ max to 5.24×10⁻¹²σ rms / 2.58×10⁻¹¹σ max. Independent 12-thread runs are not bit-identical; the residual is consistent with floating-point reduction round-off.
  • In the stacked Derive reporting xi means from an exactly nested fine grid #400 reporting check, reusing the fine product, the default-tolerance covariance-only pass took 3.95 min (seeded) / 3.76 min (saved centres) at 8 threads, excluding catalogue loading and patch-centre construction.

Layout difference in ξ± means, before and after

Depends on #394 (merge/develop-into-tomo); #400 derives reporting ξ± from a nested fine grid. The effect of changing bin_slop on jackknife covariance remains open in #401.

— Luna on behalf of Cail

@cailmdaley
cailmdaley force-pushed the fix/layout-independent-means branch from 7f5b631 to 84ce9fd Compare October 5, 2026 04:28
@cailmdaley
cailmdaley changed the base branch from merge/develop-into-tomo to feature/sp_validation-extend-to-tomography October 5, 2026 12:57
cailmdaley and others added 9 commits October 5, 2026 15:01
A target b of 0.01 gives reporting measurements a common approximation policy while retaining bin_slop=1 on finer integration grids.

Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
Tree approximations depend on the patch layout. A shared two-pass measurement keeps the full-sample means canonical while retaining patch results for covariance, including tomographic cross-pairs and aperture-mass integration. Patched text dumps cannot supply a covariance and are not reused.

Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
…changes

RhoStat and TauStat expose catalogue mappings and output tables. Measure the identical unpartitioned catalogues once, retain patched variances and joint covariance, and reuse the means across covariance draws. Record measurement settings so cached products cannot silently supply layout-dependent means.

Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
…try points

Reporting, fine-grid, and rho/tau jobs must use the configured target rather than silently falling back to a constructor default.

Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
…atched covariances

Synthetic auto/cross correlations and aperture-mass and rho/tau products retain identical single-thread means while their jackknife covariances differ. A patched-means mutation fails the layout test; fine grids retain bin_slop=1.

Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
… covariance jobs

Measurement calls retain the patch results needed for dense and derived covariances. Plotting explicitly reads the published columns and variances, at round-trip precision, so one-core figure jobs do not repeat the expensive measurement. A pinned root depth makes cached products reusable across thread counts.

Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
A direct producer rerun must retain a full jackknife covariance and patch results; cached columns are not a substitute for covariance.

Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
…he layout regression

The unpatched pass has no shared jackknife centres. Verify that it occurs once, while all three covariance draws retain distinct shared layouts and fresh catalogues.

Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
Keep b_target on the published-means pass and let TreeCorr choose its standard bin_slop for patched covariance, where the tolerance does not set the reported means. Record both configurations so caches from the shared-tolerance policy are rejected.

Co-Authored-By: GPT-6 Luna <noreply@openai.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant