Repository navigation
Retire patch logic — ShapePipe v2 has no patches #340
Description
Activity
@martinkilbinger on second thought, do we need to keep the patch logic so that older catalogs can still be read/analyzed?
and if we need to keep some backwards compatibility, how much? #343 purports to still be able to read the old layout, but also deletes some patch-based scripts...
- added 2 commits that reference this issue
on Sep 28, 2026 Answer as implemented in #343: reading old catalogues needs no patch logic. The released v1 merged catalogues are flat files (
data/data_ext, with an int8patchcolumn that just passes through), and the reader plusgrammar.adaptopen them directly. So #343 keeps only the reader and adapter for v1, and everything else speaks campaigns:JointCatmerges campaign files, and the patch-only scripts are gone (combine_results.py,compute_area.pyand the five listed above). The one thing lost is re-merging v1 from per-patch FITS intermediates, which nothing needs while the merged releases exist.One follow-up stays ShapePipe-side:
create_final_cat.pystill wraps tiles inpatches/<campaign>/. The reader walks through it, so nothing here depends on it; dropping it wants its own small ShapePipe PR at a campaign boundary.— Claude (Opus) on behalf of Cail
Reading and analysing older catalogues doesn't need the patch logic. Only rebuilding them from patch outputs does.
- Every released v1 product is one survey-wide file: the comprehensive HDF5, the per-version
cut_strucFITS, and the PSF/star FITS. Patches appear only in the inputs that build them, such asP*/sp_output/shape_catalog_comprehensive_ngmix.fitsandP*/sp_output/psf_catalog_ngmix.fits. - On develop, the patch logic is therefore build-only. It lives in
extract_infoon per-patchfinal_cat, inJointCat.get_patches/merge_catalogues(behindcreate_joint_comprehensive_cat.py), in thev1/v1.5/v1.6shortcuts ofmerge_psf_cat.py, and in the per-patch diagnostic scripts. - After Read ShapePipe v1 and v2 products side by side #343, v1 analysis still works through the grammar adapter. Recalibrating from the v1.4 comprehensive reproduces released v1.4.6.3 bit-identically on a 1M-row window, and ρ/τ on SP_v1.4.6.3 matches the earlier run.
- What Read ShapePipe v1 and v2 products side by side #343 removes is rebuilding a v1.x comprehensive catalogue from patch outputs in this repo.
extract_infono longer reads per-patchfinal_cat,create_joint_comprehensive_cat.pymerges campaign files, and the patch-only scripts are deleted.merge_psf_cat.pystill accepts an explicit-p P1+…+P9, but writes acampaigncolumn instead ofpatch.
So #343 can go in as it is. Anyone who needs to rebuild a v1.x catalogue from patch outputs can use a checkout from before #343.
— Claude Opus 5.5 on behalf of Cail
- Every released v1 product is one survey-wide file: the comprehensive HDF5, the per-version
- added a commit that references this issue
on Oct 2, 2026
ShapePipe's Snakemake workflow (CosmoStat/shapepipe#852) processes a campaign — a tile list — with no intermediate "patch" (P1…P7) unit. Its merged products (CosmoStat/shapepipe#879) are one
final_cat_<campaign>.hdf5and onefull_starcat-0000000.fitsper campaign. sp_validation still carries the patch layer throughout, and the workflow currently has to write a legacypatches/<campaign>/<tile-ID>group into the hdf5 purely socatalog.py's reader opens it.Where patch assumptions live today (rough grep, jackknife
npatchexcluded):src/sp_validation/catalog.py— hdf5 reader hardcodespatches/{name}/<ID>src/sp_validation/catalog_builders.py—get_patches()(v1/v1.5→P1..P7/8), patch→surveymerge_cataloguesover<patch>/sp_output/shape_catalog_comprehensive_*.fitssrc/sp_validation/survey.py— patch boundary coordinatessrc/sp_validation/rho_tau.py,cosmo_val/real_space.py— patch-keyed paths/labelscombine_results.py,merge_psf_cat.py,prepare_patch_for_spval.sh,plot_rho_stats_patches.py,survey_stats_all.sh,stats_tile_id_gal_counts.py,star_match_stats.py,check_tile_IDs_SP_LF.py,compute_area.pyProposed end state: sp_validation takes a campaign's two merged files directly (hdf5 with a flat per-tile layout, or a single top-level group; star cat as-is), and any cross-campaign combination is expressed as "list of campaigns", not "list of patches". Scripts that only ever made sense for the P1–P7 era get deleted rather than ported. Once the reader is flat, the
patches/key in ShapePipe'sfinal_cat_mergegoes too.— Claude (Fable) on behalf of Cail