Repository navigation
Validate the 330-tile v2 campaign (smk-p3a) and diff its shapes against v1.4.6 #405
Description
Activity
Current ShapePipe v2 products on Nibi now have one stable path:
/project/def-mjhudson/cdaley/sp-products/latest/(groupdef-mjhudson).These are the production batches from 10 Oct. They supersede smk-p3a, which ran on September code.
What Path under latest/Galaxy catalogue, all tiles final_cat_latest.hdf5: one structured dataset per tile underpatches/latest/<tile>, all rows, no cuts appliedPSF star catalogue (star vs PSFEx model, HSM) full_starcat_latest.hdf5: one dataset per exposureExposure-count and flagged-pixel maps nexp_latest.hsp,nflagged_latest.hsp(HealSparse)Per-tile catalogues; per-exposure PSFEx models ( psf/<exp>.tar)tiles/<shard>/<tile>/,exp/<shard>/<exp>/, as links into the source batchWhat's in it MANIFEST.md/MANIFEST.json: batches, code sha, counts, column conventionsCurrently: 125 tiles, 156 exposures, 5,015,937 objects (batches smk-b01–b05, code
255d4392).The path stays the same as new waves land. Each finished batch is added and the manifest updated; where a tile appears in more than one batch, the newer one is used.
N_EPOCHcounts only epochs with a validated PSF model (#952). The wave-1 files were migrated in place to that definition, and the manifest states the convention per tile.Two things persist beyond the catalogues: the PSFEx models and the PSF star catalogue. For anything else you'd want kept (SExtractor tile catalogues, star selections, …), say so here: what gets kept is a config choice, and we'd rather keep what people use.
Claude (Opus) on behalf of Cail.
The first large ShapePipe v2 campaign, smk-p3a, finished on Nibi on 1 Oct. It covers 330 contiguous tiles (about 80 deg²) with 12.5M objects, 4.47M of them in the science selection, and passes ShapePipe's standing checks. sp_validation hasn't been run on it yet.
/project/def-mjhudson/cdaley/sp-products/smk-p3aon Nibi (path inferred from the earlier campaigns; Cail can confirm).sp_catcompare.py --match sky), since v1 and v2 number objects differently.p3a predates #922/#925 (defects, UberSeg) and #934 (PSF star box). Read the blend and PSF results with that in mind.
Claude Opus 5.5 on behalf of Cail. From the 3x2pt telecon, 6 Oct 2026.