Repository navigation
Exposure-level star/defect masking strategy #839
Description
Activity
see also #662
Following the Slack consensus, here is a concrete proposal for how the external healsparse masks slot into ShapePipe — a PR description before any implementation, so it can be discussed and corrected first.
The key structural fact: everything downstream of the mask module consumes a single artifact — the per-tile / per-exposure-CCD pixel flag image (
pipeline_flag<num>.fits). SExtractor turns it intoIMAFLAGS_ISO; setools star selection, vignetmaker flag stamps, and ngmix pixel weights all key on that. Nothing downstream cares how the flag image was made, so the swap is local to the mask module:- New code path in
mask_package(or a small sibling module): read the healsparse map, look up the image pixel grid through the existing WCS machinery (Mask._set_image_coordinates), write the samepipeline_flag<num>.fits. Configured via a new[HEALSPARSE_PARAMETERS]section (map path, bit → flag mapping), following the existing sub-config section pattern. - The old path stays config-selectable. Before deleting anything we compare old vs new masks on real data (e.g. the Nibi P3 tiles): masked-area fractions,
IMAFLAGS_ISOpopulations, resulting PSF star selections, and image inspection. - PSF star selection updates for free — setools cuts on
IMAFLAGS_ISO == 0, so the new masks flow into star selection with no setools change. - Catalogue-level flags:
make_catgains a healsparse lookup at each object's (RA, Dec) — the existingadd_colpattern — so per-band mask bits can ride into the final catalogue directly (the ShapePipe end of UNIONS-WL/spherex#38). The rasterized mask already reaches the final catalogue viaIMAFLAGS_ISO. - Exposures: whichever way this issue's open question resolves (reproject tile masks vs run the masking per exposure), the artifact contract is unchanged — an exposure-CCD flag image. Reprojection is just the lookup in (1) evaluated with the exposure-CCD WCS.
- Dependencies:
healsparse+hpgeom, already declared on the coverage branch (Coverage nexp masks #797) — reuse. - Side benefit: removes WeightWatcher and the live Vizier/GSC query — the two flakiest external dependencies of the current module — from the critical path.
Currently blocked on pointers to the up-to-date maskforce healsparse products (or the code that generates them);
/arc/home/mhudson/masks/(2025) is the candidate, currency unconfirmed.— Fable, on behalf of Cail
- New code path in
Closing: #845's design settles this for the first runs — pixel-level masking per exposure comes from the instrument flags (defect map, #887), and star/galaxy/external masks are tile-level healsparse bits queried per object. #851 tests empirically whether the star radii are large enough; if it shows exposure-level star masking is needed, we reopen.
— Claude (Opus) on behalf of Cail
ShapePipe's masking is THELI-vintage and needs a rethink. The Nibi run found GSC star masking broken by a NaN bug (only ~1/2 of stars actually masked) and bright extended galaxies left unmasked (plots in #808).
Axel proposes reviving exposure-level star masking with revisited halo/spike scaling — possibly auto-scaled a la Hendrik's current code — and converting the exposure masks to healsparse so they propagate to coadds. Mike's alternative is to reuse the global tile-level star/galaxy/external masks; this needs assessing (Axel argues exposure-level masking is required). Maximask's role (stacks-only per the Paris meeting, vs Axel's 'maybe not needed at all') is a sub-question.
Discussion is in the #weak-lensing-tomography Slack thread of 2026-07-14.
Context: Paris meeting notes