Skip to content

A/B validation of off-by-default shape-measurement options #813

Description

@cailmdaley

Several ngmix-module options ship off by default, pending real-data evidence — the design contract is that production defaults flip here, on data, never in the implementing PR. Each sub-issue is one A/B question with a named metric and a named decider.

Current options awaiting a verdict:

The decisive statistics (B-modes, leakage) need real area, which is why this lives in the First run on data (Nibi) milestone: the A/B runs are consumers of the P3 footprint (#801, #808) — pipeline run twice (or more) with config variants only, matched catalogues compared through the sp_validation machinery. Complementary evidence from image simulations (calibrated m/c truth) rides UNIONS-WL/MultiBand_ImSim#1.

Methodology (area needed, patch choice, matched-object vs matched-area comparison) needs group design — to discuss on a tomography call before the runs are scheduled.

— Claude, on behalf of Cail

Activity

  1. aguinot commented on Jul 13, 2026

    @aguinot
    Contributor

    I think another statistic to monitor would be shape noise. I have seen a significant decrease in shape noise when using uberseg.

  2. removed this from the First run on data (Nibi) milestone on Sep 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions