Repository navigation

Exposure-level HealSparse maps: exposure count and flagged pixels - #887

Merged

cailmdaley merged 151 commits into

developfrom

feat/instrument-defect-map

Oct 10, 2026

Commits on Sep 9, 2026
- feat(orchestration): the keep list, and the script that acts on it
  
  authored and
  cailmdaley
  committed
  681f156
- feat(orchestration): exp_persist, between exp_psf and clean_exposure
  
  authored and
  cailmdaley
  committed
  de243a8
- docs(orchestration): exp_persist in the rule list, and why the report omits it
  
  authored and
  cailmdaley
  committed
  4cf639f
- feat(orchestration): exp_persist packs one tar per exposure, not loose copies
  
  cailmdaley
  and
  claude
  committed
  1fa6455
- fix(orchestration): persist_exp never leaves a .tmp behind on failure
  
  cailmdaley
  and
  claude
  committed
  a14a2f3
- docs(orchestration): drop references to the removed star-catalogue rules
  
  cailmdaley
  and
  claude
  committed
  d91da36
- feat(orchestration): the two campaign-level merges
  
  cailmdaley
  and
  claude
  committed
  626aaba
- fix(orchestration): seven defects in the campaign-level merges
  
  cailmdaley
  and
  claude
  committed
  fc9aa3e
- fix(create_final_cat): make the merged-catalogue writer reproducible
  
  cailmdaley
  and
  claude
  committed
  d774cc3
- perf(orchestration): size the two merges from the data, not from a guess
  
  cailmdaley
  and
  claude
  committed
  98532ce
- feat(orchestration): persist_exp keeps NAMED products, not globs
  
  cailmdaley
  and
  claude
  committed
  d0ecfdd
- perf(merge_starcat): accumulate arrays, not python lists of floats
  
  cailmdaley
  and
  claude
  committed
  3b8c7d5
- feat(orchestration): the star catalogue's inputs are not a user choice
  
  cailmdaley
  and
  claude
  committed
  0ad6403
- fix(cfis): two stale columns in final_cat.param, and no mask column at all
  
  cailmdaley
  and
  claude
  committed
  90dfb00
- perf(merge_starcat): two passes, so nothing is held twice
  
  cailmdaley
  and
  claude
  committed
  a434c9a
- feat(orchestration): final_cat_merge reconciles instead of rebuilding
  
  cailmdaley
  and
  claude
  committed
  3c29825
Commits on Sep 10, 2026
- fix(orchestration): eight defects found reviewing the merge work
  
  cailmdaley
  and
  claude
  committed
  78d2ce9
- feat(orchestration): the star catalogue becomes hdf5, reconciled like the tile one
  
  cailmdaley
  and
  claude
  committed
  0fb514b
- fix(orchestration): nine findings from the third review
  
  cailmdaley
  and
  claude
  committed
  4ed9fb0
- feat(cfis): wire the UNIONS external masks into the workflow
  
  cailmdaley
  and
  claude
  committed
  9fcf3f1
- test(unit): property-based state machines for reconcile and persist_exp
  
  cailmdaley
  and
  claude
  committed
  e86d8c8
- feat(workflow): emit the instrument flags as a healsparse defect map (#878)
  
  cailmdaley
  and
  claude
  committed
  f5c76a4
- docs(workflow): the defect map in the README, and how to join it to the ladder
  
  cailmdaley
  and
  claude
  committed
  6783f4d
- fix(defect-map): a short split is an error, a bad chip is not an OOM
  
  cailmdaley
  and
  claude
  committed
  958b562
- fix(defect-map): a reclaimed exposure is a leaf, and the merge asks for 7 GB
  
  cailmdaley
  and
  claude
  committed
  a9daa4f
- docs(defect-map): the measurements, and three per exposure rather than one
  
  cailmdaley
  and
  claude
  committed
  1f6e832
- fix(defect-map): reclamation never waits on a rasterization it cannot run
  
  cailmdaley
  and
  claude
  committed
  db4f426
- perf(defect-map): the merge allocates its accumulator once
  
  cailmdaley
  and
  claude
  committed
  f760687
- fix(persist-exp): the PSF run dir is run_sp_exp_SxSePsf
  
  cailmdaley
  and
  claude
  committed
  1a343e6
Commits on Sep 11, 2026
- workflow: image simulations as an input mode of the real-data workflow
  
  martinkilbinger
  and
  claude
  committed
  5b88f17
- workflow: image-sims merge column list, and the merge reads the workflow layout
  
  martinkilbinger
  and
  claude
  committed
  4e17fee
- workflow: campaign-unique node-local tile store for image sims; candide excludes
  
  martinkilbinger
  and
  claude
  committed
  716b160
- workflow: port the MCCD exposure chain to the workflow grammar
  
  martinkilbinger
  and
  claude
  committed
  fdc9db4
Commits on Sep 17, 2026
- workflow: exp_psf reserves 2 cores, not 8, for the MCCD chain
  
  martinkilbinger
  and
  claude
  committed
  836f91d
- mccd_interp: test for N_EPOCH among the column names
  
  martinkilbinger
  and
  claude
  committed
  ad80f23
- workflow: MCCD validated through the full chain; drop the warn-only flags
  
  martinkilbinger
  and
  claude
  committed
  2082dcf
- workflow: tile_store_root run-config key moves the tile store's bind, per campaign
  
  martinkilbinger
  and
  claude
  committed
  9085cd0
- completeness.py: atomic write for stage log/manifest JSON
  martinkilbinger
  committed
  7637b39
- workflow/config.yaml: universal template, no cluster-specific paths
  martinkilbinger
  committed
  7aababf
- workflow: retrieve mode (symlink|vos) is a run-config key, not fixed in the ini
  martinkilbinger
  committed
  d64f88e
- config.yaml: restore real nibi defaults, keep the placeholder check as a guard
  martinkilbinger
  committed
  c7bcfdd
- Snakefile: check machine: against SP_PROFILE at parse time
  martinkilbinger
  committed
  5be4c43
- config.yaml: machines: table drives per-machine (and per-input_type) defaults
  martinkilbinger
  committed
  29442d3
- workflow: one run-config resolver for Snakefile, bin/sp and container.py
  
  martinkilbinger
  and
  claude
  committed
  7293e3b
- run_config: `run:` name, expanded as $run in run-config paths
  
  martinkilbinger
  and
  claude
  committed
  084e305
- added user run config template
  martinkilbinger
  committed
  78cc0ca
- bin/sp: -c/--config-file for the run config, instead of SP_RUN_CONFIG
  
  martinkilbinger
  and
  claude
  committed
  d3b088a
- bin/sp: document -c, the two-file merge and the environment in the header
  
  martinkilbinger
  and
  claude
  committed
  746ca9c
- improved commeents
  martinkilbinger
  committed
  4e0bedc
- final_cat_merge, star_cat_merge: record code provenance in the HDF5 attributes
  
  cailmdaley
  and
  claude
  committed
  3e269c4
Commits on Sep 23, 2026
- workflow smoothed; running until hdf5 file
  martinkilbinger
  committed
  fbfdf3b
Commits on Sep 25, 2026
- bin/sp: don't swallow -c/--config meant for the delegated command
  
  cailmdaley
  and
  claude
  committed
  0b5fae7
- workflow/README: add products_dir to the image-sims run-config example
  
  cailmdaley
  and
  claude
  committed
  d22c4a0
- run_template.yaml: make it resolve
  
  cailmdaley
  and
  claude
  committed
  5ded2ca
Commits on Sep 26, 2026
- docs(astra): record the pipeline's scientific decisions in astra.yaml
  
  cailmdaley
  and
  claude
  committed
  dae9e1b
- test(astra): validate decision anchors and universe pins
  cailmdaley
  committed
  e901cc9
- Merge remote-tracking branch 'origin/develop' into feat/persist-exp-products-v2
  cailmdaley
  committed
  8e40991
- merge: feat/workflow-image-sims (#894 @5ded2ca8) + develop into feat/persist-exp-products
  
  cailmdaley
  and
  claude
  committed
  d84822f
- workflow: final_cat_merge is the one merger, for data and image sims
  
  cailmdaley
  and
  claude
  committed
  4b1e868
- workflow: the campaign name is `run:`, and `run:` is required
  
  cailmdaley
  and
  claude
  committed
  d7ce47f
- workflow: no PSF persistence under psf_model: fake
  
  cailmdaley
  and
  claude
  committed
  3386347
- test(grammar): check the image-sims final_cat.param too
  
  cailmdaley
  and
  claude
  committed
  89f00b5
- Merge remote-tracking branch 'origin/docs/astra-decision-record' into HEAD
  cailmdaley
  committed
  1685d65
- feat(cfis): wire the UNIONS external masks into the workflow
  
  cailmdaley
  and
  claude
  committed
  45a082f
- fix(file_io): recognize numpy bool arrays when picking the FITS column type
  
  cailmdaley
  and
  claude
  committed
  4454d07
- docs(astra): record the pipeline's scientific decisions in astra.yaml
  
  cailmdaley
  and
  claude
  committed
  09edd5e
- test(astra): validate decision anchors and universe pins
  cailmdaley
  committed
  206b48d
- test(astra): resolve Snakemake rule anchors; JSON report mode
  
  cailmdaley
  and
  claude
  committed
  9f9ec5e
- docs(astra): rewrite the decision record against develop
  
  cailmdaley
  and
  claude
  committed
  5368d9d
- docs(claude): point the scientific-decisions section at the anchor test
  
  cailmdaley
  and
  claude
  committed
  a6eff5e
- merge: feat/wire-external-masks-v2 (#886 on #879/#894) into feat/instrument-defect-map
  
  cailmdaley
  and
  claude
  committed
  f77959d
- feat(make_cat): fixed per-epoch slot count via N_EPOCH_SLOTS
  
  cailmdaley
  and
  claude
  committed
  62a2699
- feat(workflow): save fixed-slot per-epoch PSF data in tile make_cat
  
  cailmdaley
  and
  claude
  committed
  512e7bf
- workflow: no defect map for image simulations (MAPS_DEFECTS)
  
  cailmdaley
  and
  claude
  committed
  8d38292
- fix(workflow): five review findings on the #894 merge
  
  cailmdaley
  and
  claude
  committed
  b2a4d19
- feat(cfis): wire the UNIONS external masks into the workflow
  
  cailmdaley
  and
  claude
  committed
  168d7fc
- fix(file_io): recognize numpy bool arrays when picking the FITS column type
  
  cailmdaley
  and
  claude
  committed
  aee2ee9
- Merge branch 'feat/wire-external-masks-v2' into feat/instrument-defect-map-v2
  cailmdaley
  committed
  3bf2e12
- workflow: scientific contracts for the campaign products
  
  cailmdaley
  and
  claude
  committed
  4958d66
- test(workflow): the campaign name has one source, and products one root
  
  cailmdaley
  and
  claude
  committed
  5fcc99e
- test(final_cat_merge): columns, per-epoch slots, never-fit rows, missing names
  
  cailmdaley
  and
  claude
  committed
  e768908
- docs(astra): correct seven rationale claims against the code
  
  cailmdaley
  and
  claude
  committed
  e3f6f4a
- docs(astra): the instrument flags become a conservative healsparse defect map
  
  cailmdaley
  and
  claude
  committed
  41be84a
- Merge remote-tracking branch 'origin/docs/astra-decision-record' into feat/persist-exp-products-v2
  cailmdaley
  committed
  27b59f7
- Merge remote-tracking branch 'origin/feat/make-cat-fixed-epoch-slots' into feat/persist-exp-products-v2
  cailmdaley
  committed
  eb05d3a
- Merge remote-tracking branch 'origin/docs/astra-decision-record' into feat/persist-exp-products-v2
  cailmdaley
  committed
  d214966
- config(cfis): final_cat.param carries the 12 per-epoch slots (EXP_ID, CCD, HSM_*_PSF)
  
  cailmdaley
  and
  claude
  committed
  3358105
- docs(astra): two lints this stack resolves — IMAFLAGS_ISO is not merged, the MCCD chain runs
  
  cailmdaley
  and
  claude
  committed
  bac6eea
- test(defect-map): the fragment contains every flagged pixel
  
  cailmdaley
  and
  claude
  committed
  bc09a5a
- feat(cfis): wire the UNIONS external masks into the workflow
  
  cailmdaley
  and
  claude
  committed
  07038fd
- fix(file_io): recognize numpy bool arrays when picking the FITS column type
  
  cailmdaley
  and
  claude
  committed
  c1fbe7c
- docs(astra): the shear catalogue's default MASK_n* cut is the six r-mask bits
  
  cailmdaley
  and
  claude
  committed
  f84065d
- test(masks): final_cat.param's MASK_n* names match MASK_EXT_PATHS; the six-bit OR reproduces mask_r
  
  cailmdaley
  and
  claude
  committed
  03fd684
- Merge feat/wire-external-masks-v2 (final) into feat/instrument-defect-map-v2
  cailmdaley
  committed
  d2b4de8
- config(cfis): final_cat.param keeps NUMBER
  
  cailmdaley
  and
  claude
  committed
  df20cf4
- fix(workflow): the sims overlay resolves gauss_3.0_7x7.conv; `sp -- …` drops its separator
  
  cailmdaley
  and
  claude
  committed
  05e7381
- fix(workflow): refuse psf_model=mccd at parse time
  
  cailmdaley
  and
  claude
  committed
  45a9f1b
- fix(workflow): refuse reclamation when products_dir is run_dir
  
  cailmdaley
  and
  claude
  committed
  16df7ae
- fix(workflow): run_config expands to a fixed point and reports any leftover $
  
  cailmdaley
  and
  claude
  committed
  96abec8
- fix(defect-map): a changed fragment changes its manifest and the merge's record
  
  cailmdaley
  and
  claude
  committed
  e3d7d4f
- fix(persist_exp): the manifest records each member's sha256
  
  cailmdaley
  and
  claude
  committed
  072b705
- fix(defect-map): a resolution change rebuilds instead of passing as a no-op
  
  cailmdaley
  and
  claude
  committed
  b9882a2
- fix(defect-map): bind map and sidecar with a generation id
  
  cailmdaley
  and
  claude
  committed
  f524957
- fix(hdf5_reconcile): one type per column across a campaign
  
  cailmdaley
  and
  claude
  committed
  d179f2b
- test(hdf5_reconcile): isolate schema invalidation from source stamps
  
  cailmdaley
  and
  claude
  committed
  187968e
- test(defect-map): drive the reconcile tests through main()
  
  cailmdaley
  and
  claude
  committed
  53d4b09
- fix(build_index): readiness is the tiles row, not the retained edges
  
  cailmdaley
  and
  claude
  committed
  70d7a20
- fix(hdf5_reconcile): delete an abandoned tmp before the free-space check
  
  cailmdaley
  and
  claude
  committed
  5a61aaf
- fix(hdf5_reconcile): each provenance record replaces the last
  
  cailmdaley
  and
  claude
  committed
  47b2ec5
- fix(workflow): a reclaimed exposure's clean waits on its tar, not the manifest
  
  cailmdaley
  and
  claude
  committed
  b15d442
- test(conftest): the candide hostname test matches the whole bare name
  
  cailmdaley
  and
  claude
  committed
  5b56a68
- test(workflow): add an isolated Snakemake DAG driver
  
  cailmdaley
  and
  codex
  committed
  33391a4
- test(workflow): enforce campaign DAG scope and product custody
  
  cailmdaley
  and
  codex
  committed
  7f5ecf2
- test(workflow): pin prologues and shells at campaign boundaries
  
  cailmdaley
  and
  codex
  committed
  0e279d1
- test(workflow): the DAG harness follows the parse-time guards
  
  cailmdaley
  and
  claude
  committed
  a846e08
- feat(cfis): wire the UNIONS external masks into the workflow
  
  cailmdaley
  and
  claude
  committed
  2c12bf5
- fix(file_io): recognize numpy bool arrays when picking the FITS column type
  
  cailmdaley
  and
  claude
  committed
  3453e8f
- docs(astra): the shear catalogue's default MASK_n* cut is the six r-mask bits
  
  cailmdaley
  and
  claude
  committed
  54c569a
- test(masks): final_cat.param's MASK_n* names match MASK_EXT_PATHS; the six-bit OR reproduces mask_r
  
  cailmdaley
  and
  claude
  committed
  94e2dc1
- Merge feat/wire-external-masks-v2 (on the reviewed #879) into feat/instrument-defect-map-v2
  cailmdaley
  committed
  0e4b4f5
- test(workflow): the DAG harness knows the defect rules; the reconcile fixture carries a tiles table
  
  cailmdaley
  and
  claude
  committed
  eec9b03
- test(workflow): update the params pin — SP_INPUT_MASKS joins every data prologue
  
  cailmdaley
  and
  claude
  committed
  dc11dab
- Merge branch 'feat/wire-external-masks-v2' into feat/instrument-defect-map-v2
  cailmdaley
  committed
  5f7332d
Commits on Sep 28, 2026
- Merge origin/develop (#879 squash, #875, #854, #905, #911, #894) into feat/wire-external-masks
  
  cailmdaley
  and
  claude
  committed
  a1cb926
- Merge feat/wire-external-masks (develop merged in) into feat/instrument-defect-map
  
  cailmdaley
  and
  claude
  committed
  2d8f925
- Write the defect map under <products_dir>/defect_map/
  
  cailmdaley
  and
  claude
  committed
  be528c8
- Coverage geometry becomes a library; retire the v1.x header-scrape chain
  
  cailmdaley
  and
  claude
  committed
  2b59d35
- feat(workflow): the exposure-count map, beside the defect map
  
  cailmdaley
  and
  claude
  committed
  47b7e45
Commits on Sep 29, 2026
- feat(workflow): the exposure-count map is on for every fitted-PSF run
  
  cailmdaley
  and
  claude
  committed
  c29fd12
- fix(workflow): no exposure-count map job without a footprint record
  
  cailmdaley
  and
  claude
  committed
  e8ce5fe
- workflow: refuse retired coverage:/defect_map: config keys at parse time
  
  cailmdaley
  and
  claude
  committed
  60d9d67
- tests: key the retired-key ordering check on either defaults call
  
  cailmdaley
  and
  claude
  committed
  2e12ccd
- Clarify HSC and Pan-STARRS mask bit labels
  cailmdaley
  committed
  0a65f63
Commits on Sep 30, 2026
- Merge develop into external-mask branch
  cailmdaley
  committed
  6fe0842
- State current star halo bit labels from mask producer schema
  cailmdaley
  committed
  d43c70d
- Describe halo bit identities and guard per-map catalogue columns
  cailmdaley
  committed
  af641f8
- Clarify flag values and current mask-column comments
  cailmdaley
  committed
  f3f87db
Commits on Oct 1, 2026
- Name the mask columns MASK_<flag value>_<name>
  
  cailmdaley
  and
  claude
  committed
  a2e8068
- Merge feat/wire-external-masks (MASK_<flag value>_<name> columns, develop merged in) into feat/instrument-defect-map
  
  cailmdaley
  and
  claude
  committed
  0a44b68
- config(cfis): the defect map's label carries no flag value
  
  cailmdaley
  and
  claude
  committed
  b617e39
- config(cfis): the defect-map recipe follows the multi-line MASK_EXT_PATHS
  
  cailmdaley
  and
  claude
  committed
  9423554
Commits on Oct 5, 2026
- Merge develop into feat/wire-external-masks
  
  cailmdaley
  and
  claude
  committed
  c2b5a7d
- Merge feat/wire-external-masks (with develop) into feat/instrument-defect-map
  
  cailmdaley
  and
  claude
  committed
  6d2e9fc
- Drop star halos from the default external-mask cut
  
  cailmdaley
  and
  claude
  committed
  c28938b
- Merge feat/wire-external-masks into feat/instrument-defect-map
  cailmdaley
  committed
  d55be57
- Drop the mask_default_cut decision: ShapePipe makes no mask choice
  
  cailmdaley
  and
  claude
  committed
  baa6430
- refactor(exposure-maps): one fragment per exposure, summed into nexp and nflagged
  
  cailmdaley
  and
  claude
  committed
  9de93f7
- Merge feat/wire-external-masks into feat/instrument-defect-map
  
  cailmdaley
  and
  claude
  committed
  9b9b2a2
- docs(workflow): nflagged is a diagnostic, nexp is what randoms match
  
  cailmdaley
  and
  claude
  committed
  8f351f0
Commits on Oct 6, 2026
- Merge origin/develop (#886 squash, #940) into feat/instrument-defect-map
  
  cailmdaley
  and
  claude
  committed
  ac6523b
Commits on Oct 10, 2026
- Merge origin/develop into feat/instrument-defect-map
  
  cailmdaley
  and
  claude
  committed
  e88af8f

Commits

Commits on Sep 9, 2026

Commits on Sep 10, 2026

Commits on Sep 11, 2026

Commits on Sep 17, 2026

Commits on Sep 23, 2026

Commits on Sep 25, 2026

Commits on Sep 26, 2026

Commits on Sep 28, 2026

Commits on Sep 29, 2026

Commits on Sep 30, 2026