Repository navigation

Persist PSF products and add the two campaign-level merges (exp_persist, star_cat_merge, final_cat_merge) - #879

Merged

cailmdaley merged 95 commits into

developfrom

feat/persist-exp-products

Sep 28, 2026

Commits on Sep 9, 2026
- feat(orchestration): the keep list, and the script that acts on it
  
  authored and
  cailmdaley
  committed
  681f156
- feat(orchestration): exp_persist, between exp_psf and clean_exposure
  
  authored and
  cailmdaley
  committed
  de243a8
- docs(orchestration): exp_persist in the rule list, and why the report omits it
  
  authored and
  cailmdaley
  committed
  4cf639f
- feat(orchestration): exp_persist packs one tar per exposure, not loose copies
  
  cailmdaley
  and
  claude
  committed
  1fa6455
- fix(orchestration): persist_exp never leaves a .tmp behind on failure
  
  cailmdaley
  and
  claude
  committed
  a14a2f3
- docs(orchestration): drop references to the removed star-catalogue rules
  
  cailmdaley
  and
  claude
  committed
  d91da36
- feat(orchestration): the two campaign-level merges
  
  cailmdaley
  and
  claude
  committed
  626aaba
- fix(orchestration): seven defects in the campaign-level merges
  
  cailmdaley
  and
  claude
  committed
  fc9aa3e
- fix(create_final_cat): make the merged-catalogue writer reproducible
  
  cailmdaley
  and
  claude
  committed
  d774cc3
- perf(orchestration): size the two merges from the data, not from a guess
  
  cailmdaley
  and
  claude
  committed
  98532ce
- feat(orchestration): persist_exp keeps NAMED products, not globs
  
  cailmdaley
  and
  claude
  committed
  d0ecfdd
- perf(merge_starcat): accumulate arrays, not python lists of floats
  
  cailmdaley
  and
  claude
  committed
  3b8c7d5
- feat(orchestration): the star catalogue's inputs are not a user choice
  
  cailmdaley
  and
  claude
  committed
  0ad6403
- fix(cfis): two stale columns in final_cat.param, and no mask column at all
  
  cailmdaley
  and
  claude
  committed
  90dfb00
- perf(merge_starcat): two passes, so nothing is held twice
  
  cailmdaley
  and
  claude
  committed
  a434c9a
- feat(orchestration): final_cat_merge reconciles instead of rebuilding
  
  cailmdaley
  and
  claude
  committed
  3c29825
Commits on Sep 10, 2026
- fix(orchestration): eight defects found reviewing the merge work
  
  cailmdaley
  and
  claude
  committed
  78d2ce9
- feat(orchestration): the star catalogue becomes hdf5, reconciled like the tile one
  
  cailmdaley
  and
  claude
  committed
  0fb514b
- fix(orchestration): nine findings from the third review
  
  cailmdaley
  and
  claude
  committed
  4ed9fb0
- test(unit): property-based state machines for reconcile and persist_exp
  
  cailmdaley
  and
  claude
  committed
  e86d8c8
- fix(persist-exp): the PSF run dir is run_sp_exp_SxSePsf
  
  cailmdaley
  and
  claude
  committed
  1a343e6
Commits on Sep 11, 2026
- workflow: image simulations as an input mode of the real-data workflow
  
  martinkilbinger
  and
  claude
  committed
  5b88f17
- workflow: image-sims merge column list, and the merge reads the workflow layout
  
  martinkilbinger
  and
  claude
  committed
  4e17fee
- workflow: campaign-unique node-local tile store for image sims; candide excludes
  
  martinkilbinger
  and
  claude
  committed
  716b160
- workflow: port the MCCD exposure chain to the workflow grammar
  
  martinkilbinger
  and
  claude
  committed
  fdc9db4
Commits on Sep 17, 2026
- workflow: exp_psf reserves 2 cores, not 8, for the MCCD chain
  
  martinkilbinger
  and
  claude
  committed
  836f91d
- mccd_interp: test for N_EPOCH among the column names
  
  martinkilbinger
  and
  claude
  committed
  ad80f23
- workflow: MCCD validated through the full chain; drop the warn-only flags
  
  martinkilbinger
  and
  claude
  committed
  2082dcf
- workflow: tile_store_root run-config key moves the tile store's bind, per campaign
  
  martinkilbinger
  and
  claude
  committed
  9085cd0
- completeness.py: atomic write for stage log/manifest JSON
  martinkilbinger
  committed
  7637b39
- workflow/config.yaml: universal template, no cluster-specific paths
  martinkilbinger
  committed
  7aababf
- workflow: retrieve mode (symlink|vos) is a run-config key, not fixed in the ini
  martinkilbinger
  committed
  d64f88e
- config.yaml: restore real nibi defaults, keep the placeholder check as a guard
  martinkilbinger
  committed
  c7bcfdd
- Snakefile: check machine: against SP_PROFILE at parse time
  martinkilbinger
  committed
  5be4c43
- config.yaml: machines: table drives per-machine (and per-input_type) defaults
  martinkilbinger
  committed
  29442d3
- workflow: one run-config resolver for Snakefile, bin/sp and container.py
  
  martinkilbinger
  and
  claude
  committed
  7293e3b
- run_config: `run:` name, expanded as $run in run-config paths
  
  martinkilbinger
  and
  claude
  committed
  084e305
- added user run config template
  martinkilbinger
  committed
  78cc0ca
- bin/sp: -c/--config-file for the run config, instead of SP_RUN_CONFIG
  
  martinkilbinger
  and
  claude
  committed
  d3b088a
- bin/sp: document -c, the two-file merge and the environment in the header
  
  martinkilbinger
  and
  claude
  committed
  746ca9c
- improved commeents
  martinkilbinger
  committed
  4e0bedc
- final_cat_merge, star_cat_merge: record code provenance in the HDF5 attributes
  
  cailmdaley
  and
  claude
  committed
  3e269c4
Commits on Sep 23, 2026
- workflow smoothed; running until hdf5 file
  martinkilbinger
  committed
  fbfdf3b
Commits on Sep 25, 2026
- bin/sp: don't swallow -c/--config meant for the delegated command
  
  cailmdaley
  and
  claude
  committed
  0b5fae7
- workflow/README: add products_dir to the image-sims run-config example
  
  cailmdaley
  and
  claude
  committed
  d22c4a0
- run_template.yaml: make it resolve
  
  cailmdaley
  and
  claude
  committed
  5ded2ca
Commits on Sep 26, 2026
- docs(astra): record the pipeline's scientific decisions in astra.yaml
  
  cailmdaley
  and
  claude
  committed
  dae9e1b
- test(astra): validate decision anchors and universe pins
  cailmdaley
  committed
  e901cc9
- Merge remote-tracking branch 'origin/develop' into feat/persist-exp-products-v2
  cailmdaley
  committed
  8e40991
- merge: feat/workflow-image-sims (#894 @5ded2ca8) + develop into feat/persist-exp-products
  
  cailmdaley
  and
  claude
  committed
  d84822f
- workflow: final_cat_merge is the one merger, for data and image sims
  
  cailmdaley
  and
  claude
  committed
  4b1e868
- workflow: the campaign name is `run:`, and `run:` is required
  
  cailmdaley
  and
  claude
  committed
  d7ce47f
- workflow: no PSF persistence under psf_model: fake
  
  cailmdaley
  and
  claude
  committed
  3386347
- test(grammar): check the image-sims final_cat.param too
  
  cailmdaley
  and
  claude
  committed
  89f00b5
- Merge remote-tracking branch 'origin/docs/astra-decision-record' into HEAD
  cailmdaley
  committed
  1685d65
- docs(astra): record the pipeline's scientific decisions in astra.yaml
  
  cailmdaley
  and
  claude
  committed
  09edd5e
- test(astra): validate decision anchors and universe pins
  cailmdaley
  committed
  206b48d
- test(astra): resolve Snakemake rule anchors; JSON report mode
  
  cailmdaley
  and
  claude
  committed
  9f9ec5e
- docs(astra): rewrite the decision record against develop
  
  cailmdaley
  and
  claude
  committed
  5368d9d
- docs(claude): point the scientific-decisions section at the anchor test
  
  cailmdaley
  and
  claude
  committed
  a6eff5e
- feat(make_cat): fixed per-epoch slot count via N_EPOCH_SLOTS
  
  cailmdaley
  and
  claude
  committed
  62a2699
- feat(workflow): save fixed-slot per-epoch PSF data in tile make_cat
  
  cailmdaley
  and
  claude
  committed
  512e7bf
- fix(workflow): five review findings on the #894 merge
  
  cailmdaley
  and
  claude
  committed
  b2a4d19
- workflow: scientific contracts for the campaign products
  
  cailmdaley
  and
  claude
  committed
  4958d66
- test(workflow): the campaign name has one source, and products one root
  
  cailmdaley
  and
  claude
  committed
  5fcc99e
- test(final_cat_merge): columns, per-epoch slots, never-fit rows, missing names
  
  cailmdaley
  and
  claude
  committed
  e768908
- docs(astra): correct seven rationale claims against the code
  
  cailmdaley
  and
  claude
  committed
  e3f6f4a
- Merge remote-tracking branch 'origin/docs/astra-decision-record' into feat/persist-exp-products-v2
  cailmdaley
  committed
  27b59f7
- Merge remote-tracking branch 'origin/feat/make-cat-fixed-epoch-slots' into feat/persist-exp-products-v2
  cailmdaley
  committed
  eb05d3a
- Merge remote-tracking branch 'origin/docs/astra-decision-record' into feat/persist-exp-products-v2
  cailmdaley
  committed
  d214966
- config(cfis): final_cat.param carries the 12 per-epoch slots (EXP_ID, CCD, HSM_*_PSF)
  
  cailmdaley
  and
  claude
  committed
  3358105
- docs(astra): two lints this stack resolves — IMAFLAGS_ISO is not merged, the MCCD chain runs
  
  cailmdaley
  and
  claude
  committed
  bac6eea
- config(cfis): final_cat.param keeps NUMBER
  
  cailmdaley
  and
  claude
  committed
  df20cf4
- fix(workflow): the sims overlay resolves gauss_3.0_7x7.conv; `sp -- …` drops its separator
  
  cailmdaley
  and
  claude
  committed
  05e7381
- fix(workflow): refuse psf_model=mccd at parse time
  
  cailmdaley
  and
  claude
  committed
  45a9f1b
- fix(workflow): refuse reclamation when products_dir is run_dir
  
  cailmdaley
  and
  claude
  committed
  16df7ae
- fix(workflow): run_config expands to a fixed point and reports any leftover $
  
  cailmdaley
  and
  claude
  committed
  96abec8
- fix(persist_exp): the manifest records each member's sha256
  
  cailmdaley
  and
  claude
  committed
  072b705
- fix(hdf5_reconcile): one type per column across a campaign
  
  cailmdaley
  and
  claude
  committed
  d179f2b
- test(hdf5_reconcile): isolate schema invalidation from source stamps
  
  cailmdaley
  and
  claude
  committed
  187968e
- fix(build_index): readiness is the tiles row, not the retained edges
  
  cailmdaley
  and
  claude
  committed
  70d7a20
- fix(hdf5_reconcile): delete an abandoned tmp before the free-space check
  
  cailmdaley
  and
  claude
  committed
  5a61aaf
- fix(hdf5_reconcile): each provenance record replaces the last
  
  cailmdaley
  and
  claude
  committed
  47b2ec5
- fix(workflow): a reclaimed exposure's clean waits on its tar, not the manifest
  
  cailmdaley
  and
  claude
  committed
  b15d442
- test(conftest): the candide hostname test matches the whole bare name
  
  cailmdaley
  and
  claude
  committed
  5b56a68
- test(workflow): add an isolated Snakemake DAG driver
  
  cailmdaley
  and
  codex
  committed
  33391a4
- test(workflow): enforce campaign DAG scope and product custody
  
  cailmdaley
  and
  codex
  committed
  7f5ecf2
- test(workflow): pin prologues and shells at campaign boundaries
  
  cailmdaley
  and
  codex
  committed
  0e279d1
- test(workflow): the DAG harness follows the parse-time guards
  
  cailmdaley
  and
  claude
  committed
  a846e08
Commits on Sep 28, 2026
- Merge origin/develop into feat/make-cat-fixed-epoch-slots
  
  cailmdaley
  and
  claude
  committed
  8209e26
- Merge origin/develop into feat/persist-exp-products
  
  cailmdaley
  and
  claude
  committed
  fedbf9c
- Merge origin/feat/make-cat-fixed-epoch-slots into feat/persist-exp-products
  
  cailmdaley
  and
  claude
  committed
  a2b37a8
- test: seams and run_config follow the develop merge
  
  cailmdaley
  and
  claude
  committed
  2d833fd
- test(workflow): snakemake-driven checks run in their own CI step
  
  cailmdaley
  and
  claude
  committed
  aa458b2
- Merge origin/develop into feat/persist-exp-products
  
  cailmdaley
  and
  claude
  committed
  5891cf6

Commits

Commits on Sep 9, 2026

Commits on Sep 10, 2026

Commits on Sep 11, 2026

Commits on Sep 17, 2026

Commits on Sep 23, 2026

Commits on Sep 25, 2026

Commits on Sep 26, 2026

Commits on Sep 28, 2026