Snakemake orchestration for shapepipe

Snakemake orchestration for the real-data pipeline

Summary

Replace the shell-script orchestration of real-data ShapePipe runs (run_job_sp_canfar_v2.0.bash → job_sp_canfar_v2.0.bash, plus per-site sbatch reimplementations) with a Snakemake workflow in this repo. Rules call shapepipe_run -c <config> on the existing config chains, at the granularities the bash layer already uses; Snakemake's file-based DAG takes over the state the bash layer hand-rolls — completeness counts, done-logs, flock dedup, retry/force. Module code stays largely untouched. Once the DAG is explicit, "why did this tile fail, what is safe to rerun" become answerable from the DAG rather than reconstructed by hand. Living PRD; the eventual PR just says "implements #848." Please keep poking holes.

Pipeline shape

_{🟦 tile-level 🟧 exposure-level (shared across tiles) 🟪 index / shared catalogue 🟩 cleanup}

flowchart TD
    subgraph S1["invocation 1 · PREPARE — one chain per tile (×~7300 at DR6)"]
        Git["tile_get_images<br/>(download)"] --> Uz[uncompress weight] --> Fe["find_exposures<br/>(read header)"]
    end
    subgraph S3["invocation 2 · COMPUTE"]
        IX[("parse time: dedup union of<br/>exposure lists → index.sqlite")]
        SC[("star_catalogue: HEALPix-chunked<br/>fetch, once per footprint")]
        subgraph EX["one chain per DEDUPED exposure (×~20k) — each built once, shared by ~4 tiles"]
            Gie[exp_get_images] --> Sp[split into 40 CCDs] --> Ma[exp_mask] --> Psf["exp_psf: SExtractor → setools → PSFEx → PSF interp"]
            Esc["exp_star_cat<br/>(local cut)"] --> Ma
        end
        SC --> Esc
        Gie --> Esc
        subgraph TQ["one chain per tile (×~7300)"]
            Mh[tile_merge_headers] --> Sxt["tile_detect<br/>(SExtractor)"] --> Vi["tile_vignets<br/>(PSF interp + stamps)<br/>🗑 vignettes after make_cat"]
            Vi --> Ng1["ngmix chunk 1"]
            Vi --> Ngd["⋯"]
            Vi --> NgN["ngmix chunk N"]
            Ng1 --> Ms["tile_merge_cats — gather<br/>🗑 chunks after merge"]
            Ngd --> Ms
            NgN --> Ms
            Ms --> Mc["make_cat → final_cat ×1 per tile"]
        end
        Vi --> Cl["🗑 clean_exposure ×1 per exposure:<br/>fires when ALL its tiles' vignettes exist"]
    end
    Fe --> IX
    IX -.->|defines the job set| Gie
    Psf -->|"PSF models, stamps"| Vi
    Sp -->|headers| Mh
    Psf -.-> Cl

    classDef tile fill:#cfe2f3,stroke:#4a86c8,color:#1a3a5c
    classDef exp fill:#fce5cd,stroke:#e69138,color:#5c3a1a
    classDef idx fill:#d9d2e9,stroke:#8e7cc3,color:#3a2a5c
    classDef clean fill:#d9ead3,stroke:#6aa84f,color:#2a4a1a
    class Git,Uz,Fe,Mh,Sxt,Vi,Ng1,Ngd,NgN,Ms,Mc tile
    class Gie,Sp,Ma,Psf,Esc exp
    class IX,SC idx
    class Cl clean

Each box is one Snakemake job: one shapepipe_run on one unit (tile or exposure); modules parallelize over CCDs/objects inside the job. Arrows are declared input: edges; the scheduler orders everything from them, including the tile↔exposure fan and the rolling cleanup. Deletion (🗑) rolls forward with the run — each product is removed when its last reader finishes: the exposure store once its tiles' vignettes exist, the vignettes once make_cat has read them, the chunks at the merge (D5). Only clean_exposure is a rule; the rest is temp() bookkeeping by the scheduler, not jobs. The prepare products and final_cat persist.

Design

D1. Execution model: `sp run` is the entry point

A run is declared by a tile list. sp run brings the products on disk up to date with it. It is two snakemake invocations, because the exposure job set is determined from tile headers and a Snakemake DAG is fixed at parse time:

sp run [--tiles tiles.txt] [--profile nibi] [-n] [snakemake passthrough…]
│
├─ 1. PREPARE   snakemake prepare_all_tiles
│      Static per-tile DAG, known from the tile list alone:
│      download image, unzip weight, find_exposures.
│      Cheap, wide, idempotent — already-prepared tiles are no-ops.
│
└─ 2. COMPUTE   snakemake all
       At parse time: read each tile's exposure list (paths determined by
       the tile list), dedup the union into run_index.sqlite; tiles whose list
       is missing land in missing.json and are skipped. Then the DAG:
       exposure chains, tile chains, ngmix scatter/gather, final_cat per tile.

Snakemake flags (--jobs, -n, --forcerun, …) pass through to both invocations. sp run alone exports SP_PHASE; a bare snakemake invocation fails at parse time rather than silently building an empty DAG.

The two invocations are joined by a tile ↔ exposure index, built at parse time and persisted as index/run_index.sqlite. It accumulates across invocations, so the clean_exposure consumer sets span the whole campaign, not just the current tile list. The build runs only in the head process's compute parse (workflow.is_main_process): the SLURM executor re-parses the Snakefile inside every submitted job, and hundreds of concurrent sqlite writers on Lustre would otherwise contend on the one file. Job parses only read, and every connection uses a 60 s lock timeout — Lustre lock handoffs outlive sqlite's 5 s default.

Each exposure is processed exactly once, and cleaned automatically once its tiles have extracted what they need. In miniature — a run requesting tiles A and B, where A's header lists exposures 101, 102 and B's lists 102, 103:

flowchart LR
    subgraph P["PREPARE"]
        TA["tile A prepared"]
        TB["tile B prepared"]
    end
    TA -->|"101, 102"| IX
    TB -->|"102, 103"| IX
    IX[("index:<br/>101, 102, 103")]
    subgraph C["COMPUTE"]
        E1[exp 101 chain] --> A[tile A chain] --> FA[final_cat A]
        E2["exp 102 chain (shared)"] --> A
        E2 --> B[tile B chain] --> FB[final_cat B]
        E3[exp 103 chain] --> B
        A --> C1[clean exp 101]
        A --> C2[clean exp 102]
        B --> C2
        B --> C3[clean exp 103]
    end
    IX -.-> E1
    IX -.-> E2
    IX -.-> E3

    classDef tile fill:#cfe2f3,stroke:#4a86c8,color:#1a3a5c
    classDef exp fill:#fce5cd,stroke:#e69138,color:#5c3a1a
    classDef idx fill:#d9d2e9,stroke:#8e7cc3,color:#3a2a5c
    classDef clean fill:#d9ead3,stroke:#6aa84f,color:#2a4a1a
    class TA,TB,A,B,FA,FB tile
    class E1,E2,E3 exp
    class IX idx
    class C1,C2,C3 clean

Two tiles in, three exposure chains out (not four — 102 is deduplicated), built once each, read back by their tiles, cleaned once every consuming tile has extracted its stamps (D5).

D2. Rules: one `shapepipe_run` per unit, one manifest out

The atom. A rule runs one ShapePipe stage on one unit (a tile or an exposure). Its shell is one line:

SP_UNIT_NUM=-2605805 shapepipe_run -c $SP_CONFIG/config_exp_Ma.ini -b {threads} \
    && completeness.py check exp_mask {output}

completeness.py verifies the products and writes the manifest (D3).

Manifests are the DAG's currency. Each rule declares one output: its manifest — a success sentinel recording the products verified present, written only when the stage's verdict is ok, so its existence is "this stage succeeded" and a resume can trust the disk without opening a file. The rule's log: carries the full structured verdict — counts against floors, failure reasons parsed from the module logs — written on every run; Snakemake preserves logs across failures and retries, so failure evidence survives without being a DAG token. Downstream rules declare the upstream manifest as input:. The product files themselves are not declared, for two reasons: a missing CCD is often legitimate (masked, too few stars) and Snakemake hard-fails any missing declared output; and at DR6 scale per-CCD declaration means millions of paths. A job clears its own run directory at start, so a rerun never sees stale products.

Product paths are fixed and known in advance. RUN_DATETIME = False removes the timestamp from ShapePipe's output directories: stage X of unit U always writes <unit dir>/output/run_sp_X/, and the committed configs write every INPUT_DIR as such a fixed path — consumers read products directly, no run-log resolution. Three environment variables carry the per-unit values: SP_RUN (the unit's directory), SP_UNIT_NUM (the unit ID), SP_CONFIG (the committed config directory); configs interpolate them and fail loudly when one is unset. -b {threads} sets the module's worker count. No wrapper script, no generated configs, no run-log. merge_sep_cats locates chunk k by substituting the chunk number inside the run-directory name of chunk 1's path (a small module fix shipped with this PR), so the rule holds with no exceptions.

The tile and exposure stores are sharded — tiles/<2-digit-prefix>/<ID>/, exp/<prefix>/<expbase>/ — so no directory exceeds ~1k entries at full-UNIONS scale. Every path the DAG uses comes from the index; the shard exists for humans and the filesystem's metadata server.

The rules, at the existing config boundaries ((PREPARE) rows run in the PREPARE invocation):

rule	unit	config (example/cfis)	product path
`tile_get_images`, `tile_uncompress`, `tile_find_exposures` (PREPARE)	tile	`config_tile_Git_*`, `Uz`, `Fe`	`run_sp_tile_Git/`, `run_sp_tile_Uz/`, `run_sp_tile_Fe/.../exp_numbers-<IDra>-<IDdec>.txt`
(index)	run	`build_index.py`	`index/run_index.sqlite`, `index/missing.json`
`exp_get_images`	exposure	`config_exp_Gie_*`	`run_sp_exp_Gie/`
`exp_split`	exposure	`config_exp_Sp.ini`	`run_sp_exp_Sp/`
`star_catalogue`	footprint	(HEALPix-chunked Vizier fetch)	`star_cats/<chunk>.fits` — durable, run-independent
`exp_star_cat`	exposure	(local cut from chunks)	per-exposure catalogue + `star_cat_exp/` per-CCD farm
`exp_mask`	exposure	`config_exp_Ma_onthefly.ini`	`run_sp_exp_Ma/`
`exp_psf`	exposure	`config_exp_psfex.ini`	`run_sp_exp_SxSePsfPi/`
`tile_exp_forest`	tile	(symlink curation)	per-tile `$SP_EXP` view — convenience, not a DAG edge
`tile_merge_headers`	tile	`config_tile_Mh_exp.ini`	`.../log_exp_headers-<IDra>-<IDdec>.sqlite`
`tile_detect`	tile	`sx_nomask` (committed); `uc` and `sx`+mask unwired	`run_sp_tile_<Uc\|Sx>/`
`tile_vignets`	tile	`config_tile_PiViVi_*.ini`	`run_sp_tile_PiViVi/`
`tile_ngmix`	tile × chunk	`config_tile_Ng` template, env ID range	`run_sp_tile_ngmix_Ng<k>u/`
`tile_merge_cats`	tile	`config_merge_sep_cats.ini`	`run_sp_tile_msc/`
`tile_make_cat`	tile	`config_tile_Mc_*.ini`	`final_cat-<IDra>-<IDdec>.fits`

Notes, in decreasing order of importance:

DAG edges point at real exposure manifests. Rules that read exposure products (tile_merge_headers, tile_vignets) declare the producing exposures' manifests as input:. tile_exp_forest builds a per-tile symlink view and points $SP_EXP at it purely so ShapePipe configs use one path — a convenience, never an edge.
Star catalogues are produced by the DAG, network scaling with sky area rather than exposure count. star_catalogue fetches GSC 2.3 once per footprint as HEALPix-chunked FITS files in a run-independent store — chunks already on disk are skipped, so a growing campaign fetches only new sky, and the assembled catalogue is a durable science product in its own right. exp_star_cat cuts each exposure's catalogue locally from the chunks and builds a real per-CCD symlink farm (the mask module intersects image numbers across a config's input directories, so the farm must contain only this exposure's CCDs); config_exp_Ma.ini reads it unchanged.
The migration work is a config sweep: ~13 committed configs rewritten with fixed INPUT_DIRs, committed RUN_DATETIME, env-interpolated values. One optional module addition, --set SECTION.KEY=VALUE on shapepipe_run (~15 lines), would give a uniform override surface; the design does not depend on it.
NUMBER_LIST = $SP_UNIT_NUM selects the unit, and is set only for stages whose numbering scheme is the unit ID; the *_get_images/exp_mask/exp_psf configs omit it.
The committed tile_detect chain is the sx_nomask variant — SExtractor with no mask at detection, matching the validated bash baseline; survey masks are applied downstream at catalogue level. Whether DR6 detects masked or unmasked is deliberately undecided here and deferred to the masking rework; the masked variant (one config, one rule, a tile-side star-catalogue cut) and the external-catalogue uc variant are unwired until then. The data/image_sims config axis is kept; sims parity is a named follow-up.

D3. Failure handling: count floors

completeness.py check — the second half of every rule's shell line — counts outputs against per-runner floors and exits 0 above the floor, nonzero below. Zero output is always fatal. The thresholds are the ported bash complete_check table, committed in completeness.py (stdlib-only, imported by the check and the report), with counts corrected to the nibi flavor where verified against real trees (SExtractor writes 3 files/CCD → 120/exposure; vignetmaker run-2 writes 5 sqlites/tile); below is its human rendering.

stage	check
tile get_images	`get_images_runner:2` (nibi symlink flavor: image+weight; canfar vos was 4)
tile uncompress	`uncompress_fits_runner:1`
tile find_exposures	`find_exposures_runner:1`
exp get_images	`get_images_runner:3` (nibi symlink flavor: image+weight+flag; canfar vos was 6)
exp split	`split_exp_runner:121` (40 CCD × 3 + 1 header)
exp mask	`mask_runner:40`
exp psf	`sextractor_runner:120 psfex_runner:80 psfex_interp_runner:40::warn setools_runner:80:rand_split`
tile merge_headers	`merge_headers_runner:1`
tile detect (sx)	`sextractor_runner:2`
tile detect (uc, unwired)	`get_images_runner:2 read_ext_sexcat_runner:1`
tile vignets (data)	`psfex_interp_runner:1 vignetmaker_runner_run_1:1 vignetmaker_runner_run_2:5` (`:3` sims)
tile ngmix	`ngmix_runner:1`
tile make_cat	`make_cat_runner:1`

:warn means a short count does not fail the unit (expected sparse-CCD PSF loss); :subpath counts a nested dir. So exp_psf tolerates <40 PSF-interp CCDs but fails hard on 0-of-40, a short SExtractor/PSFEx count, or a nonzero exit.

Three mechanisms cover the three real situations:

Transient (vcp hiccups, node flakiness, OOM): per-rule retries: with attempt-scaled resources (mem_mb = base * attempt). A deterministic failure exhausts retries and fails.
Hard (real bug): the job exits nonzero. With keep-going: true, everything independent continues; only the failed downstream cone waits.
Below-floor-but-expected (too few stars on some CCDs): absorbed by :warn, recorded in the manifest. No separate class, no whitelist.

Rerun triggers are the v9 defaults minus input — {mtime, params, software-env, code} — so a changed config, param, or rule shell line reruns its cone natively; input is pruned as part of the cleanup design (D5).

sp report reads the index, the manifests (success detail), and the log: verdicts (failure detail) and emits the success/failure tables — per tile, per exposure, per CCD, with a failure-reason column — the successor to summary_tiles.py. It is a standalone verb, runnable at any time including mid-run, so it can describe failures rather than be blocked by them; onsuccess/onerror handlers call it so every invocation ends with one. Job-level statistics (runtimes, provenance) come from Snakemake's own --report. The verdict composes ok = (job exit code == 0) and (every mandatory runner ≥ its floor) — the exit code passed explicitly, since a runner can crash after its siblings have already met the floor. A failed stage leaves no manifest (Snakemake's default cleanup, which is the wanted semantics) and a complete log.

D4. Execution: the SLURM executor, one profile

Snakemake submits one SLURM job per rule (--executor slurm), each with its rule's exact resources — cpus_per_task = threads, mem_mb, runtime — and each wrapped in apptainer exec by the SDM. SLURM enforces limits per job, so attempt-scaled retries are real requests, and the cluster can backfill around our jobs, and we hold no idle cores. The profile's jobs: cap sits at the cluster's per-user submission limit (queried, not invented); snakemake feeds the queue as jobs finish, so DR6's ~170k total jobs never need to queue at once. Multi-node scaling is inherent. This mode ran end-to-end in a probe on nibi.

Two consequences to manage:

Short rules pay queue latency per job. Chunky rules (exp_psf, ngmix) amortize it; the minutes-short ones (get_images, uncompress, split, merges — all under ~3 min measured) fuse into their neighbours via group: labels in the rule files, grouping only rules with similar modest resource requests. At DR6 scale this is required, not optional: unfused, the campaign submits ~50k sub-3-minute jobs, and Alliance guidance asks for bundling below ~15 min.
The orchestrator is a light, long-lived process. The full DR6 DAG parses in ~11 min / ~4 GB, so snakemake runs under tmux on the login node or in a 1-core long-walltime shepherd job. Ctrl-C scancels outstanding jobs (executor behaviour); an orphaned run's jobs are one manual scancel.

threads == SMP fork width. A rule's threads must equal ShapePipe's internal fork width, or the SLURM request decouples from the true process count. Enforced mechanically: every rule passes -b {threads}, so the two are one number by construction. mem_mb = threads × measured per-worker footprint (ngmix ≈ 3–4 GB/worker after #843).

ngmix scatter/gather (the fix for the 13-hour straggler chunk):

Chunking uses Snakemake's native scattergather directive. Chunk count is a parse-time constant, overridable with --set-scatter; N=1 recovers one ngmix job per tile.
Each chunk's rule computes its ID range at execution time from the tile's own object count: equal shares, closed ranges, no open-ended final chunk (the old root cause: ID_OBJ_MAX=-1 on the last chunk). A params: function cannot do this — params evaluate before the sexcat exists. The range enters the committed template as $NGMIX_ID_MIN/$NGMIX_ID_MAX; ngmix_runner reads these fields with plain getint today, so a two-line change reads them env-expanded (verified: config.py expands only via getexpanded) — the second named module touch.
Chunks write nothing shared; each writes its own run_sp_tile_ngmix_Ng<k>u/, and tile_merge_cats — DAG-serialized after all chunks — is the gather. The per-chunk run-dir name comes from the committed template's RUN_NAME interpolating $SP_NGMIX_CHUNK; RUN_NAME was read with plain get (never env-expanded), so a one-word getexpanded change in run.py is a third, trivial module touch.
Thread/arena hygiene: MALLOC_ARENA_MAX=2, MALLOC_TRIM_THRESHOLD_=0, OMP/OPENBLAS/MKL_NUM_THREADS=1.

RNG. Chunking is only safe with SEED_FROM_POSITION=True (#796): each object's RNG derives from its own (ra, dec, ccd), so results are bit-identical under any chunking — N is a pure throughput knob and every rerun reproduces. It becomes the production mode; the tile-seed mode (one ordered stream across the tile, results dependent on chunk boundaries) is retired. Chunking is on by default: smaller per-job memory and disk high-water mark, better packing, N tuned against measured runtimes.

benchmark: on the heavy rules (ngmix, exp_psf) gives per-unit wall/RSS/CPU TSVs — the feed for the measured mem_mb values (ngmix's is measured; exp_psf's is pending). Benchmark files live outside the manifest directory, which cleanup deletes wholesale.

# profiles/nibi/config.yaml
executor: slurm
jobs: <per-user MaxSubmitJobs>       # query the cluster; do not invent
default-resources: {mem_mb: 2000, runtime: 2h, slurm_account: def-mjhudson_cpu}  # the association name, not the group name — sbatch rejects the latter
software-deployment-method: apptainer
apptainer-args: "--cleanenv --env OMP_NUM_THREADS=1 --env MALLOC_ARENA_MAX=2 --env PYTHONPATH=<prod worktree>/src --home /home/cdaley --bind /project --bind /scratch"
keep-going: true
show-failed-logs: true
rerun-incomplete: true
latency-wait: 60
rerun-triggers: [mtime, params, code, software-env]   # input pruned — D5

Snakemake wraps every job in apptainer exec itself; the workflow never calls apptainer. Rules point container: at the local /project .sif, never docker:// — no registry pull mid-campaign. This retires the hand-rolled in_container machinery.

The apptainer-args line carries the proven container recipe: $HOME for the CADC cert, thread/arena caps (one openblas pool per worker), and a PYTHONPATH pin at a frozen commit; production rebuilds the sif at the validated commit and drops the pin. The committed launcher (workflow/bin/sp) runs module load apptainer before exec-ing snakemake. Profile values are defaults — any of them can be overridden on the command line.

D5. Storage and cleanup

Storage placement. Two roots, both workflow/config.yaml keys. products_dir is the persistent root — the final catalogues, the index and report, the star-catalogue sky store; run_dir is the scratch root for bulk intermediates, each batch sized to finish inside the purge window so purge never deletes what a rerun would need. products_dir defaults to run_dir, so a fixture or a smoke test still needs exactly one path. What decides the split is not size. The sky store is durable because it is bought with catalogue-server queries — ~1.5k at DR6 — the one product local compute cannot re-buy. final_cat is durable because it is also the tile-finished marker the exposure-edge cut keys on (below): on scratch a purge would not merely lose catalogues, it would make finished tiles re-declare inputs against exposure stores reclamation had already deleted — the rerun avalanche the cut exists to prevent, arriving by another door. Snakemake's own state is the counter-example and stays on scratch: one small, hot metadata file per output, rewritten every job, >170k files of churn at DR6 against a backed-up and inode-limited filesystem — and reconstructible, since losing it costs a re-parse, not a rerun.

Disk economy. Un-cleaned intermediates measure ~34 GB/tile — at DR6 (~7300 tiles), beyond any scratch allocation — so intermediates are cleaned up during the campaign. Two tiers, split by who sees each product's full consumer set:

Intra-tile intermediates → temp(). Their consumer set closes inside one invocation, so native temp() deletes them when the last reader finishes. --notemp keeps them for debugging. Mechanically, the two bulk stores (the vignette store, the ngmix chunk dirs) ride as temp(directory()) secondary outputs alongside their manifests — the one scoped exception to "no directory() outputs"; the manifest stays the DAG edge, the directory exists purely so temp() fires at the right moment.
Cross-tile exposure store → in-DAG clean_exposure rules. The index lists every consuming tile in the campaign, so one wildcarded clean_exposure rule — one job per exposure, inputs generated from the index — declares every consumer's vignette output as input:, deletes the exposure's intermediates, and leaves a tombstone. Vignetting is the last stage that reads exposure products; everything after it reads tile-level files. Writer, then readers, then cleaner — DAG-ordered, race-free, rolling mid-run while ngmix still grinds; only exposures at the processing frontier stay pinned. It is temp() semantics applied to the campaign's consumer set instead of one invocation's.

A clean: config flag gates the clean rules (parsed with strtobool — any non-empty string is truthy in Python, so a naive parse would read --config clean=false as on); flipping it later cleans up retroactively, since the missing tombstones schedule exactly the clean jobs.
A tile appended after its exposures were cleaned reschedules those chains — the accepted cost of a late append, and it stays bounded to the appended tile (below).
clean_ignore_tiles: drops a permanently-failed tile from the consumer sets, so it stops pinning its ~8 exposures for the life of the campaign.

Cleanup without rerun cascades. Cleaning an exposure deletes its manifests — files every consuming tile declares as input:.
To a file-based DAG, a missing input means a pending producer: on any later invocation (a resume, a retry, an append), every finished tile sharing the exposure would rebuild its chain, and the rerun would propagate through the exposure-sharing graph at ~8 tiles per exposure.
This is unavoidable in kind, not a timing bug — the deletion is always legitimate under the consumer-set rule, but file state alone cannot distinguish a deliberate deletion from missing work.
DES and Rubin resolved the same problem by moving the source of truth into a database (DESDM, the Butler registry); here the index and the tombstones carry that record, and three input declarations keep the DAG quiet:

A tile whose final_cat exists drops non-existent exposure manifests from its input list — ancient() cannot cover those, since it only suppresses staleness for files that exist — and wraps the rest in ancient().
clean_exposure declares only in-scope consumers' manifests; out-of-scope consumers are existence-checked at parse time instead of declared.
The profile prunes input from rerun-triggers ([mtime, params, code, software-env]), because a conditional input list otherwise reads as "the input set changed". Content changes still propagate through mtimes and params.

Measured on a four-tile fixture, retrying one damaged tile: 82 jobs scheduled without these declarations, 28 with them — exactly the damaged tile's own chain.
One accepted cost: --forcerun on a finished tile will not rebuild its cleaned exposures, because those edges are no longer in the DAG; delete its final_cat first.

D6. Repo layout and migration

workflow/
  Snakefile              # parse-time index load; onsuccess/onerror → sp report; includes rule files
  rules/{prepare,exposure,tile}.smk
  scripts/{build_index.py, build_forest.py, run_report.py, completeness.py}
  bin/sp                 # launcher: module load apptainer → snakemake --profile profiles/nibi
profiles/nibi/config.yaml

Pin snakemake>=9,<10 plus snakemake-executor-plugin-slurm. The v8→v9 breaks are minor here: --use-singularity → --sdm, executors became plugins, full rerun-triggers is the default.

The line-count check: four small scripts (index, forest, report, completeness table) plus committed configs replace ~2000 lines across three bash levels, each site's sbatch wrapper, and the run-log machinery. Less code, in one place, testable, with the DAG delegated to a tool whose whole job is DAGs. If an implementation PR does not deliver that, this design has failed.

Migration runs in phases, each independently useful, bash layers untouched until Martin retires them (validation phases use their own output tree, never sharing exp/ with a bash-driven run):

Prepare + exposure stages on a 4-tile P3 subset. Delivered — see the results comment below.
Full tile chain through final_cat; retire job_p3_batch1.sh for new runs. Validation was a one-time acceptance test against the bash baseline at final_cat — join on object ID, science columns within tolerances, never bit parity (the RNG mode deliberately changes, D4). Parity is retired as a gate. It caught none of the orchestration defects this branch actually had (those came from dry-runs, job-count fixtures, count floors and review), it certifies nothing about defects the two sides share, and the baseline is drifted, unvalidated code. What replaces it is cheaper and stronger: self-reproducibility — one exposure chain and one tile chain run twice into separate roots and byte-diffed — which fix(psf): deterministic star train/validation split; align tile STAR_THRESH with the 80% convention #873's seeded star split makes possible for the first time, plus directed assertions inside the workflow (unit-ID checks on outputs, the declared-versus-header exposure set) for the routing failures reproducibility cannot see.
Scale: batched invocations sized to scratch purge windows (50-tile → P3 → ~7300 tiles); exercise the count-floor, retries, rerun propagation, and rolling clean_exposure cleanup for real; tune jobs: and group: fusion against measured queue behaviour.

Out of scope for the first pass: image_sims parity (a named follow-up — the sims im_pipeline step will call these same per-unit rules, converging the two pipelines on one workflow).

Round 2 posted — the body above has been rewritten in place (round 1 is preserved in the edit history). What happened between rounds:

Cail's feedback on round 1, whose through-line is simplify: one execution mode (the one-big-allocation), one profile, Snakemake owns the containers (--sdm apptainer — nothing in the workflow ever calls apptainer itself), full default rerun-triggers, a single run-level report, snakemake>=9,<10, CANFAR deferred to a future profile.
A second research pass that overturned two round-1 assumptions: nibi compute nodes do have outbound internet (verified empirically from a compute node), so downloads run in-DAG; and the failure-handling machinery collapsed to the count-floor that the bash complete_check already encodes — the three-class taxonomy, whitelist, and verdict-classifier are gone. Per-CCD stages now declare deterministic directory() outputs (RUN_DATETIME=False per-unit), with an in-job per-runner count-floor from a committed expected_counts.yaml.
A second four-lens adversarial review of the revised design (30 raw findings → 22 verified, 4 blockers), all folded in. The blockers were seam bugs between Snakemake's state model and ShapePipe's: timestamped run dirs vs. declared outputs, the apptainer SDM silently dropping the proven container environment (--home, thread caps), and temp() on shared exposure products destroying the "finished tiles stay finished" invariant on the workflow's most routine operation (appending tiles). That last one is why cleanup is now an explicit index-driven verb instead of temp() on the exposure store.
Empirical grounding on nibi: the prepare phase already runs green on a 4-tile P3 subset (deterministic run dirs, run_index.sqlite with 4 tiles / 30 exposures), and both execution paths were probed live (the one-allocation container mechanics, and a slurm-executor probe job that completed end-to-end — kept as a deferred scaling option).

A draft PR with the implementation skeleton follows shortly; this issue stays the living PRD and will be updated as implementation teaches us things. The open questions at the foot are still the discussion agenda.

— Claude (Fable) on behalf of Cail

🤖 Generated with Claude Code

What the first real runs taught (P0 night 1)

The design has now met real data on nibi and delivered. Every compute attempt failed on a genuine defect, each one at the seam between Snakemake's state model and ShapePipe's directory-as-database bookkeeping — never a design error, always a concrete v2.0 mechanism the abstraction hadn't yet been told about. The recovery run then drove the full tile chain through all eight ngmix chunks on every tile: job 17808445 COMPLETED in 3h47, four final_cats on disk, and all four tiles PASS scientific-equivalence validation against the bash baseline. This is what the runs taught, organized by the design section each learning touches.

P0 delivered — four final_cats, validation PASS

Job 17808445 COMPLETED (3h47): 4/4 final_cats on disk (46/42/47/44 MB, 173 columns). Tile 210.296 = 34,321 rows — exactly its SExtractor object count: the closed-range ngmix chunking covered every object once, no straggler drop, no double-count.
run_report.json: 4 tiles / 19 exposures, no missing tiles, zero failures at exposure granularity. (Attrition granularity caveat in D3.)
Append-invariant test PASSED. Appending the 196 quad to the tile list and dry-running scheduled exactly 65 jobs, all for the new tiles (every rule count = 4, ngmix 32 = 4×8) and zero for the finished quad; no exposure jobs until the index is rebuilt (parse-time index doing its job). Run under --rerun-triggers mtime — the first default-trigger invocation would storm the DAG on the script-hash param change (metadata reconciliation, D3).
Smoke gates green in-job: shapepipe.__file__ = pinned worktree src in both driver logs; OMP pool 32→1; ulimit -u = 3.09 M (no nproc ceiling).
Validation: all four tiles PASS scientific equivalence to the p3-batch1 bash baseline — table and the RNG caveat in the Validation section below. (The caveat was originally attributed to PSFEx; corrected 2026-08-29 to the unseeded star split.)

D1 — Staged execution / the index

The index must carry the exposure's original name, not just the dedup key. The exposure store keeps CFHT's p suffix (2086319p), while the DAG keys exposures by bare base id. The base id is the right dedup key, but get_images needs the original name: the index carries both, and the wrapper writes the original name verbatim into the fabricated per-unit exp_numbers list. This confirms the posted design's own note ("carries both bare exposure ids and original names, 2605805p") — it's now load-bearing in code, not just prose, and the write must be unconditional (an exists-guard had pinned a stale pre-fix copy of the list file across a rerun).

Structural dedup is real and observed. prepare came up green on the 210/211 overlap quad: 34 tile→exposure edges collapsed to 19 unique exposure chains, each scheduled exactly once. The dedup-within-an-invocation contract holds against actual overlap geometry.

D2 — Wrapper fabrication, forest, star-cats, count flavors

Four D2 mechanisms needed correction against the running modules:

The exposure forest must be sharded, not flat. exp_utils.get_exp_output_files hardwires the v2.0 sharded layout (forest/<2-digit-prefix>/<base>/output) into its $SP_EXP glob, so a flat forest fails every tile gather stage. The design's flat-store decision remains correct — nothing globs the store, the index drives every path — but the module-facing view the forest presents must carry the shard. This is a refinement of the posted text, not a contradiction: the store is flat, the forest view is sharded.
Star cats are consumed as directories, not per-unit files. They're per-CCD directories (~40/exposure) reached via $SP_RUN/star_cat_{exp,tiles} dir symlinks — the v2.0 mechanism the wrapper already promises to reproduce. The per-unit star-cat file rules I had sketched linked names that don't exist and nothing read them; they're deleted. The wrapper's dir-symlink step (finding 11b) is the whole mechanism.
get_images completeness counts are config-flavor-dependent. The nibi symlink flavor produces tile 2 / exp 3 files, versus the canfar vos table's 4 / 6. The committed count table now follows the actual configs in use, verified against the p3-batch1 baseline (100 files / 50 tiles). This directly touches the D3 count table (see below).
Shell-string placeholders don't reach into params:. {threads}/{output} survive literally into the process if they sit inside a params: value — Snakemake formats the shell: string once. Placeholders must live in shell:. This is a wrapper-plumbing fix, but it validates the D4 insistence that SMP_BATCH_SIZE={threads} be written by the wrapper into the config copy, not passed as a shell placeholder.

D3 — Failure policy, observed live + the quota postmortem

The count-floor works, watched in real job exits. attempt 3's log shows both OK lines (above floor → exit 0) and FATAL breaches (below floor → nonzero) — the P0 failure-policy bullet now has live evidence, not just a ported table.

The report counter needs to be per-CCD, not per-exposure. Validation exposed a granularity gap in run_report: its "zero attrition" is exposure-granularity only. The real signal lives per-CCD — psfex_interp (validation_psf) attrition is 46/760 CCDs = 6.05% (smk) vs 41/760 = 5.39% (bash), ~30× the 0.2% lore figure, on both sides. The run is healthy; the counter is just aggregated one level too coarse to show where the loss happens. This is the DES-blacklist granularity exactly (per-CCD, not per-exposure), so the report should count at that grain. FOLLOW-UP: make the run_report attrition counter per-CCD.

The DES/Rubin precedent is citable. The count-floor + standalone report is the field's converged pattern, and I've verified the citations: DES's per-CCD PSF blacklist (Y3: <25 stars → blacklisted, ~2% of data removed, survey proceeds — Jarvis et al. 2021 §4.2; also 2016, Y6 2025) is our sparse-CCD setools case almost literally; Rubin's Butler quantum registry + pipetask report (Jenness 2022, Lust 2023) is the same "record per-unit, gate on a floor, report and continue" shape at survey scale. Both avoid a workflow engine's per-file targets; Snakemake forces the same choice (#4016, no optional-output primitive). Worth adding these citations to the D3 precedent sentence.

Quota-kill postmortem — two independent failure layers. attempt 3 (job 17798771) computed the full chain of tile 210.296 — all 8 ngmix chunks passed with in-job closed ID ranges (n_obj=34321 → chunk 8 = [30031, 34321], no open-ended last chunk: the straggler fix working) — then died writing Snakemake job metadata, because /project def-mjhudson had gone hard-full (27/27 TiB). Two layers:

The forest-staleness layer, which is a D3 rerun-trigger finding: three of four tiles FATAL'd at tile_merge_headers on stale flat forests from attempt 2. The forest rule never re-ran, because the code trigger tracks the shell: string but not the external sp_rule.py it invokes — exactly the tradeoff the design accepts to keep wrapper edits from invalidating the whole DAG. The fix folds a wrapper-script content hash into every rule's params:, closing the gap. The one-time cost: a params mismatch on every already-recorded job → a full-DAG storm at the next default-trigger invocation. So the recovery (attempt 4) runs --rerun-triggers mtime deliberately, with the 3 stale forests deleted by hand, and defers metadata reconciliation until after green.
The quota layer is a real-world instance of D6's storage story, below.

D5 — Profile env, state placement, execution mode

PYTHONPATH pin and container env held up — the smoke gate (attempt 1, job 17794661) confirmed the pinned src resolves and threadpools cap at 1, exactly as D5 specifies.

State placement had to move off /project under duress. The posted design places the Snakemake workdir and index_db on /project (so scratch purge can't break the mtime chain). /project going hard-full forced the opposite tonight: the working repo moved to /home/cdaley/shapepipe-smk (clone @ a273e763, pushed), bin/sp runs Snakemake with --directory <run_dir>-state on /scratch, the Snakefile resolves its configfile relative to itself, and index_db moved to scratch. This is a forced deviation, not a design change — but it means the D6 "durable products + .snakemake/ on /project" placement is currently blocked and should be flagged as such until group-level /project reclamation happens. The design intent stands; the environment doesn't currently permit it.

One-allocation mode delivered — attempt 4 = job 17808445, one deliberate invocation, COMPLETED in 3h47. Confirms the D4 single-allocation model in practice, start to four catalogues.

D6 — Bench numbers, storage reality, precedent

DAG-scale gate cleared, and it reframes P2. Synthetic index (realistic ~2.7 exp/tile overlap), snakemake -n, nibi login, Snakemake 9.23.1, parse-time dict-lookup input functions:

tiles	jobs	DAG build	peak RSS
250	6,689	14.6 s	238 MB
1,000	26,689	77 s	616 MB
2,000	53,357	163 s	1.1 GB
4,000	106,689	351 s	2.1 GB
DR6 (~7,300)	~195k	~11–12 min	~4 GB

Near-linear (~3.3 ms, ~20 KB per job). All-of-DR6 builds in ~11–12 min at ~4 GB — viable on a login node or allocation head. Consequence: the D5/D6 batching-for-scale rationale should say batching is an operational choice (walltime, storage), not a DAG-scale necessity. The group:-fusion escape hatch stays in reserve, unforced.

Storage: ~34 GB/tile of un-cleaned intermediates (smk-p0: 136 G/4 tiles; p3-batch1: 1.6 T/50 tiles). This is the concrete number behind the sp clean-exposures verb — at DR6 the intermediate footprint alone exceeds any single allocation's scratch, so index-driven cleanup isn't optional at scale. And it's exactly what bit us: /project hard-full (27/27 TiB) killed attempt 3's metadata write; scratch is over its soft quota (1693/1024 GiB) even after a 357 G reclaim tonight. The storage story in D6 is now measured, not projected.

Validation — scientific equivalence to the bash baseline

All four tiles PASS scientific equivalence to the p3-batch1 bash baseline (verdict: /home/cdaley/scratch-validation/validation_verdict-p0night1.json). Rows are exactly equal on every tile (100% NUMBER crossmatch), column sets are identical, and every deterministic (non-NGMIX) column is bit-identical: SExtractor 760/760 CCD cells exact, setools star counts exact (max delta 0), WCS headers max-diff 0.0 across all 19 exposures. Shape statistics agree — per-object g1/g2 correlation 0.964–0.989, g1/g2/T/SNR/FLUX medians within 1–2%, NGMIX_FLAGS_NOSHEAR==0 at a rate of exactly 1.00000 both sides, all tiles.

tile	rows	g-corr	T rel-Δ	SNR rel-Δ
210.296	34,321	0.988–0.989	0.4%	1.1%
210.297	30,637	0.964	0.2%	0.2%
211.296	35,037	0.983–0.984	0.2%	0.04%
211.297	32,712	0.987–0.988	0.9%	1.7%

Finding — the residual shape differences are our own unseeded RNG, not orchestration and not PSFEx. (Corrected 2026-08-29; the original attribution below was wrong. The follow-up it opened has been done and it falsifies the claim.) The g-correlations sit at 0.96–0.99 rather than 1.00 because SETools._make_rand_split drew the 80/20 PSF train/validation split from unseeded np.random, so the two sides trained PSFEx on different star samples. setools.py, psfex_runner.py, psfex_interp.py, default.psfex and the container are byte-identical across the two code pins, which leaves that split as the only nondeterminism in the path — there is no PSFEx config drift to attribute anything to.

The original reading does not survive scrutiny either. It rested on PSFEx LOADED matching 120/120, taken to mean PSFEx ingested identical stars. But LOADED is n_keep = ceil(cat_size × ratio) — a deterministic function of the input catalogue, invariant under which stars land in which half. ACCEPTED matched only 83/120, and that is the quantity the split moves. Same for the rest of the green column: star_split_ratio_{20,80} was compared by filename, and Nb stars is defined over the pre-split mask. Every quantity that agreed was one the RNG cannot touch.

The downstream numbers stand as measured — CCD 0 attrition on 6 exposures (in smk, not bash; the direction was reported backwards), propagating through NGMIX_N_EPOCH on 0.02–13.1% of objects and surfacing as a few 2–2.6σ paired g2/SNR offsets at the ~1e-3 level. Only the cause changes.

Two consequences, replacing the originals. (1) The PSFEx-config-parity follow-up is closed as answered: the configs were always identical. #873 seeds the split, so from that commit on the pipeline reproduces itself and this class of divergence is gone. (2) A real result was buried here: per-object T_corr was 0.669–0.756 while g1/g2_corr was 0.96–0.99. A different random split of the same stars decorrelates per-object size measurements to r≈0.72 and shifts the PSF model at galaxy positions by ~1%. Seeding freezes the coin flip; it does not remove the sensitivity, which makes it a DR6 systematic-budget term worth measuring on purpose.

Naming divergence to reconcile

The posted D3 text names the count table two ways: the wrapper's committed table is workflow/scripts/completeness.py (a stdlib-only Python table), while the standalone report is described as reading expected_counts.yaml (D3 final paragraph and the finding-8 report paragraph). Implementation settled on the Python table (completeness.py) as the single source both wrapper and report import in-container. The PRD should pick one name — completeness.py — and drop the expected_counts.yaml references so there's one table, imported, not a YAML the report re-reads.

PRD edits needed (checklist for tightening #848 tomorrow)

Status note for the issue: P0 is delivered — job 17808445 COMPLETED (3h47), four final_cats on disk, all four tiles PASS scientific-equivalence validation against the bash baseline, and the append-invariant regression passed. The open items are the four PRD edits' follow-ups (per-CCD attrition counter, PSFEx config parity — since closed as falsified, metadata reconciliation, /project reclamation — since done), none of which block the Gate-2 exit. This comment records what the runs proved.

— Claude (Fable) on behalf of Cail

🤖 Generated with Claude Code

Looks nice!

Comments:

Level 3 (curl_canfar_local.sh) this is obsolete and already no longer used (was used up to the skaha python canfar update in 2025).
Setting RUN_DATETIME=False is fine as long as Snakemake handles re-runs of failed modules (shapepipe would exit with an error that output dir already exists)
exposure index file: Is it flexible that exposures can be added if we run more tiles? E.g. do a test run of a few tiles, then add more for a larger run?
tiles/exposures/tile_exposures: I am still a bit worried that having all tiles (exposures) in one root dir will overload the file system. Even with index sql files we still want to type ls, find, wc, etc. Are or order 50k subdirectories ok? And what is tiles_exposures?
star_cat_exp, star catalogue for masking: This will be no longer used once healsparse masks replace ShapePipe-internal masking
config_merge_sep_cats.ini: running ngmix as parallel jobs on object batches and the merging in the next module has been removed for early v2.0 runs; this was fairly unstable and difficult to manage in v1.4. Make sure this is done properly in the new Snakemake setup.
I'd like to implement a global statistics of tiles and exposures, how many succeeded and how many failed, and for what reason. See previous script shapepipe/scripts/python/summary_tiles.py.
When running tiles in parallel, the code has to make sure that not more than one job accesses the same exposure; this should be already implemented in v2.0, but check.

Thanks Martin — great set of questions, several of them sharpened the design. Point by point:

Level 3 (curl_canfar_local.sh) obsolete. Good to know — removed from the PRD; the survey notes keep the history.

RUN_DATETIME=False and re-runs of failed modules. Handled, and more simply than P0's implementation: RUN_DATETIME=False is committed in the configs, so every stage's run dir is a deterministic path, declared as the rule's output — Snakemake itself deletes and recreates it before every re-run, so a retried job never sees "output dir already exists." (No wrapper needed here; more below.) Exercised for real — the P0 run took four attempts and every re-run went through cleanly.

Exposure index — adding tiles later. Yes, and it's tested on real data: after the 4-tile P0 run went green, we appended 196 tiles and re-invoked — Snakemake scheduled exactly the 65 jobs belonging to the new tiles, zero for the finished quad. The index rebuild is incremental by construction, and it's folded into the single sp run entry point, so "add tiles, run again" is the whole workflow.

Flat tiles/ and exp/ directories at ~50k entries. Agreed — the stores get 2-digit-prefix sharding (exp/<prefix>/<expnum>/, tiles/<prefix>/<tile-id>/), same idea as v2.0's layout. Snakemake never globs the stores (every path comes from the index), so the shard is purely for humans and the filesystem's metadata server; at full-UNIONS scale no directory exceeds ~1k entries, comfortably ls/find-able. We'll pick the prefix digits for uniformity against the real exposure list (sequential numbering can cluster; trailing digits if so).

tile_exposures is not a directory — it's the tile↔exposure join table in the sqlite index (each tile → its ~7–10 covering exposures). Renamed nothing on disk; the PRD will say "DB table" explicitly.

star_cat_exp obsolete once healsparse masks land. Turned out to be simpler than that: the committed on-the-fly mask configs (USE_EXT_STAR = False) never read the star-cat rules or symlinks at all — they're unused in the current chain. Dropped from the design outright rather than kept transitional; nothing depended on them. An external-star variant, if it's ever wanted, is a single INPUT_DIR pointing at the shared cat pool — no per-unit plumbing.

ngmix parallel chunks + merge. Reinstated using Snakemake's native scattergather directive, with the v1.4 failure modes removed: chunks compute closed ID ranges in-job from the tile's sexcat (no open-ended last chunk) and write nothing shared — the merge is a plain gather and the single sync point; a failed chunk retries alone. Chunks still parallelize internally via SMP, and chunk count is a runtime knob (--set-scatter; N=1 recovers one ngmix job per tile). Validated row-exact against the unchunked bash baseline on all 4 P0 tiles.

Global tile/exposure statistics (summary_tiles.py). Job-level stats (ran/succeeded/failed, runtimes) come free from Snakemake's metadata and --report. Below job level, each job writes a small manifest file (sub-units present/missing, with reasons parsed from the module logs), and sp report aggregates manifests into the summary_tiles.py-style table — per-tile, per-exposure, and per-CCD, with a failure-reason column.

Same-exposure concurrency. This stops being a locking problem entirely: each exposure's module chain is a single node in the DAG (one job per exposure, no matter how many tiles need it), and tiles consume its products as declared input: — writer-before-readers by construction, no lock needed. The DAG edge is the lock. Observed in P0: 34 tile→exposure needs deduplicated to 19 exposure chains, each built exactly once.

— Claude (Fable) on behalf of Cail

Some more comments:

Can we make sure that the workflow of running sp on data and simulations is still as consistent as possible, using a maximum of the same machinery? That was the idea of a single run script run_job_sp_canfar_v2.0.bash, which I believe is now no longer used. That's fine if this can be done with overlapping snakemake workflow scripts. Maybe in a future PR?
cleaning up exposure run dirs: We might not want to do this in case more tile IDs are added to an ongoing run, in which case deleted exposure files need to be re-run.

@martinkilbinger

@martinkilbinger The design above went through a careful rework with Cail — simpler in several places, and it stands on its own now. We are starting to implement it. Comments remain welcome, and the design will keep being revised as they land.

— Claude (Fable) on behalf of Cail

feat(ngmix): env-expanded chunk fields; position-seeded RNG becomes t…

feat(orchestration): committed workflow configs — no wrapper, no runt…

feat(orchestration): manifests become the DAG's currency — completene…

feat(orchestration): the rules rewrite — manifests as DAG currency, s…

feat(orchestration): clean_exposure becomes an in-DAG rule with tombs…

feat(orchestration): nibi profile goes slurm-executor (#848 D4, S6)

fix(orchestration): stop the clean_exposure cascade; report reads tom…

fix(configs): the workflow's own PiViVi config takes STAR_THRESH 22 too

Navigation Menu

Repository navigation

Snakemake orchestration for shapepipe #848

Snakemake orchestration for the real-data pipeline

Summary

Pipeline shape

Design

D1. Execution model: `sp run` is the entry point

D2. Rules: one `shapepipe_run` per unit, one manifest out

D3. Failure handling: count floors

D4. Execution: the SLURM executor, one profile

D5. Storage and cleanup

D6. Repo layout and migration

What the first real runs taught (P0 night 1)

P0 delivered — four final_cats, validation PASS

D1 — Staged execution / the index

D2 — Wrapper fabrication, forest, star-cats, count flavors

D3 — Failure policy, observed live + the quota postmortem

D5 — Profile env, state placement, execution mode

D6 — Bench numbers, storage reality, precedent

Validation — scientific equivalence to the bash baseline

Naming divergence to reconcile

PRD edits needed (checklist for tightening #848 tomorrow)

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Snakemake orchestration for shapepipe #848

Description

Snakemake orchestration for the real-data pipeline

Summary

Pipeline shape

Design

D1. Execution model: sp run is the entry point

D2. Rules: one shapepipe_run per unit, one manifest out

D3. Failure handling: count floors

D4. Execution: the SLURM executor, one profile

D5. Storage and cleanup

D6. Repo layout and migration

Activity

cailmdaley commented on Jul 16, 2026

cailmdaley commented on Jul 17, 2026

What the first real runs taught (P0 night 1)

P0 delivered — four final_cats, validation PASS

D1 — Staged execution / the index

D2 — Wrapper fabrication, forest, star-cats, count flavors

D3 — Failure policy, observed live + the quota postmortem

D5 — Profile env, state placement, execution mode

D6 — Bench numbers, storage reality, precedent

Validation — scientific equivalence to the bash baseline

Naming divergence to reconcile

PRD edits needed (checklist for tightening #848 tomorrow)

martinkilbinger commented on Jul 17, 2026

cailmdaley commented on Jul 21, 2026

martinkilbinger commented on Jul 30, 2026

cailmdaley commented on Jul 31, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

D1. Execution model: `sp run` is the entry point

D2. Rules: one `shapepipe_run` per unit, one manifest out