Skip to content

Design docs: the drift PR #58 missed, and a Grid re-walk now that realignment shipped - #59

Merged
JamesKane merged 2 commits into
mainfrom
chore/design-drift-followups-2
Aug 24, 2026
Merged

Design docs: the drift PR #58 missed, and a Grid re-walk now that realignment shipped#59
JamesKane merged 2 commits into
mainfrom
chore/design-drift-followups-2

Conversation

@JamesKane

Copy link
Copy Markdown
Owner

Documentation only — no code changed, so nothing here alters behavior.

Two threads, in two commits.

1. The drift the last audit missed

PR #58 audited all 32 design docs and corrected five headers. A pass the following
day found three more claims it re-affirmed in place without checking. All three run
the same direction as PR #58's findings — the doc claims less than the tree has.

Doc Said Actually
RustRewrite_Plan.md §4e "not yet built" Built 04728ba, 2026-06-03 — two days after the plan was drafted
project-block-tree.md §11.4 + BACKLOG.md §3.3 "no GUI trigger for the private-Y batch" The chore table, 331e8cb (PR #47)
project-block-tree.md §11 + BACKLOG.md project_report debt unpaid c0994d0, the day before

The §4e case is the interesting one: the section's body already carried the
measured result
(13.9 Mb callable chrY short-read vs 1.1 Mb HiFi, the three
private-Y bucket sizes). Only the sub-heading and the "Missing" list contradicted it.
The section disagreed with itself, which is how two consecutive audits read past it.
A status marker that sits away from the evidence is the thing that goes stale.

PR #47 is also the second confirmed instance of a substantial PR shipping and
touching no design doc at all — PR #42 was the first, found by the last audit.

Four docs carried no status header. Three are Scala-era and were missed by PR #58
because they live in documents/, not documents/design/. The fourth,
DecodingUsTreeProvider.md, had no status line of any kind despite being fully
implemented with DecodingUs as the default provider.

2. The Grid re-walk

distributed-compute-grid.md was drafted while realignment was still a plan, and
depended on it. Realignment merged 2026-08-14 — which unblocked the Grid and
overtook one of its four locked decisions. New §11 is the reconnaissance
refresh; D1 and §7.3 are corrected in place, because a locked decision the tree
has already overruled otherwise reads as still binding.

The design's shape survives intact. The cost moved down:

  • D1 locked minimap2 via FFI. navigator-align ships minimap2-pure-rs — pure
    Rust, no C toolchain, 99.74 % byte-identical, zero disagreements at MAPQ > 0. §7.3
    split the fleet by OS on that assumption. Windows nodes can take FASTQ units on day
    one; /grid/claim should filter on RAM/disk/threads; the P4 Windows milestone is gone.
  • §7.1's "prerequisite refactor" is mostly builtApp::analyze_biosample from
    PR Make batch analysis tractable, and fix the Y re-placement bugs it uncovered #47 is already the headless, cancellable, resumable, preflighted per-unit driver.
  • mt_terminal is a contradiction, not a gap. The digest demands an exact match;
    analyze_biosample declines to assign mtDNA because it "is not final on CHM13" —
    and the Grid realigns to CHM13. §11.3 gives three ways out and recommends one.
    This is the decision to settle before any P1 code.
  • The AppView already resolves ENA at run level and curates it
    EnaClient::run_files + du-jobs/crawl_project.rs, already splitting BAM/CRAM from
    index files, which is D3's passthrough selector.

Two traps recorded: SV sat in the §7.2 driver at 2–5 h per WGS (vs ~1 h for the whole
rest of the stack) for a signal the digest never compares; and ena.rs cannot mirror
refgenome::download, which sends no Range and hashes SHA-256 where ENA publishes
md5 on multi-GB files.

Net: the critical path is the AppView halfgrid.work_unit, the SKIP LOCKED
claim, the validator.

Checks

  • scripts/ste-check.py (no arguments, per the known gotcha): 0 files with violations
  • No Rust touched, so fmt/clippy/test are unaffected
  • Every claim verified by grep against the tree, not by reading a status header

🤖 Generated with Claude Code

https://claude.ai/code/session_01SequryftjcFkLnjQRMKCx3

JamesKane and others added 2 commits August 24, 2026 13:04
…cs with no header at all

A pass over all 38 design documents, one day after the PR #58 audit, looking for
anything that pass re-affirmed in place without checking. Three claims were wrong,
all three in the same direction as PR #58's: the doc claims *less* than the tree has.

- `RustRewrite_Plan.md` §4e said "not yet built". It was built on 2026-06-03
  (`04728ba`), two days after the plan was drafted. The section's *body* already
  carried the measured result — 13.9 Mb callable chrY short-read vs 1.1 Mb HiFi, the
  three private-Y bucket sizes — so the section disagreed with itself, which is how
  two audits read past it. The sub-heading and the "Missing" list are corrected, and
  the one genuinely unbuilt piece (the BED is never persisted as a per-alignment
  artifact) is now stated where a reader will find it.

- `project-block-tree.md` §11.4 and `BACKLOG.md` §3.3 both said the private-Y batch
  has no GUI trigger. The workspace chore table (`maintenance.rs` +
  `central.rs::maintenance_section`) gave all three CLI-only chores one shared
  surface in `331e8cb` (PR #47) — which is exactly the "one answer, not a third
  separate button" §11.4 asked for. Like PR #42 before it, that PR touched no design
  doc at all; this is now the second confirmed instance of the pattern.

- `project-block-tree.md` §11 and `BACKLOG.md` still listed the `project_report`
  artifact-payload debt. `c0994d0` paid it the day before.

BACKLOG §1.2 is re-verified genuinely open — `genotype_panel_for_subject` still has
exactly two callers — and now names the chore table as its obvious home.

Four documents carried no status header. Three are Scala-era and were missed by PR
#58 because they live in `documents/` rather than `documents/design/`; each now says
what survived into Rust and where the current design lives. The fourth,
`DecodingUsTreeProvider.md`, had no status line of any kind despite being fully
implemented with DecodingUs as the default provider and all three of its "Decisions
to confirm" resolved.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SequryftjcFkLnjQRMKCx3
…ent has shipped

The Grid was drafted while realignment was still a plan, and it depended on that
module. Realignment merged 2026-08-14, which unblocked the Grid and, in the same
stroke, overtook one of its four locked decisions. §11 is the reconnaissance
refresh; D1 and §7.3 are corrected in place rather than annotated, because a locked
decision the tree has already overruled will otherwise be read as still binding.

The design's shape survives intact — staged phases, adaptive replication, digest
comparison, the lease state machine. What moved is the cost, and it moved down.

- D1 locked minimap2 through the `minimap2-rs` FFI. `navigator-align` ships
  `minimap2-pure-rs` instead: pure Rust, no C toolchain, 99.74 % byte-identical with
  zero disagreements at MAPQ > 0. §7.3 built a two-tier fleet on that FFI assumption
  — macOS+Linux realign, Windows passthrough-only until a spike lands — and none of
  it is true. Windows nodes take FASTQ units the day P2 opens, `/grid/claim` should
  filter on RAM/disk/threads rather than OS, and the P4 Windows milestone is gone.

- §7.1 asked someone to lift `run_full_analysis_streaming` out of the UI worker.
  `App::analyze_biosample` already is that: PR #47 built it for batch analysis,
  headless, cancellable mid-sample, resumable, preflighted before any I/O.

- It deliberately omits mtDNA, because that value "is not final on CHM13" — and the
  digest in §5.2 wants an exact `mt_terminal` match while the Grid realigns *to*
  CHM13. That is a contradiction, not a wiring gap, and it is the one decision worth
  settling before P1 code. §11.3 lays out three ways out and recommends the cheapest.

- §2 said the AppView pulls ENA study metadata only. `EnaClient::run_files` resolves
  the run-level filereport with precisely the columns a work unit needs, and
  `du-jobs/crawl_project.rs` already groups runs by sample and materializes files,
  splitting BAM/CRAM from the index files — which is D3's passthrough selector.

Two traps recorded: SV was in the §7.2 driver at 2–5 h per WGS against ~1 h for the
whole rest of the stack, for a signal the digest never compares (removed); and
`ena.rs` cannot mirror `refgenome::download`, which sends no `Range` and hashes
SHA-256 where ENA publishes md5 on multi-GB files.

Still greenfield, re-verified: the `grid` schema (`0075`, not the `0059` guessed),
the claim SQL, `Provenance`, subject≠contributor, credit, the validator. The
`fed.pds_*` scaffold is still dormant. The critical path is the AppView half.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SequryftjcFkLnjQRMKCx3
@JamesKane
JamesKane merged commit fa15569 into main Aug 24, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant