Design-doc audit: correct five stale status headers, and the two one-file fixes it surfaced

crates/navigator-analysis/src/archaic_match.rs

@@ -103,7 +103,10 @@
 //! **But the reported extent puts the two populations in the wrong order.** The truth puts the
 //! archaic extent of East Asia at **1.217x** that of Europe. The extent that this caller reports
 //! is **0.937x**. A user would read that an East Asian carries *less* archaic ancestry than a
-//! European. That is the wrong way round, and it is the one reason that this module stays gated.
+//! European. That is the wrong way round.
+//!
+//! This is the reason that the shipped report is a **within-population** measure, and the reason
+//! that the UI states the limit under the number. See *Where this landed* at the end.
 //!
 //! Here is the cause. The reported extent is the true positives *plus* the false positives. The
 //! load of false positives depends on the population, at a precision of 32.2 % against 41.9 %.
@@ -176,14 +179,40 @@
 //! number that you can compare across populations is not possible this way at present.** The
 //! caller is defensible inside one population, and not between two.
 //!
-//! **This is still not enough to turn the module on.** Beyond the order of the populations, there
-//! are three more reasons.
+//! ## What this left open, and what answered it
+//!
+//! The module was still off at this point. Three things counted against it. Two of them now have
+//! an answer, and the third stands.
+//!
+//! **The cohort was chr21 and chr22 alone.** The genome-wide run answers this. Read *Across the
+//! genome* above: three Europeans, all 22 autosomes, sensitivity of 40 to 43 % and precision of
+//! about 46 %. Both are better than the two-chromosome figures. So those figures are careful, and
+//! the caller keeps its accuracy outside the two chromosomes that the fit used.
 //!
-//! The precision is 34.9 % without the filter, on held-out Europeans. The cohort is **chr21 and
-//! chr22 alone**. And the reference callset itself has weak support: the tracts of hmmix show an
+//! **The precision was 34.9 % without the filter.** The path that ships applies the filter. The
+//! app calls [`filter_by_concordance`] at [`MIN_CONCORDANCE`] after [`call_from_observations`],
+//! and that configuration measures 90 %.
+//!
+//! **The reference callset has weak support.** This one stands. The tracts of hmmix show an
 //! enrichment of only 1.84x for their own archaic SNPs. Agreement with that callset then stops
 //! well below 100 %, even for a caller that is correct. F1 alone can not tell you when this work
 //! reaches its end.
+//!
+//! ## Where this landed
+//!
+//! **The module is on.** `ARCHAIC_SEGMENTS_ENABLED` in `navigator-app` is `true`, and Tier B
+//! shipped in `v0.1.0-alpha.15`. The sections above read as a record of the work, and they stop
+//! before that decision, so read this one for the result.
+//!
+//! It reports a **within-population** measure, for the reason that *A concordance filter* gives
+//! above. The UI states that limit under the number, and gives no universal percentage. **You
+//! must not use this number to compare people of different ancestries.**
+//!
+//! Tier A is a different measure, and its rule does not change. Tier A reports a **count** of
+//! marker copies, and never a percentage.
+//!
+//! § 11 of `documents/design/ArchaicAncestry_Design.md` records the same history for a reader who
+//! starts from the design. This doc-comment stays the detailed record.
 
 use std::collections::BTreeMap;
 

crates/navigator-analysis/src/phasing.rs

@@ -17,8 +17,9 @@
 //! beam width `B`, and the candidate count `M`.
 //!
 //! The [`Phaser`] trait holds the seam steady. Two other phasers can then go in, and no caller
-//! changes. One is a Mendelian [`TrioPhaser`], for a workspace that holds a parent sample. The
-//! other is a full PBWT phaser.
+//! changes. One is a Mendelian trio phaser, for a workspace that holds a parent sample. The other
+//! is a full PBWT phaser. **Nobody has written either one.** [`ReferencePhaser`] is the only
+//! implementation today, and the beam search above is what stands in for the PBWT.
 
 use std::collections::HashMap;
 
@@ -49,8 +50,8 @@ pub struct PhasedGenotypes {
     pub sites: Vec<PhasedSite>,
 }
 
-/// The phasing strategy. Reference-based statistical phasing by default; a Mendelian trio phaser
-/// when a parent is available (see [`TrioPhaser`]).
+/// The phasing strategy. [`ReferencePhaser`] is the only implementation. A Mendelian trio phaser,
+/// for a workspace that holds a parent sample, goes behind this same trait when someone writes it.
 pub trait Phaser {
     /// Phase the sample's genotypes into two consistent parental sides.
     fn phase(&self, genotypes: &[SiteGenotype]) -> PhasedGenotypes;

crates/navigator-app/src/queries.rs

@@ -11,7 +11,12 @@ use super::*;
 ///
 /// A project report reads five kinds of artifact for each alignment of each member. So the earlier
 /// form, one read for each cell, sent thousands of queries to open one tab. This type reads each
-/// artifact in one `IN` query and stats each BAM file one time.
+/// wanted kind in one `IN` query and stats each BAM file one time.
+///
+/// The load asks for the kinds that the caller names, and for no other kind. A read of every kind
+/// is not correct here. Some payloads are very large. A `tree-genotype` row runs to megabytes, and
+/// a cohort of them is gigabytes of JSON that no report builder reads. One narrow query for each
+/// kind costs less than one wide query, because the wide query selects every payload.
 ///
 /// The rule for an old result is the rule of [`App::load_analysis`]. A cached payload is absent when
 /// the `mtime:size` value of the source file changed after the calculation.
@@ -25,13 +30,16 @@ struct AlignmentArtifacts {
 }
 
 impl AlignmentArtifacts {
-    async fn load(store: &Store, alignments: &[&Alignment]) -> Result<Self, AppError> {
+    /// `kinds` holds the `(kind, algorithm version)` pairs to read. A pair that the caller does not
+    /// name is absent from the result, and [`Self::raw`] gives `None` for it.
+    async fn load(store: &Store, alignments: &[&Alignment], kinds: &[(&str, &str)]) -> Result<Self, AppError> {
         let ids: Vec<i64> = alignments.iter().map(|a| a.id).collect();
-        let by_key = artifact::list_for_alignments(store.pool(), &ids)
-            .await?
-            .into_iter()
-            .map(|a| ((a.alignment_id, a.kind.clone(), a.algorithm_version.clone()), a))
-            .collect();
+        let mut by_key = HashMap::new();
+        for (kind, version) in kinds {
+            for a in artifact::list_for_alignments_of_kind(store.pool(), &ids, kind, version).await? {
+                by_key.insert((a.alignment_id, a.kind.clone(), a.algorithm_version.clone()), a);
+            }
+        }
         // One stat call for each alignment, from the row that the code already holds. The code
         // does not make one stat call for each artifact.
         let sigs = alignments
@@ -72,6 +80,18 @@ impl AlignmentArtifacts {
     }
 }
 
+/// Each `(kind, algorithm version)` pair that [`App::project_report`] reads.
+///
+/// Keep this list and the body of that method together. A cell that reads a kind which is absent
+/// here gets `None` at all times. The column then stays empty for each member of the project.
+const PROJECT_REPORT_KINDS: &[(&str, &str)] = &[
+    ("coverage", coverage::COVERAGE_VERSION),
+    ("sex", "1"),
+    ("read_metrics", "1"),
+    ("sv", "1"),
+    (ERROR_KIND, ERROR_VERSION),
+];
+
 impl App {
     // ---- queries -----------------------------------------------------------
 
@@ -570,7 +590,7 @@ impl App {
             by_subject.entry(guid).or_default().push(aln);
         }
         let all_alignments: Vec<&Alignment> = by_subject.values().flatten().collect();
-        let artifacts = AlignmentArtifacts::load(&self.store, &all_alignments).await?;
+        let artifacts = AlignmentArtifacts::load(&self.store, &all_alignments, PROJECT_REPORT_KINDS).await?;
         // The order is the same as the order in `haplogroup_consensus`. It is the vote of each
         // run, then the placed label, then a value that the user set.
         let terminals = self.haplogroup_terminals().await?;

crates/navigator-store/src/artifact.rs

@@ -154,6 +154,12 @@ pub async fn list_kinds(pool: &SqlitePool, alignment_id: i64) -> Result<Vec<Stri
 /// Every artifact of every id in `alignment_ids`, in one query. The caller indexes the result by
 /// `(alignment_id, kind)` itself. This replaces a `get` for each (alignment, kind) pair, which gave
 /// a project report one round trip for each cell. An empty `alignment_ids` runs no query.
+///
+/// **No caller uses this today, and a new caller almost certainly wants
+/// [`list_for_alignments_of_kind`].** This query selects the payload of *every* artifact, and some
+/// payloads run to megabytes. The project report used this form and pulled gigabytes of
+/// `tree-genotype` JSON to read five small kinds. Use this one only when the caller genuinely wants
+/// each kind that an alignment holds, and does not know the kinds in advance.
 pub async fn list_for_alignments(
     pool: &SqlitePool,
     alignment_ids: &[i64],

documents/BACKLOG.md

@@ -23,10 +23,13 @@ Code exists or the design is settled; these are the near-term threads.
 
 ### 1.1 Archaic ancestry (Neanderthal / Denisovan)
 - **Design:** [`design/ArchaicAncestry_Design.md`](design/ArchaicAncestry_Design.md)
-- **Status (corrected 2026-08-02):** **Tier A shipped** (`230353b`, `#34`) and reports a *count*,
-  never a % Neanderthal. **Tier B is built but gated OFF** (`#35`, `#40`) — the diagnosis is that it
-  measured the wrong observable, not that the HMM is broken; read `#41`/`#42` before reopening. The
-  "design draft, no code" status below was already stale when this file was written.
+- **Status (corrected 2026-08-23):** **Both tiers shipped and ON.** Tier A (`230353b`, `#34`)
+  reports a *count*, never a % Neanderthal. **Tier B was rebuilt and re-enabled** in `#42`
+  (`906b9ee`, released in `v0.1.0-alpha.15`): `ARCHAIC_SEGMENTS_ENABLED = true`. The v1 density
+  caller was gated off in `#40` for measuring the wrong observable; v2 (`archaic_match`) matches the
+  archaic genomes directly and reaches r = +0.710 on held-out Europeans against −0.018 for v1. It is
+  a **within-population** measure only — our four archaic genomes under-represent East Asian archaic
+  diversity, so cross-ancestry comparison orders populations backwards. See design §11.
 - **Scope:** Phase 1 = compute our own marker panel (EVA archaic VCFs + Ensembl-75 ancestral alleles
   + 1kGP AFR outgroup) + Tier A `count_archaic_markers` + domain/store/UI card — the 23andMe
   equivalent, for chip *and* WGS, reusing the ancestry-panel machinery. Phase 2 = Tier B segment HMM
@@ -96,72 +99,42 @@ Code exists or the design is settled; these are the near-term threads.
 - **Design:** [`design/packaging-and-release.md`](design/packaging-and-release.md)
 - **Status:** Shipping (all four installers build on a `v*` tag; assets fetched on demand from the
   GitHub asset release).
-- **Scope:** code signing + notarization (Apple Developer ID $99/yr; a Windows cert) — deferred for
-  alpha with a documented Gatekeeper work-around; the Linux glibc-2.28 container CI is authored but
-  **has never been run**; `default_reference_sha` is still `None` for all four builds
-  (`navigator-refgenome/src/registry.rs:172`), awaiting confirmed publisher checksums.
+- **Scope:** **macOS signing + notarization are DONE** (2026-08-15) — only **Windows** code signing
+  (Authenticode / Azure Trusted Signing) is still open, deliberately deferred for alpha with a
+  documented SmartScreen work-around; the Linux glibc-2.28 container CI is authored but **has never
+  been run**; `default_reference_sha` is still `None` for all four builds
+  (`navigator-refgenome/src/registry.rs:191`), awaiting confirmed publisher checksums.
 
 ---
 
 ## Tier 2 — Designed, not started
 
-Verified 2026-07-26 to have no implementation in the tree.
-
-### 2.1 Realignment module — **in progress** (phase 1 landed 2026-08-08)
-- **Design:** [`design/realignment-module.md`](design/realignment-module.md) — revised 2026-08-08
-  after a phase 0 spike that **retracted the module's motivating premise** (ancestry is *not*
-  build-locked; off-build samples already estimate ancestry through the multi-build IBD panel) and
-  reversed the backend decision to pure-Rust `minimap2-pure-rs`. Read the correction blocks before
-  planning further work — whether the remaining payoff justifies the module is an open product
-  question.
-- **Scope:** revert + realign GRCh37/38 vendor WGS to CHM13v2 / hs1; aligner-index cache in
-  `navigator-refgenome`, job orchestration + provenance, opt-in background job with warnings.
-- **Done:** phase 0 spikes; **phase 1** (stage A, `navigator-analysis/src/revert/`) — primaries-only
-  revert with orientation restore and `OQ` preference, a disk-backed external merge sort that
-  collates by read name, synchronized paired-FASTQ output; **phase 2** (stage B,
-  `navigator-align`) — pure-Rust minimap2 backend, preset selection, RAM-sized part-by-part index
-  build and map with cross-part merge, single- and paired-end, BAM/CRAM output via noodles;
-  **phase 3** (stages C and D) — coordinate sort, short-read duplicate marking, CRAM + `.crai`,
-  and the provenance migration with registration in `navigator-app::realign`.
-  **Phase 4** — the cancellable job with preflight, the per-alignment and per-project cards, the
-  realigned badge, and selector preference — is built too.
-- **Phase 5 in progress, and it found something.** Backend parity on 168k real WGS229 chrY reads:
-  99.2% identical placements, but `minimap2-pure-rs` reports **systematically higher MAPQ** than C
-  minimap2 (1,157 up vs 47 down, median +10), concentrated on chrY at 15x the off-chrY rate, and
-  111 reads per 168k cross the MQ>=20 callable threshold upward. That is the input to the
-  private-Y filter stack, so it is material. **Paired-end re-run against upstream 2.31 widened the
-  gap** (1.09% of records differ in MAPQ vs 0.72% single-end; 96.6% of differences are Rust
-  higher), and does not reproduce the upstream crate's own claim of exact `sr` PAF parity.
-  **But it does not reach the output**: run through the shipped stage C and de-novo caller over
-  the worst-case window, upstream produced 232 calls and the Rust backend 233 — every upstream
-  call reproduced, one extra at depth 3 that the `depth >= 4` callable gate removes. Decision 1
-  re-settled on the pure-Rust backend with the divergence documented.
-- **First WGS-scale run (2026-08-12): 10 h 41 m, and it did not finish.** WGS229's 17.3 GB CRAM
-  (615.6M reads) died in stage 7 of 8 on a CRAM-encoding panic — a secondary alignment carries
-  `SEQ: *`, which is legal SAM and what minimap2 emits, and CRAM cannot store a read it has no
-  bases for. Fixed: such non-primary records are dropped and counted, a primary of that shape
-  errors. Two other findings: **the sort is the most expensive stage** (4 h 44 m, 44% of wall
-  clock — more than mapping's 3 h 40 m), and **stage A runs on one core** because `open_seq` only
-  threads the BAM path, fixable by reverting per-contig in parallel. The run also proved
-  resumability is not optional — `JobScratch` discarded seven working stages on the way out, so
-  `NAVIGATOR_REALIGN_KEEP_SCRATCH=1` now inverts that for pipeline work.
-- **Purpose:** Y-chromosome variant discovery. A private-Y call set is only usable on CHM13 —
-  the callable mask, non-PAR restriction, recurrent blocklist, and de-novo tree are all defined
-  there — and liftover cannot help with *discovery*, where the variant is not in any site list
-  yet. Autosomal fixed-site matching was never the motivation; two earlier revisions of the
-  design said otherwise and have been corrected.
-- **Settled:** whole-genome is the only correct scope. A Y-only mode was proposed and withdrawn —
-  the reads that need realigning are the ones GRCh38 placed wrongly, so selecting them by source
-  coordinate requires the answer being computed. Revert reads the whole file anyway.
+Verified 2026-08-23 to have no implementation in the tree, **except 2.1, which shipped** and is kept
+here under its old number so that existing references still resolve.
+
+### 2.1 Realignment module — **SHIPPED** (no longer a backlog item)
+- **Design:** [`design/realignment-module.md`](design/realignment-module.md) — the full record:
+  phase 0 spike, backend parity, the three whole-genome runs, and the phase 5 acceptance result.
+- **Status:** merged to `main` as `bf576ab` (2026-08-14), released in **`v0.1.0-alpha.17`**. All
+  five phases are done: revert (stage A), pure-Rust minimap2 mapping (stage B), sort + duplicate
+  marking + CRAM (stages C/D), the cancellable job with preflight and provenance, and end-to-end
+  validation on WGS229 (private-Y 438 → 11, chrY breadth 41 % → 98 %). Simple-mode offer and the
+  Windows disk preflight came with it.
+- **Purpose:** Y-chromosome variant discovery. A private-Y call set is only usable on CHM13, and
+  liftover cannot help with *discovery*, where the variant is in no site list yet. Two earlier
+  revisions of the design argued it from autosomal ancestry and were wrong; they are corrected in
+  place.
+- **Eligibility lives in one place** — `realignable_for_subject`. Do not re-derive it at a call site.
 - **Do not confuse** with `navigator-analysis/src/realign.rs`, which is *indel local realignment*
-  (plan §4b) and is a different thing entirely.
+  (plan §4b) and a different thing entirely.
+- **Unblocks 2.2**, which depended on this module.
 
 ### 2.2 Distributed compute grid
 - **Design:** [`design/distributed-compute-grid.md`](design/distributed-compute-grid.md)
 - **Scope:** a Seti@Home-style layer — the AppView publishes public-ENA work units, Navigator
   instances reserve a lease, fetch, realign to CHM13, run the analysis stack, submit signed results,
   and earn capped compute credit. Cross-repo (Navigator worker + AppView coordinator + shared wire
-  records). Depends on 2.1.
+  records). **Depended on 2.1, which has now shipped — this is unblocked for the first time.**
 
 ### 2.3 Academic / public-dataset (ENA) import
 - **Design:** [`design/academic-ena-import.md`](design/academic-ena-import.md)

documents/Edge_Client_Implementation_Status.md

@@ -1,8 +1,21 @@
 # Edge Client Implementation Status
 
+> **STALE — do not trust the numbers below (header added 2026-08-23).** This is a **Scala-era**
+> document, last updated 2025-12-08, more than six months before the Rust cutover (`0dee32c`,
+> 2026-06-19). Every percentage and every file path in it refers to code that no longer exists.
+>
+> Atmosphere lexicon alignment **completed** (phases A–D) during the rewrite, and the federation
+> surface has moved well past what this file describes: `sync_outbox` with idempotent putRecord at
+> TID and PULL reconcile, signed IBD attestations over an X3DH/AES-GCM exchange channel, feed posts,
+> peer DMs, and recruitment invitations are all built and merged.
+>
+> For the current picture use [`BACKLOG.md`](BACKLOG.md) (the "Social layer — deferred slices" and
+> "AppView-side" sections) and the per-topic agent memory. Kept only for the historical record of
+> what the Scala edge client had reached.
+
 Navigator Desktop implementation status against the Atmosphere Lexicon specification.
 
-**Overall Completion: ~55%**
+**Overall Completion: ~55%** *(as of 2025-12-08, Scala codebase — see the warning above)*
 
 Last updated: 2025-12-08