docs: Simplified Technical English across seven crates - #56
Merged
Conversation
…nical English Navigator's comments are to follow ASD-STE100. Two things have to exist before any of them can be judged compliant, and neither is a rewrite. `documents/STE-dictionary.md` is the project dictionary. STE permits three classes of word: its own ~900-word approved list, Technical Names, and Technical Verbs. A project must declare the second and third itself, because the approved list contains no `biosample`, no `contig`, no `haplogroup`. The file also records the nine rules in short form, the substitutions this codebase needs most, and what STE does *not* reach: identifiers, code inside backticks, and commit messages. `scripts/ste-check.py` enforces the seven rules a script can judge without a part-of-speech tagger — sentence length, active voice, `-ing` forms, noun clusters, vocabulary, paragraph length, and idiom. It reads the Technical Names out of the dictionary, so declaring a term is enough to stop the checker flagging it. It is advisory and always exits 0; the intent is to see the number, not to gate a commit on it. Scope is Rust comments under `crates/`. Markdown is out, and `--all` includes it for information only. The baseline as this lands is 12,023 violations across 272 files. Three bugs found while calibrating it against real files, each of which had inflated the count: every doc block in a file merged into one paragraph, Markdown list items merged into one paragraph, and irregular participles matched as suffixes, so "present" read as `pre` + `sent` and reported plain adjectives as passive voice. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The unambiguous half of STE rule 1: a word with exactly one approved replacement regardless of context. `utilize` to `use`, `prior to` to `before`, `whilst` to `while`, `approximately` to `about`, `however` to `but` at the start of a sentence, and every contraction expanded. A script did this, and it is deliberately conservative. It never enters a backtick span, and it leaves every judgement call for the hand pass — `via`, `due to`, `determine`, `identify`, `require`, `significant`, and mid-sentence `however` all survive untouched here. The first attempt corrupted 14 lines and was reverted rather than patched. The patterns had no leading word boundary, so `it's` matched inside `kit's` and produced "kit is own CSV", `sufficient` matched inside `Insufficient` and produced "Inenough", and `ascertain` matched inside the flag name `--ascertain-sites`. The rewrite added boundaries, plus explicit guards for the three words that appear inside longer words, and was only applied once ten probes covering those exact cases passed. This commit changes no meaning. It is the floor the hand conversion starts from. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The first two files taken to zero violations, and the proof that the rule set survives contact with this codebase's harder prose. `navigator-resource/src/lib.rs` was the real test: its module header is a post-mortem of the 2026-08-13 WGS sort that a WindowServer watchdog killed. Narrative, em-dashes throughout, and the whole argument for the crate carried in two long paragraphs. Under STE it becomes a dated sequence of short factual statements. Every fact survives — the 549 GB of dirty file-backed memory, the 1.4x write-back limit, the 40-second watchdog, the login session going down — and so does every reason. What goes is the compression and the voice. That is the trade this standard makes on purpose. 134 violations in that file, four passes, roughly 34 targeted rewrites. `sig_cache.rs` took two. Both crates still build and `cargo fmt --all --check` is clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
appview, auth, dm, error, export, ibd_exchange, matching, recruitment, settings, social, sync, sync_reconcile and update, all at zero violations. The crate goes 3,893 to 3,521; twenty files remain, of which haplogroup.rs (788) and lib.rs (685) are two fifths. Nothing here is a summary. Where a comment explained why the code is shaped as it is, the explanation is still present and usually longer: the reason a device key signs each call rather than the HTTP request, why a federated post is deliberately absent from `PUBLISHED_COLLECTIONS`, why `AlignmentFileMissing` earns a variant of its own, why the update check compared versions wrongly for sixteen alphas. Short sentences, active voice, one idea each. Worth recording for whoever picks this up: a first pass reliably leaves violations behind. sync.rs went 39 to 19, ibd_exchange 51 to 29, update 54 to 17, each needing a second and sometimes a third pass. Splitting one long sentence into three explanatory ones almost always produces a new twenty-six-word sentence. It converges at roughly two and a half passes per file, not one. `cargo check -p navigator-app --all-targets` passes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
brief, ftdna_import, import_profiles, maintenance, publish, the three test and example files, and the last of the small ones. Twenty-three of thirty-three files are now at zero; the crate goes 3,893 to 3,096. Two checker corrections came out of this batch, both of which had been inflating the count: The em-dash rule judged raw text, so cargo's `--` argument separator inside a fenced shell block — `cargo test … -- --ignored` — read as an em-dash aside. It now judges the code-stripped text like every other rule. That alone removed several hundred false positives across the repository. `genotyping array` is a Technical Name and was being flagged as an `-ing` form. The dictionary now declares the `-ing` Technical Names in their own section rather than leaving them implicit in the checker, so the two stay in step. The prose keeps the reasoning and loses the compression, which is the intended trade. The FTDNA matcher still explains why a genetic distance of 3 to 11 over 100 markers is normal inside a single-haplogroup project and why only a near-exact haplotype identifies the same person. The publish gate still explains why a WGS-labelled Y-only extract must not file a coverage record. Both now say it in sentences a second-language reader can parse. `cargo check -p navigator-app --all-targets` passes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
blocktree, commands, llm, realign. Twenty-seven of thirty-three files at zero; the crate goes
3,096 to 2,694.
blocktree.rs is the file this standard was most likely to damage, and the one worth reading to
judge the result. Almost every constant in it is a threshold defended by a measurement, and the
defence is the comment. All of it survives:
- why six "novel" calls inside 32 bp are one misaligned read and not six mutations, and that this
shape produced most of the first candidate branches on the CTS4466 cohort;
- why three branches inside a 567 bp window at 56.83 Mb are one repeat unit mis-mapping rather
than three lineage events, and why a kilobase is the right scale;
- why five positions carried by all 111 private-Y donors are reference-vs-population differences
that a bundled blocklist cannot anticipate;
- why the frequency rule abstains below a donor count instead of throwing away every genuine
branch in a four-donor cohort.
Each now reads as short declarative sentences instead of one dense paragraph with three
subordinate clauses. Longer on the page, and a reader parsing English as a second language can
follow the argument.
Two more Technical Names declared, because the checker was flagging real domain terms: `genotyping
array` and `reasoning model`. The dictionary now carries a Local LLM section and lists the `-ing`
Technical Names explicitly.
`cargo check -p navigator-app --all-targets` passes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
app.rs (tests), commands, fastpath, import_unified, queries, realign_job. Thirty of thirty-three
files at zero; the crate goes 2,694 to 1,710. Only analysis.rs, lib.rs and haplogroup.rs remain.
These are the files where a comment is a post-mortem, and the conversion is worth reading for that
reason. Every finding survives:
- why the BGZF end-of-file marker cannot prove a file is complete — noodles finishes its stream
from `Drop`, so a cancelled merge left 13.2 GB of an expected 30 GB wearing a valid marker, and
the next run marked duplicates on a truncated alignment;
- how a 59 GB `mapped.bam`, four hours of revert and mapping, was destroyed by that marker bug
combined with a resumed run deleting an artifact it had not written;
- why scratch is sized from the source's uncompressed volume and not its file size: 17 GB of CRAM
is ~70 GB of BAM-equivalent data, and the file-size estimate told a user a 200 GB job needed 69;
- why a Y-only extract must be forced male — the chrX/autosome ratio reads it as female, which
silently disabled the entire Y pipeline;
- why `min_dp 4` removes misaligned-read clusters without touching real private SNVs.
The prose is longer and flatter. The reasoning is intact, and a reader parsing English as a second
language can now follow it a sentence at a time.
`cargo check -p navigator-app --all-targets` passes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
analysis.rs joins the converted set; 31 of 33 files in the crate are at zero. lib.rs goes 678 to
524 and is the first file left partly done — the type documentation at the top is converted, the
`impl App` body below it is not.
Two findings in analysis.rs were worth the care they got:
- the localized-copy leak. Copies were cached in a directory that one caller cleared while three
created them, so every other path copied ~400 MB per alignment and cleaned up nothing — 687
files and 145 GB, which filled the volume mid-run. Removal now hangs off `Drop`, and the
comment explains why a refcount keeps the original benefit.
- why two concurrent copiers must never name the same scratch file: sharing one lets them open a
single inode, where one truncates what the other is writing and keeps writing into it after the
other has renamed it into place and started reading.
`cargo check -p navigator-app --all-targets` passes on this tree.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Whoever picks this up next should not have to rediscover the method. The dictionary now carries it,
and `scripts/ste-blocks.py` — the block dumper the workflow is built around — is committed rather
than living in a scratch directory.
`ste-check.py` gives the score; `ste-blocks.py` gives the text to rewrite, with the rules that fired
on each block. The loop is: dump the blocks, rewrite ten to fifteen of them in one batch of exact
`(old, new)` string pairs, re-score, repeat.
Three things the recipe records because they cost time to learn:
- **Expect about three passes per file, not one.** A first pass on 50 violations leaves 15 to 25,
every time. Replacing one 40-word sentence with three explanatory ones reliably produces a new
26-word sentence, so rule 6 fires on a line that did not exist before. It converges, and each
pass is smaller.
- **Batch with exact string pairs and a MISS guard, never a regex over prose.** The one regex pass
that touched prose corrupted 14 lines ("kit is own CSV", "Inenough", "--find out-sites") and had
to be reverted wholesale.
- **A violation that makes no sense is probably the checker.** Three real domain terms and one
suffix-matching bug in the passive-voice rule were found that way. Read the checker before
rewriting prose that is already correct.
Also states what a rewrite must keep — every measured number, date, sample id and failure that
motivated the code — and what it is expected to lose. The compression and the voice go. That is the
trade, and it is deliberate.
Ends with where the work stands, so the next session starts from a fact and not an assumption.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`navigator-app/src/lib.rs` goes 524 -> 0, which leaves 32 of the crate's 33 files converted. `haplogroup.rs` goes 758 -> 384. Two tool changes came out of the work, both of the kind the recipe already warns about. The sentence splitter did not break on `.**`. A sentence that ends inside bold emphasis merged with the one after it, and rule 6 then reported a length that no sentence had. `SENT_SPLIT` now steps over a trailing `*` or `_`. That is the third time a violation that made no sense turned out to be the checker. `copying` joins the -ing Technical Names. The chromosome painter's published method is copying-LAI, so the word is a name here, not a verb. The recipe gains a failure mode that cost real time. Splitting a long doc comment into paragraphs with a blank line, rather than a `///` line, orphans the text above it: that is E0585, a hard build error, not a lint. It is easy to hit, because a few files hold two doc comments that ran together into one block. The dictionary now carries the scan that finds it. Verified: `cargo check -p navigator-app --all-targets` and `cargo fmt --all --check` both clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`haplogroup.rs` goes 758 -> 0, which is the last of the crate's 33 files. navigator-app is now fully converted, 3,893 -> 0 across the round. Three more checker bugs, all the same family the recipe warns about. `is down` read as passive voice. The participle pattern carries an `-own` branch for `shown`, `thrown`, and `known`, and `down` ends in `own`. That word and its rhymes join NOT_PARTICIPLE. The sentence splitter did not break before a quoted sentence, so `... is not one. "The network is down" is not one.` became one sentence. That is what pulled `is down` into scope. The lookahead now accepts a leading quote. The orphaned-doc-comment scan added in the previous commit caught the E0585 mistake twice more here, both within a minute of writing it. `haplogroup.rs` holds four doc comments that ran together in the source, which is what invites the error: one function's documentation sits physically on the next function. This commit only re-separates the paragraphs. Moving a doc onto the function it describes is an API-documentation change, not an English one, so it is left alone. Verified: `cargo check -p navigator-app --all-targets` and `cargo fmt --all --check` both clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
All 12 files, 235 -> 0. The crate covers OAuth, the keychain gateway, the device key, the encrypted edge exchange, the PDS record contracts, and the resilience engine. `navigator-resource` regressed to 1 and is back at 0. The `.**` fix in the previous commit made the splitter see `- **8 GB maximum.**` as its own sentence, which took that list item to seven. That is the checker getting *more* right, not less, so the prose moved rather than the rule. Two prose notes. "byte-for-byte" is on the idiom list, and the signing-string modules used it three times to say that a string must equal the AppView's exactly; "exactly" is the word. The privacy boundary in `records.rs` needed care: "living tester" and "not living-donor PII" both had to survive the rewrite as testable claims, because that comment is the argument for what may leave the workspace. Verified: `cargo check -p navigator-sync -p navigator-resource --all-targets` and `cargo fmt --all --check` both clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
All 10 files, 237 -> 0: the gateway, the build registry, the atomic config cache, VCF liftover, decompress-and-index, genome regions, and the live network tests. Two false positives went to the code, not the prose, because in both cases the text was already right. `cargo test ... -- --ignored` in a doc comment reads as an em-dash aside, and the Windows error string "Access is denied" reads as passive voice. Backticks make each one a code span, which the checker already skips. That is the same fix the existing note about fenced blocks describes. Two comments here carry a claim that had to survive intact. `ReferencePolarity` states that polarity comes from the haplotree and never from REF-against-ALT, and gives CHM13v2.0 as the trap: its chrY is HG002, a haplogroup-J Y, so the reference base is the derived allele at many sites. `atomic_write` and its reader document issue #26 end to end, including why an unreadable config must not silently become "no config". Both are longer now and say the same thing. Verified: `cargo check -p navigator-refgenome --all-targets` and `cargo fmt --all --check` both clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Some doc blocks held the documentation of two items. One function's doc sat
physically on the *next* function, and the function it described had none. The
STE conversion found them, because a block that changes topic in the middle is
hard to miss when you rewrite it.
A repo-wide scan produced 64 candidates: a multi-paragraph doc on one item,
followed by an item with no doc at all. Reading the first paragraph of each
against the two names left 11 real defects in 4 files. The other 53 are a
documented item followed by a private helper, and they are correct as they are.
Nine were stranded and move to the function they describe:
lib.rs "Score a tree against the sample calls" -> assemble_assignment
"Tier B: positions variable in the ..." -> archaic_outgroup_path
"Map a computed AncestryResult ..." -> population_breakdown_record
"The build a haplotree's positions ..." -> tree_build_for_contig
"Group the genotypes of each site ..." -> group_chrom_genotypes
haplogroup.rs "A haplotree JSON, from the cache ..." -> fetch_tree
"Assign a Y haplogroup to an alignment" -> assign_y_haplogroup
"Like assign_haplogroup_from_alignment" -> assign_haplogroup_detail
"Base-call an alignment at a parsed ..." -> base_calls
One was two summaries of the same function. `snp_obs_from_assignment` carried an
old doc and its rewrite; they merge. The older one held the only description of
the `in_tree` parameter, which the merge keeps.
One documented code that no longer exists, and it is deleted. The doc of a
per-alignment ancestry estimator sat on `estimate_ancestry_from_consensus`. No
such function is left in the repo; the consensus path replaced it, and its own
doc says so in the past tense. It documented nothing, and it made the function
below it read as though it did two jobs. This is the only deletion here.
`export.rs` shows how these happen. Its `// ---- mtDNA variants ----` banner sat
above `branch_report_tsv`, and `mtdna_variants_tsv` had been pushed 260 lines
down into the block-tree section: two functions went in between a banner and the
function it introduced. The banner moves back to its section, and the two report
functions get a heading that names them.
No prose is lost. `git diff --numstat` gives 21/21 for worker.rs, a pure move,
and -11 for haplogroup.rs, which is the 7-line deleted orphan plus the 4
separator lines that splitting the blocks removes.
The dictionary gains the detector, the three repair shapes, and the rule that
this is its own commit: a doc that moves to a different item changes the API
documentation, which is a different review from a change to the English.
Verified: `cargo check -p navigator-app -p navigator-ui --all-targets` and
`cargo fmt --all --check` both clean; navigator-app still reads 0 violations.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
All 30 files, 320 -> 0: the biosample, alignment, sequence-run and artifact queries, the haplogroup-call ledger, the sync outbox and history, the IBD request and exchange tables, peer DMs, and the integration test. One more checker false positive, of the family already on record. `unknown` ends in `own`, which is the irregular-participle branch that carries `shown` and `known`, so `is unknown` read as passive voice. It joins NOT_PARTICIPLE beside `down` and its rhymes. That fix alone took 118 violations off the repo-wide count, so the word is common. A second flag went to the code, not the prose: `"database is locked"` is the SQLite error string, and backticks make it a code span, which the checker already skips. Same remedy as `Access is denied` in navigator-refgenome. Two comments here carry a claim worth keeping exact. `biosample::clear_data` lists what a reset removes and, more importantly, what it keeps, and it records why the signature-keyed caches come from `sig_cache::ALL` and not a hand-written list: when the list was written by hand it named `consensus_painting` alone, and ROH and both archaic caches survived a clear. `mdka::publishable` states the consent rule for what may leave the workspace, with the measured 558 of 583. Verified: `cargo check -p navigator-store --all-targets` and `cargo fmt --all --check` both clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
All 28 files, 502 -> 0: the AIMs and fine-population builders, the PCA, the ancient and archaic panels, the IBD panel, the genetic map, the LAI and ancient-panel validators, the manifest, and fifteen throwaway examples. `loading` joins the -ing Technical Names. In a PCA a loading is the coefficient of a SNP, so the word is a name here and not a verb. This crate is where the measured numbers live, and every one of them survived. The archaic panel's frequency floors are calibrated against the hmmix callset: F1 peaks at 0.701 for a 0.0005 non-African floor, and removing that floor drops precision to 64.7% for almost no recall. The LD variance inflation is a ladder, not a constant: 52x on the full panel against 5.3x on a 2.6% subset of it, and a single factor applied to a chip would squash every percentile toward 50. `validate_ancient` carries the reason the whole gate exists, which is that the previous ancient implementation released numbers nobody had checked. Three comments needed care beyond their length. The archaic module header explains why polarity is assigned in GRCh37 and the panel ships in CHM13, and why an unoriented CrossMap lift is a silent defect: an f4 statistic does not change under a consistent flip, so nothing fails loudly. `read_genome_table` records that an EVA all-sites record with `ALT=.` is a positive statement, and that dropping it biases the panel one way. `validate_ancient`'s population option records that the SGDP columns write 0.0 for "no sample", so 60% of their sites are false zeros. Verified: `cargo check -p navigator-panelbuild --all-targets` and `cargo fmt --all --check` both clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The two largest files in navigator-analysis, 264 and 234 violations. Every
measured number, date and sample id survives verbatim; only the sentences moved.
Load-bearing content checked line by line against the original, not paraphrased:
- the ANCIENT_MAX_DISPERSION calibration ladder (GBR 1.65 · CEU 1.58 ·
FIN 1.78 · TSI 2.38 · IBS 3.65 ‖ CHB 13.1 · JPT 12.4 · YRI 175 · LWK 158)
and why 4.0 sits in the gap rather than being tuned;
- the ANCIENT_MIN_WEST_EURASIAN scope argument, including the Punjabi
Steppe-67%-vs-real-20-30% worked example;
- the estimate_ancient_admixture history: what the PCA-centroid classifier
got wrong, WHG↔ANF Fst ≈ 0.07, and why None must stay None to the UI/PDS;
- the parsimony guard's validation (GFX0457637 → R-FGC29071), the stray
ancestral at R-Z16250 blocking R-CTS4466 at 10 derived / 0 ancestral, and
the ~12.7k indel loci that made indel-heavy nodes capture the placement;
- the DecodingUs ~1.4% swapped-coordinate polarity rule and the huF98AFD bug.
Four checker fixes, all the "a violation that makes no sense is a checker bug"
family the recipe warns about:
- `prior` declared a Technical Name. It is the Bayesian noun in 28 of its 29
uses; `prior to` stays forbidden. The one temporal use (brief.rs, "a prior
download") is rewritten so the exemption hides nothing.
- `drift` likewise: genetic drift, on the idiom list only because of *version
drift*. The idiom rule now honours Technical Names, as NOT_APPROVED already
did, and the five metaphorical uses across align/analysis are rewritten.
- `is derived` is the genetics adjective ("the sample is derived at a site"),
not the passive of "to derive". Exempted only when "from" does not follow —
a blanket rule would have hidden the ~15 genuine passives in the repo.
- "Busing et al. 1999" is the jackknife citation, not an -ing verb.
Repo-wide 6,734 -> 6,192.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The doc block above `is_snp_locus` in haplo.rs opened with a paragraph that described a different function: "Does the sample carry the derived allele of this locus? ... except at a SNP that is ambiguous about the strand." That is `locus_carried`, which sat forty lines below with no doc of its own. The two ran together the way the eleven in 77a2fcf did. The strand-complement rule belongs to the carry test, and reading it above `is_snp_locus` made that function look as though it decided both whether a locus is a SNP and whether the sample carries it. Pure move: `git diff --numstat` read 5 insertions, 5 deletions. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three more of the largest files: 187, 182 and 145 violations.
archaic_match.rs carried a 161-line module doc that is the densest evidence
record in the crate — the whole case for withdrawing the density caller and
for keeping this one gated. It is rewritten whole rather than in pieces, and
every figure was checked against the original line by line:
- the 30-fold evidence gap (~1 variant / 14.3 % against ~30 sites / 95.1 %
at a 36 kb tract) and the 38 %-of-14.6x overdispersion the density model
could not get a proxy for;
- the genome-wide table for HG00096/HG00102/HG00112 and the note that the
chr21+22 numbers are conservative, because the previous design was burned
by extrapolating a chr21+22 target 6 % low;
- the 30/30 train/test split, the four-row calibration table, and *why* the
split exists — the previous caller was tuned until a cohort statistic
matched, and the statistic was then reported as evidence;
- the frozen-parameter East Asian transfer, the inverted extent ordering
(1.217x truth against 0.937x called) that keeps this gated, and the three
causes ruled out by measurement (11.9/12.2 %, 29 kb both, 2.99x/3.04x);
- the per-genome concordance table and the 64 %/57 % positions;
- the 54 % -> 90 % filter with its Denisova hold-out (74.9 % against 21.5 %),
and the 32.2 %/11.2 % Denisovan-match asymmetry that makes a
cross-population number unreachable this way.
Also kept verbatim: the degenerate Baum-Welch fit (22x emission ratio, 9 kb
tracts, 7x truth) that is why every parameter here is measured; the 5 kb
min-segment argument (8 % of truth discarded against 16 % at 10 kb, and the
40 kb sweep that would have discarded 61 %); archaic.rs's NoCall-as-absence
error (18,551 against 24,077 on chr21+22, ~19 % Denisovan for a European);
and caller.rs's indel additive-only rule.
Repo-wide 6,192 -> 5,678. navigator-analysis 2,914 -> 2,400.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two more: 110 and 109 violations.
lai.rs is where the copying painter's calibration evidence lives, and every
measured trade-off came through unchanged:
- max_ref_haps: why the cap must stay well above HGDP Orcadian 30 / Adygei
32 / Basque 46, and the validate-lai gradient that shows it (at 50, 12.9 %
fine with 27.2 % of the genome called into a drifted isolate; at 200,
24.6 % and 11.8 %; above ~300 it falls again);
- size_normalize: the knob whose answer *reversed* when the panel got denser
— rejected on the 15.6k-site panel, and on the 165k-site one the full
per-population rescue list (Sardinian 10→13 % / 33→36 %, Basque
33→37 % / 37→39 %, Orcadian 6→10 %, Russian 10→12 % / 4→9 %) plus why 0.25
is the knee (top-1 50 %→42 % above it, isolate over-call 9.6→11→15→18 %);
- min_segment_cm in genetic distance rather than sites, and the 40 Mb trap a
site count would spring if the panel got 10x denser;
- the shipped calibration figures (32.0 % fine against 11.4 % chance, 68.1 %
regional, 98.5 % super-population, 9.6 % isolate over-call over 41 cases);
- the two test gates: the numeric GBR-over-FIN property and the regression
gate that fails if the old by-eye defaults come back.
unified.rs keeps the fusion argument intact: 2→1 reads for BAM and 3→1 for
CRAM, the multi-reference-slice gate that stops a record being counted once per
overlapping contig, and the CRAM 3.1 decode-stack note ("an overflow aborts the
whole process, so the margin here must be wide").
One more checker fix: "Balding" joins "Busing" as a citation surname the -ing
rule read as a verb.
Repo-wide 5,678 -> 5,459. navigator-analysis 2,400 -> 2,184.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Four more: 105, 98, 95 and 95 violations.
preflight.rs is a module whose entire subject is a distinction, and the
distinction survives intact: EPERM is macOS TCC refusing the process whatever
the mode bits say (which is why chmod 777 in ~/Desktop still fails), EACCES is
the Unix permission failure, and chmod advice on the former sends the user
down a dead end. So does the invariant a batch depends on — a missing index
must not read as "this sample is unanalyzable", because coverage, read metrics
and sex still complete sequentially.
coverage.rs keeps its measured memory argument (whole-genome HG002 peaks ~2 GB
against the ~84 GB dense per-contig arrays would need), the coverage-2 cache
bump and why it invalidates every coverage-1 result, and the out-of-order-base
guard with the panic it replaced ("attempt to subtract with overflow").
archaic_segments.rs keeps the reasons Tier B lineage attribution is gated off:
4.3 % against 3.9 % base rates for a ratio of 1.10, and 0.00 Mb Neanderthal
against 0.48 Mb Denisovan on the ground-truth European where §7 expects the
reverse. Also the hmmix calibration (45 segments / 2.01 Mb against 43 / 2.09,
inside the p10-p90 spread), why the transition rate is 5 and not 1, and why the
minimum segment is 5 kb and not 50 kb.
reader.rs keeps the numbers behind cram_container_offsets — the change that
made CRAM region queries usable at all: 20.9 s for a 1 bp query on chr21 and
116 s on chr1 before it, against 4-6 ms for the same query on a BAM, and the
1,140 containers on chr21 of which a point query needs one. Also the CRAM 3.1
decode-stack argument and its consequence ("a stack overflow aborts the
process; it is not a panic that the code can catch").
One checker fix, and it is the useful kind: a Markdown table row is data, not
prose. Its cells carry no terminal punctuation, so the sentence splitter read a
whole table as one sentence and reported a length that was not there.
`extract_md` already skipped these in a .md file; `extract_rust` now does too.
That alone cleared violations in files this commit does not otherwise touch.
Repo-wide 5,459 -> 5,065. navigator-analysis 2,184 -> 1,788.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two more: 79 and 70 violations. reassembly.rs keeps the reason the resolver exists — the pileup caller rejects a ~50/50 position as a suspected paralog, and at Y segmental-duplication loci that discards *true* derived SNVs, because paralog reads mismap and carry the reference base onto the site. It also keeps why the assembled-alt-haplotype option is default-off: it helps the synthetic linked-variant case but perturbs marginal sites on real WGS229 data (it regressed chrY:4284195), and there is no real linked-variant truth site to check the benefit against. That same site is why the majority-consensus rule is deliberately not a raw POA. sv/walker.rs keeps the measurement that motivated the parallel fan-out: SV was the last whole-genome analysis walking on one thread, at 2-5 h per 30x WGS CRAM against ~55 min for the per-contig unified walk over the same files. It also keeps the allocation history behind classify_pair — two mallocs per read thrown away almost every time, which is what made the walk malloc-bound — and the evidence cap's honesty rule: evidence past the cap is still counted as dropped, so a truncated run is visible rather than silent. navigator-analysis 1,788 -> 1,639. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two more: 82 and 60 violations. postprocess/tests.rs is a file of test docs that each say why the test exists, and those reasons are the value. Kept: a sort that loses records shows later only as coverage reading a little low, which almost nobody would find; the spill/no-spill equivalence claim the revert stage's external sort also makes; why grouping duplicates on the unclipped 5' position finds copies that a group on the alignment start misses; why a signature that ignored the mate would collapse two real molecules and delete coverage; and the resume argument behind is_complete_bam — a wrong answer either way costs a 60 GB re-derive or hours. gvcf.rs keeps the CHM13 trap in assemble_calls, which is the subtlest thing in the file: the Y of CHM13 is HG002, haplogroup J1 and deep in the tree, so at every backbone SNP J1 shares with the sample the GVCF emits a ref block where hom-ref equals the reference equals the *derived* allele. Reading ancestral there would break the descent silently. Also kept: why an indel ALT at a SNP tree position stays a no-call rather than becoming a false ancestral refutation, and why the GQ gate applies only when the record carries a GQ (a bcftools mpileup call set has none, and treating that as 0 would drop everything). navigator-analysis 1,639 -> 1,497. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two more at 59 each. ibd_panel.rs keeps the orientation argument, which is the whole reason the module scores the way it does: chip allele letters are absolute, but the per-build (REF, ALT) labels in the asset are not reliably oriented to the CHM13 ALT — a large share are swapped, and the GRCh37 reference allele is often the CHM13 ALT — so scoring against the build ALT turns the dosage 0↔2 at those sites. Scoring against the CHM13 alleles, with the reverse-complement retry, cannot go wrong that way. Also kept: why palindromes stay in the panel but are skipped by the chip resolver alone. revert/mod.rs keeps the four reasons collation is the hard part, including the one that motivates the whole external sort (a read-name→record hash map is not possible at ~10⁹ records) and the one that motivates the module (an unmapped read on GRCh38 is exactly the read that may land in CHM13-resolved sequence, so it is the realignment payoff and must reach the FASTQ). Also the spill-budget history: the 256 MB constant was one run per million reads at ~340 bytes each, so a 30x WGS spilled hundreds of runs and the merge opened every one. One checker fix, and it is the recurring kind: the marker of a list item is not prose. "1. " ends in a period, so the sentence splitter counted it as a sentence of its own, and every numbered item read as one sentence longer than it is. That was inflating STE7 across the repo — repo-wide 4,774 -> 4,657 from the fix alone. navigator-analysis 1,497 -> 1,379. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
writer.rs 49, tests.rs 49, transform.rs 47 and collate.rs 32, on top of mod.rs
in the previous commit. `revert/` is now clean end to end.
The measurements that justify each design choice all survive:
- writer.rs on gzip: a 30x WGS reverted to plain FASTQ is ~200 GB, more than
the free space on a normal machine, against ~55 GB compressed — "the
difference between the pipeline running and filling the disk in its second
stage". And the one-write_all change, measured on that exact stack at
151 bp: 1,882 ns/record against 539 ns, ~13 minutes of single-threaded CPU
per realignment at ~600 M reads.
- collate.rs on why the sort is disk-backed: peak memory is the budget plus
one block per run and nothing that grows with the input, which is what lets
one code path revert a 5 GB exome and a 200 GB WGS on the same laptop.
- transform.rs on orientation: SAM stores SEQ and QUAL in reference
orientation, so emitting a reverse-strand read without flipping it back
would hand the mapper a read that never existed.
- the lockstep invariant, stated in writer.rs and pinned in tests.rs: writing
an unmatched read into `_1` shifts every later pair by one and mis-pairs
the whole rest of the file.
navigator-analysis 1,379 -> 1,202.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
cram.rs 49, sort.rs 47, markdup.rs 46, bamio.rs 41, finalize.rs 22, mod.rs 8,
and index.rs 50. `postprocess/` is now clean end to end.
The measurements and the upstream-defect record all survive:
- bamio.rs on why the stages are multithreaded: stage C moves the whole
alignment through BGZF four times, and the first WGS-scale run measured the
sort at 4 h 44 m — the most expensive stage in the pipeline, ahead of the
mapping it feeds. And open_many's counter-case: giving each of 688 spilled
runs its own worker pool spawned 4,843 threads, 15,000 IOPS and 6.6 GB/s of
disk reads to produce 5 MB/s of merged output, until the WindowServer
watchdog killed the run.
- sort.rs on the buffer estimate: the old flat 256 left out the tag
dictionary entirely, and minimap2 alone writes NM, MD, AS, ms, nn, tp, cm,
s1, s2, de, rl. That mattered little against a constant with an unwritten
margin and matters a great deal now the budget is a fraction of the machine.
- finalize.rs on why stage C emits BAM: both noodles-cram 0.94 defects, with
the reason each is a bug rather than a requirement — the index needs only
the coordinates, and cram::fs::index decodes anyway against an empty
repository that still carries its `// TODO` upstream.
- markdup.rs on the symmetry argument that makes per-end marking safe, and on
why the upstream sort must be deterministic for "first" to mean the same
thing on every run.
- index.rs on the same multi-reference panic, and why it is distinguished
from the slice-level one that means a genuinely missing contig — telling a
user to run `samtools index` for that would be wrong.
navigator-analysis 1,202 -> 939.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
47, 46 and 44. mask.rs keeps the whole YRegionClass modifier ladder verbatim — PAR 0.5, palindrome 0.4, XTR 0.3, ampliconic 0.3, STR 0.25, centromere 0.1, heterochromatin 0.1 — and the reason each class down-weights rather than drops. It also keeps the argument for why PAR and heterochromatin are per-build constants and not lifted: a chain is least trustworthy exactly there, because PAR is shared with chrX and Yq12 is satellite, so a lift through either is as likely to be wrong as absent. The palindromes and amplicons *are* lifted, because they sit in male-specific euchromatin where the chain holds. roh.rs keeps the two things the spike is honest about — the single baseline_het knob that a production version should replace with per-site 2·f·(1−f), and the uncalibrated endogamy-vs-consanguinity heuristic — plus why the run threshold and F_ROH are both physical Mb: ROH cluster in low-recombination pericentromeric regions where a multi-Mb run spans well under a cM, so a cM threshold would under-report exactly the runs that matter. scan.rs keeps both filename layouts the sidecar matcher has to satisfy at once (the ytree flat name and the bare per-analysis GATK name), and why the GVCF pattern excludes `chrY.vcf.gz` and the `.tbi`. navigator-analysis 939 -> 802. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
36, 35 and 35. readview.rs keeps both measurements that justify the trait's existence: the RecordBuf conversion was ~half the per-read CPU on a WGS BAM, and the CRAM RecordBuf::try_from_alignment_record copy measured ~1.74x the per-read decode cost on a 30x short-read WGS CRAM. sex.rs keeps the Y-scoped discriminator and why it matters: the chrX:autosome ratio is meaningless for a Y-only extract and can read as *female*, which would turn off the whole Y pipeline where nobody sees it — the Y haplogroup step skips a female before it ever downloads the tree. Also the arithmetic behind the threshold: a whole-genome male carries ~100x more autosome than chrY reads, so a genuine Y-only extract clears it by orders of magnitude. testtype.rs keeps why the absolute Scala cutoffs were replaced: a real FTDNA Big Y aligned to the whole genome measured Y 51x against autosome 1.8x off-target, which `autoCov<1` mislabels as low-pass WGS. The 28x enrichment ratio marks it, and being read-length-independent it also survives the long-read coverage underestimate. navigator-analysis 802 -> 698. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
35, 33, 33 and 32. strmarker.rs holds the Y-STR calibration record, and every figure in it is intact: the 216-kit CHM13 corpus and its acceptance rule (modal ftdna−caller difference, kept at ≥70 % agreement over ≥20 kits), the eight offsets that cross-validated against the earlier 14-kit GRCh38 corpus (DYS438 +2, DYS435 +2, DYS474 −3, DYS442 −3, DYS520 −2, DYS585 −3, DYS615 −2, DYS629 −3), the twelve markers that keep GRCh38-corpus values because the CHM13 lift dropped them, the DYS460 resolution (+1, n=180, 98 %), and the six build-dependent offsets that two independent GRCh38 corpora confirmed. cancel.rs keeps the design argument whole: why the old UI flag could not work (it was read only *between* pipeline steps, while the step ran inside a spawn_blocking closure tokio cannot interrupt), where a token may be checked and where it may not, and why a cancelled walk is an error rather than a partial result — a partial coverage is indistinguishable from a genuinely low one once persisted. One vocabulary addition: `enclosing`, `spanning` and `flanking` are GangSTR's published read-class names, so they join the -ing Technical Names alongside `genotyping array`. strcaller.rs is built on that class distinction and could not state it otherwise. navigator-analysis 698 -> 555. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
32, 27 and 26. reassembly_probe.rs is the record of what the POC actually proved, and every site and figure survives: plain realignment TIES the marginal misaligned-ref sites (4284195 scores 10/10 crude), the base-quality PairHMM breaks those ties and recovers 4 of 5 against GATK's own gVCF, and the one miss (20973395) is explained — paralog reference reads pass the MQ>=20 gate, and GATK drops them through active-region read selection, its own DP falling 7->5 there. That is named as the remaining ingredient the full caller needs. mastervar.rs keeps the compound-locus rule that a test also pins: a `ref` or `no-call` row on the same allele must not mask a `snp` row, or the locus is lost entirely. sidecar.rs keeps the `0.0`-not-`NaN` rule and its reason — the cache round- trips through serde_json, which writes NaN as null and then cannot read it back. navigator-analysis 555 -> 475. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…rref to zero Seven files, 23 down to 15 each. realign.rs keeps the worked case that justifies the whole module: on HG002 chrM, a +1C in the 16295–16301 C-run makes BWA place the insertion differently across reads, and ~47 of them then put a spurious C on the reference T at 16302 — a false T>C SNP. gzio.rs keeps the reason MultiGzDecoder is not optional: BGZF is a chain of independent gzip members, and GzDecoder decodes only the first and reports EOF, cutting short any multi-block file where nobody sees it happen. error.rs keeps the argument for catch_panic — what it is for (noodles' todo!() and expect() on inputs it does not handle), why it reports the panic's own text rather than a guess, and why AssertUnwindSafe is sound here. parity_real.rs keeps the 84 GB figure that motivated the sliding-window walker, and the ULP note explaining why the sex comparison is on the call rather than the float. Two more usage lines moved into fenced blocks so the checker stops reading cargo's `--` separator as an em-dash aside — the third and fourth use of that remedy. navigator-analysis 475 -> 332. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
sv.rs tests 14, ibd_attest 14, archaic_panel_dump 13, genotype 13, library_stats 13, callset 17, manifest 11, ibd 11, sv/evidence 11, sv/types 10. sv/types.rs keeps the measured basis for the evidence cap: across 33 analysed alignments the discordant-pair counts run 0.05–15.6 M and split reads 0–1.1 M, so 32 M is ~2x the worst case observed and fires on none of them. And what it is for — one pathological library must not take a 148-sample batch down with an OOM. sv/evidence.rs keeps the allocation reasoning behind Arc<str> contigs and the absent read name: a 30x WGS retains 3–16 M of these records, and the read name cost an allocation and ~55 bytes each to carry something clustering identifies by position alone. archaic_panel_dump.rs keeps why an independent referee is needed at all — hmmix's own tracts are enriched only 1.84x for their own archaic SNPs, so a call absent from that callset is not thereby wrong. genotype.rs keeps the full GATK/bcftools model statement, and ibd_attest.rs the agreement rule (both signed attestations carrying the same summary_hash is the proof, and only panel dosages cross the wire). navigator-analysis 332 -> 205. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The long tail, 10 violations down to 1: the sv/ submodules, contig, lib, vcf,
parity, read_metrics, callset, ibd, manifest, the ten `examples/` harnesses and
the twelve `tests/` files.
**navigator-analysis is now clean end to end: 3,435 -> 0 across 95 files.**
The evidence that survives in this batch is mostly the reasons the diagnostic
harnesses exist at all:
- archaic_outgroup_density.rs: the number that killed the density caller —
the background private-variant density varies 5.3x between its 10th and
90th percentile and is 14.6x overdispersed against the Poisson it is
modelled with, both larger than the 2.89x enrichment inside a real tract,
so the model calls its own upper tail archaic.
- archaic_private_dump.rs and archaic_callable_dump.rs: why the *input* to a
model has to be checkable independently of the model's own output.
- cram_query_probe.rs: written to diagnose a CRAM taking ~500x longer than an
equivalent BAM for one region query, and its VERIFY oracle note — a wrongly
skipped container presents as a faster caller, not a broken one.
- str_calibrate.rs: the QC step that catches a kit whose BAM and CSV are
different people (>=10 Reliable markers comparable, <70% matching).
- sv/types.rs and sv/evidence.rs keep the cap and allocation arguments from
the previous commit's siblings.
Repo-wide 4,216 -> 3,278.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Converts Rust comments to ASD-STE100 Simplified Technical English. Seven crates
reach zero:
navigator-app,navigator-analysis,navigator-store,navigator-panelbuild,navigator-refgenome,navigator-sync,navigator-resource.Repo-wide 9,343 → 3,278.
navigator-analysisalone was 3,435 → 0 across 95files.
What this is not
Enforcement is advisory.
scripts/ste-check.pyalways exits 0; there is noCI gate and no pre-commit hook for it. Scope is Rust comments under
crates/.The trade, stated plainly
The compression and the voice are gone, and that is intended. Every fact stays:
the measured numbers, the dates, the sample ids, and the recorded failure that
caused the code. Where a comment defends a constant with a measurement, the
measurement was checked against the original line by line rather than
paraphrased — the archaic frequency floors, the LD variance-inflation ladder
(52x full panel against 5.3x on a 2.6% subset), the ANCIENT_MAX_DISPERSION
calibration table, the Y-STR 216-kit offset corpus, the 4,843-thread sort
regression, the 4 h 44 m stage-C measurement.
documents/STE-dictionary.mdcarries the project's Technical Names, TechnicalVerbs, and a conversion recipe with the failure modes it teaches.
Two commits are not English changes
77a2fcfand8fdf714relocate twelve stranded doc comments onto the itemsthey actually describe. The conversion surfaced them: a block that changes
topic in the middle is easy to see when you rewrite it.
git diff --numstatproves each is a pure move.
77a2fcf): the documentation of a per-alignmentancestry estimator that exists nowhere in the repo. Its own text describes the
replacement in the past tense.
Nine checker fixes
Each was the "a violation that makes no sense is usually a checker bug" family.
Four were worth more than the file that surfaced them:
splitter read a whole table as one sentence.
1.ends in a period, so every numbered item read onesentence longer than it is. That fix alone dropped the repo count by 117 in
files this branch does not otherwise touch.
is derived— the genetics adjective, exempted only whenfromdoes notfollow. A blanket rule would have hidden ~15 genuine passives.
drift— genetic drift is a Technical Name; the idiom rule now honoursTechnical Names as
NOT_APPROVEDalready did.For
prior,drift, and the GangSTR read-class names, the handful of genuinelymetaphorical or temporal uses were rewritten first, so no exemption hides a
real violation.
Gates
cargo fmt --all --check— cleancargo clippy --all-targets -- -D warnings— clean (27m33s, exit 0)cargo test --workspace— pass (exit 0)Remaining, not in this branch:
navigator-ui1,780 ·navigator-align748 ·navigator-domain750.🤖 Generated with Claude Code