Skip to content

Remove dead code; stdout writers are the file writers on another output - #365

Open
BenjaminDEMAILLE wants to merge 17 commits into
mainfrom
chore/dead-code
Open

BenjaminDEMAILLE wants to merge 17 commits into
mainfrom
chore/dead-code

Conversation

@BenjaminDEMAILLE

Copy link
Copy Markdown
Contributor

Part of #354 (finding 20).

Stacked on #346. perf/hotpaths is on a fork, so GitHub cannot use it as the base of a PR here; the base is main, and until #346 lands the diff shows its commits too. Only the last commit (bcddd31) is this PR: review with git diff aeba641 bcddd31. Retarget or rebase after #346 merges.

Problem

The tree carried code nothing runs. One block of it cost time: the stitcher scored every seed before extension for every window, then discarded the result with let _ = best_pre_score. The stdout writers were copies of the file writers that had drifted: SamStdoutWriter lacked the [SAM-MISMATCH] check.

Change

12 files, 265 insertions, 1056 deletions. Each item was grepped across src, tests, examples, benches and scripts, then confirmed by the compiler once removed.

Removed How it was verified dead
Stitcher block ending in let _ = best_pre_score, and the pre_ext_score field it filled Result discarded; no other reader of the field
Reverse-strand branches of the stitcher stitch_seeds_core returns stitch_cluster with is_reverse: false, the only cluster stitch_recurse, stitch_align_in_place and finalize_transcript receive. is_reverse goes from extend_alignment(_cached), score_region, count_mismatches_in_region
Reverse-strand branches of the junction scan (not named in the finding, same cause) find_best_junction_position_cached is called only from stitch.rs, always forward. is_reverse, n_genome, genome_offset go from find_best_junction_position(_cached) and ScanArgs; score_gap_with_strand merged into score_gap. examples/jrbench.rs passed these arguments and is updated
BufferedBamRecords, SamWriter::write_alignment and its test No production caller. test_sam_output_includes_rg_header_and_tag used write_alignment; it now uses build_unmapped_record and write_batch, and still checks the RG header and tag
Seed::find_paired_seeds, Seed::get_genome_positions, combined_transcript_for_projection Called only by their own tests, which go with them
parse_gtf, extract_junctions_from_exons Wrappers used only by tests, which now call parse_gtf_configured and extract_junctions_configured
word_comp_length No caller
Duplicate in src/index/mod.rs The transcript tables and sjdb files were written by two copies; both paths now call write_transcript_tables and write_sjdb_files
SamStdoutWriter, BamStdoutWriter, SortedBamStdoutWriter Replaced by SamWriter<W>, BamWriter<W> (default type parameter is the file writer, so existing call sites are unchanged) and SortedBamWriter::create_stdout

Every item was dead in production; none had to be kept. stitch_is_reverse and original_is_reverse, which carry the read's real strand, are untouched.

These were pub items of the library crate, so this is an API removal, marked in CHANGELOG.md.

One intended difference: --outStd SAM now has the file writer's [SAM-MISMATCH] check. No run below triggers that check, so the check itself on stdout is not exercised; the output of those runs is identical.

Conflict with #355

git merge-tree reports conflicts with fix/output-completeness (#355) in CHANGELOG.md, src/io/sam.rs and src/lib.rs. This branch does not add finish, because #355 has not landed and this PR changes no behaviour. Whichever lands second must keep the flush: #355's SamWriter::finish goes on the generic impl<W> SamWriter<W>, the AlignmentWriter impl on impl<W> AlignmentWriter for SamWriter<W>, and its SamStdoutWriter::finish is dropped along with that type. I did not carry out or test this resolution.

Measured

macOS, aarch64, 16 cores. Baseline: clean build of aeba641. Comparison: bcddd31.

Output. 84 scenarios, each at --runThreadN 1, 4 and 8, every binary under the same argv[0] and relative prefix so that command lines in headers match. md5 of every output file, raw BAM and SAM included, plus decoded BAM records and Log.final.out without its clock lines. Log.out, Log.progress.out and stderr are not in the md5s.

Set Scenarios Rows compared Rows differing
Generated fixture (20 000 reads) and tiny fixtures 75 2367 4 (below)
Earlier fixtures, including rejected inputs 9 132 0

The 4 rows are Solo.out/{Gene,GeneFull}/CellReads.stats at 4 and 8 threads in one scenario. This is existing behaviour: two runs of the baseline differ from each other in the same file, with the same md5 once sorted (64136fbd), so only the row order changes between runs. At 1 thread the file is identical. Not touched here.

Mode Scenarios cover
Single-end SAM, BAM unsorted and sorted, --outStd in all three forms, Within, TranscriptomeSAM, GeneCounts, Fastx, chimeric junctions and WithinBAM, BySJout, fixed clip, clip to empty, adapter, CellRanger4, bedGraph, WASP, splicing tag, per-file RG, NoQS, None, 2-pass, read limit
Paired-end The same where it applies, plus half-mapped pairs
Solo SAM, BAM, Gene, GeneFull, SJ, Velocyto, EM, CB_samTagOut, clip, CR4, RG, None
Solo paired Third barcode file and --soloBarcodeMate 1, half-mapped, CB_samTagOut, clip, RG, None

The fixture was checked to reach these paths on the baseline: SE 15 965 unique, 1614 multi, 1417 too short, 1003 other, of 20 000 reads; a second fixture gives 2151 (SE) and 2152 (PE) chimeric reads, with as many supplementary records under WithinBAM; relaxed --outFilter*OverLread gives 92 records of half-mapped pairs in PE and in solo paired.

Scenarios the program refuses: sp_tag_m (one invalid parameter combination of mine, all 3 thread counts) and 5 of the 9 earlier scenarios (input files the reader rejects; the 3 I compared give the same error text). Same exit code 1 on both binaries; they prove little beyond that.

Exit codes, decoded records and non-header bytes are also identical across 1, 4 and 8 threads within each binary, apart from the same CellReads.stats rows.

Speed. --runThreadN 8. The machine was not idle: unrelated bwa-mem2 and salmon jobs were running, so wall time is noisy (the same binary varies between 4 and 16 s), and I compare instructions retired and CPU time. From /usr/bin/time -l, binaries alternated.

Case Input Instructions (median), base this PR Change CPU s (median of 5), base this PR
SE, largest fixture B200.fq, 801 000 reads, 176 MB 103.47 G 93.32 G −9.8 % 10.94 10.29
SE 800 000 reads 376.35 G 344.33 G −8.5 % 30.48 28.67
PE 800 000 pairs 1466.2 G 1400.2 G −4.5 % 110.70 106.69
Solo 800 000 reads 382.25 G 350.08 G −8.4 % 30.34 28.52
Solo paired 800 000 pairs 1472.7 G 1407.1 G −4.5 % 111.88 106.33

Instructions: 3 runs for SE and solo, 1 run for PE and solo paired. The reads are synthetic and short; expect a different figure on real data.

Tests

cargo test --release passes (740 library tests and all integration tests, tests/determinism.rs and tests/phase9_threading.rs included). cargo clippy --all-targets has no warnings; cargo fmt applied. No test was added: this removes code.

Not checked

  • I did not run test/parity_suite.sh, and did not compare against STAR here; the claim is "same output as aeba641".
  • Real sequencing data, gzip input, Linux.
  • docs-old still names pre_ext_score and get_genome_positions; it is historic and left alone.

Also dead but not named in the finding, left in place: AlignmentScorer::score_gap is called only by tests, and stitch_seeds / stitch_seeds_with_jdb have no production caller beyond a re-export and each other.

🤖 Generated with Claude Code

BenjaminDEMAILLE and others added 17 commits October 29, 2026 13:40
STAR's trA.nMM is summed while the transcript is built and never recounted. When a junction scan moves the junction back into exon A (jR < 0), STAR counts those bases again against the acceptor without taking back the mismatches A had already counted (stitchAlignToTranscript.cpp, the commented-out nMM--). finalize_transcript now reports the stitched count plus its extensions; the two halves of a pair carry the pair's count once. Fixes nM on chr21 ERR188021.1648513/1662773 and 21 fusion pairs (DIVERGENCE.md 5).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
STAR's mate loop starts at exons[0], whose EX_iFrag is trOut.Str (ReadAlign_outputTranscriptSAM.cpp, ReadAlign_alignBAM.cpp): mate 1 when it is forward, mate 2 otherwise. rustar-aligner always wrote mate 1 first, about half of STAR's pairs; the per-read multiset comparison of the parity suite did not see it. The transcriptome path already did this.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
It has no counterpart in STAR, whose window dedup (blocksOverlap in stitchWindowAligns.cpp) compares the whole pair. It collapsed two alignments with the same mate positions but a different indel or splice placement inside mate 1 (fusion fustut000009a03e, fustut000009a03f, fustut0000167ff6).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
STAR's assignAlignToWindow recomputes WALrec over the non-anchors of a full window before it looks at the new align; with none it sets MARKER_TOO_MANY_ANCHORS_PER_WINDOW and nW=0, so the read is 'other'. rustar-aligner skipped an incoming non-anchor instead, so the read could map (chr21 ERR188021.1789477) or count as 'too short' (yeast 1, chr21 17, fusion 73 reads). New integration test in tests/unmapped_buckets.rs; it fails on the previous code.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
STAR's gEem starts from unique plus uniform counts, zeroes values below 0.01 at the start of each round, and stops when no gene moves by 0.01 or after 101 rounds, so it can keep a value below 0.01 (SoloFeature_collapseUMIall.cpp). The UniqueAndMult matrices also list every gene of the cell's unique or multi-gene counts, a zero included (SoloFeature_outputResults.cpp).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
STAR records them with their whitelist candidates and resolves the barcode at counting time, as for a unique-gene read (SoloReadFeature_inputRecords.cpp); one it cannot resolve counts as noTooManyWLmatches. rustar-aligner dropped them, which changed GeneFull Features.stats and Summary.csv on the mouse chr19 fixture.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
STAR's collapseUMIall filters multi-gene UMIs on every feature. For SJ, a read crossing two junctions is one UMI on two features, a tie MultiGeneUMI_CR drops (4 UMIs on the mouse chr19 fixture). The gene path's filter-then-dedup is now shared by both.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
DIVERGENCE.md 5, the README comparison tables and residuals, and CHANGELOG.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
SpliceJunctionDb is only looked up (is_annotated), never iterated, so the hasher has no effect on output. It is consulted on every junction stitch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every leaf of the recursion compared itself with each transcript already recorded in the window, recomputing their final score (a log2) and mapped length each time. They are now computed once, when the transcript is recorded.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
score1 changes by at most +1 per position, so when score1 plus the positions left plus the best motif score cannot beat the best score so far, the remaining positions cannot be chosen. Ties count for the reverse-strand deletion case only, as in the scan itself. The chosen position, motif and repeat lengths are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Within one window the read, strand and mismatch limits are fixed, so an extension is determined by its start, direction, length cap and the transcript's mismatches and length so far; the recursion's leaves repeat many of them. Windows with fewer than 25 seeds skip the memo, where hashing cost more than it saved (+1.7% instructions on chr21 without the threshold).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The exact-duplicate filters (SE transcripts, PE pairs) formatted every CIGAR into a String and hashed it with SipHash. They now key on the positions with FxHashMap and compare the CIGAR ops of transcripts at the same position; the first occurrence is kept, as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
stitch_recurse used to clone the working transcript at every include branch (and stitch_align_to_transcript cloned it again), so the exclude branch could keep the original. It now stitches in place on a single transcript and undoes the change afterwards: a stitch only pushes exons and junctions, changes the last exon and updates scalar fields, and a leaf's extensions also change the first exon, so a small WtMark restores it. A leaf takes its mark only when an extension applies, and a recorded transcript is cloned once, when it enters the window list. stitch_align_to_transcript remains, for tests, as a copy-then-stitch wrapper; ExonBlock is now Copy.

Output is unchanged: alignments, SJ.out.tab, Chimeric.out.junction, Log.final.out and Solo.out are identical on the yeast, chr21, fusion and STARsolo runs, and the full fusion set still matches STAR 2.7.11b. Single-threaded instructions: fusion 30k pairs -22%, yeast 100k PE -2.7%, chr21 100k SE unchanged (+0.4%, within noise).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Phase 2 of find_best_junction_position read the read base, the upstream and downstream genome bases and the four motif bases position by position, through the genome view. It now copies them into stack buffers first (SeqView::bases_into, one copy per run; the reverse strand's donor run is read backwards), computes the score1 steps in one branch-free pass, and scans the arrays: same positions, same order, same tie rules, same early stop. Positions off the read or the genome become the out-of-range sentinel, which every test there treats as STAR's skip. Scans longer than 512 positions, or whose coordinates would leave the genome, take the position-by-position path, kept as scan_phase2_scalar.

buffered_junction_scan_matches_the_scalar_one checks the two on 20,000 random scans (both strands, deletions and introns, N bases, genome ends). Output is unchanged on the yeast, chr21, fusion and STARsolo runs, and the full fusion set and chr21 PE still match STAR 2.7.11b. Single-threaded instructions: chr21 100k SE -16%, yeast 100k PE -13%, fusion -1%.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The extension memo added 4 to 6 MB of private memory on read sets with many
large windows, though no table exceeded 4,191 entries (about 500 KB). Creating
and freeing a table per window is the likely cause, through heap fragmentation.
The tables now live for the thread and are cleared between windows, and an
extension entry takes 40 bytes instead of 64. As the extension table now
outlives the window, it stops storing at 14,336 entries.

Fusion PE, 150k pairs, 8 threads: 51 -> 39 MB of private memory (44 MB before
the memo), with 0.7 % fewer instructions.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Removes, each verified unused in production:
- the stitcher's scoreSeedBest block, ending in `let _ = best_pre_score`,
  and the `pre_ext_score` field it filled
- the reverse-strand branches of the stitcher and the junction scan:
  `stitch_cluster.is_reverse` is always false, so `is_reverse` / `n_genome` /
  `genome_offset` go from those functions
- BufferedBamRecords, SamWriter::write_alignment, Seed::find_paired_seeds,
  Seed::get_genome_positions, combined_transcript_for_projection, parse_gtf,
  extract_junctions_from_exons, score_gap_with_strand, word_comp_length
- the transcriptome/sjdb file tail duplicated in index/mod.rs

SamStdoutWriter, BamStdoutWriter and SortedBamStdoutWriter are replaced by
SamWriter<W>, BamWriter<W> and SortedBamWriter::create_stdout. --outStd SAM
gains the file writer's [SAM-MISMATCH] check.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant