Simplify + align marker step: gene filter in canonicalize, default-param Wilcoxon, drop COSG (#36)

[Marius1311]

Closes #36.

Aligns the marker step with the scanpy clustering tutorial + sc-best-practices annotation chapter: one DE/specificity source, one fewer dependency, no new user-facing options.

Changes

Change 1 — upfront gene filter in canonicalization (canonicalize.py, constants.py)

• sc.pp.filter_genes(slim, min_cells=GENE_FILTER_MIN_CELLS=5) right before the slim is written, so every downstream step inherits one clean, consistent gene set.
• Absolute, not relative-to-n_obs on purpose: a relative threshold (e.g. 0.1%×500k=500) would delete genuine rare-cluster markers (the 27-cell-microglia / P2RY12 case). The floor is bounded by the smallest cluster you want to annotate, so it stays low and absolute. This is a noise-cleanliness step, not the atlas-speed lever.
• Filter result recorded in canonicalization.json (gene_filter.min_cells / .dropped).

Change 2 — marker method (compute_markers.py)

• Default-param Wilcoxon: drop tie_correct=True → scanpy default False (keep pts=True, use_raw=False).
• Remove COSG and the --no-cosg/--cosg flag; specificity now via sc.tl.filter_rank_genes_groups → markers_filtered_<level>.csv (cluster, rank, gene, logfoldchange, score, pct_in, pct_out).

Change 3 — COSG blast radius

• cluster_confidence.py: proxy re-expressed on log2FC (lfc_top1, lfc_margin, n_strong); CONFIDENCE_BANDS recalibrated to the logFC scale; --strong-thresh default 0.02 → 1.0. Writes a stable header even when no markers survive.
• build_report.py, report_plots.py, scaffold_cards.py: repointed from cosg_<level>.csv to markers_filtered_<level>.csv.
• score_signatures.py: the rarely-hit fallback DE aligned to default-param (no tie_correct) so it can't drift from the canonical markers table. (DE still computed once in the normal flow — filter_rank_genes_groups reuses the rank_genes_groups result, and score_signatures reuses the written markers_<level>.csv.)
• pixi.toml / pixi.lock: cosg removed (lock regenerated cleanly).
• Docs: SKILL.md + reference/{differential_expression,marker_tools,methodology,artifacts,tool_registry}.md, incl. the "not measured vs not expressed" panel-gating note and canonicalize documented as a standard Stage-1 step.
• tests/test_smoke.py: drop --no-cosg, assert the markers_filtered schema.

Naming note

The issue refers to prep_slim.py / cta data prep-slim; that step is now canonicalize.py / cta data canonicalize (its docstring says it "replaces the old prep-slim"), so Change 1 landed there.

For review (judgment calls)

• Recalibrated bands: ((3.0,5,0.9),(2.0,3,0.8),(1.0,1,0.7),(0.5,0,0.6)), base 0.45; --strong-thresh 1.0 (log2FC≥1 = 2-fold). Best-defensible mapping from COSG's 0–1 cosine scale to logFC — easy to retune.

Verification

• Smoke tests green (4/4).
• Full canonicalize → markers → confidence → scaffold-cards chain on the benchmark harness's blinded, log-normalized pbmc3k (2638 cells, 8 clusters): textbook PBMC markers per cluster (T/B/NK/mono/DC/platelet), the logFC confidence proxy populates, and both rare clusters (15 & 37 cells) keep 50 specific markers each — the 15-cell platelet cluster (PPBP/PF4) has the highest specificity (lfc_top1=11.7), validating the rare-marker rationale.
• On real HNOCA (499,996 × 35,725): the gene filter drops 3,657 noise-tail genes (→32,068) and P2RY12 survives (the 27-cell Microglia marker). This also confirmed the issue's framing that the filter is noise-cleanliness, not the speed lever — the DE itself stays the bottleneck (→ follow-up GPU-accelerate atlas-scale marker DE via rapids-singlecell (routed by compute profile) #38, GPU routing).

Follow-up

Companion #38 (GPU-route atlas-scale DE via rapids-singlecell) lands separately on top of this.

🤖 Generated with Claude Code