Skip to content

Central scanpy↔rapids-singlecell GPU routing; route marker DE (#38)#40

Merged
Marius1311 merged 1 commit into
mainfrom
claude/issue-38-gpu-markers
Jun 30, 2026
Merged

Central scanpy↔rapids-singlecell GPU routing; route marker DE (#38)#40
Marius1311 merged 1 commit into
mainfrom
claude/issue-38-gpu-markers

Conversation

@Marius1311

@Marius1311 Marius1311 commented Jun 26, 2026

Copy link
Copy Markdown
Member

Closes #38. Follow-up to #36 (merged).

Introduces one central mechanism for the scanpy↔rapids-singlecell compute-profile split, and routes the step where GPU genuinely helps (full-object marker DE). Built to be forwards-compatible: adding a future GPU-accelerable step is a two-callable call, not a re-implementation of the env check / device transfer / fallback.

The mechanism — cta/gpu.py

gpu.run(adata, cpu=<scanpy fn>, gpu=<rapids fn>, label=…, equivalent=…)
  • gpu_available() — single source of truth for CPU/GPU: CUDA reachable and rapids-singlecell importable (cached). Importing rsc is the real gate (it can be absent or fail to load a CUDA-mismatched kernel).
  • Moves the object to the device, runs the GPU fn, always restores to host, and falls back to scanpy on any failure — GPU is an accelerator, never a correctness dependency. No new user knob.
  • equivalent is per-op and forwards-compatible: marker DE is bit-identical CPU/GPU; a future divergent op (Leiden/UMAP — different backend) passes equivalent=False so the divergence is logged.

Routed onto it

Step op note
markers compute Wilcoxon rank_genes_groups (replaces #38's ad-hoc helper) atlas-scale, bit-identical
markers score overlap fallback DE same full-object DE op

Deliberately kept on CPU (GPU doesn't make sense there): cluster subcluster — runs on a small subset, so no real speedup, and cuGraph Leiden would diverge from scanpy igraph for no gain; and HVG (train_reference_model's seurat_v3+subset defines the model's gene set; the 30-gene subcluster HVG is trivial). The mechanism is ready to route these later if that ever changes.

GPU enablement (found by verifying on hardware — not the clean drop-in the issue assumed)

  • rapids-singlecell[rapids-cu12] (its full RAPIDS stack: cugraph + cuvs) + dask (imported at load, only declared under doc/test extras).
  • cuda-cudart = "13.*" + a gpu-only [activation.env] LD_LIBRARY_PATH=$CONDA_PREFIX/lib — rapids-singlecell 0.15.2's compiled DE kernel is CUDA-13 (libcudart.so.13) while the stack is CUDA-12; without the cu13 cudart on the loader path the kernel silently no-ops to CPU. The two cudart majors coexist; torch/cupy/cuml still import. (Latent in other lab repos too.)
  • docs: tool_registry.md + compute_environments.md (central mechanism + per-op equivalence).

Verification (Euler RTX 4090)

  • Markers: GPU bit-identical to CPU — hypomap (30,754×51,676, 66 cl) Jaccard 1.000 / logFC corr 1.0; pbmc3k 373 shared markers identical. Atlas HNOCA (499,996×32,068): GPU 4.0 min vs CPU >27 min.
  • CPU paths unchanged — smoke tests green; routing falls through to scanpy off-GPU.

🤖 Generated with Claude Code

@Marius1311 Marius1311 force-pushed the claude/issue-38-gpu-markers branch from 4e70002 to 027d563 Compare June 26, 2026 15:17
@Marius1311 Marius1311 changed the title GPU-route atlas-scale marker DE via rapids-singlecell (#38) Central scanpy↔rapids-singlecell GPU routing; route marker DE + sub-cluster (#38) Jun 26, 2026
Introduce one central mechanism for the compute-profile split instead of per-command
plumbing, and route the steps where GPU genuinely helps (full-object marker DE).

`cta/gpu.py` owns: the CPU/GPU decision (`gpu_available()` — CUDA reachable AND
rapids-singlecell importable, cached), the host<->device transfer, and the CPU fallback.
A GPU-accelerable step is then one call:

    gpu.run(adata, cpu=<scanpy>, gpu=<rapids>, label=..., equivalent=...)

No new user knob; GPU is an accelerator, never a correctness dependency (any GPU failure
logs and falls back to scanpy). `equivalent` is per-op (forwards-compatible): marker DE is
bit-identical CPU/GPU; a future divergent op (Leiden/UMAP — different backend) would pass
`equivalent=False` to log the divergence.

Routed onto the mechanism:
- compute_markers.py — Wilcoxon `rank_genes_groups` (atlas-scale: seconds vs ~minutes;
  replaces #38's ad-hoc helper).
- score_signatures.py — the overlap fallback DE (same full-object DE op).
Deliberately CPU (where GPU doesn't make sense): sub-clustering (runs on a small subset — no
real speedup, and cuGraph Leiden would diverge from scanpy igraph for no gain) and HVG
(train_reference_model's seurat_v3+subset defines the model's gene set; the 30-gene subcluster
HVG is trivial). The mechanism is ready to route them later if that changes.

GPU enablement (rapids-singlecell on the CUDA-12 gpu env) — found by verifying on hardware:
- rapids-singlecell[rapids-cu12] (full RAPIDS: cugraph/cuvs) + dask (imported at load, only
  declared under doc/test extras).
- cuda-cudart = "13.*" + a gpu-only [activation.env] LD_LIBRARY_PATH=$CONDA_PREFIX/lib:
  rapids-singlecell 0.15.2's compiled DE kernel is CUDA-13 (libcudart.so.13) while the stack is
  CUDA-12; without the cu13 cudart on the loader path the kernel silently no-ops to CPU. The two
  cudart majors coexist; torch/cupy/cuml still import.
- docs: tool_registry.md + compute_environments.md (central mechanism, per-op equivalence).

Verified on Euler RTX 4090:
- markers: GPU bit-identical to CPU (hypomap Jaccard 1.000 / logFC corr 1.0; pbmc3k 373
  shared markers identical). Atlas (HNOCA 499996x32068): GPU 4.0 min vs CPU >27 min.
- CPU paths unchanged (smoke tests green; routing falls through to scanpy off-GPU).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@Marius1311 Marius1311 force-pushed the claude/issue-38-gpu-markers branch from 027d563 to 9e51800 Compare June 30, 2026 10:49
@Marius1311 Marius1311 changed the title Central scanpy↔rapids-singlecell GPU routing; route marker DE + sub-cluster (#38) Central scanpy↔rapids-singlecell GPU routing; route marker DE (#38) Jun 30, 2026
@Marius1311 Marius1311 merged commit fa6a143 into main Jun 30, 2026
0 of 2 checks passed
@Marius1311 Marius1311 deleted the claude/issue-38-gpu-markers branch June 30, 2026 11:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

GPU-accelerate atlas-scale marker DE via rapids-singlecell (routed by compute profile)

1 participant