RareSim

A rare disease similarity pipeline for phenotype-driven diagnosis support. Given a patient's HPO terms or clinical text, RareSim ranks diseases by phenotypic similarity using multiple methods — semantic IC-based, set-based, TF-IDF, transformer, and LLM.

---

Overview

Patient HPO terms / clinical text
        │
        ▼
  HPO Extraction          <-dictionary, NER, FastHPOCR, GPT, PhenoBrain
        │
        ▼
  Similarity Pipeline     <-semantic (Resnik/Lin/JC), set-based, TF-IDF, transformer, LLM
        │
        ▼
  Ranked Disease Results  <-scored against ~10,000 rare diseases

Disease knowledge is built from four ontologies: HP, ORDO, MONDO, and HOOM, merged and canonicalized to ORPHA identifiers.

---

Repository Structure

RareSim/
  packages/
    raresim-core/          <-installable Python package (core logic)
    raresim-api/           <-FastAPI backend (wraps raresim-core)
    raresim-frontend/      <-Vue 3 web interface

  scripts/
    setup/                 <-one-time setup: ontologies, artifacts, third-party tools
    evaluation/             <-evaluate pipeline performance
    validation_tools/      <-run and compare existing tools (LIRICAL etc.)
    experiments/            <-ad-hoc experiments
    analysis/               <-analyse outputs, automated reporting

  tests/
    unit/
    integration/
    validation_tool/
    evaluation/

  data/                    <-gitignored large files (see data/README.md)
    ontologies/             <-hp.owl, ordo.owl, mondo.owl, hoom.owl
    datasets/                <-HMS.json, MME.json, LIRICAL.json, phenopackets/

  outputs/                 <-gitignored, generated at runtime
    artifacts/              <-precomputed profiles, IC, ancestors, labels
    transformer/
    semantic/
    evaluation/
    validation/
    gui/

  third_party/             <-externally cloned tools (gitignored)
    fast_hpo_cr/

  wiki/                    <-VitePress documentation site (see "Project Wiki" below)
    .vitepress/
      config.mts

  docs/
    notebooks/

  pyproject.toml           <-root: dev tooling + uv workspace
  setup.sh                 <-bootstrap script: third-party tools, ontologies, artifacts
  .env                     <-local paths (not committed)
  README.md

---

Quickstart

1. Prerequisites

• Python 3.11+
• Node.js 18+ (frontend only)
• uv (recommended) or pip

2. Clone the repo

git clone https://github.com/your-org/RareSim.git
cd RareSim

3. Set up environment

# create and activate virtual environment
python -m venv .venv
source .venv/bin/activate

# install all packages (uv workspace)
uv sync

# or with pip
pip install -e packages/raresim-core
pip install -e packages/raresim-backend

4. Configure paths

Create a .env file at the repo root:

RARESIM_ROOT=/path/to/RareSim
OPENAI_API_KEY=sk-...        # optional — only needed for GPT extraction

5. Run the setup script

./setup.sh

This bootstraps RareSim from a fresh clone to a runnable state, and all three steps are required; RareSim will not run without them:

• Clones third-party tools (FastHPOCR) into third_party/, skipping anything already cloned.
• Downloads the ontology sources into data/ontologies/.
• Builds the shared artifacts into outputs/artifacts/.

Running the steps individually

Useful for debugging a single step, or if you only need to rebuild artifacts after an ontology update.

Set up third-party tools

python -m raresim.build.setup_third_party

Clones FastHPOCR into third_party/. Skips anything already cloned.

Download ontologies

python -m raresim.build.load_ontologies_to_local

Downloads the following files into data/ontologies/ (see data/README.md for exact versions and sources):

File	Source
hp.owl	hpo.jax.org
ordo.owl	BioPortal
mondo.owl	GitHub
hoom.owl	BioPortal
phenotype.hpoa	hpo.jax.org
en_product4_HPO.xml	Orphadata
disease_to_phenotypic_feature_association.all.tsv.gz	Monarch

Build shared artifacts

python -m raresim.build.build_shared_artifacts

Generates precomputed files in outputs/artifacts/:

• canonical_disease_profiles.json
• hpo_labels.json
• hpo_ancestors.json
• information_content.json
• alias_to_canonical.json

6. Standardize phenopackets (optional)

Only needed if you are using phenopacket datasets for evaluation.

python scripts/evaluation/data_prep/standardize_phenopackets.py

Converts raw phenopackets from data/datasets/phenopackets/raw/ into a standardized [[HP terms], [disease codes]] format, saved per release folder under data/datasets/phenopackets/standardized_to_json/.

7. Run the pipeline

Terminal interface:

# from HPO terms
python packages/raresim_cli/app.py --hpo HP:0001251,HP:0000545

# from a specific similarity method
python packages/raresim_core/similarity_methods/semantic/pipeline.py

Web interface:

# terminal 1 — backend
uvicorn raresim_api.main:app --reload --port 8000

# With GPU: 
CUDA_VISIBLE_DEVICES=4,5 uvicorn raresim_api.main:app --reload --port 8000

# terminal 2 — frontend
cd packages/raresim-frontend
npm install
npm run dev

Open http://localhost:3000.

---

Similarity Methods

Method	Type	Description
semantic_resnik_bma	Semantic	Resnik IC-based Best Match Average
semantic_lin_bma	Semantic	Lin normalized IC similarity
semantic_jiang_conrath_bma	Semantic	Jiang-Conrath distance-based
set_cosine	Set-based	Cosine similarity over HPO term vectors
set_jaccard	Set-based	Jaccard index
set_dice	Set-based	Sørensen-Dice coefficient
set_overlap	Set-based	Szymkiewicz-Simpson overlap
tfidf	TF-IDF	Term frequency-inverse disease frequency
transformer	Embedding	Sentence transformer cosine similarity
llm	LLM	GPT-based disease ranking

---

HPO Extraction Methods

Method	Description
dictionary	Exact HPO label matching (fast baseline)
biomedical_ner	d4data/biomedical-ner-all transformer NER
fast_hpo_cr	FastHPOCR morphological token cluster matching
chatgpt	GPT-4o-mini prompted extraction
phenobrain_api	PhenoBrain public BERT-based API

The phenotype extraction pipeline supports multiple methods. Some require additional setup:

Dictionary

No setup required. Uses regex matching against HPO labels.

Biomedical NER (d4data)

Requires transformers and torch — already in requirements.txt. The model (d4data/biomedical-ner-all) will be downloaded automatically from HuggingFace on first use.

FastHPOCR

Morphological HPO concept recognition (recommended): The index will be built automatically on first use (~12 min) and cached to outputs/fast_hpo_cr_index/.

ChatGPT Extraction

Requires an OpenAI API key. Add to your .env file: OPENAI_API_KEY=sk-...

PhenoBrain API

No API key required. Uses the public PhenoBrain endpoint. Requires pip install requests (already in requirements.txt).

---

Development

Running tests

pytest tests/

Code style

ruff check .
ruff format .

Adding a new similarity method

1. Create packages/raresim-core/src/raresim/similarity_methods/<name>/
2. Add methods.py and pipeline.py following existing patterns
3. Register the method name in scripts/run_pipeline.py

---

Package Architecture

raresim-core            <-pure logic, no HTTP
    │  Python import
raresim-api             <-FastAPI, wraps raresim-core
    │  HTTP /api/*
raresim-frontend        <-Vue 3, calls raresim-api

raresim-core has zero knowledge of the API or frontend. Scripts and tests import directly from raresim-core.

---

Project Wiki

Project documentation (method write-ups, setup guides, validation-tool notes, etc.) lives in a VitePress site under the wiki/ folder of this repository, rather than in GitLab's built-in wiki. The existing GitLab wiki pages have already been migrated into wiki/; continue updating them there, or add new pages following the steps below.

One-time setup

Install Node.js (if not already installed):

nvm install 24

From the repository root, install the project dependencies:

npm install

This installs VitePress and only needs to be run once (or again after wiki/.vitepress/config.mts or its dependencies change).

Adding or editing a page

1. Create a new Markdown file inside wiki/, named Category_PageName.md (e.g. SimilarityMethods_SetBased.md).

2. Write the page content in that file.

3. Open wiki/.vitepress/config.mts and add the new page under the correct sidebar category, using a link that matches the filename without the .md extension:

   { text: 'Set-based', link: '/SimilarityMethods_SetBased' },

To edit an existing page, just edit its Markdown file directly; no config changes are needed unless you are moving it to a different sidebar category.

Previewing locally

npm run wiki:dev

Open the local URL printed in the terminal to preview the site with your changes.

---

Data Sources

See data/README.md for full provenance, versions, and download instructions.

---

License

MIT