AI-assisted diabetic retinopathy screening. Upload a retinal photograph; the app grades it 0–4 on the international clinical scale, shows where the model looked via Grad-CAM, and drafts both a patient explanation and a clinical referral note.
This is a triage tool that extends an ophthalmologist's reach — not a diagnostic replacement. Research prototype. Not a certified medical device. Always consult an ophthalmologist.
./start.shThat's the whole thing. On first run it creates the Python venv, installs both dependency sets, then starts the API and the frontend together. Ctrl-C stops both.
Open the URL Vite prints (usually http://localhost:5173).
The script picks a free port for the API starting at 8001 and points the Vite proxy at whichever one it got, so a busy port doesn't break the run.
# terminal 1 — API
python3 -m venv .venv
.venv/bin/python -m pip install -r backend/requirements.txt
.venv/bin/python -m uvicorn backend.app.main:app --host 127.0.0.1 --port 8001
# terminal 2 — frontend
cd frontend
npm install
npm run devIf you run the API on a port other than 8001, tell Vite where it is:
VITE_API_TARGET=http://127.0.0.1:9000 npm run dev- Python 3.11+ (verified on 3.14). CPU-only — no GPU needed.
- Node 20+ (verified on 24).
- A
.envat the repo root withGROQ_API_KEY=.... Without it the app still works; reports fall back to a deterministic template and are labelledoffline templatein the UI.
.
├── start.sh one command, both servers
├── dr_efficientnet_b0.pth trained weights (EfficientNet-B0, 5 classes)
├── sample_scans/ 20 held-out fundus photographs, grade in filename
├── evaluation/ Grad-CAM panels written by backend/evaluate.py
├── reference/ training script + the original hero design
├── backend/
│ ├── requirements.txt
│ ├── evaluate.py honest baseline evaluation over sample_scans/
│ ├── calibration.py the confidence-calibration investigation
│ ├── build_reference.py rebuilds the domain gate's embedding bank
│ └── app/
│ ├── main.py FastAPI routes
│ ├── preprocessing.py VERBATIM copy of the training preprocessing
│ ├── inference.py model singleton, prediction, Grad-CAM
│ ├── fundus.py domain gate — refuses non-retinal uploads
│ ├── quality.py image-quality gate
│ ├── report.py Groq report layer + offline fallback
│ ├── config.py paths, thresholds, env
│ └── schemas.py response models
└── frontend/
└── src/
├── pages/ Landing, Screening, Batch, HowItWorks
├── components/ Nav, Dropzone, Results, Skeletons, Disclaimer
├── api.ts typed client
└── index.css design tokens, liquid-glass, all animation
Base URL http://127.0.0.1:8001. Interactive docs at /docs.
| Endpoint | Method | Purpose |
|---|---|---|
/health |
GET | Liveness, model-loaded flag, thresholds |
/predict |
POST | One image → grade, confidence, Grad-CAM, report |
/predict-batch |
POST | Many images → urgency-ranked worklist |
curl -X POST http://127.0.0.1:8001/predict \
-F "file=@sample_scans/grade4_ff8a0b45c789.png"/predict returns grade (0–4), grade_label, confidence (max softmax),
referable (grade ≥ 2), uncertain (confidence < 0.70), probabilities,
gradcam (base64 PNG), hotspots, preprocessed, quality, fundus,
inference_ms, and report.
hotspots are the strongest attention points as normalised {x, y, weight},
computed from the raw activation map so the UI can mark them on the overlay.
Finding them client-side would not work — the overlay is a 50/50 blend with
retinal tissue, which is itself red.
report carries a one-sentence headline, a patient_explanation and a
referral_note. Decision-carrying words arrive wrapped in **double asterisks**;
the client renders those as emphasis and nothing else in those strings is markup.
/predict-batch sorts by the order a screening clinic would actually work
through: referable first, then anything needing human review, then by descending
grade, then by descending confidence.
Uploads that are not retinas are refused, not graded. /predict answers
422 with {"detail": {"code": "not_fundus", "message": ..., "metrics": ...}};
/predict-batch puts them in errors with code: "not_fundus" and counts them
in summary.not_fundus, so a stray photograph never lands in a clinical
worklist. See "The domain gate" below.
This is the part that silently breaks everything if it drifts.
crop_image_from_gray and preprocess_image in
backend/app/preprocessing.py are copied
verbatim from reference/dr_train_kaggle_v2.py, Cell 2. Uploads arrive as
bytes but the verbatim function takes a path, so uploads are spooled to a temp
file and passed to the original function rather than reimplemented against a
buffer. The I/O cost is nothing next to the guarantee.
The pipeline, per image: BGR→RGB → crop the black border at intensity > 7 →
resize to 224×224 → Ben Graham contrast
(addWeighted(img, 4, GaussianBlur(img, sigma=22.4), -4, 128)) → ToTensor →
normalise to ImageNet mean/std.
Re-run backend/evaluate.py after touching anything near this. A preprocessing
mismatch shows up there as a collapsed prediction spread long before it shows up
in the UI.
.venv/bin/python backend/evaluate.py --panelsGrades all 20 held-out scans and writes labelled Grad-CAM panels to
evaluation/. Measured baseline:
| Metric | Value |
|---|---|
| Quadratic Weighted Kappa | 0.925 |
| Within one grade | 100% (20/20) |
| Exact-grade agreement | 70% (14/20) |
| Referable sensitivity | 100% (12/12, zero missed referrals) |
| Referable specificity | 87.5% (1 over-call) |
| Off by ≥2 grades | none |
Training-time validation QWK was 0.84. The 0.925 here comes from 20 scans balanced four per grade; real screening populations are ~70% grade 0, so 0.84 is the more honest estimate of field performance.
The model is systematically overconfident on "No DR". In evaluation, a true
grade 1 scan was called No DR at confidence 1.00 — the uncertain flag did
not fire and could not have.
We investigated (backend/calibration.py):
- Temperature scaling — the training validation split isn't available, so it can only be fitted leave-one-out on 20 points. ECE moves 0.157 → 0.112, but the bootstrap CIs ([0.052, 0.367] vs [0.058, 0.335]) overlap almost entirely. Worse, AUROC for error detection drops from 0.714 to 0.655. The failing scan has a logit gap of 9.21 — inside the range of correct grade-0 predictions — so no single global rescaling can separate it.
- A low-grade probability-mass rule (flag argmax 0 when P(grade≥1) is high) — strictly dominated. A correctly-classified healthy retina carries more mass on grades 1+ (2.73e-4) than the missed grade 1 (1.66e-4). No threshold catches the error without first flagging a true negative.
Neither was shipped. A grade 0 result means no referable disease was detected in this image — it is not an all-clear, and the UI and generated reports are worded accordingly.
P(grade≥2) as a referable score gives AUROC 1.0 on this sample, so the ordinal
structure is sound; the miscalibration is confined to top-1 confidence. A
threshold in (0.748, 0.991) would separate all 20 perfectly, but that gap is
defined by exactly one scan on each side, so fitting it would be overfitting to
n=20 and we left it alone.
Other limits: one dataset and one camera population; a single field photograph with no clinical context, no visual acuity, no HbA1c, no OCT; no regulatory clearance and no prospective validation.
A 5-class softmax has no "none of the above" outlet. Fed a group photo, this model does not get confused — it answers a different question, confidently:
| upload | grade returned | confidence |
|---|---|---|
| photograph of a person | 2 | 0.995 |
| landscape wallpaper | 2 | 0.996 |
| poster / graphic | 2 | 0.535 |
The uncertain flag never fires on those, because 0.995 is not uncertain. So
the check cannot live downstream of the model — app/fundus.py runs on the
image first, and a failure is refused rather than graded.
Two independent tests, either sufficient:
- Aperture. Every fundus camera images the retina through a circular aperture, leaving a lit disc on black. The lit region's boundary is fitted with a circle (Kåsa least squares) and scored by IoU. Real captures score 0.958–0.995; the closest non-fundus image in the test set — a photograph of a lit laptop screen — scores 0.891. The threshold is 0.925, inside that gap. This property survives across cameras, datasets, colour balance and crop, and no ordinary photograph has it.
- Colour plus model-space agreement. Some fundus images arrive cropped
inside the disc with no black border left to measure (1 of the 20 sample
scans). Those fall back to red dominance and cosine similarity between the
pooled EfficientNet embedding and a bank of reference-scan embeddings
(
app/fundus_reference.npy, 20 × 1280 float32, 100 KB).
The second half of that fallback is not decoration. Colour alone cannot carry it: a red wall scores 4.73 red dominance and a perfect 1.00 channel ordering — better than most real retinas. Embedding similarity is what separates them:
| no-aperture upload | red dominance | embedding | verdict |
|---|---|---|---|
| fundus cropped inside the disc | 2.32 | 0.396 | graded |
| red wall | 4.73 | 0.098 | refused |
| sunset | 3.18 | 0.158 | refused |
| person under warm light | 5.18 | 0.215 | refused |
Embedding similarity is used only on that branch, because that is the only place it was measured to separate cleanly. Applied to every image it overlaps badly (lowest fundus 0.178 against highest non-fundus 0.217) and would be worthless as a global rule.
Reproduce with, where DIR holds images that are not retinas:
.venv/bin/python backend/evaluate.py --fundus-gate DIR [--fundus-extra FUNDUS_DIR]Scored 23/23 fundus photographs accepted, 16/16 non-fundus refused — the latter covering photographs of people, landscapes, a document, a UI screenshot, posters, a photographed laptop screen, a synthetic red wall, a red car, a sunset and a warm-lit portrait. Sample scans are scored leave-one-out, with their own embedding removed from the bank first.
Residual limitation, deliberately not papered over. A photograph deliberately masked into a circle on black still gets through: it satisfies the aperture test by construction, and colour cannot separate a warm-toned face from a hazy real retina (1.39 vs 1.02 red dominance). Closing it would need an embedding threshold with 0.01 of margin fitted to a single image, which would break on the next unusual camera. Two synthetic constructions in the test set fail this way; no ordinary upload does.
- Model:
timmEfficientNet-B0,num_classes=5, loaded once at startup into a module-level singleton ineval()mode. A lock serialises access because Grad-CAM runs a backward pass through the shared module. - Grad-CAM: implemented in
app/gradcam.py, target layermodel.blocks[-1], computed against the predicted class from the same forward pass that produces the grade. It replacespytorch-grad-cam, whose import chain cost 120 MB of RSS and caused OOM kills on a 512 MB instance;evaluate.py --parityverifies the two are bit-for-bit identical. - Domain gate: blocking. Non-retinal uploads are refused before the model runs — see "The domain gate" above.
- Image-quality gate: advisory, not blocking, and only reached by images already confirmed retinal. Focus is measured on a size-normalised 512px crop because variance-of-Laplacian scales with resolution — on raw input a sharp 3216px scan scores lower than a soft 1050px one. Thresholds sit in the gap between real scans (focus 15–160) and genuine blur (<9). All 20 sample scans pass; blurred and badly exposed images are caught.
- Frontend: React 19 + TypeScript + Vite + Tailwind v4,
lucide-reactfor icons. The hero is ported fromreference/RetinaGuard Hero.html— geometry, gradients, vessel paths, lesion positions and the 6s SMIL timings unchanged; the Geist woff2 files and background video were extracted from that bundle. All motion is CSS/SVG. There is no animation library in this project. - Groq only, no OpenAI. Model is configurable via
GROQ_MODEL, with a fallback chain and a deterministic offline template if every model fails.