PracticeOps Agents

A two-agent LLM system that analyzes medical practice operational metrics, identifies workflow problems, and recommends predefined operational actions — running entirely on a local Ollama model (gemma2:9b), no API key or GPU required.

Main impact: replaces manual dashboard-reading with an automated read → diagnose → recommend pipeline — turning six raw operational metrics into a specific, catalog-backed action plan in one pass, at 0.88 F1 on issue detection and 1.00 F1 on action selection (see Evaluation), entirely on a free local model with no PHI ever leaving the machine.

Problem statement

Medical practices collect operational metrics (no-show rate, portal adoption, check-in time, phone-call volume, claim denial rate, provider utilization) but a traditional dashboard just displays the numbers and leaves interpretation to the practice manager. PracticeOps Agents uses two constrained LLM agents to turn those raw metrics into structured, actionable recommendations. It does not diagnose patients, process protected health information, or make clinical decisions.

Architecture

Practice Metrics
      |
      v
LLM Agent 1
Diagnostic Agent
      |
      v
Structured Issue Labels
      |
      v
LLM Agent 2
Action Agent
      |
      v
Structured Action Labels
      |
      v
Streamlit Dashboard

The agents communicate through structured JSON. There is no memory system, retrieval pipeline, vector database, or autonomous planning loop.

Agents

Agent 1 — Diagnostic Agent

Reads specialty, provider count, appointment volume, and the six operational metrics. Selects zero or more issues from a fixed label set by applying the same documented threshold rules used to build the evaluation ground truth:

Rule	Issue
no_show_rate >= 0.15	high_no_show
portal_adoption_rate < 0.40	low_portal_use
average_check_in_minutes >= 15	long_check_in
weekly_phone_calls >= 600	front_desk_overload
claim_denial_rate >= 0.08	billing_risk
provider_utilization < 0.70	low_provider_utilization

Output: {"issues": ["high_no_show", ...]}

Agent 2 — Action Agent

Receives Agent 1's issue labels and selects the matching actions from a fixed catalog using a 1:1 controlled mapping:

Issue	Action
high_no_show	send_appointment_reminders
low_portal_use	promote_portal_enrollment
long_check_in	enable_digital_intake
front_desk_overload	review_front_desk_workflow
billing_risk	review_claim_denials
low_provider_utilization	rebalance_provider_schedule

Output: {"actions": ["send_appointment_reminders", ...]}

Both agents are given their allowed label list and rules directly in the prompt, are called with temperature: 0 and Ollama's format: "json" mode, and any label outside the allowed catalog is stripped and counted toward the unsupported-label rate rather than passed through.

Setup

# 1. Clone the repo and move into it
git clone https://github.com/RexManglicmot/practiceops-agent.git
cd practiceops-agent

# 2. Create/activate the virtual environment
python3 -m venv venv_praticeops_agent
source venv_praticeops_agent/bin/activate

# 3. Install dependencies
pip install -r requirements.txt

# 4. Pull and serve the local model (only needs to be done once; ollama serve
#    may already be running as a background service)
ollama pull gemma2:9b
ollama serve

# 5. Configure environment (defaults already match a standard local Ollama install)
cp .env.example .env

# 6. Run the evaluation suite
python evaluate.py

# 7. Run the interactive dashboard
streamlit run app.py

Synthetic dataset

test_cases.json is a hand-built, synthetic set of 30 fictional practices (P001–P030) across a range of medical specialties (Dermatology, Family Medicine, Cardiology, Orthopedics, Psychiatry, Pediatrics, and others) — no real practice or patient data is used anywhere in this project. Each case pairs a plausible metrics profile with hand-labeled ground truth:

• metrics — the nine input variables described below, fed to the Diagnostic Agent.
• expected_issues — ground truth for the Diagnostic Agent: the issues that actually apply to that practice.
• expected_actions — ground truth for the Action Agent: the actions that actually apply to that practice.

The nine input variables

Every metrics object has exactly these fields. Three are contextual only (no rule reads them); six each feed exactly one threshold rule:

• specialty — the practice's medical specialty (e.g. "Dermatology").
• providers — how many doctors/clinicians work at the practice.
• monthly_appointments — total patient appointments booked per month.
• no_show_rate — the fraction of scheduled appointments patients simply don't show up for.
• portal_adoption_rate — the fraction of patients actively using the online patient portal.
• average_check_in_minutes — how long, on average, a patient waits at check-in before being seen.
• weekly_phone_calls — how many phone calls the front desk fields in a typical week.
• claim_denial_rate — the fraction of submitted insurance claims that get denied.
• provider_utilization — how much of a provider's available appointment capacity is actually booked.

Variable	Type	Dataset range	Rule it feeds	Used by a rule?
specialty	string	28 specialties (Dermatology, Cardiology, …)	—	No — context only
providers	integer, count	4–20	—	No — context only
monthly_appointments	integer, count	800–6,500	—	No — context only
no_show_rate	float, 0–1	0.05–0.25	>= 0.15 → high_no_show	Yes
portal_adoption_rate	float, 0–1	0.20–0.95	< 0.40 → low_portal_use	Yes
average_check_in_minutes	integer, minutes	7–22	>= 15 → long_check_in	Yes
weekly_phone_calls	integer, count	200–1,100	>= 600 → front_desk_overload	Yes
claim_denial_rate	float, 0–1	0.01–0.15	>= 0.08 → billing_risk	Yes
provider_utilization	float, 0–1	0.52–0.95	< 0.70 → low_provider_utilization	Yes

specialty, providers, and monthly_appointments exist purely to make each case read like a real practice — the Diagnostic Agent's prompt never applies a rule to them, and changing them alone can't change a case's expected issues. Every other variable was specifically chosen, per case, to land on both sides of its threshold across the dataset (e.g. some cases have claim_denial_rate just under 0.08, others just over) so the evaluation exercises boundary behavior, not just obviously-healthy or obviously-broken practices.

Because both ground-truth fields are generated deterministically from the same threshold and mapping rules given to the agents (rather than sourced from real EHR/PM system exports), the dataset is best read as a controlled unit-test suite for rule-following and JSON-formatting behavior, not a benchmark of real-world diagnostic difficulty.

Evaluation

evaluate.py runs both agents against all 30 cases in test_cases.json and scores each one separately, using multi-label precision, recall, and F1 (explained below):

• Diagnostic Agent — given each case's metrics, compare its predicted issues against expected_issues.
• Action Agent — given each case's expected_issues (the ground truth, not whatever the Diagnostic Agent actually predicted), compare its predicted actions against expected_actions.

Feeding the Action Agent ground-truth issues instead of the Diagnostic Agent's real output is deliberate: it isolates the Action Agent's own accuracy. If the Diagnostic Agent missed an issue, that mistake would otherwise also count against the Action Agent, even though the Action Agent behaved correctly given the input it received.

This is different from the live Streamlit dashboard, which chains the real pipeline — the Diagnostic Agent's actual predicted issues become the Action Agent's actual input — since that's what a real user experiences.

How precision, recall, and F1 are calculated

Scoring is multi-label and micro-averaged: for each test case, evaluate.py compares the agent's predicted label set against the expected set and adds to three running totals — tp (correctly predicted labels), fp (predicted but not expected), fn (expected but not predicted) — across all 30 cases, then divides only once at the end:

precision = tp / (tp + fp)
recall    = tp / (tp + fn)
f1        = 2 * precision * recall / (precision + recall)

Concrete example, using three real cases from a live gemma2:9b Diagnostic Agent run (test_cases.json / report.html):

Case	expected_issues	predicted_issues	TP	FP	FN
P001	high_no_show, low_portal_use, long_check_in, front_desk_overload	same, all 4	4	0	0
P002	high_no_show, front_desk_overload, billing_risk	high_no_show, front_desk_overload (missed billing_risk, whose claim_denial_rate of 0.09 only just clears the 0.08 threshold)	2	0	1
P008	(none — healthy practice)	low_portal_use (hallucinated — this practice's portal_adoption_rate is 0.90, well above the 0.40 threshold)	0	1	0

Pooling just these three cases: tp = 4+2+0 = 6, fp = 0+0+1 = 1, fn = 0+1+0 = 1, so precision = 6/7 ≈ 0.86, recall = 6/7 ≈ 0.86, f1 ≈ 0.86 — over all 30 cases the same arithmetic produces the 0.90/0.86/0.88 in the table below. Two edge cases in the code: tp+fp == 0 (agent predicted nothing) defaults precision to 1.0, and tp+fn == 0 (nothing was expected) defaults recall to 1.0 — so a correctly-predicted empty set, like a healthy practice with no hallucinated labels, doesn't get penalized.

gemma2:9b is the model used by default (see OLLAMA_MODEL in .env), chosen after comparing it against other locally-available models on the same test suite:

Model	Agent	Precision	Recall	F1
gemma2:9b (default)	Diagnostic Agent	0.90	0.86	0.88
gemma2:9b (default)	Action Agent	1.00	1.00	1.00
llama3.1:latest	Diagnostic Agent	0.88	0.67	0.76
llama3.1:latest	Action Agent	0.99	1.00	0.99

The gap between models is almost entirely a recall gap on the Diagnostic Agent (0.86 vs. 0.67) — gemma2:9b catches more of the true issue labels per practice, while both models hold similarly high precision. The Action Agent is near-parity either way, since mapping a given set of issues to actions is a simpler, more mechanical task than reading thresholds off raw metrics. Regenerate this chart with python generate_charts.py (needs pip install matplotlib) after any change to the numbers above.

Confusion matrices

This is multi-label classification — each practice can have zero to six issues at once — so there's no single N×N confusion matrix the way there would be for a single-label classifier. Instead each label gets its own independent 2×2 (present/absent) matrix, from the same live gemma2:9b run as the table above:

Summing all six Diagnostic Agent matrices reproduces the aggregate precision/recall in the table above (57 TP, 6 FP, 9 FN → precision 57/63 ≈ 0.90, recall 57/66 ≈ 0.86) — but the per-label view surfaces something the aggregate hides: billing_risk is the model's weak spot, missed 7 of 10 times (TP 3, FN 7), while every other issue label sits at perfect or near-perfect recall. low_portal_use is the main source of false positives (FP 5) — the model over-triggers it more than any other label. The Action Agent's matrices are all clean diagonals (zero FP/FN on every label), matching its 1.00/1.00/1.00 score, since it's only ever scored on correct input. Regenerate both with python generate_confusion_matrix.py (needs pip install matplotlib; makes one live Ollama call per test case per agent).

Supporting reliability metrics:

Model	JSON validity rate	Unsupported-label rate
gemma2:9b (default)	100%	1% (target 0%)
llama3.1:latest	100%	0%

gemma2:9b clearly leads llama3.1:latest on the Diagnostic Agent's threshold-reasoning task (F1 0.88 vs. 0.76, mainly a recall gap — 0.86 vs. 0.67), at the cost of somewhat higher latency (5.4GB vs. 4.7GB model) and an occasional out-of-catalog label that gets filtered before display. mistral:latest was also tried and trailed both at F1 0.49 on the Diagnostic Agent. Swap the OLLAMA_MODEL value in .env to compare further — both tables above were produced by re-running evaluate.py with OLLAMA_MODEL set to each model.

Limitations

• Does not diagnose patients, process PHI, or make clinical decisions — operational workflow signals only.
• Single practice, single page, no authentication, no database, no persistence.
• Label catalogs are fixed by design; the agents cannot invent new issues or actions.
• Diagnostic accuracy depends on the local model — results will vary between Ollama models.

Next steps for deployment

This project currently runs as a local, single-user Streamlit prototype. Moving it toward a real deployment would mean addressing, roughly in order:

1. Real data validation — replace the synthetic test_cases.json with metrics sampled or derived from actual practice-management/EHR exports, and re-run the evaluation to see whether accuracy holds outside the hand-built threshold cases.
2. Auth & multi-practice support — add login and per-practice data isolation; today the app has no authentication and holds one practice's metrics in memory per session.
3. Human-in-the-loop review — given this still touches practice operations, add a confirm/override step before any recommended action is treated as taken, rather than presenting agent output as a final decision.