⚡ CodePulse

AI-powered GitHub code review with structured findings, developer analytics, and empirical evaluation.

CodePulse is an end-to-end AI code-review system for GitHub pull requests. It verifies webhooks, enqueues durable review jobs, analyzes diffs with structured LLM tool-calling, posts exact-line review comments, and persists findings for analytics. A pre-Groq injection-defense gate scans untrusted PR content; an adversarial eval harness and /security dashboard measure catch rates. Separate offline eval under server/eval/ tracks review-model quality. It is designed as a production-style system—not a thin chatbot wrapper around a model API.

→ Live app · GitHub

Why CodePulse

Capability	What it provides
Automated PR reviews	Reviews on `opened`, `synchronize`, and `reopened`
Async queue + worker	BullMQ/Redis jobs; API acknowledges fast; worker runs the pipeline
Structured LLM analysis	Two-pass triage + chunked review via `GroqAnalysisService.analyzeDiff()`
Exact-line GitHub comments	Inline findings with category, severity, explanation, and suggestion
Injection defense	Pre-Groq gate on PR title/body/filenames/diff — allow / flag / block
Adversarial security eval	Taxonomy runner + catch/miss metrics; `/security` dashboard
Jobs + Trace Viewer	`/jobs` queue health and per-job step timeline (`TraceEvent`)
Multi-tenant isolation	Data scoped to each GitHub App installation
Installation authorization	Users can link only installations they can access on GitHub
HMAC webhook verification	Every delivery verified with HMAC-SHA256
PR-head idempotency	Same repository + PR + head SHA is not reviewed twice
Lifecycle tracking	Persists `open`, `closed`, and `merged` from GitHub payloads
Developer / repo analytics	Findings stored for dashboard charts and history
Weekly digest	Opt-in Resend email summarizing recent issue categories and file hotspots
Offline review evaluation	Labeled TypeScript benchmark for prompt/model regression (`server/eval/`)
Production deployment	Vercel frontend, Azure API + worker, Neon, Upstash Redis, digest cron

Analytics reflect historical findings per developer and repository. CodePulse does not fine-tune or retrain a model from team data.

How it works

A GitHub App webhook delivers a pull-request event.
The API verifies the HMAC signature, returns 202 Accepted quickly (ACK before heavy Neon/Redis work where configured), inserts a ReviewJob (unique on repo + PR + head SHA), and enqueues it on BullMQ/Redis.
A separate worker consumes the queue:
- fetch/parse diff
- injection_scan (optional, flag-gated) — OpenAI embeddings + logistic classifier; block skips Groq and posts a short security skip comment
- Groq triage + chunked analysis → GitHub comments → Postgres
Failed jobs retry with exponential backoff (3 attempts) then land in dead, visible on Jobs.
Each pipeline step emits a TraceEvent (Trace Viewer on /jobs/:id).
Dashboard analytics + opt-in weekly digests (Resend). Security overview on /security.

The production analysis prompt is the first evidence-based refinement (see Evaluation). A second experimental prompt was tried and reverted.

Architecture

GitHub PR event
        │
        ▼
GitHub webhook + HMAC-SHA256 verification
        │
        ▼
API (enqueue only)
  • action routing (review vs lifecycle-only)
  • ReviewJob insert + @@unique(repoId, prNumber, headSha)
  • BullMQ enqueue → 202 Accepted (fast ACK)
        │
        ▼
Worker process (BullMQ consumer)
  • fetch/parse diff
  • injection_scan (flag + allowlist) → allow | flag | block
  • Groq triage → chunked review → Zod-validated findings
      (skipped on block)
  • GitHub review comments + PostgreSQL persistence
  • TraceEvent per step · retries → dead letter
        │
        ▼
Dashboard / Jobs / Security / Digest APIs
        │
        ▼
Resend weekly email (opt-in)

Webhooks target the Azure API host directly. The Vercel frontend proxies /api/v1/* to the backend so session cookies remain same-origin.

Processes: API App Service (thecodepulse) + Worker App Service (thecodepulse-worker, startup node dist/worker.js or npm run worker) + Redis (e.g. Upstash rediss://) + Neon Postgres.

Injection defense & security eval

Piece	Location / notes
Gate module	`server/src/defense/` — embeddings (`text-embedding-3-small`) + logistic scorer (`artifacts/logistic.json`); centroids fallback
Pipeline step	`injection_scan` after `fetch_diff`, before `analyze_diff`
Outcomes	`allow` / `flag` (continue) / `block` (skip Groq; security skip comment; `InjectionDecision`)
Flags	`INJECTION_DEFENSE_ENABLED` (default off); optional installation allowlist for test rollout
Rebuild classifier	`npm run defense:build-classifier` (needs `OPENAI_API_KEY`)
Adversarial harness	`server/src/eval-harness/` — `npm run eval-harness:run` → catch/miss by category
UI	`/security` — live decisions + last harness results
ADR	`docs/architecture/adr/006-injection-defense-gate.md`

Separate from review-model eval in server/eval/ (Groq precision/recall on labeled diffs).

Optional refactor PRs (Phase 4, org flag OFF by default)

For maintainability findings (code-quality, best-practices) only, an org may opt in to a second verified PR:

Caps checked (one attempt per finding per headSha, per-PR + daily org caps) before any Groq patch call or sandbox start.
Patch verified in an ephemeral Docker container (no worker secrets; registry-oriented egress; cloud metadata 169.254.169.254 / link-local IMDS explicitly blocked).
On gate failure → TraceEvent + RefactorAttempt row; no GitHub PR.
On success → separate codepulse/refactor-* branch/PR with rationale linking the source finding.

See docs/architecture/adr/004-refactor-pr-verification-gate.md.

Engineering & reliability

Installation authorization

Client-supplied installation_id values are not trusted alone.
The installation must belong to this GitHub App.
Personal installations must match the authenticated GitHub user.
Organization installations require active org membership, verified with a short OAuth flow (read:org) and signed OAuth state.

Webhook security & processing

HMAC-SHA256 signature verification on every delivery.
Review pipeline actions: opened, synchronize, reopened.
closed (including merges) updates lifecycle state without running another AI review.
Response code: 202 Accepted after enqueue (or idempotent duplicate); ACK is prioritized so GitHub does not hit delivery deadlines.
Idempotency: Postgres UNIQUE (repoId, prNumber, headSha) on ReviewJob — not check-then-insert.
Worker retries 3× with exponential backoff; permanent failures → dead (visible on /jobs).

Local Redis (BullMQ)

docker compose up -d   # Redis on localhost:6380 (LabCrew-consistent)

Set REDIS_URL=redis://127.0.0.1:6380 in server/.env.

Run API + worker:

cd server
npm run dev          # API :3001
npm run worker:dev   # BullMQ consumer

Review idempotency

Uses existing PullRequest.headSha as the success marker (written only after a successful review pipeline).
Logical key: repository + PR number + head SHA.
Duplicate delivery of an already processed SHA skips Groq, GitHub comments, and Issue inserts.
A new head SHA triggers a new review.
Failed runs do not mark the SHA as processed.

PR lifecycle

State is derived from the GitHub payload (open / closed / merged via state, merged, and merged_at).
Lifecycle updates are persisted for dashboard accuracy.

Digest email HTML safety

Dynamic string values in digest HTML (weekRange, severity, category, file path) are escaped at the render boundary.
Attacker- or LLM-influenced file names cannot break out of HTML elements in the generated email.
This describes the implemented escaping for those interpolated strings—not a claim that every email path is broadly sanitized.

Multi-tenant isolation

Tenant = GitHub App installation → Organization (organizationId).
Enforcement = tenantRepository(organizationId) — every dashboard/job query for repositories, PRs, issues, developers, review jobs, and traces must go through this wrapper. Empty organizationId throws.
Coverage: npm run test:isolation seeds two installations and asserts A cannot read B’s rows for each tenant-scoped model.
See docs/architecture/adr/003-tenant-isolation-model.md.

Multi-Tenancy & Data Isolation

Concern	Detail
Tenant definition	GitHub App installation → Organization
Enforcement point	`server/src/services/tenantRepository.ts`
Scoped models	Repository, PullRequest, Issue, Developer, ReviewJob, TraceEvent, InjectionDecision
Test coverage	`npm run test:isolation` (2 seeded installations, cross-tenant null/empty assertions)
Acceptance fixtures	`npm run cleanup:acceptance` removes Phase 1 fake-install test rows

Evaluation

Two separate evaluation tracks:

Track	Path	Purpose
Review quality	`server/eval/`	Groq precision/recall on labeled TypeScript diffs
Injection defense	`server/src/eval-harness/`	Catch/miss on adversarial taxonomy (`npm run eval-harness:run`)

Review quality (`server/eval/`)

Offline benchmark under server/eval/ (dataset + runner). Production analysis model: openai/gpt-oss-20b. Comparison runs can set EVAL_COMPARE_MODEL (e.g. openai/gpt-oss-120b) for side-by-side checks.

Dataset

Property	v1 (historical)	v2 (current)
Labeled TypeScript diffs	20	50
Known defects (positive)	12	30
Clean cases	8	20

Historical results (v1 dataset, 20 cases) — kept for honest iteration

Reported production configuration on v1 = baseline + first evidence-based prompt refinement (model reported at the time: openai/gpt-oss-120b):

Stage	Recall	Precision	F1	FP findings
Baseline	100%	30.0%	46.2%	28
Evidence-based prompt refinement	100%	46.2%	63.2%	14

On that refined configuration: TP = 12, FN = 0. False-positive findings fell from 28 → 14 while recall stayed at 100%.

A second experimental prompt was tried and reverted; it is not the production system result.

v2 precision gap — diagnosis (historical)

Early v2 runs on 2026-08-19 showed openai/gpt-oss-120b at 31.5% precision / F1 47.1, below the README’s v1 refined claim (46.2% / 63.2%), while openai/gpt-oss-20b beat it on precision and F1.

1. Prompt drift (real regression, fixed):
main was still on the May baseline SYSTEM_PROMPT (git blame → 47df2e7). The evidence-based refined prompt lived only on divergent commit fab2cbf and was never an ancestor of HEAD. Refined prompt text is restored in groqAnalysisService.ts.

2. Clean-case pressure: v2 has 20 negatives (v1 had 8). Eager FP-prone behavior shows up harder on the larger clean set.

3. Eval-methodology: v2 P/F1 are not directly comparable to v1’s 46.2%. Report v1 and v2 separately; only compare models within the same suite + same prompt + same harness.

v2 fair same-day comparison (50 cases, 2026-08-21) — production lock

Fair same-day --fresh runs under the fixed harness (empty-intent tool recovery; rate limits never counted as analysis_failed; optional EVAL_GROQ_API_KEYS rotation). Both models 50/50 scored, analysis_failed=0 — not rate-limit contaminated.

Model	Precision	Recall	F1	TP	FP	FN	Clean any-FP rate
openai/gpt-oss-20b (production)	56.5%	86.7%	68.4%	26	20	4	40% (8/20)
openai/gpt-oss-120b (comparison)	48.2%	90.0%	62.8%	27	29	3	60% (12/20)

Lock decision: GROQ_MODEL = openai/gpt-oss-20b. 20b was chosen for lower false-positive rate on clean code (40% vs 60%), accepting a ~3-point recall tradeoff (86.7% vs 90.0%), because precision/clean-FP matter more than marginal recall for a production review tool — every false positive costs reviewer trust, and CodePulse’s own evaluation work already indicates AI-authored changes tend to draw more human scrutiny than average (so a few missed findings are less costly than systematic over-flagging).

v2 results under drifted baseline prompt (historical only)

Captured before prompt restore:

Model	Precision	Recall	F1	TP	FP	FN
openai/gpt-oss-120b (baseline prompt)	31.5%	93.3%	47.1%	28	61	2
openai/gpt-oss-20b (baseline prompt)	50.0%	76.7%	60.5%	23	23	7

Full tables: server/eval/results/. Re-run with npm run eval:offline / npm run eval:compare.

Reproduce dataset plumbing (no Groq):

cd server && npm run eval:smoke

Full offline LLM eval (requires configured Groq API credentials):

cd server && npm run eval:offline
cd server && npm run eval:compare

Generated reports under server/eval/results/ (latest.json, latest-<model>.md, comparison.md) are gitignored except examples. See server/eval/README.md for matching rules.

Limitations / future work: Even at 50 cases this remains a focused TypeScript detection suite—useful for regression and prompt/model iteration, not statistically conclusive production proof. A 40% clean-case false-positive rate is still high on its own terms and is the biggest opportunity for further prompt refinement (same framing as the original baseline→refined prompt work).

Injection defense harness (`server/src/eval-harness/`)

cd server
npm run defense:build-classifier   # rebuild logistic + centroids (OPENAI_API_KEY)
npm run eval-harness:run           # taxonomy catch/miss → results/latest.json

See server/src/eval-harness/README.md. Dashboard: /security.

Database migrations (Neon)

Never prisma db push against shared/prod Neon. Generate reviewable SQL (migrate dev --create-only or hand-authored files), read for unexpected DROPs, then migrate deploy. See docs/architecture/neon-migration-process.md — same rule for LabCrew.

Tech stack

Layer	Technologies
Backend	TypeScript, Node.js, Express, Prisma, PostgreSQL (Neon), Groq, OpenAI embeddings, BullMQ, Octokit, Resend
Frontend	TypeScript, React, TanStack Router, Tailwind CSS, Recharts
Platform	GitHub App, GitHub OAuth, GitHub Actions, Azure App Service (API + worker), Vercel, Redis

Product views

View	Scope
Dashboard (`/dashboard`)	Installation-wide stats, recent reviews, connected repositories
Repositories (`/repos/{owner}/{repo}`)	Single-repo health, severity trends, PR list
Developers	Per-developer issue trends (recent window)
Jobs (`/jobs`)	Queue counts, recent ReviewJobs, Trace Viewer per job
Security (`/security`)	Injection decisions + eval-harness catch rates
Weekly digest (`/digest`)	Digest preview and email opt-in

Getting started

Prerequisites

Node.js 20+ (22 recommended; matches Azure/CI)
PostgreSQL (Neon works well)
Redis (Docker Compose on :6380, or Upstash for cloud)
GitHub App + OAuth App (GitHub Apps docs)
Groq API key (console.groq.com)
OpenAI API key if enabling injection defense (platform.openai.com)
Resend API key (resend.com) for weekly digests

1. Clone & install

git clone https://github.com/ahmadmustafa02/CodePulse
cd CodePulse

cd server && npm install
cd ../web && npm install

2. Configure & run the server

cd server
cp .env.example .env
# Fill in values from the environment tables below
npx prisma migrate deploy
npm run dev

API: http://localhost:3001

3. Configure & run the web app

cd web
cp .env.example .env.local
npm run dev

Dashboard: http://localhost:8080

4. GitHub App / OAuth setup

Setting	Local	Production
OAuth callback	`http://localhost:3001/api/v1/auth/github/callback`	`https://getcodepulse.vercel.app/api/v1/auth/github/callback` (Vercel proxy) or your API host
Webhook URL	ngrok → `/api/v1/webhooks/github`	`https://your-api-host/api/v1/webhooks/github`
Webhook events	Pull request	Pull request

Minimum GitHub App permissions:

Permission	Access
Repository metadata	Read
Contents	Read
Pull requests	Read & write

Post-install callback (production example):
https://getcodepulse.vercel.app/api/v1/auth/installation/callback

5. Verify PR review locally

docker compose up -d and set REDIS_URL.
Run API (npm run dev) and worker (npm run worker:dev).
Sign in at http://localhost:8080 and install the GitHub App on a test repo.
Open a PR with a real code change (not only lockfiles).
Expect inline review comments within a few minutes; check /jobs for status/traces.
Refresh the dashboard — the PR should appear under recent reviews.

Debug deliveries: GitHub → App → Advanced → Recent Deliveries (look for accepted responses).

6. Weekly digest

Set server env: RESEND_API_KEY, DIGEST_FROM_EMAIL, DIGEST_CRON_SECRET.
Set repository secrets:
- CODEPULSE_API_URL — Azure API base URL, no trailing slash
- DIGEST_CRON_SECRET — same value as the server
Workflow .github/workflows/weekly-digest.yml runs Sundays 09:00 UTC (manual trigger available).
Users opt in on /digest after signing in (GitHub email required).

Manual trigger:

curl -X POST https://your-api-host/api/v1/digest/trigger \
  -H "Content-Type: application/json" \
  -H "x-digest-secret: YOUR_DIGEST_CRON_SECRET" \
  -d "{}"

7. Evaluation

# Review-model quality (Groq)
cd server && npm run eval:offline

# Injection-defense catch rates (OpenAI embeddings)
cd server && npm run eval-harness:run

Environment variables

Server · server/.env

Variable	Description
`DATABASE_URL`	Neon / PostgreSQL connection string
`GITHUB_APP_ID`	GitHub App ID
`GITHUB_PRIVATE_KEY`	App private key (PEM; `\n` escaped in `.env`)
`GITHUB_WEBHOOK_SECRET`	Webhook secret (min 20 chars)
`GITHUB_OAUTH_CLIENT_ID`	OAuth App client ID
`GITHUB_OAUTH_CLIENT_SECRET`	OAuth App client secret
`GITHUB_OAUTH_CALLBACK_URL`	Must match OAuth app callback exactly
`GROQ_API_KEY`	Groq API key
`REDIS_URL`	BullMQ Redis URL (local default `redis://127.0.0.1:6380`; production often `rediss://…`)
`AUTH_SECRET`	Session JWT signing secret (min 32 chars)
`WEB_APP_URL`	Frontend origin for CORS and redirects
`RESEND_API_KEY`	Resend API key
`DIGEST_FROM_EMAIL`	Sender address for digest emails
`DIGEST_CRON_SECRET`	Protects `POST /api/v1/digest/trigger` (min 20 chars)
`INJECTION_DEFENSE_ENABLED`	`true` to run pre-Groq injection gate (default `false`)
`OPENAI_API_KEY`	Required when injection defense is enabled (embeddings)
`INJECTION_DEFENSE_INSTALLATION_ALLOWLIST`	Optional comma-separated GitHub App installation IDs for test-only rollout
`INJECTION_BLOCK_THRESHOLD`	Malicious score ≥ this → block (default `0.7`; set on worker)
`INJECTION_FLAG_THRESHOLD`	Malicious score ≥ this → flag (default `0.35`; set on worker)

Web · web/.env.local

Variable	Description
`VITE_API_URL`	Local API base (`http://localhost:3001/api/v1`). Leave unset in production; the app uses same-origin `/api/v1` via Vercel rewrites.

GitHub Actions · repository secrets

Secret	Description
`CODEPULSE_API_URL`	Production API host (no trailing slash)
`DIGEST_CRON_SECRET`	Same as server `DIGEST_CRON_SECRET`

Scripts

# Server (from server/)
npm run dev         # nodemon + ts-node (API)
npm run worker:dev  # BullMQ worker
npm run worker      # production worker (dist/)
npm run build       # compile TypeScript (+ copy defense/eval-harness artifacts)
npm run start       # node dist/index.js
npm run typecheck
npm run lint
npm run eval:smoke     # dataset + diff parse (no Groq)
npm run eval:offline   # offline review benchmark
npm run eval:compare   # production model vs EVAL_COMPARE_MODEL
npm run defense:build-classifier  # rebuild logistic + centroids
npm run defense:smoke             # quick scorer smoke
npm run eval-harness:run          # adversarial injection catch rates

# Root
docker compose up -d   # Redis :6380

# Web (from web/)
npm run dev         # Vite (port 8080)
npm run build
npm run lint

Deployment

Layer	Host
Frontend	Vercel
API	Azure App Service (`thecodepulse`)
Worker	Azure App Service (`thecodepulse-worker`, startup `node dist/worker.js`)
Queue	Redis / Upstash (`REDIS_URL`, TLS `rediss://` in cloud)
Database	Neon PostgreSQL
Weekly digest cron	GitHub Actions (`weekly-digest.yml`)

See ADRs: 001, 002, 006 (injection defense).

GitHub webhooks must point to the API host, not the Vercel frontend URL. Injection thresholds and OPENAI_API_KEY for the gate belong on the worker app.

Limitations & future work

The offline review eval set is 50 labeled TypeScript diffs (v2; was 20 in v1).
Review benchmark focuses on TypeScript detection, not large-scale acceptance of suggested fixes.
Injection defense defaults off and may use an installation allowlist during rollout; tune thresholds from live InjectionDecision data.
Optional LLM-as-judge for the adversarial harness is reserved/not implemented.
Sandbox/IMDS hardening for refactor PRs is strongest on hosts with Docker/iptables (limited on plain App Service).

These are scope boundaries for the current system, not blockers for the production review pipeline described above.

CodePulse — structured AI review for GitHub, with async workers, injection defense, persistence, and measurable evaluation.

Live App · Issues · Repository

Name		Name	Last commit message	Last commit date
.github/workflows		.github/workflows
docs		docs
server		server
web		web
.eslintrc.js		.eslintrc.js
.gitignore		.gitignore
.prettierrc		.prettierrc
README.md		README.md
docker-compose.yml		docker-compose.yml

Latest commit

History

Folders and files

Repository files navigation