VU AI Service

AI-powered interview engine for the VU platform — real-time voice interviews, LLM-based scoring, CV analysis, cheat detection, and text-to-speech.

Tech Stack

Component	Technology	Purpose
Framework	FastAPI + Uvicorn	HTTP server, WebSocket, async
LLM	Gemini 2.0 Flash or Groq (llama-3.3-70b-versatile)	Question generation, answer evaluation, CV analysis, score adjustment
STT	AssemblyAI (Universal Streaming v3)	Real-time speech-to-text
TTS	edge-tts (Microsoft neural voices)	AI speech → base64 MP3
Video	OpenCV + YuNet (ONNX)	Face detection, gaze tracking, eye contact
CV Parsing	pdfplumber + python-docx	PDF and DOCX text extraction
HTTP Client	httpx	Async calls to NestJS backend
Validation	Pydantic	Request/response models, LLM output validation

Quick Start

Prerequisites

• Python 3.13+
• An LLM API key (Gemini or Groq)
• An AssemblyAI API key

Install

git clone https://github.com/vu-app-dev/vu-ai.git
cd vu-ai
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
cp .env.example .env   # edit with your API keys

Run

python main.py
# Runs on http://localhost:8000

Run with Docker

docker build -t vu-ai .
docker run -p 8000:8000 --env-file .env vu-ai

For the full stack (AI + backend + frontend + PostgreSQL), see the vu-app orchestration repo.

LLM Provider Configuration

The service supports two LLM providers via the LLM_PROVIDER env var:

Provider	LLM_PROVIDER	Model	API Key Env Var	Free Tier
Gemini	gemini	gemini-2.0-flash	GEMINI_API_KEY	1500 RPD, 1M TPM
Groq	groq	llama-3.3-70b-versatile	GROQ_API_KEY	30 RPM, 12K TPM

Switch providers by changing one env var — no code changes needed. The LLMService abstraction handles the rest.

LLM_PROVIDER=gemini    # or groq
GEMINI_API_KEY=your_key
GROQ_API_KEY=your_key  # only needed if LLM_PROVIDER=groq

API Endpoints

REST

Method	Path	Description
GET	/health	Health check
POST	/api/stt/transcribe/url?url=<url>	Transcribe audio from URL
POST	/api/stt/transcribe/file	Transcribe uploaded audio file
POST	/api/cv/analyze	Analyze CV — skills, summary, BARS score
POST	/api/interview/start	Start interview session
POST	/api/interview/end/{sessionId}	End session, return full performance

WebSocket

Path	Description
WS /api/stt/realtime	Real-time speech-to-text (AssemblyAI)
WS /api/interview/session/{sessionId}?token=<token>	Interview control (answers, video frames, cheat events)

Scoring

The AI evaluates candidates across 5 BARS-anchored dimensions:

Dimension	Source	Weight
Technical	LLM (transcript)	36%
Communication	LLM (transcript)	20%
Problem Solving	LLM (transcript)	20%
Structured Thinking	LLM (transcript)	14%
Confidence	Audio analysis (WPM + fillers)	10%

Overall = weighted average + LLM adjustment (±10)

Scores use BARS (Behaviorally Anchored Rating Scales) — LLM rates 1-5 with behavioral descriptors, converted to 0-100.

Cheat Detection

5 signals monitored throughout the interview:

Signal	Threshold	Result
Tab switches	3-5 / 6+	Flagged / Critical
No face detected	>20% frames	Flagged
Multiple faces	>10% frames	Flagged
Gaze away	>40% frames	Flagged
Second speaker (diarization)	>5% / >15%	Flagged / Critical

Two+ flagged signals = Critical. Cheat status is stored separately from the score — score = competence, cheat = integrity.

CV Analysis

CVs are scored across 4 BARS dimensions (1-5 scale, weighted to 0-100):

Dimension	Weight
skillsMatch	40%
experienceDepth	25%
projectRelevance	20%
educationFit	15%

Supports PDF and DOCX. The score is computed deterministically from the LLM's dimension ratings — not generated directly.

Environment Variables

Variable	Required	Default	Description
LLM_PROVIDER	Yes	gemini	gemini or groq
GEMINI_API_KEY	If gemini	—	Google Gemini API key
GROQ_API_KEY	If groq	—	Groq API key
LLM_MODEL	No	Provider default	Override LLM model
ASSEMBLYAI_API_KEY	Yes	—	AssemblyAI API key for STT
BACKEND_URL	Yes	http://localhost:3000	NestJS backend URL
BACKEND_API_KEY	Yes	—	Shared secret for backend auth
FRONTEND_URL	Yes	http://localhost:5173	Frontend URL (CORS)
HOST	No	0.0.0.0	Server host
PORT	No	8000	Server port
SESSION_TIMEOUT_SECONDS	No	120	Session inactivity timeout
TTS_VOICE	No	en-US-AriaNeural	edge-tts voice
TTS_RATE	No	—	Speech rate (e.g. +10%)
TTS_VOLUME	No	—	Volume (e.g. +0%)
TTS_PITCH	No	—	Pitch (e.g. -2Hz)
TTS_PROXY	No	—	HTTP proxy for edge-tts

Tests

.venv/bin/python -m pytest tests/ -v
# 347 tests, ~65s

ARM64 / Production Notes

• MediaPipe is not used — replaced with YuNet (ONNX, 232KB) for face detection
• opencv-python-headless — no GUI/libGL dependencies, works in slim containers
• Graceful degradation: if OpenCV/YuNet unavailable, video analysis is skipped (interview continues)
• The Dockerfile is based on python:3.13-slim with minimal system deps

Project Structure

vu-ai/
├── main.py                     # FastAPI app, CORS, routers
├── config/settings.py          # Env vars + validation
├── routers/                    # STT, interview, CV endpoints
├── services/
│   ├── llm/                    # LLMService (Gemini/Groq abstraction)
│   ├── stt/                    # AssemblyAI streaming
│   ├── tts/                    # edge-tts
│   ├── interview/              # Session manager, question generator, cheat detector
│   ├── cv/                     # CV analyzer (PDF/DOCX)
│   ├── scoring/                # Transcript, audio, video scorers + aggregator
│   └── video/                  # YuNet face analyzer
├── models/                     # Pydantic models
├── prompts/                    # Prompt templates + BARS anchors
├── clients/backend_client.py   # httpx client to NestJS
└── tests/                      # 347 unit tests

Related Repos

• vu-app — Full-stack deployment (docker-compose)
• vu-backend — NestJS backend
• vu-frontend — React frontend

License

MIT