Mantis Agent Assist v2.0

Real-time AI-powered agent assist dashboard. Mantis runs two parallel inference pipelines — a local HuBERT acoustic emotion model and a cloud LLaMA 3 tactical directive engine — to give customer service agents live guidance during calls.

---

🖥️ Cockpit & Assist Interface

Mantis Copilot Cockpit (Standby): Audio pipeline awaiting stream, HuBERT vocal classifier meters, speaker ratio KPIs, and tactical AI directive console.

Live Customer Triage Session: Real-time speech transcription with [refund RISK] tagging, HuBERT acoustic emotion detection (SAD), silence warning (68%), and context-aware LLaMA-3 de-escalation directive.

---

Architecture

Browser (Chrome / Edge)
│
├── Web Audio API ──► PCM bytes ──► WebSocket ──► FastAPI Backend
│                                                       │
│                                               HuBERT CNN (local)
│                                               Emotion inference
│                                                       │
├── Web Speech API ──► Transcript text ──► POST /api/advice
│                                                       │
│                                               Groq LLaMA-3-8b
│                                               Tactical directive
│                                                       │
└──────────────────── JSON response ◄───────────────────┘
                    (emotion + action displayed in UI)

Dual-Pipeline Design

Pipeline	Transport	Model	Output
Acoustic	WebSocket (raw PCM)	HuBERT + RMS/ZCR fusion	Emotion label + energy level
Tactical	HTTP POST	Groq LLaMA-3-8b-instant	One-line agent directive

---

Features

• 🎙️ Real-Time Emotion Inference — Live HuBERT acoustic model fused with RMS energy analysis. Classifies: Angry, Frustrated, Sad, Happy, Content, Neutral
• ⚡ Tactical AI Directives — Context-aware, de-escalation suggestions generated by LLaMA 3 on every sentence boundary
• 🗣️ Speaker Diarisation — Hold Shift to toggle the active speaker between Customer and Agent in the live transcript
• 🚨 Keyword Escalation Alerts — Automated slide-in notification on critical keywords (manager, sue, cancel, etc.) or sustained negative emotion
• 📊 Sentiment Telemetry — Click the ● widget to switch from waveform to a live sentiment graph
• 📋 Post-Call Analytics — Auto-generated call summary, disposition label (RESOLVED / ESCALATED / CHURN_RISK), and dead-air ratio on call end
• 🔇 Dead-Air Tracking — Real-time silence ratio metric to flag engagement drop-off

---

Tech Stack

Layer	Technology
Frontend	React 19 (Vite), Tailwind CSS v4, Web Audio API, Web Speech API
Backend	Python 3.11, FastAPI, WebSockets, Uvicorn
Acoustic ML	PyTorch, HuBERT (superb/hubert-large-superb-er), Librosa
LLM Engine	Groq API — llama-3.1-8b-instant
Containerisation	Docker, Docker Compose, Nginx
CI/CD	GitHub Actions

---

Quick Start (Windows)

Prerequisites

• Python 3.10+ — python.org
• Node.js v18+ — nodejs.org
• Groq API Key — free at console.groq.com
• Chrome or Edge (required for Web Speech API)

1. Clone

git clone https://github.com/Ares19v/Mantis.git
cd Mantis

2. Install

INSTALL.bat

This creates the Python venv, installs all dependencies, runs npm install, and creates .env from the template.

3. Configure

Open .env and add your key:

GROQ_API_KEY=your_groq_api_key_here

4. Run

Run_Project.bat

Two terminal windows open (backend + frontend) and the dashboard launches automatically at http://localhost:5173.

Uninstall

UNINSTALL.bat

---

Docker

Requires Docker Desktop with Compose V2.

Note: The HuBERT model (~1.5 GB) downloads from HuggingFace on first boot and is cached in a Docker volume. Subsequent starts are fast.

# Copy and configure environment
cp .env.example .env
# Edit .env and add GROQ_API_KEY

# Build and launch
docker compose up --build

Service	URL
Dashboard	http://localhost
API / Docs	http://localhost:8000/docs

---

Project Structure

Mantis/
├── main.py                  # FastAPI server — WebSocket + HTTP endpoints
├── transformer_brain.py     # HuBERT emotion inference + acoustic fusion
├── emotion_cnn.py           # Custom CNN architecture (MFCC-based)
├── audio_processor.py       # PCM → MFCC feature extraction
├── trainer.py               # CNN training script (RAVDESS dataset)
├── setup_data.py            # Dataset download utility
├── requirements.txt
├── Dockerfile               # Backend container
├── docker-compose.yml
│
├── mantis_ui/               # React / Vite frontend
│   ├── src/
│   │   └── App.jsx          # Main dashboard component
│   ├── Dockerfile           # Frontend container (nginx)
│   └── nginx.conf
│
├── .github/
│   └── workflows/
│       └── ci.yml           # GitHub Actions — lint + build
│
├── INSTALL.bat
├── Run_Project.bat
└── UNINSTALL.bat

---

Keyboard Shortcuts

Key	Action
Hold Shift	Speak as Agent (release to return to Customer)
Click ● widget	Toggle waveform ↔ sentiment graph
Click notification	Dismiss escalation alert

---

Retraining the Emotion CNN

The repository ships with a pre-trained model (emotion_model.pth — gitignored due to size, 4.3 MB). To retrain on RAVDESS:

# Download RAVDESS dataset
python setup_data.py

# Train for 25 epochs
python trainer.py

---

---

🤗 Model on Hugging Face

The trained model is available on Hugging Face: devanshty/Mantis

Download

from huggingface_hub import hf_hub_download
model_path = hf_hub_download(repo_id='devanshty/Mantis', filename='emotion_model.pth')

---

© 2026 Devansh Tyagi (Ares19v). All Rights Reserved.