Voice Assistant

On-device iOS voice assistant with local LLM inference. Speech stays on the phone (CoreML TTS + Apple Speech ASR), and the LLM runs on your Mac over a WebSocket — no cloud, no API keys, no data leaving your network.

Architecture

┌──────────────────────────────┐         ┌──────────────────────────────┐
│         iOS App              │         │        Mac Server            │
│                              │   WSS   │                              │
│  Apple Speech (ASR)  ────────┼────────►│  FastAPI WebSocket           │
│                              │  JWT    │  LLM: gpt-oss-20b (MLX)      │
│  Kokoro-82M CoreML (TTS) ◄───┼─────────┤  Auth: PBKDF2 + JWT          │
│                              │         │  Knowledge Base: RAG         │
│  SwiftUI Hold-to-Talk UI     │         │                              │
└──────────────────────────────┘         └──────────────────────────────┘

What runs where:

Component	Location	Purpose
Speech-to-text	iOS device	Apple Speech framework, on-device
Text-to-speech	iOS device	Kokoro-82M converted to CoreML
LLM inference	Mac	gpt-oss-20b via mlx-lm (streaming)
Auth	Mac	JWT with PBKDF2 (210k iterations), refresh rotation
RAG / Knowledge base	Mac	Cosine similarity over embeddings

Prerequisites

Hardware:

• Apple Silicon Mac (M1/M2/M3/M4) with ~20GB RAM
• iPhone running iOS 26.0+
• Both on the same Wi-Fi network (or connected via Tailscale)

Software:

• Xcode 26+
• Python 3.10+ (for the server and conversion scripts)
• Apple Developer account (free tier works for local device builds)

Setup

1. Clone and install server dependencies

git clone https://github.com/maxpar1/voice-assistant.git
cd voice-assistant/server

python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

2. Get the CoreML TTS models

The kokoro_acoustic.mlpackage and kokoro_decoder.mlpackage files are not in the repo (too large for GitHub — 310MB combined).

Option A — Download prebuilt from Hugging Face (recommended):

pip install huggingface_hub
python scripts/download_models.py

This fetches the CoreML packages from maxpar1/kokoro-82m-coreml and places them in ios/VoiceAssistant/.

Option B — Generate from source:

cd scripts
python convert_kokoro82m_to_coreml.py
python export_two_models.py
cp -r outputs/kokoro_acoustic.mlpackage ../ios/VoiceAssistant/
cp -r outputs/kokoro_decoder.mlpackage ../ios/VoiceAssistant/

This downloads Kokoro-82M weights, patches the PyTorch model (split bidirectional LSTMs, iSTFT fixes, manual interpolation, Box-Muller noise), and exports two CoreML packages. See scripts/COREML_CONVERSION_STATUS.md for the full conversion story.

After either option, open ios/VoiceAssistant.xcodeproj in Xcode and confirm both .mlpackage directories are added to the VoiceAssistant target (they should auto-sync).

3. Start the server

Optional: copy and edit environment variables:

cp server/.env.example server/.env
# Set ASSISTANT_JWT_SECRET to a random string for production use.
# For local dev you can leave defaults.

Start the server:

cd voice-assistant/server
python voice_assistant_ws_server.py \
  --host 0.0.0.0 \
  --port 8765 \
  --kb-enabled \
  --kb-embedding-model Qwen/Qwen3-Embedding-0.6B

First run downloads the 20B LLM (~12GB to ~/.cache/huggingface/) and the 0.6B embedding model (~600MB). Subsequent runs load from cache (~30 seconds).

4. Register a user

curl -X POST http://127.0.0.1:8765/auth/register \
  -H "Content-Type: application/json" \
  -d '{"email":"you@example.com","password":"your-password"}'

5. Update the iOS WebSocket URL

Find your Mac's LAN IP:

ipconfig getifaddr en0

Edit ios/VoiceAssistant/VoiceSessionService.swift, line 9:

static let defaultWSURL = "ws://YOUR_MAC_IP:8765/ws/assistant"

6. Build and run the iOS app

Open ios/VoiceAssistant.xcodeproj in Xcode:

• Select your iPhone as the build target
• Sign with your Apple Developer account (Signing & Capabilities tab)
• Build and run (⌘R)

iOS App Transport Security note: The app connects over plain ws:// for local dev, which iOS blocks by default. The project's Info.plist includes NSAppTransportSecurity → NSAllowsArbitraryLoads = YES to permit this. For production, use wss:// with a TLS-terminating proxy.

First launch:

1. Log in with the credentials you registered
2. Tap Prepare Kokoro Model (loads and compiles CoreML models, ~10 seconds)
3. Tap Connect
4. Hold Hold to Talk, speak, release

Notable Engineering

CoreML Conversion

Converting Kokoro-82M to CoreML required solving several problems that aren't in the standard TTS export path:

• Bidirectional LSTM splitting — pack_padded_sequence is untraceable by torch.jit.trace, so bidirectional LSTMs were split into forward/backward unidirectional passes.
• iSTFT reconstruction — the original iSTFT implementation was missing one-sided spectrum doubling and COLA normalization.
• Dynamic interpolation — F.interpolate with dynamic sizes fails coremltools export. Replaced with manual arange/gather/lerp.
• Deterministic noise — torch.randn is non-deterministic at export time. Replaced with Box-Muller Gaussian approximation from a seed.
• espeak-ng bridge — for phoneme generation, dlopen/dlsym to a bundled .xcframework rather than bridging through Swift.

Final result: correlation 0.9826 vs original PyTorch output, SNR 15dB.

See scripts/COREML_CONVERSION_STATUS.md for the full debugging diary.

Auth

Hand-rolled JWT auth (no frameworks):

• PBKDF2-SHA256 with 210,000 iterations
• HS256 JWTs with typed access/refresh distinction
• Refresh token rotation with revocation chains
• Constant-time comparisons, email normalization, cascade deletes
• Thread-safe SQLite with WAL mode

See server/assistant_auth.py (800 lines, pure stdlib + sqlite3).

Knowledge Base (RAG)

Embedding-based retrieval for context-aware responses:

• Qwen3-Embedding-0.6B for document/query embeddings
• Cosine similarity search, configurable top_k
• Document ingestion via /kb/text and /kb/documents endpoints

Project Structure

voice-assistant/
├── ios/                                # iOS app (Swift, SwiftUI)
│   ├── VoiceAssistant.xcodeproj/
│   └── VoiceAssistant/
│       ├── AIView.swift                # Main voice UI
│       ├── VoiceSessionService.swift   # WebSocket + ASR + TTS orchestration
│       ├── KokoroTTSService.swift      # CoreML TTS + espeak-ng bridge
│       ├── VoiceWSMessages.swift       # WebSocket message models
│       ├── HarComputer.swift           # Phonemic analysis
│       ├── Info.plist
│       ├── kokoro_assets/              # TTS voices, lexicons, runtime
│       └── vendor/
│           └── EspeakNg.xcframework    # espeak-ng prebuilt framework
│
├── server/                             # Python backend
│   ├── voice_assistant_ws_server.py    # FastAPI WebSocket server
│   ├── assistant_auth.py               # JWT auth with PBKDF2
│   ├── assistant_kb.py                 # Knowledge base with embedding search
│   ├── assistant_web.html              # Browser client for testing
│   ├── KB_API.md                       # Knowledge base API docs
│   ├── requirements.txt
│   └── .env.example
│
├── scripts/                            # CoreML conversion + utilities
│   ├── convert_kokoro82m_to_coreml.py  # Main conversion pipeline
│   ├── export_two_models.py            # Split acoustic + decoder export
│   ├── validate_har_computer.py        # Correlation/SNR validation
│   └── COREML_CONVERSION_STATUS.md     # Conversion debugging diary
│
├── LICENSE                             # Apache 2.0
├── .gitignore
└── README.md

Tech Stack

iOS: Swift, SwiftUI, Apple Speech, CoreML, AVFoundation, WebSocket Server: Python, FastAPI, uvicorn, mlx-lm, websockets, wsproto TTS: Kokoro-82M (custom CoreML conversion with iSTFT + manual interpolation patches) LLM: gpt-oss-20b (MLX, 4-bit MXFP4 quantization) Embedding: Qwen3-Embedding-0.6B Auth: PBKDF2-SHA256, JWT HS256 Phoneme generation: espeak-ng via dlopen/dlsym from xcframework

Troubleshooting

"Could not connect to server" on iOS:

• Ensure server started with --host 0.0.0.0 (not 127.0.0.1)
• Verify iPhone can reach Mac: open http://MAC_IP:8765/health in mobile Safari
• Check Mac firewall allows incoming connections on port 8765

"Library not loaded: libespeak-ng.1.dylib":

• The app bundle's Frameworks/ is missing the espeak dylib
• Clean build folder (⌘⇧K) and rebuild
• Verify ENABLE_USER_SCRIPT_SANDBOXING = NO in build settings (required for the embed script)

"Kokoro frontend manifest not found":

• kokoro_assets/ is not bundled as a folder reference
• In Xcode, remove and re-add kokoro_assets/ as a folder reference (blue folder icon)

Server shows "No supported WebSocket library detected":

• Activate the venv before running the server: source .venv/bin/activate
• Reinstall: pip install 'uvicorn[standard]' websockets wsproto

License

Apache 2.0 — see LICENSE.