Skip to content
Open
28 changes: 28 additions & 0 deletions OLLAMA_QUICKSTART.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
================================================================
NEVERENDINGQUEST - OLLAMA QUICK START
================================================================

1. INSTALL OLLAMA https://ollama.com/download
2. PULL ONE MODEL ollama pull llama3.1:8b-instruct-q4_K_M
3. LAUNCH Windows: run_with_ollama_direct.bat
Mac/Lin: ./run_with_ollama_direct.sh

That's it. The launcher auto-creates the aliases the game needs on
first run, pointing both of the game's internal model tiers at your
one pulled model (same behavior as LM Studio -- no VRAM thrashing).

Prerequisite:
config.py must exist.
cp config_template.py config.py
(Ollama ignores the API key value, but the file must be importable.)

To pick between multiple pulled models:
OLLAMA_MODEL=<tag> ./run_with_ollama_direct.sh

Known gaps:
- Image generation (NPC/monster portraits) and TTS will 404.
Disable TTS in the UI; avoid toolkit image features.
- Requires Ollama >= 0.4.0 for function/tool calling.

Full guide: OLLAMA_SETUP.md
================================================================
127 changes: 127 additions & 0 deletions OLLAMA_SETUP.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,127 @@
# Ollama Setup Guide

Run NeverEndingQuest using your local [Ollama](https://ollama.com) installation instead of OpenAI's API. No API costs, no network dependency after pulling a model, and full offline play.

**Status: EXPERIMENTAL.** NeverEndingQuest's prompts are tuned for GPT-4/GPT-4.1. Local models may produce malformed JSON or incorrect combat decisions — see *Known Issues* below.

## How It Works

```
NeverEndingQuest -> Ollama (port 11434)
```

Direct connection, zero overhead. The launcher sets `OPENAI_BASE_URL` so the OpenAI Python SDK talks to Ollama's OpenAI-compatible endpoint instead of `api.openai.com`.

The game's source code contains two hardcoded OpenAI model identifiers (`gpt-4.1-2025-04-14` for heavy calls and `gpt-4.1-mini-2025-04-14` for light ones). LM Studio ignores these and serves whatever you've loaded — effectively single-model. Ollama **validates** the identifier, so the launcher creates two Ollama aliases on first run, both pointing at the same user-chosen model. Single model, no VRAM thrashing, LM-Studio-equivalent behavior.

## Prerequisites

1. **`config.py` must exist.** Every API callsite in NeverEndingQuest is coded as `OpenAI(api_key=config.OPENAI_API_KEY)`. If the file is missing, the game crashes on import before talking to Ollama. Ollama ignores the key value, but the file is mandatory:

```bash
cp config_template.py config.py
```

Leave the placeholder key in place or put any non-empty string — it's sent to Ollama and discarded.

2. **Ollama 0.4 or newer.** Earlier versions lack reliable function/tool calling on the OpenAI-compatible endpoint, which silently breaks combat and validation. Check with `ollama --version`.

## 1. Install Ollama

Download from [ollama.com/download](https://ollama.com/download). On macOS/Windows the installer starts the daemon automatically; on Linux, start it with `ollama serve` (as a systemd unit for persistent setups).

Verify:
```bash
curl http://localhost:11434/api/tags
```
Expected: JSON response (possibly `{"models":[]}` if nothing is pulled yet).

## 2. Pull One Model

Pick exactly one model — the same model will handle both the game's heavy and light calls. Recommended:

| Model tag | Notes |
|-----------------------------------------|------------------------------------|
| `llama3.1:8b-instruct-q4_K_M` | 128K context, ~5 GB disk/VRAM |
| `mistral:7b-instruct-q4_K_M` | 32K context, ~4 GB disk/VRAM |
| `mistral-nemo:12b-instruct-q4_K_M` | 128K context, stronger prose |

Example:
```bash
ollama pull llama3.1:8b-instruct-q4_K_M
```

You can pull more than one and pick between them with the `OLLAMA_MODEL` env var (see *Switching models* below). But the **active** game session will always use a single model, just like LM Studio.

## 3. Launch the Game

- **Windows:** double-click `run_with_ollama_direct.bat`
- **macOS / Linux:** `./run_with_ollama_direct.sh`

On **first run** the launcher:
1. Verifies `config.py` and the Ollama daemon
2. Finds your pulled model (or uses `$OLLAMA_MODEL` if set)
3. Creates the aliases `gpt-4.1-2025-04-14` and `gpt-4.1-mini-2025-04-14`, both pointing at your model
4. Starts the web server on `http://localhost:8357`

On **subsequent runs** the launcher sees the aliases already exist and jumps straight to step 4.

If you've pulled more than one model, the launcher refuses to guess and prints the list. Pick one:

```bash
OLLAMA_MODEL=llama3.1:8b-instruct-q4_K_M ./run_with_ollama_direct.sh
```

## Switching Models

```bash
ollama pull <new-tag>
ollama rm gpt-4.1-2025-04-14 gpt-4.1-mini-2025-04-14
./run_with_ollama_direct.sh # recreates aliases against whatever is pulled
```

Or force a specific model without deleting aliases first:

```bash
OLLAMA_MODEL=<new-tag> ./run_with_ollama_direct.sh
```

When `OLLAMA_MODEL` is set, the launcher re-points both aliases at the specified model even if they already exist — no `ollama rm` needed.

## Verifying It Works

1. Open `http://localhost:8357`
2. Start a new game and take any action
3. Watch the Ollama log: requests should hit `POST /v1/chat/completions`
4. If you see a 404 for `gpt-4.1-2025-04-14`, the alias didn't land — check `ollama list` and re-run the launcher

## Known Issues

**Ollama-specific (not LM Studio):**
- **Image generation is broken.** Ollama has no `/v1/images/generations` endpoint. NPC/monster portrait generation from the toolkit will 404.
- **Text-to-speech is broken.** Ollama has no `/v1/audio/speech` endpoint. Disable TTS in the UI before starting a session.
- **Function/tool calling requires Ollama 0.4+.** Older versions produce free-form text instead of JSON action structures and break combat silently.

**Shared with LM Studio:**
- JSON parsing errors in combat with smaller models (try `q5_K_M` quantization or a larger model)
- Inconsistent action detection vs GPT-4
- Slow response on CPU-only systems
- `@TAG` compression notation is sometimes ignored

## Troubleshooting

| Symptom | Fix |
|------------------------------------------------------|----------------------------------------------------------|
| `connection refused: localhost:11434` | Start Ollama (`ollama serve` / open the app) |
| `No Ollama models are pulled` | `ollama pull <tag>` and re-run the launcher |
| `Multiple models pulled; can't auto-pick` | Set `OLLAMA_MODEL=<tag>` and re-run |
| `OLLAMA_MODEL='X' is not pulled` | Typo or missing pull — check `ollama list` |
| `model 'gpt-4.1-2025-04-14' not found` at runtime | Aliases were deleted externally — re-run the launcher |
| Combat actions fail silently | Try a larger quantization or compare with LM Studio |

## Configuration Reference

- **Endpoint:** `http://localhost:11434/v1` (set by launcher via `OPENAI_BASE_URL`)
- **Aliases:** `gpt-4.1-2025-04-14` and `gpt-4.1-mini-2025-04-14` (both point at your chosen model)
- **Override source model:** set `OLLAMA_MODEL` before running the launcher
- **Compression:** enabled by default in `model_config.py` (`COMPRESSION_ENABLED = True`)
62 changes: 57 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -454,11 +454,18 @@ The compression system enables deployment with popular open-source models:
- **Batch Processing**: Support for multiple concurrent games

#### Setup for Local Models
1. Install local model runtime (Ollama, llama.cpp, etc.)
2. Enable compression in `config.py`
3. Configure model endpoint in `config.py`
4. Adjust context window settings for your model
5. Run game normally - compression handles adaptation
NeverEndingQuest ships with two supported local-model runtimes: **LM Studio** and **Ollama**. Both present an OpenAI-compatible HTTP endpoint that the game reaches via the `OPENAI_BASE_URL` environment variable, which the launcher scripts set for you. You do **not** need to edit `config.py` to switch endpoints — `config.py` only holds your OpenAI API key (used for cloud mode; ignored by local runtimes but the file must exist) and the modules directory.

1. Install one of the supported runtimes:
- **LM Studio:** [lmstudio.ai](https://lmstudio.ai) — see `LMSTUDIO_SETUP.md`
- **Ollama:** [ollama.com](https://ollama.com) — see `OLLAMA_SETUP.md`
2. Load or pull a model.
3. Start the local server (LM Studio's "Start Server" button, or `ollama serve` — auto on macOS/Windows).
4. Launch the game using the runtime's dedicated script:
- LM Studio: `run_with_lmstudio_direct.bat`
- Ollama: `run_with_ollama_direct.bat` (or `.sh` on macOS/Linux)

Compression is enabled by default in `model_config.py` (`COMPRESSION_ENABLED = True`) and needs no tuning.

#### 🧪 EXPERIMENTAL: LM Studio Integration

Expand Down Expand Up @@ -545,6 +552,51 @@ Context Length: 32768 or higher

**Support**: LM Studio mode is provided as-is for experimentation. For production gameplay, we recommend using OpenAI's API with the compression system for optimal experience.

#### 🧪 EXPERIMENTAL: Ollama Integration

**Status:** EXPERIMENTAL — same prompt-compatibility caveats as the LM Studio integration above.

NeverEndingQuest supports [Ollama](https://ollama.com) via its OpenAI-compatible endpoint (`http://localhost:11434/v1`). Ollama tends to be simpler to install on headless machines than LM Studio and has a larger model catalog via `ollama pull`.

**UX is identical to LM Studio:**

1. Install Ollama from [ollama.com/download](https://ollama.com/download).
2. Pull **one** model (used for every game request):
```
ollama pull llama3.1:8b-instruct-q4_K_M
```
3. Double-click `run_with_ollama_direct.bat` (Windows) or run `./run_with_ollama_direct.sh` (macOS/Linux).

The launcher automatically creates the Ollama aliases the game requires on first run. You only ever pick one model — the launcher points both of the game's internal tier names at that single model, just like LM Studio does implicitly. No VRAM thrashing from loading multiple models between requests.

**To switch models later:**

```bash
ollama pull <new-model>
ollama rm gpt-4.1-2025-04-14 gpt-4.1-mini-2025-04-14
# then re-launch: the launcher will recreate aliases against whatever is pulled
```

Or set `OLLAMA_MODEL=<new-model>` before running the launcher to force re-aliasing.

**Recommended Models:**
- `llama3.1:8b-instruct-q4_K_M` — 128K context, strong general performance
- `mistral:7b-instruct-q4_K_M` — 32K context, fast and lightweight
- `mistral-nemo:12b-instruct-q4_K_M` — 128K context, strong storytelling

**Feature gaps:** Ollama has no image-generation or text-to-speech endpoints — NPC/monster portrait generation and spoken narration will 404 in Ollama mode. Disable TTS in the UI and avoid the toolkit image features. Also requires **Ollama 0.4+** for reliable function/tool calling.

**Known Issues** (shared with LM Studio):
- JSON parsing errors during complex combat
- Inconsistent action detection compared to GPT-4
- Slower response times on CPU-only systems

**Documentation:**
- Complete setup guide: `OLLAMA_SETUP.md`
- Quick reference: `OLLAMA_QUICKSTART.txt`

**Support:** Ollama mode is provided as-is. For production gameplay, we recommend OpenAI's API with the compression system.

### Performance Metrics Summary

| Component | Original Size | Compressed Size | Reduction |
Expand Down
124 changes: 124 additions & 0 deletions run_with_ollama_direct.bat
Original file line number Diff line number Diff line change
@@ -0,0 +1,124 @@
@echo off
REM Run NeverEndingQuest with Ollama (Direct Connection - No Proxy)
REM
REM Creates two Ollama aliases on first run pointing at a single user-chosen
REM model, then starts the game. Matches LM Studio's single-model behavior.

setlocal EnableDelayedExpansion
cd /d %~dp0

set "ALIAS_FULL=gpt-4.1-2025-04-14"
set "ALIAS_MINI=gpt-4.1-mini-2025-04-14"

echo.
echo ========================================================================
echo NEVERENDINGQUEST - OLLAMA MODE (DIRECT)
echo ========================================================================
echo.

REM --- Prerequisite 1: config.py must exist --------------------------------
if not exist config.py (
echo [ERROR] config.py not found. Copy config_template.py to config.py first:
echo copy config_template.py config.py
echo Ollama ignores the OPENAI_API_KEY value, but the file must exist.
pause
exit /b 1
)

REM --- Prerequisite 2: Ollama daemon reachable -----------------------------
netstat -an | find "11434" | find "LISTENING" >nul 2>&1
if %ERRORLEVEL% NEQ 0 (
echo [ERROR] Ollama daemon is not listening on port 11434.
echo Launch the Ollama app or run 'ollama serve' in another terminal.
pause
exit /b 1
)

REM --- Alias setup ----------------------------------------------------------
REM If OLLAMA_MODEL is set, it is an explicit override -- (re)point the
REM aliases to that model even if they already exist. Otherwise only create
REM aliases when missing.
if defined OLLAMA_MODEL (
set "source_model=%OLLAMA_MODEL%"
ollama list | findstr /B /L /C:"!source_model!" >nul 2>&1
if errorlevel 1 (
echo [ERROR] OLLAMA_MODEL='!source_model!' is not pulled.
echo Run: ollama pull !source_model!
pause
exit /b 1
)
echo [INFO] OLLAMA_MODEL set; re-pointing aliases at '!source_model!'.
ollama cp "!source_model!" "%ALIAS_FULL%"
if errorlevel 1 goto :alias_fail
ollama cp "!source_model!" "%ALIAS_MINI%"
if errorlevel 1 goto :alias_fail
echo [INFO] Aliases updated.
goto :launch
)

REM findstr /B /L /C:"..." = literal (non-regex) prefix match; handles "name" and "name:latest".
set "have_full=0"
set "have_mini=0"
ollama list | findstr /B /L /C:"%ALIAS_FULL%" >nul 2>&1 && set "have_full=1"
ollama list | findstr /B /L /C:"%ALIAS_MINI%" >nul 2>&1 && set "have_mini=1"

if "%have_full%"=="1" if "%have_mini%"=="1" goto :launch

echo [INFO] Ollama aliases missing. Creating them now...

REM Build a candidates list: all names in `ollama list`, skipping the
REM header row and the alias names themselves.
set "count=0"
set "first_candidate="
for /f "skip=1 tokens=1" %%N in ('ollama list') do (
set "name=%%N"
set "base=!name!"
if /i "!base:~-7!"==":latest" set "base=!base:~0,-7!"
if /i not "!base!"=="%ALIAS_FULL%" if /i not "!base!"=="%ALIAS_MINI%" (
set /a count+=1
if not defined first_candidate set "first_candidate=!name!"
set "cand_!count!=!name!"
)
)
if !count! EQU 0 (
echo [ERROR] No Ollama models are pulled.
echo Pull one first, e.g.: ollama pull llama3.1:8b-instruct-q4_K_M
pause
exit /b 1
)
if !count! GTR 1 (
echo [ERROR] Multiple models pulled; can't auto-pick.
echo Set OLLAMA_MODEL to one of the following and re-run:
for /L %%I in (1,1,!count!) do echo !cand_%%I!
pause
exit /b 1
)
set "source_model=!first_candidate!"

echo [INFO] Using '!source_model!' for both full and mini tiers.
ollama cp "!source_model!" "%ALIAS_FULL%"
if errorlevel 1 goto :alias_fail
ollama cp "!source_model!" "%ALIAS_MINI%"
if errorlevel 1 goto :alias_fail
echo [INFO] Aliases created.

:launch
echo [INFO] Redirecting OpenAI SDK to Ollama (localhost:11434)...
set OPENAI_BASE_URL=http://localhost:11434/v1
set OPENAI_API_KEY=ollama

REM Prefer `python`, fall back to the Windows launcher `py -3` for installs
REM that lack `python` on PATH but have the official launcher.
where python >nul 2>&1
if %ERRORLEVEL% EQU 0 (
python run_web.py
) else (
py -3 run_web.py
)
pause
exit /b 0

:alias_fail
echo [ERROR] ollama cp failed.
pause
exit /b 1
Loading