diff --git a/OLLAMA_QUICKSTART.txt b/OLLAMA_QUICKSTART.txt new file mode 100644 index 00000000..6a621b3e --- /dev/null +++ b/OLLAMA_QUICKSTART.txt @@ -0,0 +1,28 @@ +================================================================ +NEVERENDINGQUEST - OLLAMA QUICK START +================================================================ + +1. INSTALL OLLAMA https://ollama.com/download +2. PULL ONE MODEL ollama pull llama3.1:8b-instruct-q4_K_M +3. LAUNCH Windows: run_with_ollama_direct.bat + Mac/Lin: ./run_with_ollama_direct.sh + +That's it. The launcher auto-creates the aliases the game needs on +first run, pointing both of the game's internal model tiers at your +one pulled model (same behavior as LM Studio -- no VRAM thrashing). + +Prerequisite: + config.py must exist. + cp config_template.py config.py + (Ollama ignores the API key value, but the file must be importable.) + +To pick between multiple pulled models: + OLLAMA_MODEL= ./run_with_ollama_direct.sh + +Known gaps: + - Image generation (NPC/monster portraits) and TTS will 404. + Disable TTS in the UI; avoid toolkit image features. + - Requires Ollama >= 0.4.0 for function/tool calling. + +Full guide: OLLAMA_SETUP.md +================================================================ diff --git a/OLLAMA_SETUP.md b/OLLAMA_SETUP.md new file mode 100644 index 00000000..c0f25f6a --- /dev/null +++ b/OLLAMA_SETUP.md @@ -0,0 +1,127 @@ +# Ollama Setup Guide + +Run NeverEndingQuest using your local [Ollama](https://ollama.com) installation instead of OpenAI's API. No API costs, no network dependency after pulling a model, and full offline play. + +**Status: EXPERIMENTAL.** NeverEndingQuest's prompts are tuned for GPT-4/GPT-4.1. Local models may produce malformed JSON or incorrect combat decisions — see *Known Issues* below. + +## How It Works + +``` +NeverEndingQuest -> Ollama (port 11434) +``` + +Direct connection, zero overhead. The launcher sets `OPENAI_BASE_URL` so the OpenAI Python SDK talks to Ollama's OpenAI-compatible endpoint instead of `api.openai.com`. + +The game's source code contains two hardcoded OpenAI model identifiers (`gpt-4.1-2025-04-14` for heavy calls and `gpt-4.1-mini-2025-04-14` for light ones). LM Studio ignores these and serves whatever you've loaded — effectively single-model. Ollama **validates** the identifier, so the launcher creates two Ollama aliases on first run, both pointing at the same user-chosen model. Single model, no VRAM thrashing, LM-Studio-equivalent behavior. + +## Prerequisites + +1. **`config.py` must exist.** Every API callsite in NeverEndingQuest is coded as `OpenAI(api_key=config.OPENAI_API_KEY)`. If the file is missing, the game crashes on import before talking to Ollama. Ollama ignores the key value, but the file is mandatory: + + ```bash + cp config_template.py config.py + ``` + + Leave the placeholder key in place or put any non-empty string — it's sent to Ollama and discarded. + +2. **Ollama 0.4 or newer.** Earlier versions lack reliable function/tool calling on the OpenAI-compatible endpoint, which silently breaks combat and validation. Check with `ollama --version`. + +## 1. Install Ollama + +Download from [ollama.com/download](https://ollama.com/download). On macOS/Windows the installer starts the daemon automatically; on Linux, start it with `ollama serve` (as a systemd unit for persistent setups). + +Verify: +```bash +curl http://localhost:11434/api/tags +``` +Expected: JSON response (possibly `{"models":[]}` if nothing is pulled yet). + +## 2. Pull One Model + +Pick exactly one model — the same model will handle both the game's heavy and light calls. Recommended: + +| Model tag | Notes | +|-----------------------------------------|------------------------------------| +| `llama3.1:8b-instruct-q4_K_M` | 128K context, ~5 GB disk/VRAM | +| `mistral:7b-instruct-q4_K_M` | 32K context, ~4 GB disk/VRAM | +| `mistral-nemo:12b-instruct-q4_K_M` | 128K context, stronger prose | + +Example: +```bash +ollama pull llama3.1:8b-instruct-q4_K_M +``` + +You can pull more than one and pick between them with the `OLLAMA_MODEL` env var (see *Switching models* below). But the **active** game session will always use a single model, just like LM Studio. + +## 3. Launch the Game + +- **Windows:** double-click `run_with_ollama_direct.bat` +- **macOS / Linux:** `./run_with_ollama_direct.sh` + +On **first run** the launcher: +1. Verifies `config.py` and the Ollama daemon +2. Finds your pulled model (or uses `$OLLAMA_MODEL` if set) +3. Creates the aliases `gpt-4.1-2025-04-14` and `gpt-4.1-mini-2025-04-14`, both pointing at your model +4. Starts the web server on `http://localhost:8357` + +On **subsequent runs** the launcher sees the aliases already exist and jumps straight to step 4. + +If you've pulled more than one model, the launcher refuses to guess and prints the list. Pick one: + +```bash +OLLAMA_MODEL=llama3.1:8b-instruct-q4_K_M ./run_with_ollama_direct.sh +``` + +## Switching Models + +```bash +ollama pull +ollama rm gpt-4.1-2025-04-14 gpt-4.1-mini-2025-04-14 +./run_with_ollama_direct.sh # recreates aliases against whatever is pulled +``` + +Or force a specific model without deleting aliases first: + +```bash +OLLAMA_MODEL= ./run_with_ollama_direct.sh +``` + +When `OLLAMA_MODEL` is set, the launcher re-points both aliases at the specified model even if they already exist — no `ollama rm` needed. + +## Verifying It Works + +1. Open `http://localhost:8357` +2. Start a new game and take any action +3. Watch the Ollama log: requests should hit `POST /v1/chat/completions` +4. If you see a 404 for `gpt-4.1-2025-04-14`, the alias didn't land — check `ollama list` and re-run the launcher + +## Known Issues + +**Ollama-specific (not LM Studio):** +- **Image generation is broken.** Ollama has no `/v1/images/generations` endpoint. NPC/monster portrait generation from the toolkit will 404. +- **Text-to-speech is broken.** Ollama has no `/v1/audio/speech` endpoint. Disable TTS in the UI before starting a session. +- **Function/tool calling requires Ollama 0.4+.** Older versions produce free-form text instead of JSON action structures and break combat silently. + +**Shared with LM Studio:** +- JSON parsing errors in combat with smaller models (try `q5_K_M` quantization or a larger model) +- Inconsistent action detection vs GPT-4 +- Slow response on CPU-only systems +- `@TAG` compression notation is sometimes ignored + +## Troubleshooting + +| Symptom | Fix | +|------------------------------------------------------|----------------------------------------------------------| +| `connection refused: localhost:11434` | Start Ollama (`ollama serve` / open the app) | +| `No Ollama models are pulled` | `ollama pull ` and re-run the launcher | +| `Multiple models pulled; can't auto-pick` | Set `OLLAMA_MODEL=` and re-run | +| `OLLAMA_MODEL='X' is not pulled` | Typo or missing pull — check `ollama list` | +| `model 'gpt-4.1-2025-04-14' not found` at runtime | Aliases were deleted externally — re-run the launcher | +| Combat actions fail silently | Try a larger quantization or compare with LM Studio | + +## Configuration Reference + +- **Endpoint:** `http://localhost:11434/v1` (set by launcher via `OPENAI_BASE_URL`) +- **Aliases:** `gpt-4.1-2025-04-14` and `gpt-4.1-mini-2025-04-14` (both point at your chosen model) +- **Override source model:** set `OLLAMA_MODEL` before running the launcher +- **Compression:** enabled by default in `model_config.py` (`COMPRESSION_ENABLED = True`) diff --git a/README.md b/README.md index 85a58e8d..c9efb413 100644 --- a/README.md +++ b/README.md @@ -454,11 +454,18 @@ The compression system enables deployment with popular open-source models: - **Batch Processing**: Support for multiple concurrent games #### Setup for Local Models -1. Install local model runtime (Ollama, llama.cpp, etc.) -2. Enable compression in `config.py` -3. Configure model endpoint in `config.py` -4. Adjust context window settings for your model -5. Run game normally - compression handles adaptation +NeverEndingQuest ships with two supported local-model runtimes: **LM Studio** and **Ollama**. Both present an OpenAI-compatible HTTP endpoint that the game reaches via the `OPENAI_BASE_URL` environment variable, which the launcher scripts set for you. You do **not** need to edit `config.py` to switch endpoints — `config.py` only holds your OpenAI API key (used for cloud mode; ignored by local runtimes but the file must exist) and the modules directory. + +1. Install one of the supported runtimes: + - **LM Studio:** [lmstudio.ai](https://lmstudio.ai) — see `LMSTUDIO_SETUP.md` + - **Ollama:** [ollama.com](https://ollama.com) — see `OLLAMA_SETUP.md` +2. Load or pull a model. +3. Start the local server (LM Studio's "Start Server" button, or `ollama serve` — auto on macOS/Windows). +4. Launch the game using the runtime's dedicated script: + - LM Studio: `run_with_lmstudio_direct.bat` + - Ollama: `run_with_ollama_direct.bat` (or `.sh` on macOS/Linux) + +Compression is enabled by default in `model_config.py` (`COMPRESSION_ENABLED = True`) and needs no tuning. #### 🧪 EXPERIMENTAL: LM Studio Integration @@ -545,6 +552,51 @@ Context Length: 32768 or higher **Support**: LM Studio mode is provided as-is for experimentation. For production gameplay, we recommend using OpenAI's API with the compression system for optimal experience. +#### 🧪 EXPERIMENTAL: Ollama Integration + +**Status:** EXPERIMENTAL — same prompt-compatibility caveats as the LM Studio integration above. + +NeverEndingQuest supports [Ollama](https://ollama.com) via its OpenAI-compatible endpoint (`http://localhost:11434/v1`). Ollama tends to be simpler to install on headless machines than LM Studio and has a larger model catalog via `ollama pull`. + +**UX is identical to LM Studio:** + +1. Install Ollama from [ollama.com/download](https://ollama.com/download). +2. Pull **one** model (used for every game request): + ``` + ollama pull llama3.1:8b-instruct-q4_K_M + ``` +3. Double-click `run_with_ollama_direct.bat` (Windows) or run `./run_with_ollama_direct.sh` (macOS/Linux). + +The launcher automatically creates the Ollama aliases the game requires on first run. You only ever pick one model — the launcher points both of the game's internal tier names at that single model, just like LM Studio does implicitly. No VRAM thrashing from loading multiple models between requests. + +**To switch models later:** + +```bash +ollama pull +ollama rm gpt-4.1-2025-04-14 gpt-4.1-mini-2025-04-14 +# then re-launch: the launcher will recreate aliases against whatever is pulled +``` + +Or set `OLLAMA_MODEL=` before running the launcher to force re-aliasing. + +**Recommended Models:** +- `llama3.1:8b-instruct-q4_K_M` — 128K context, strong general performance +- `mistral:7b-instruct-q4_K_M` — 32K context, fast and lightweight +- `mistral-nemo:12b-instruct-q4_K_M` — 128K context, strong storytelling + +**Feature gaps:** Ollama has no image-generation or text-to-speech endpoints — NPC/monster portrait generation and spoken narration will 404 in Ollama mode. Disable TTS in the UI and avoid the toolkit image features. Also requires **Ollama 0.4+** for reliable function/tool calling. + +**Known Issues** (shared with LM Studio): +- JSON parsing errors during complex combat +- Inconsistent action detection compared to GPT-4 +- Slower response times on CPU-only systems + +**Documentation:** +- Complete setup guide: `OLLAMA_SETUP.md` +- Quick reference: `OLLAMA_QUICKSTART.txt` + +**Support:** Ollama mode is provided as-is. For production gameplay, we recommend OpenAI's API with the compression system. + ### Performance Metrics Summary | Component | Original Size | Compressed Size | Reduction | diff --git a/run_with_ollama_direct.bat b/run_with_ollama_direct.bat new file mode 100644 index 00000000..9d93048e --- /dev/null +++ b/run_with_ollama_direct.bat @@ -0,0 +1,124 @@ +@echo off +REM Run NeverEndingQuest with Ollama (Direct Connection - No Proxy) +REM +REM Creates two Ollama aliases on first run pointing at a single user-chosen +REM model, then starts the game. Matches LM Studio's single-model behavior. + +setlocal EnableDelayedExpansion +cd /d %~dp0 + +set "ALIAS_FULL=gpt-4.1-2025-04-14" +set "ALIAS_MINI=gpt-4.1-mini-2025-04-14" + +echo. +echo ======================================================================== +echo NEVERENDINGQUEST - OLLAMA MODE (DIRECT) +echo ======================================================================== +echo. + +REM --- Prerequisite 1: config.py must exist -------------------------------- +if not exist config.py ( + echo [ERROR] config.py not found. Copy config_template.py to config.py first: + echo copy config_template.py config.py + echo Ollama ignores the OPENAI_API_KEY value, but the file must exist. + pause + exit /b 1 +) + +REM --- Prerequisite 2: Ollama daemon reachable ----------------------------- +netstat -an | find "11434" | find "LISTENING" >nul 2>&1 +if %ERRORLEVEL% NEQ 0 ( + echo [ERROR] Ollama daemon is not listening on port 11434. + echo Launch the Ollama app or run 'ollama serve' in another terminal. + pause + exit /b 1 +) + +REM --- Alias setup ---------------------------------------------------------- +REM If OLLAMA_MODEL is set, it is an explicit override -- (re)point the +REM aliases to that model even if they already exist. Otherwise only create +REM aliases when missing. +if defined OLLAMA_MODEL ( + set "source_model=%OLLAMA_MODEL%" + ollama list | findstr /B /L /C:"!source_model!" >nul 2>&1 + if errorlevel 1 ( + echo [ERROR] OLLAMA_MODEL='!source_model!' is not pulled. + echo Run: ollama pull !source_model! + pause + exit /b 1 + ) + echo [INFO] OLLAMA_MODEL set; re-pointing aliases at '!source_model!'. + ollama cp "!source_model!" "%ALIAS_FULL%" + if errorlevel 1 goto :alias_fail + ollama cp "!source_model!" "%ALIAS_MINI%" + if errorlevel 1 goto :alias_fail + echo [INFO] Aliases updated. + goto :launch +) + +REM findstr /B /L /C:"..." = literal (non-regex) prefix match; handles "name" and "name:latest". +set "have_full=0" +set "have_mini=0" +ollama list | findstr /B /L /C:"%ALIAS_FULL%" >nul 2>&1 && set "have_full=1" +ollama list | findstr /B /L /C:"%ALIAS_MINI%" >nul 2>&1 && set "have_mini=1" + +if "%have_full%"=="1" if "%have_mini%"=="1" goto :launch + +echo [INFO] Ollama aliases missing. Creating them now... + +REM Build a candidates list: all names in `ollama list`, skipping the +REM header row and the alias names themselves. +set "count=0" +set "first_candidate=" +for /f "skip=1 tokens=1" %%N in ('ollama list') do ( + set "name=%%N" + set "base=!name!" + if /i "!base:~-7!"==":latest" set "base=!base:~0,-7!" + if /i not "!base!"=="%ALIAS_FULL%" if /i not "!base!"=="%ALIAS_MINI%" ( + set /a count+=1 + if not defined first_candidate set "first_candidate=!name!" + set "cand_!count!=!name!" + ) +) +if !count! EQU 0 ( + echo [ERROR] No Ollama models are pulled. + echo Pull one first, e.g.: ollama pull llama3.1:8b-instruct-q4_K_M + pause + exit /b 1 +) +if !count! GTR 1 ( + echo [ERROR] Multiple models pulled; can't auto-pick. + echo Set OLLAMA_MODEL to one of the following and re-run: + for /L %%I in (1,1,!count!) do echo !cand_%%I! + pause + exit /b 1 +) +set "source_model=!first_candidate!" + +echo [INFO] Using '!source_model!' for both full and mini tiers. +ollama cp "!source_model!" "%ALIAS_FULL%" +if errorlevel 1 goto :alias_fail +ollama cp "!source_model!" "%ALIAS_MINI%" +if errorlevel 1 goto :alias_fail +echo [INFO] Aliases created. + +:launch +echo [INFO] Redirecting OpenAI SDK to Ollama (localhost:11434)... +set OPENAI_BASE_URL=http://localhost:11434/v1 +set OPENAI_API_KEY=ollama + +REM Prefer `python`, fall back to the Windows launcher `py -3` for installs +REM that lack `python` on PATH but have the official launcher. +where python >nul 2>&1 +if %ERRORLEVEL% EQU 0 ( + python run_web.py +) else ( + py -3 run_web.py +) +pause +exit /b 0 + +:alias_fail +echo [ERROR] ollama cp failed. +pause +exit /b 1 diff --git a/run_with_ollama_direct.sh b/run_with_ollama_direct.sh new file mode 100644 index 00000000..da621c8e --- /dev/null +++ b/run_with_ollama_direct.sh @@ -0,0 +1,122 @@ +#!/usr/bin/env bash +# Run NeverEndingQuest with Ollama (Direct Connection - No Proxy) +# +# The game sends hardcoded OpenAI model names in every request. LM Studio +# ignores them; Ollama validates them and 404s unknown tags. This launcher +# creates two Ollama aliases on first run that both point at a single +# user-chosen model -- matching LM Studio's single-model behavior and +# avoiding VRAM thrashing between tiers. + +set -euo pipefail +cd "$(dirname "$0")" + +readonly ALIAS_FULL="gpt-4.1-2025-04-14" +readonly ALIAS_MINI="gpt-4.1-mini-2025-04-14" + +echo +echo "========================================================================" +echo "NEVERENDINGQUEST - OLLAMA MODE (DIRECT)" +echo "========================================================================" +echo + +# --- Prerequisite 1: config.py must exist ------------------------------------ +# Every callsite does OpenAI(api_key=config.OPENAI_API_KEY). Import fails +# before any Ollama request if config.py is missing. +if [ ! -f config.py ]; then + echo "[ERROR] config.py not found. Copy the template first:" + echo " cp config_template.py config.py" + echo "Ollama ignores the OPENAI_API_KEY value, but the file must exist." + exit 1 +fi + +# --- Prerequisite 2: Ollama CLI + daemon -------------------------------------- +if ! command -v ollama >/dev/null 2>&1; then + echo "[ERROR] ollama CLI not found on PATH. Install from https://ollama.com/download" + exit 1 +fi +if ! curl -sf --connect-timeout 3 --max-time 5 http://localhost:11434/api/tags >/dev/null 2>&1; then + echo "[ERROR] Ollama daemon is not reachable on localhost:11434." + echo "Start it with 'ollama serve' (Linux) or open the Ollama app (macOS)." + exit 1 +fi + +# --- Helper: match a model name in `ollama list` ------------------------------ +# `ollama list` prints names as either "name" or "name:tag". Use awk equality +# (not regex) to avoid metacharacter pitfalls with dots/colons in user tags. +_ollama_has_model() { + local target="$1" + ollama list | awk -v t="$target" ' + NR > 1 { + if ($1 == t) { found = 1; exit } + if ($1 == t ":latest") { found = 1; exit } + if (index($1, t ":") == 1) { found = 1; exit } + } + END { exit !found } + ' +} + +# --- Alias setup ------------------------------------------------------------- +# Normally: create aliases only if missing. If OLLAMA_MODEL is set, treat it +# as an explicit override and (re)point the aliases to that model even if +# they already exist. +_aliases_present() { + _ollama_has_model "$ALIAS_FULL" && _ollama_has_model "$ALIAS_MINI" +} + +if [ -n "${OLLAMA_MODEL:-}" ]; then + source_model="$OLLAMA_MODEL" + if ! _ollama_has_model "$source_model"; then + echo "[ERROR] OLLAMA_MODEL='$source_model' is not pulled." + echo "Run: ollama pull $source_model" + exit 1 + fi + echo "[INFO] OLLAMA_MODEL set; re-pointing aliases at '$source_model'." + ollama cp "$source_model" "$ALIAS_FULL" + ollama cp "$source_model" "$ALIAS_MINI" + echo "[INFO] Aliases updated." +elif _aliases_present; then + echo "[INFO] Aliases already present; skipping setup." +else + echo "[INFO] Ollama aliases missing. Creating them now..." + + # Candidates = column 1 of `ollama list`, excluding header and the + # two alias names themselves (so re-runs don't count them). + # Portable to bash 3.2 (macOS default) -- no `mapfile`. + candidates=() + while IFS= read -r _name; do + [ -n "$_name" ] && candidates+=("$_name") + done < <(ollama list | awk -v a="$ALIAS_FULL" -v b="$ALIAS_MINI" ' + NR > 1 { + name = $1 + # Strip :latest suffix for dedupe comparisons only + base = name + sub(/:latest$/, "", base) + if (base == a || name == a) next + if (base == b || name == b) next + print name + }') + if [ "${#candidates[@]}" -eq 0 ]; then + echo "[ERROR] No Ollama models are pulled." + echo "Pull one first, e.g.: ollama pull llama3.1:8b-instruct-q4_K_M" + exit 1 + elif [ "${#candidates[@]}" -gt 1 ]; then + echo "[ERROR] Multiple models pulled; can't auto-pick." + echo "Set OLLAMA_MODEL to one of the following and re-run:" + printf " %s\n" "${candidates[@]}" + exit 1 + fi + source_model="${candidates[0]}" + + echo "[INFO] Using '$source_model' for both full and mini tiers." + ollama cp "$source_model" "$ALIAS_FULL" + ollama cp "$source_model" "$ALIAS_MINI" + echo "[INFO] Aliases created." +fi + +# --- Launch ------------------------------------------------------------------ +echo "[INFO] Redirecting OpenAI SDK to Ollama (localhost:11434)..." +export OPENAI_BASE_URL="http://localhost:11434/v1" +export OPENAI_API_KEY="ollama" + +PYTHON_BIN="${PYTHON:-python3}" +"$PYTHON_BIN" run_web.py