diff --git a/CHANGELOG.md b/CHANGELOG.md index 8b0cd7bf..70a57c85 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -109,6 +109,13 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 conversation under a new resumable session id; prompt history persists in `.godspeed/history` with Ctrl-R reverse search. +- **docs(llamacpp): Qwen3.8-27B profile now carries measurements, not estimates** — + `scripts/settings_local_llm_qwen38_27b.yaml` claimed ~65-75 tok/s plain and + 1.3x-2.8x from MTP; measured on an RTX 5070 Ti it is 54.6 tok/s plain and + ~1.4x (78 tok/s) on real agent requests, with `spec_draft_n_max: 4` the best + of 2-6. New `docs/local_qwen38_27b.md` (methods, tables, caveats) and + `scripts/serve_qwen38_27b_llamacpp.sh` (the measured launch command). + ### Fixed - **fix(llm): Qwen3.8 was not recognised as thinking-capable** — the model diff --git a/docs/local_qwen38_27b.md b/docs/local_qwen38_27b.md new file mode 100644 index 00000000..107460fa --- /dev/null +++ b/docs/local_qwen38_27b.md @@ -0,0 +1,87 @@ +# Local Qwen3.8-27B on a 16 GB GPU: what was measured + +Measured on **one machine** on 2026-09-24: RTX 5070 Ti (16 GB, 16,303 MiB), Ryzen 5 7600, WSL2 Ubuntu, +llama.cpp `3173a56` (built 2026-08-29), model +[`ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF`](https://huggingface.co/ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF) +`Qwen3.8-27B-GSQ-RCO-IQ3_XXS-mtp.gguf` (10.44 GB, Apache-2.0). Treat every number as "what this box did", +not a promise for other hardware. VRAM figures include the ~1.3-1.9 GB the Windows desktop was using. + +## Launch + +`scripts/serve_qwen38_27b_llamacpp.sh` starts the exact configuration below. Godspeed on Windows cannot +launch a Linux binary, so start it in WSL and Godspeed attaches to `127.0.0.1:8080`; then use +`scripts/settings_local_llm_qwen38_27b.yaml`. + +Key flags: `-c 32768 -fa on -np 1 --cache-type-k q8_0 --cache-type-v q8_0 -ub 512` +`--spec-type draft-mtp --spec-draft-n-max 4 --jinja`. MTP needs `-np 1` (one slot). + +## Speed and VRAM (32K context, q8_0 KV) + +| config | decode tok/s, real agent requests | decode tok/s, synthetic prompts | peak VRAM (MiB) | +|---|---|---|---| +| no speculation | 54.6 | 54.5 | 12,960 | +| MTP n=3 | 76.4 | 94.7 | 14,203 | +| **MTP n=4** | **78.2** | 101.0 | 14,128 | +| DFlash2 drafter, n=7 | 78.8 | 117.1 | 15,124 (`-ub 256`) | + +* "Real agent requests" = 69 requests captured from an agent-in-loop run and replayed against each + configuration (identical requests, at most 300 new tokens each). The live evaluations measured the same + thing: 77.5 tok/s and 0.51 draft acceptance with MTP n=4 over 249 requests. +* "Synthetic prompts" (code, prose, tool-call and reasoning prompts written for the benchmark) overstate the + speed-up: real output is mostly medium-effort reasoning prose, which the drafters accept poorly (DFlash2 + acceptance 0.36 on real requests vs 0.58 synthetic). Do not choose a speculation config from synthetic + numbers alone. +* DFlash2 needs a separate drafter file and about 1 GB more VRAM for no measurable gain on real traffic, so + MTP n=4 is the recommended setting. +* KV `q4_0` saves about 1 GiB with no speed change; its effect on quality was **not** measured. + +## Prefix cache + +The model is a Gated-DeltaNet hybrid, whose recurrent state cannot be rolled back. Measured on the real +Godspeed path (requests captured by a logging proxy): one system message per request, `reasoning_content` is +not sent back, and after the first request (about 7.9K tokens of system prompt + tool schemas in the full +harness) each agent step re-processes only 250-570 new tokens. Prefill is about 1.3K tok/s, so a full cache +miss at 24K context would cost about 17 s. + +## Quality screen (small, read the caveats) + +Godspeed's agent-in-loop SWE-bench runner (`experiments/swebench_lite`), 8 SWE-bench Lite **dev** instances, +`reasoning_effort: medium`, 40 iterations / 15 minutes per task, scored afterwards with the official swebench +harness. The 8 tasks were drawn in a seeded random order after excluding dev instances that the gold patch +itself cannot resolve in the published Docker images (8 of 23: all five pvlib ones - NumPy 2 removed +`np.Inf`; pyvista - missing `libGL.so.1`; two pydicom - test-infrastructure error). + +| arm | draws | resolved | +|---|---|---| +| Qwen3.8-27B IQ3_XXS (this profile) | 3 (8 tasks each) | 7/8, 6/8, 6/8 = 19/24 (79%, Wilson 95% CI 60-91%) | +| KAT-Coder REAP-50 Q4_K_M, plain (32K, 32K, 64K context) | 3 (8 tasks each) | 4/8, 4/8, 4/8 = 12/24 (50%, CI 31-69%) | + +* The agent can call the hidden-test harness as a tool (up to 5 times per task), so these scores are **not + comparable to leaderboard numbers**. The draws share the same 8 tasks and are not independent. Per task, + the 27B solved at least as many of its 3 draws as KAT on every task and strictly more on 4 of 8 (the other + 4 tie; task-level sign test, two-sided p = 0.125): suggestive, not established. KAT's two draws with network access (32K + and 64K) solved exactly the same four tasks, so its ceiling here is not the context window. +* Protocol differences between draws: the first 27B draw ran before the filesystem sandbox existed (its + command log was audited: 1 of 165 shell commands touched anything outside the task workspace), and the + first KAT draw ran in a sandbox that accidentally had no DNS. The other draws (27B: two, KAT: two) share + one protocol: sandboxed, with network. +* `sqlfluff-1733` was unsolved by every draw of both arms; two other tasks flip between draws (about one + task of noise per draw). +* Mean wall time per 8-task pass: 26 min (27B, three draws) vs 22 min (KAT, three draws), i.e. about 14.5 + vs 11.0 tasks solved per hour of agent time. KAT's plain decode measured 3.4x the 27B's plain and about + 2.4x its MTP n=4 speed on synthetic prompts (its real-workload speed was not measured separately), but its + trajectories are longer and it solves fewer tasks: at 32K with network access 5 of its 8 tasks ended at the + 40-iteration cap (2 of 8 at 64K). + +## Known issues + +* **Context estimate.** Godspeed's token estimate (tiktoken cl100k) ran 1.29x below the server's true count on + real Qwen3.8 requests (median), mostly because assistant `tool_calls` were not counted; with a hard 32K + window compaction at 0.8 (0.8 x 1.29 is about 1.03 of the window) fired only when the real context was + already at or past the limit (reply truncated mid tool call, llama-server HTTP 500, session lost). Fixed by counting `tool_calls` (median 1.11x after); + until that is in your build, consider `compaction_threshold: 0.7`. +* **Benchmark hygiene.** The agent shell shares the host filesystem: during one run a model searched the host + for hidden tests and gold data. Run benchmark agents in an isolated mount/PID namespace (or container) and + give each task its own throwaway virtualenv; do not let the agent `pip install` into the harness venv. +* `pvlib` and a few other SWE-bench Lite instances cannot pass even with the gold patch in current images: + gold-check an instance set before trusting a score on it. diff --git a/scripts/serve_qwen38_27b_llamacpp.sh b/scripts/serve_qwen38_27b_llamacpp.sh new file mode 100755 index 00000000..704c8714 --- /dev/null +++ b/scripts/serve_qwen38_27b_llamacpp.sh @@ -0,0 +1,25 @@ +#!/usr/bin/env bash +# Start llama-server for Qwen3.8-27B (ISTA GSQ-RCO IQ3_XXS with the embedded MTP head) on a 16 GB GPU. +# The configuration measured in docs/local_qwen38_27b.md (RTX 5070 Ti, WSL2, llama.cpp 3173a56): +# ~78 tok/s on real agent requests (54.6 without speculation), peak VRAM 14,128 MiB of 16,303 MiB. +# +# Needs a llama.cpp build that has MTP (upstream PR #22673). MTP requires exactly one slot (-np 1). +# Override paths with LLAMA_SERVER / MODEL; PORT defaults to 8080 (where Godspeed attaches). +set -euo pipefail + +LLAMA_SERVER="${LLAMA_SERVER:-$HOME/llama.cpp/build/bin/llama-server}" +MODEL="${MODEL:-$HOME/models/Qwen3.8-27B-GSQ-RCO-IQ3_XXS-mtp.gguf}" +PORT="${PORT:-8080}" +CTX="${CTX:-32768}" + +[ -x "$LLAMA_SERVER" ] || { echo "llama-server not found: $LLAMA_SERVER (set LLAMA_SERVER)" >&2; exit 1; } +[ -f "$MODEL" ] || { echo "model not found: $MODEL (set MODEL)" >&2; exit 1; } + +exec "$LLAMA_SERVER" \ + --model "$MODEL" \ + -c "$CTX" -ngl 999 -fa on -np 1 \ + --cache-type-k q8_0 --cache-type-v q8_0 \ + -b 2048 -ub 512 -t 4 \ + --spec-type draft-mtp --spec-draft-n-max 4 \ + --ctx-checkpoints 32 --cache-ram 8192 \ + --jinja --no-webui --host 127.0.0.1 --port "$PORT" diff --git a/scripts/settings_local_llm_qwen38_27b.yaml b/scripts/settings_local_llm_qwen38_27b.yaml index 519a56fe..1bdbdab5 100644 --- a/scripts/settings_local_llm_qwen38_27b.yaml +++ b/scripts/settings_local_llm_qwen38_27b.yaml @@ -1,10 +1,9 @@ # Godspeed settings for local Qwen3.8-27B (ISTA GSQ-RCO IQ3, MTP) via llama.cpp # -# STATUS: UNVALIDATED on this codebase. Nothing below was benchmarked in -# Godspeed; every performance number is either quoted from a source or an -# estimate, and is labelled as such. Run scripts/validate_driver.py against a -# live server before adding a driver_catalog.yaml entry (that catalog requires -# a passing validate_driver run). +# STATUS: measured on ONE machine on 2026-09-24 (RTX 5070 Ti 16 GB, WSL2, llama.cpp +# 3173a56); methods, tables and caveats are in docs/local_qwen38_27b.md. Numbers +# below are what that box did, not promises for other hardware. The catalog still +# requires a passing scripts/validate_driver.py run before an entry is added. # # Model file (Apache-2.0), 16 GB-class sizes, from # https://huggingface.co/ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF @@ -15,9 +14,10 @@ # 61.7, Terminal-Bench 2.1 73.0 (vendor scaffolds; not comparable to Godspeed's). # The "-mtp" files embed the trained multi-token-prediction head (+~0.35 GB). # -# Speed: NOT MEASURED here. Estimate only: bandwidth-bound decode on a 896 GB/s -# card at ~73-84% efficiency (measured on two other cards) gives ~65-75 tok/s -# plain for IQ3_XXS; MTP multipliers reported by others range 1.3x-2.8x. +# Speed (measured, q8_0 KV, 32K context): plain decode 54.6 tok/s; MTP n=4 gives +# ~78 tok/s on real agent requests (1.4x). Synthetic prompts show 101 tok/s and +# overstate it: real output is mostly reasoning prose, which drafts accept +# poorly. Peak VRAM 14,128 MiB of 16,303 MiB (includes the desktop's ~1.3-1.9 GB). # # llama.cpp: needs a build with MTP (upstream PR #22673, merged 2026-05-16). # The pinned b9066 builds predate it and are refused by start_server() with a @@ -36,13 +36,17 @@ strong_model: openai/qwen3.8-27b cheap_model: openai/qwen3.8-27b # Qwen3.8 thinks by default at effort "xhigh" (its chat template default). -# medium is a starting point, UNMEASURED. Change at runtime with -# /effort none|low|medium|high (none = thinking off). +# medium is what every measured run used (docs/local_qwen38_27b.md); other efforts +# were not compared. Change at runtime with /effort none|low|medium|high +# (none = thinking off). reasoning_effort: medium # Context: the working range for a 16 GB card. KV for this model is small # (16 full-attention layers x 4 KV heads x 256 dim ~= 34 KiB/token at q8_0, -# computed from the model card => ~1.1 GB at 32K). Compact at 80%. +# computed from the model card => ~1.1 GB at 32K). Compact at 80%: Godspeed's token +# estimate ran ~1.3x below the server's count on real requests (tool_calls were +# not counted), so on a hard 32K window consider compaction_threshold: 0.7 until +# your build counts them (see docs/local_qwen38_27b.md). max_context_tokens: 32768 compaction_threshold: 0.8 @@ -50,17 +54,17 @@ llamacpp: # server_bin: /path/to/llama-server # or export GODSPEED_LLAMA_SERVER model_path: ~/.llamacpp/models/Qwen3.8-27B-GSQ-RCO-IQ3_XXS-mtp.gguf context: 32768 - # KV on the GPU: the KV cache is small for this hybrid model, so the old - # "KV in system RAM" tuning (made for a 14B dense model) is probably not - # needed. UNMEASURED - verify with nvidia-smi on first run; if VRAM is tight - # (model 10.4 GB + KV + MTP ~1-2 GB + compute buffers + desktop), lower - # context to 24576 or set no_kv_offload: true. + # KV on the GPU: measured peak 14,128 MiB at 32K / q8_0 KV with MTP n=4 (of + # 16,303 MiB). If VRAM is tighter on your machine, lower context to 24576 or + # set kv_cache_type: q4_0 (saves ~1 GiB; quality effect NOT measured). no_kv_offload: false kv_cache_type: q8_0 # Built-in speculation using the MTP head embedded in the GGUF. start_server # also passes --parallel 1 (the speed-up disappears with more slots). spec_type: draft-mtp - spec_draft_n_max: 2 # community-tuned for 16-24 GB cards + # Measured on real agent requests: n=3 -> 76.4 tok/s, n=4 -> 78.2 tok/s. A + # separate DFlash2 drafter ties n=4 on real traffic for ~1 GB more VRAM. + spec_draft_n_max: 4 # reasoning_budget: 4096 # optional server-wide cap on thinking tokens # n_cpu_moe: 0 # MoE models only