t-timms · t-timms · Sep 24, 2026 · Sep 24, 2026 · Sep 24, 2026 · Sep 24, 2026
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -109,6 +109,13 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
   conversation under a new resumable session id; prompt history persists
   in `.godspeed/history` with Ctrl-R reverse search.
 
+- **docs(llamacpp): Qwen3.8-27B profile now carries measurements, not estimates** —
+  `scripts/settings_local_llm_qwen38_27b.yaml` claimed ~65-75 tok/s plain and
+  1.3x-2.8x from MTP; measured on an RTX 5070 Ti it is 54.6 tok/s plain and
+  ~1.4x (78 tok/s) on real agent requests, with `spec_draft_n_max: 4` the best
+  of 2-6. New `docs/local_qwen38_27b.md` (methods, tables, caveats) and
+  `scripts/serve_qwen38_27b_llamacpp.sh` (the measured launch command).
+
 ### Fixed
 
 - **fix(llm): Qwen3.8 was not recognised as thinking-capable** — the model

diff --git a/docs/local_qwen38_27b.md b/docs/local_qwen38_27b.md
@@ -0,0 +1,87 @@
+# Local Qwen3.8-27B on a 16 GB GPU: what was measured
+
+Measured on **one machine** on 2026-09-24: RTX 5070 Ti (16 GB, 16,303 MiB), Ryzen 5 7600, WSL2 Ubuntu,
+llama.cpp `3173a56` (built 2026-08-29), model
+[`ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF`](https://huggingface.co/ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF)
+`Qwen3.8-27B-GSQ-RCO-IQ3_XXS-mtp.gguf` (10.44 GB, Apache-2.0). Treat every number as "what this box did",
+not a promise for other hardware. VRAM figures include the ~1.3-1.9 GB the Windows desktop was using.
+
+## Launch
+
+`scripts/serve_qwen38_27b_llamacpp.sh` starts the exact configuration below. Godspeed on Windows cannot
+launch a Linux binary, so start it in WSL and Godspeed attaches to `127.0.0.1:8080`; then use
+`scripts/settings_local_llm_qwen38_27b.yaml`.
+
+Key flags: `-c 32768 -fa on -np 1 --cache-type-k q8_0 --cache-type-v q8_0 -ub 512`
+`--spec-type draft-mtp --spec-draft-n-max 4 --jinja`. MTP needs `-np 1` (one slot).
+
+## Speed and VRAM (32K context, q8_0 KV)
+
+| config | decode tok/s, real agent requests | decode tok/s, synthetic prompts | peak VRAM (MiB) |
+|---|---|---|---|
+| no speculation | 54.6 | 54.5 | 12,960 |
+| MTP n=3 | 76.4 | 94.7 | 14,203 |
+| **MTP n=4** | **78.2** | 101.0 | 14,128 |
+| DFlash2 drafter, n=7 | 78.8 | 117.1 | 15,124 (`-ub 256`) |
+
+* "Real agent requests" = 69 requests captured from an agent-in-loop run and replayed against each
+  configuration (identical requests, at most 300 new tokens each). The live evaluations measured the same
+  thing: 77.5 tok/s and 0.51 draft acceptance with MTP n=4 over 249 requests.
+* "Synthetic prompts" (code, prose, tool-call and reasoning prompts written for the benchmark) overstate the
+  speed-up: real output is mostly medium-effort reasoning prose, which the drafters accept poorly (DFlash2
+  acceptance 0.36 on real requests vs 0.58 synthetic). Do not choose a speculation config from synthetic
+  numbers alone.
+* DFlash2 needs a separate drafter file and about 1 GB more VRAM for no measurable gain on real traffic, so
+  MTP n=4 is the recommended setting.
+* KV `q4_0` saves about 1 GiB with no speed change; its effect on quality was **not** measured.
+
+## Prefix cache
+
+The model is a Gated-DeltaNet hybrid, whose recurrent state cannot be rolled back. Measured on the real
+Godspeed path (requests captured by a logging proxy): one system message per request, `reasoning_content` is
+not sent back, and after the first request (about 7.9K tokens of system prompt + tool schemas in the full
+harness) each agent step re-processes only 250-570 new tokens. Prefill is about 1.3K tok/s, so a full cache
+miss at 24K context would cost about 17 s.
+
+## Quality screen (small, read the caveats)
+
+Godspeed's agent-in-loop SWE-bench runner (`experiments/swebench_lite`), 8 SWE-bench Lite **dev** instances,
+`reasoning_effort: medium`, 40 iterations / 15 minutes per task, scored afterwards with the official swebench
+harness. The 8 tasks were drawn in a seeded random order after excluding dev instances that the gold patch
+itself cannot resolve in the published Docker images (8 of 23: all five pvlib ones - NumPy 2 removed
+`np.Inf`; pyvista - missing `libGL.so.1`; two pydicom - test-infrastructure error).
+
+| arm | draws | resolved |
+|---|---|---|
+| Qwen3.8-27B IQ3_XXS (this profile) | 3 (8 tasks each) | 7/8, 6/8, 6/8 = 19/24 (79%, Wilson 95% CI 60-91%) |
+| KAT-Coder REAP-50 Q4_K_M, plain (32K, 32K, 64K context) | 3 (8 tasks each) | 4/8, 4/8, 4/8 = 12/24 (50%, CI 31-69%) |
+
+* The agent can call the hidden-test harness as a tool (up to 5 times per task), so these scores are **not
+  comparable to leaderboard numbers**. The draws share the same 8 tasks and are not independent. Per task,
+  the 27B solved at least as many of its 3 draws as KAT on every task and strictly more on 4 of 8 (the other
+  4 tie; task-level sign test, two-sided p = 0.125): suggestive, not established. KAT's two draws with network access (32K
+  and 64K) solved exactly the same four tasks, so its ceiling here is not the context window.
+* Protocol differences between draws: the first 27B draw ran before the filesystem sandbox existed (its
+  command log was audited: 1 of 165 shell commands touched anything outside the task workspace), and the
+  first KAT draw ran in a sandbox that accidentally had no DNS. The other draws (27B: two, KAT: two) share
+  one protocol: sandboxed, with network.
+* `sqlfluff-1733` was unsolved by every draw of both arms; two other tasks flip between draws (about one
+  task of noise per draw).
+* Mean wall time per 8-task pass: 26 min (27B, three draws) vs 22 min (KAT, three draws), i.e. about 14.5
+  vs 11.0 tasks solved per hour of agent time. KAT's plain decode measured 3.4x the 27B's plain and about
+  2.4x its MTP n=4 speed on synthetic prompts (its real-workload speed was not measured separately), but its
+  trajectories are longer and it solves fewer tasks: at 32K with network access 5 of its 8 tasks ended at the
+  40-iteration cap (2 of 8 at 64K).
+
+## Known issues
+
+* **Context estimate.** Godspeed's token estimate (tiktoken cl100k) ran 1.29x below the server's true count on
+  real Qwen3.8 requests (median), mostly because assistant `tool_calls` were not counted; with a hard 32K
+  window compaction at 0.8 (0.8 x 1.29 is about 1.03 of the window) fired only when the real context was
+  already at or past the limit (reply truncated mid tool call, llama-server HTTP 500, session lost). Fixed by counting `tool_calls` (median 1.11x after);
+  until that is in your build, consider `compaction_threshold: 0.7`.
+* **Benchmark hygiene.** The agent shell shares the host filesystem: during one run a model searched the host
+  for hidden tests and gold data. Run benchmark agents in an isolated mount/PID namespace (or container) and
+  give each task its own throwaway virtualenv; do not let the agent `pip install` into the harness venv.
+* `pvlib` and a few other SWE-bench Lite instances cannot pass even with the gold patch in current images:
+  gold-check an instance set before trusting a score on it.
diff --git a/scripts/serve_qwen38_27b_llamacpp.sh b/scripts/serve_qwen38_27b_llamacpp.sh
@@ -0,0 +1,25 @@
+#!/usr/bin/env bash
+# Start llama-server for Qwen3.8-27B (ISTA GSQ-RCO IQ3_XXS with the embedded MTP head) on a 16 GB GPU.
+# The configuration measured in docs/local_qwen38_27b.md (RTX 5070 Ti, WSL2, llama.cpp 3173a56):
+# ~78 tok/s on real agent requests (54.6 without speculation), peak VRAM 14,128 MiB of 16,303 MiB.
+#
+# Needs a llama.cpp build that has MTP (upstream PR #22673). MTP requires exactly one slot (-np 1).
+# Override paths with LLAMA_SERVER / MODEL; PORT defaults to 8080 (where Godspeed attaches).
+set -euo pipefail
+
+LLAMA_SERVER="${LLAMA_SERVER:-$HOME/llama.cpp/build/bin/llama-server}"
+MODEL="${MODEL:-$HOME/models/Qwen3.8-27B-GSQ-RCO-IQ3_XXS-mtp.gguf}"
+PORT="${PORT:-8080}"
+CTX="${CTX:-32768}"
+
+[ -x "$LLAMA_SERVER" ] || { echo "llama-server not found: $LLAMA_SERVER (set LLAMA_SERVER)" >&2; exit 1; }
+[ -f "$MODEL" ] || { echo "model not found: $MODEL (set MODEL)" >&2; exit 1; }
+
+exec "$LLAMA_SERVER" \
+  --model "$MODEL" \
+  -c "$CTX" -ngl 999 -fa on -np 1 \
+  --cache-type-k q8_0 --cache-type-v q8_0 \
+  -b 2048 -ub 512 -t 4 \
+  --spec-type draft-mtp --spec-draft-n-max 4 \
+  --ctx-checkpoints 32 --cache-ram 8192 \
+  --jinja --no-webui --host 127.0.0.1 --port "$PORT"
diff --git a/scripts/settings_local_llm_qwen38_27b.yaml b/scripts/settings_local_llm_qwen38_27b.yaml
@@ -1,10 +1,9 @@
 # Godspeed settings for local Qwen3.8-27B (ISTA GSQ-RCO IQ3, MTP) via llama.cpp
 #
-# STATUS: UNVALIDATED on this codebase. Nothing below was benchmarked in
-# Godspeed; every performance number is either quoted from a source or an
-# estimate, and is labelled as such. Run scripts/validate_driver.py against a
-# live server before adding a driver_catalog.yaml entry (that catalog requires
-# a passing validate_driver run).
+# STATUS: measured on ONE machine on 2026-09-24 (RTX 5070 Ti 16 GB, WSL2, llama.cpp
+# 3173a56); methods, tables and caveats are in docs/local_qwen38_27b.md. Numbers
+# below are what that box did, not promises for other hardware. The catalog still
+# requires a passing scripts/validate_driver.py run before an entry is added.
 #
 # Model file (Apache-2.0), 16 GB-class sizes, from
 #   https://huggingface.co/ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF
@@ -15,9 +14,10 @@
 # 61.7, Terminal-Bench 2.1 73.0 (vendor scaffolds; not comparable to Godspeed's).
 # The "-mtp" files embed the trained multi-token-prediction head (+~0.35 GB).
 #
-# Speed: NOT MEASURED here. Estimate only: bandwidth-bound decode on a 896 GB/s
-# card at ~73-84% efficiency (measured on two other cards) gives ~65-75 tok/s
-# plain for IQ3_XXS; MTP multipliers reported by others range 1.3x-2.8x.
+# Speed (measured, q8_0 KV, 32K context): plain decode 54.6 tok/s; MTP n=4 gives
+# ~78 tok/s on real agent requests (1.4x). Synthetic prompts show 101 tok/s and
+# overstate it: real output is mostly reasoning prose, which drafts accept
+# poorly. Peak VRAM 14,128 MiB of 16,303 MiB (includes the desktop's ~1.3-1.9 GB).
 #
 # llama.cpp: needs a build with MTP (upstream PR #22673, merged 2026-05-16).
 # The pinned b9066 builds predate it and are refused by start_server() with a
@@ -36,31 +36,35 @@ strong_model: openai/qwen3.8-27b
 cheap_model: openai/qwen3.8-27b
 
 # Qwen3.8 thinks by default at effort "xhigh" (its chat template default).
-# medium is a starting point, UNMEASURED. Change at runtime with
-# /effort none|low|medium|high  (none = thinking off).
+# medium is what every measured run used (docs/local_qwen38_27b.md); other efforts
+# were not compared. Change at runtime with /effort none|low|medium|high
+# (none = thinking off).
 reasoning_effort: medium
 
 # Context: the working range for a 16 GB card. KV for this model is small
 # (16 full-attention layers x 4 KV heads x 256 dim ~= 34 KiB/token at q8_0,
-# computed from the model card => ~1.1 GB at 32K). Compact at 80%.
+# computed from the model card => ~1.1 GB at 32K). Compact at 80%: Godspeed's token
+# estimate ran ~1.3x below the server's count on real requests (tool_calls were
+# not counted), so on a hard 32K window consider compaction_threshold: 0.7 until
+# your build counts them (see docs/local_qwen38_27b.md).
 max_context_tokens: 32768
 compaction_threshold: 0.8
 
 llamacpp:
   # server_bin: /path/to/llama-server   # or export GODSPEED_LLAMA_SERVER
   model_path: ~/.llamacpp/models/Qwen3.8-27B-GSQ-RCO-IQ3_XXS-mtp.gguf
   context: 32768
-  # KV on the GPU: the KV cache is small for this hybrid model, so the old
-  # "KV in system RAM" tuning (made for a 14B dense model) is probably not
-  # needed. UNMEASURED - verify with nvidia-smi on first run; if VRAM is tight
-  # (model 10.4 GB + KV + MTP ~1-2 GB + compute buffers + desktop), lower
-  # context to 24576 or set no_kv_offload: true.
+  # KV on the GPU: measured peak 14,128 MiB at 32K / q8_0 KV with MTP n=4 (of
+  # 16,303 MiB). If VRAM is tighter on your machine, lower context to 24576 or
+  # set kv_cache_type: q4_0 (saves ~1 GiB; quality effect NOT measured).
   no_kv_offload: false
   kv_cache_type: q8_0
   # Built-in speculation using the MTP head embedded in the GGUF. start_server
   # also passes --parallel 1 (the speed-up disappears with more slots).
   spec_type: draft-mtp
-  spec_draft_n_max: 2   # community-tuned for 16-24 GB cards
+  # Measured on real agent requests: n=3 -> 76.4 tok/s, n=4 -> 78.2 tok/s. A
+  # separate DFlash2 drafter ties n=4 on real traffic for ~1 GB more VRAM.
+  spec_draft_n_max: 4
   # reasoning_budget: 4096   # optional server-wide cap on thinking tokens
   # n_cpu_moe: 0             # MoE models only