Add llamacpp launch settings (MTP, MoE offload) and Qwen3.5+ thinking control - #228
Merged
Merged
Conversation
… control llamacpp_manager could only launch the 14B + draft-model setup it was tuned for, and the client did not recognise Qwen3.8 as a thinking model. llama.cpp launch - New `llamacpp:` settings section (LlamaCppSettings) wired through _ensure_llamacpp from both the CLI and the Textual app: server_bin, model_path, context, no_kv_offload, kv_cache_type, spec_type, spec_draft_n_max, n_cpu_moe, reasoning_budget, extra_args. - spec_type=draft-mtp uses the MTP head embedded in Qwen3.5+ GGUFs instead of a separate draft model (skips draft auto-detect, greedy sampling and the 24576 context cap, and passes --parallel 1). - $GODSPEED_LLAMA_SERVER overrides binary detection. - start_server probes `llama-server --help` and refuses to launch with a flag the binary does not know. The pinned b9066 builds predate MTP (upstream PR #22673), so this fails early with a clear message instead of after launch. - Defaults reproduce the previous command line exactly. Thinking control - The model match covered only qwen3.6 and qwen3-, so qwen3.8-* got no thinking control while its chat template thinks by default at effort xhigh. - qwen3.5..qwen3.9 now get chat_template_kwargs (enable_thinking, reasoning_effort) from reasoning_effort or thinking_budget. The Qwen3.8 template raises on any effort outside xhigh|medium|low, so raw values are normalised and never forwarded to these models. - /effort accepts `none` to turn thinking off. - The duplicated thinking blocks in _call and the streaming path are now one helper. Also: example profile scripts/settings_local_llm_qwen38_27b.yaml, docs and CHANGELOG entries, and a caveat on the unverified qwen3.6-27b speed claim in driver_catalog.yaml.
🤖 Godspeed Review
SecurityNo secrets detected in changed files. |
This was referenced Sep 24, 2026
t-timms
added a commit
that referenced
this pull request
Sep 24, 2026
…easoning tier (#229) Ternary-Bonsai-2-27B's KNOWN_ISSUES page documents chat-template failures that also apply to any Qwen3.8 build (the template is shared). Checking Godspeed against them found three exposures. - Resumed sessions appended a mid-conversation `system` message (Conversation.add_system_message, called from the TUI resume bootstrap). Qwen3.5+ templates raise "System message must be at the beginning" -> HTTP 500. It now merges into the single leading system prompt; once messages exist it adds a labelled `user` note instead. - Empty / None tool-call arguments were stored as-is in history, and the templates reject them (HTTP 500/400). add_assistant_message now stores "{}". - _parse_tool_call called json.loads("") on an empty argument string and dropped a valid zero-argument call as malformed. Empty/None now parse as no arguments. Malformed JSON is still rejected. - The derived reasoning-effort tiers picked "low" for thinking_budget <= 2048 and for the "minimal" alias, but Prism ML measured "low" using about as many reasoning tokens as the default xhigh on Qwen3.8. Derived tiers now pick "medium" (budget <= 8192) or "xhigh"; an explicit `reasoning_effort: low` is still honoured. Follows up #228. Tests: tests/test_qwen38_template_safety.py (new), and the two resume tests that asserted the old two-system-message behavior now pin the fixed behavior.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Makes Godspeed able to launch and drive Qwen3.8-27B-class models on a 16 GB card, based on a desk review of what the harness could not do.
llama.cpp launch (
llamacpp_manager,config,cli,tui/textual_app)llamacpp:settings section:server_bin,model_path,context,no_kv_offload,kv_cache_type,spec_type,spec_draft_n_max,n_cpu_moe,reasoning_budget,extra_args.spec_type: draft-mtpuses the multi-token-prediction head embedded in Qwen3.5+ MTP GGUFs (no separate draft model; also--parallel 1).$GODSPEED_LLAMA_SERVERoverrides binary detection.start_server()checks the binary's--helpand refuses flags it does not know. The pinnedb9066builds predate MTP (upstream PR #22673, 2026-05-16), so this now fails early with a clear message.Thinking control (
llm/client.py,/effort)_THINKING_CAPABLE_PREFIXESmatched onlyqwen3.6/qwen3-, soqwen3.8-*got no thinking control. Its chat template thinks by default atxhigh.qwen3.5..qwen3.9now sendchat_template_kwargs(enable_thinking,reasoning_effort), the per-request mechanism llama-server documents. Rawreasoning_effortis no longer forwarded to these models because the Qwen3.8 template raises on anything outsidexhigh|medium|low./effort nonedisables thinking.Behavior changes / compatibility
llamacpp:section the launch command line is identical (covered by a test), and with noreasoning_effort/thinking_budgetset nothing new is sent.thinking_budget > 0now also sendschat_template_kwargsin addition to the previousextra_bodykeys (kept for compatibility)./effortacceptsnonefor every model; for non-Qwen3.5+ models it is forwarded like the other values.Verification
ruff check .,ruff format --check .,mypy src/(CI flags),bandit(CI flags): clean.test_llamacpp_manager,test_thinking,test_config_llamacpp,test_llm_client,test_cli_more): 338 passed, 1 skipped (the skip is the pre-existing model-fileskipif).--parallel 1, and treatingreasoning_budget=0as falsy each fail a specific new test.Not verified (needs a GPU / live server)
--n-cpu-moe,--reasoning-budgetand--spec-type draft-mtpwere confirmed present in a local WSL build's--help, not run.scripts/settings_local_llm_qwen38_27b.yamlis labelled UNVALIDATED; its speed figures are sourced or estimated and say so.no_kv_offload: falsefor the hybrid model is an unmeasured choice.driver_catalog.yamlentries added: the catalog requires a passingscripts/validate_driver.pyrun, which needs the model served. Only added a caveat to the existing unverifiedqwen3.6-27bspeed claim.wsl.exeis a possible follow-up.Why these numbers/choices
chat_template.jinja.chat_template_kwargsandreasoning_effort; a per-request thinking-token budget is not documented, hence the server-widereasoning_budget.