Bump llama.cpp to 4801e3c56 (b10362), release v0.8.43 - #84
Merged
Conversation
Submodule 61881b1f7 -> 4801e3c56 (82 commits, tag b10280 -> b10362). Pinned to the release tag rather than origin/master, which was 14 commits past it. Unlike the v0.8.42 range this one does not break the upstream C API and needed no NIF change. One additive change was a near miss worth recording: - `llama_context_params` gained `n_outputs_max_per_seq`, defaulting to 1 (not 0) in `llama_context_default_params()`, and `llama_decode` now enforces it. That is the same shape as the `load_mtp` field that broke MTP in v0.8.41 --- a restrictive default inherited silently from the defaults struct --- and this binding does request logits at every position of a sequence during MTP prefill, exactly what the limit forbids. It is inert here because the check is gated on `!sampling.samplers.empty()` (src/llama-context.cpp:1664), i.e. on backend samplers registered through `llama_context_params.samplers`, which this NIF never sets: it samples host-side via `llama_sampler_chain`. Because reading the diff is how v0.8.41 got this wrong, this was also confirmed by running MTP end-to-end against a real MTP GGUF. If backend sampling is ever adopted, the field must be set from `common_speculative_get_output_limits`. - `llama_sampler_i` gained `backend_reset`/`copy_state` and `backend_init` gained a parameter; only custom sampler vtables are affected, and there are none here. - Grammar: a repetition bound >= 2000 now degrades to unbounded instead of raising (#26613), reachable from Schema/Grammar via a large `maxItems`. Fixed: max_tokens was not an upper bound under MTP. The verify loop emits up to `1 + n_draft` tokens per iteration but checked the caller's budget only on iteration entry, so the final iteration could run past it. `max_tokens: 16` returned 16, 17 or 18 tokens for the same prompt under greedy decoding, measured across six successive generate/3 calls on one session. The overshoot is not constant: it tracks how many drafts the target accepts in the last iteration, and acceptance varies between runs on a reused session (11, 10, 12, 11, 10, 12 of 15 drafted). The token sequence was deterministic throughout; only the stopping point moved, which is why it surfaced as stream/3 and generate/3 returning different-length prefixes of the same continuation --- misleading, since generate/3 is stream_events/3 joined. The loop now re-checks the budget per token; breaking mid-iteration leaves positions decoded but not emitted, which the existing partial-accept rollback already discards. Pre-existing, not from this bump: the loop last changed in #79 (v0.8.39). It survived because the :mtp suite had only ever run against one model, where the boundary happened to land consistently. Tests: the :mtp and :embeddings suites now run against real models per release. `unsloth/Qwen3.5-0.8B-MTP-GGUF` (~0.8 GB) makes MTP cheap to exercise --- the previously documented MTP GGUFs start at ~21 GB, which is why that suite went unrun and the bound above stayed broken. max_tokens is now pinned as an exact bound at 1, 4 and 16 instead of being covered only by a stream-vs-generate comparison that could not tell a streaming bug from a budget bug. Verified on an M1 Max, source builds with both LLAMA_BACKEND=metal and LLAMA_BACKEND=cpu, each running generation, embedding and MTP suites against real GGUFs: 520 passed, 11 excluded on both (:slow and the known-broken :mtp_cancel). `mix credo --strict`, `mix dialyzer` and `mix format --check-formatted` are clean. A source build from the Hex tarball clones 4801e3c56 and links, confirming the Makefile pin agrees with the submodule.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Submodule
61881b1f7→4801e3c56(82 commits, tag b10280 → b10362). Pinned to therelease tag rather than
origin/master, which was 14 commits past it.Unlike the v0.8.42 range this one does not break the upstream C API — every
change in it is additive and no NIF source change was required for the bump itself.
Upstream API review
One additive change is a near miss worth recording rather than rediscovering.
llama_context_paramsgainedn_outputs_max_per_seq, defaulting to1(not0)in
llama_context_default_params(), andllama_decodenow enforces it, returning-1withbackend sampling supports at most %u outputs per sequence.That is exactly the shape of the
load_mtpfield that broke MTP in v0.8.41 — arestrictive default inherited silently from the defaults struct — and this binding
does request logits at every position of a sequence during MTP prefill, which is
the pattern the limit forbids.
It is nonetheless inert here: the check is gated on
!sampling.samplers.empty()(
src/llama-context.cpp:1664), i.e. on backend samplers registered throughllama_context_params.samplers. This binding never sets that field — it sampleshost-side via
llama_sampler_chain/llama_sampler_sample— so the map is emptyand the limit never applies. Because reading the diff is how v0.8.41 got this wrong,
this was also confirmed by running MTP end-to-end against a real MTP GGUF.
Also in the range:
llama_sampler_igainedbackend_reset/copy_state, andbackend_initgainedan
n_outputs_max_per_seqparameter. Only affects custom sampler vtables; thereare none here.
>= 2000now degrades to unboundedinstead of raising (#26613). Reachable from
LlamaCppEx.Schema/Grammar— aschema whose
maxItemsexceeded the threshold used to fail compilation and nowcompiles.
minItemsover the threshold still raises.aarch64-apple-darwinartifact:
NORM/RMS_NORMfor partial simdgroups (#26708),kernel_lightning_indexer(#26646), contiguous-src
ROLL(#25928).to the
LLAMA_PORTABLE=1artifacts.Makefilepasses was renamed orremoved, so the build configuration is unchanged.
Fixed:
max_tokenswas not an upper bound under MTPThe verify loop emits up to
1 + n_drafttokens per iteration but checked thecaller's budget only on iteration entry, so the final iteration could run past it.
max_tokens: 16, same prompt,temp: 0.0,seed: 7, six successivegenerate/3calls on one session:
" 1, 2, 3, 4, 5.\nCount"" 1, 2, 3, 4, 5.\n"" 1, 2, 3, 4, 5.\nCount to"The overshoot is not constant because it tracks how many drafts the target accepts
in the last iteration, and acceptance varies between runs on a reused session
(11, 10, 12, 11, 10, 12 accepted of 15 drafted — unchanged by this fix, it just no
longer leaks into the token budget). The token sequence was deterministic
throughout; only the stopping point moved.
That is what made it look like something it wasn't:
stream/3andgenerate/3returned different-length prefixes of the same continuation, which reads as a
streaming bug — but
generate/3isstream_events/3joined, so it was always abudget bug.
The loop now re-checks the budget per token. Breaking mid-iteration leaves positions
decoded but not emitted, which the existing partial-accept rollback already discards,
so nothing else moved.
Pre-existing, not from this bump — the loop last changed in #79 (v0.8.39). It
survived because the
:mtpsuite had only ever run against one model, where theboundary happened to land consistently.
Tests
:mtpand:embeddingsnow run against real models per release.unsloth/Qwen3.5-0.8B-MTP-GGUF(~0.8 GB) makes MTP cheap enough to exercise everytime — the previously documented MTP GGUFs start at ~21 GB, which is precisely why
that suite went unrun and the bound above stayed broken. Both small models are now
listed in the README.
max_tokensis pinned as an exact bound at 1, 4 and 16, instead of being coveredonly by a
stream/3-vs-generate/3comparison that could not distinguish astreaming bug from a budget bug.
Verification
M1 Max, source builds with both backends, each running generation, embedding and
MTP suites against real GGUFs:
LLAMA_BACKEND=metal+ smoke/embeddings/mtpLLAMA_BACKEND=cpu+ smoke/embeddings/mtpmix credo --strict/mix dialyzer/mix format --check-formatted4801e3c56, links.soExclusions are
:slowand the known-broken:mtp_cancel. The Hex source buildconfirms
Makefile'sLLAMA_COMMITagrees with the submodule — the drift this repohas been bitten by before.
CUDA was verified separately on 2× NVIDIA DGX Spark (GB10,
sm_121a, aarch64,CUDA 13.0.2) at the previous base (b10280): loads, reports
backend: "CUDA", offloads31/31 layers, 528 tests / 0 failures.
Notes
checksum.exsis still at 0.8.42 by design; thechecksumCI job regenerates andcommits it during the tagged release run.
v0.8.43is pushed frommasterafter this merges.