Skip to content

Bump llama.cpp to 4801e3c56 (b10362), release v0.8.43 - #84

Merged
nyo16 merged 1 commit into
masterfrom
bump-llama-cpp-b10362
Aug 12, 2026
Merged

Bump llama.cpp to 4801e3c56 (b10362), release v0.8.43#84
nyo16 merged 1 commit into
masterfrom
bump-llama-cpp-b10362

Conversation

@nyo16

@nyo16 nyo16 commented Aug 12, 2026

Copy link
Copy Markdown
Owner

Submodule 61881b1f74801e3c56 (82 commits, tag b10280 → b10362). Pinned to the
release tag rather than origin/master, which was 14 commits past it.

Unlike the v0.8.42 range this one does not break the upstream C API — every
change in it is additive and no NIF source change was required for the bump itself.

Upstream API review

One additive change is a near miss worth recording rather than rediscovering.

llama_context_params gained n_outputs_max_per_seq, defaulting to 1 (not 0)
in llama_context_default_params(), and llama_decode now enforces it, returning
-1 with backend sampling supports at most %u outputs per sequence.

That is exactly the shape of the load_mtp field that broke MTP in v0.8.41 — a
restrictive default inherited silently from the defaults struct — and this binding
does request logits at every position of a sequence during MTP prefill, which is
the pattern the limit forbids.

It is nonetheless inert here: the check is gated on !sampling.samplers.empty()
(src/llama-context.cpp:1664), i.e. on backend samplers registered through
llama_context_params.samplers. This binding never sets that field — it samples
host-side via llama_sampler_chain / llama_sampler_sample — so the map is empty
and the limit never applies. Because reading the diff is how v0.8.41 got this wrong,
this was also confirmed by running MTP end-to-end against a real MTP GGUF.

If backend sampling is ever adopted here, n_outputs_max_per_seq becomes
load-bearing and must be set from common_speculative_get_output_limits (new in
this range), as upstream's server and speculative-simple now do.

Also in the range:

  • llama_sampler_i gained backend_reset / copy_state, and backend_init gained
    an n_outputs_max_per_seq parameter. Only affects custom sampler vtables; there
    are none here.
  • Grammar semantics: a repetition bound >= 2000 now degrades to unbounded
    instead of raising (#26613). Reachable from LlamaCppEx.Schema / Grammar — a
    schema whose maxItems exceeded the threshold used to fail compilation and now
    compiles. minItems over the threshold still raises.
  • Metal correctness fixes on the backend that ships in the aarch64-apple-darwin
    artifact: NORM/RMS_NORM for partial simdgroups (#26708), kernel_lightning_indexer
    (#26646), contiguous-src ROLL (#25928).
  • CPU/aarch64: HWCAP fallbacks and fp16 variant detection (#25554), which matters
    to the LLAMA_PORTABLE=1 artifacts.
  • MTP: upstream memory-allocation fix for MTP layers (#26605), Nemotron MTP (#26725).
  • ggml 0.18.1 → 0.19.0. No cmake option this Makefile passes was renamed or
    removed, so the build configuration is unchanged.

Fixed: max_tokens was not an upper bound under MTP

The verify loop emits up to 1 + n_draft tokens per iteration but checked the
caller's budget only on iteration entry, so the final iteration could run past it.

max_tokens: 16, same prompt, temp: 0.0, seed: 7, six successive generate/3
calls on one session:

Run Before After
1 17 tok — " 1, 2, 3, 4, 5.\nCount" 16 tok — " 1, 2, 3, 4, 5.\n"
2 16 tok 16 tok
3 18 tok — " 1, 2, 3, 4, 5.\nCount to" 16 tok
4 17 tok 16 tok
5 16 tok 16 tok
6 18 tok 16 tok

The overshoot is not constant because it tracks how many drafts the target accepts
in the last iteration, and acceptance varies between runs on a reused session
(11, 10, 12, 11, 10, 12 accepted of 15 drafted — unchanged by this fix, it just no
longer leaks into the token budget). The token sequence was deterministic
throughout; only the stopping point moved.

That is what made it look like something it wasn't: stream/3 and generate/3
returned different-length prefixes of the same continuation, which reads as a
streaming bug — but generate/3 is stream_events/3 joined, so it was always a
budget bug.

The loop now re-checks the budget per token. Breaking mid-iteration leaves positions
decoded but not emitted, which the existing partial-accept rollback already discards,
so nothing else moved.

Pre-existing, not from this bump — the loop last changed in #79 (v0.8.39). It
survived because the :mtp suite had only ever run against one model, where the
boundary happened to land consistently.

Tests

  • :mtp and :embeddings now run against real models per release.
    unsloth/Qwen3.5-0.8B-MTP-GGUF (~0.8 GB) makes MTP cheap enough to exercise every
    time — the previously documented MTP GGUFs start at ~21 GB, which is precisely why
    that suite went unrun and the bound above stayed broken. Both small models are now
    listed in the README.
  • max_tokens is pinned as an exact bound at 1, 4 and 16, instead of being covered
    only by a stream/3-vs-generate/3 comparison that could not distinguish a
    streaming bug from a budget bug.

Verification

M1 Max, source builds with both backends, each running generation, embedding and
MTP suites against real GGUFs:

Gate Result
LLAMA_BACKEND=metal + smoke/embeddings/mtp 520 passed, 11 excluded
LLAMA_BACKEND=cpu + smoke/embeddings/mtp 520 passed, 11 excluded
mix credo --strict / mix dialyzer / mix format --check-formatted clean
Hex tarball source build clones 4801e3c56, links .so

Exclusions are :slow and the known-broken :mtp_cancel. The Hex source build
confirms Makefile's LLAMA_COMMIT agrees with the submodule — the drift this repo
has been bitten by before.

CUDA was verified separately on 2× NVIDIA DGX Spark (GB10, sm_121a, aarch64,
CUDA 13.0.2) at the previous base (b10280): loads, reports backend: "CUDA", offloads
31/31 layers, 528 tests / 0 failures.

Notes

  • checksum.exs is still at 0.8.42 by design; the checksum CI job regenerates and
    commits it during the tagged release run.
  • No tag pushed — v0.8.43 reaches Hex only when v0.8.43 is pushed from master
    after this merges.

Submodule 61881b1f7 -> 4801e3c56 (82 commits, tag b10280 -> b10362). Pinned to the
release tag rather than origin/master, which was 14 commits past it. Unlike the
v0.8.42 range this one does not break the upstream C API and needed no NIF change.

One additive change was a near miss worth recording:

- `llama_context_params` gained `n_outputs_max_per_seq`, defaulting to 1 (not 0)
  in `llama_context_default_params()`, and `llama_decode` now enforces it. That
  is the same shape as the `load_mtp` field that broke MTP in v0.8.41 --- a
  restrictive default inherited silently from the defaults struct --- and this
  binding does request logits at every position of a sequence during MTP
  prefill, exactly what the limit forbids. It is inert here because the check is
  gated on `!sampling.samplers.empty()` (src/llama-context.cpp:1664), i.e. on
  backend samplers registered through `llama_context_params.samplers`, which
  this NIF never sets: it samples host-side via `llama_sampler_chain`. Because
  reading the diff is how v0.8.41 got this wrong, this was also confirmed by
  running MTP end-to-end against a real MTP GGUF. If backend sampling is ever
  adopted, the field must be set from `common_speculative_get_output_limits`.
- `llama_sampler_i` gained `backend_reset`/`copy_state` and `backend_init` gained
  a parameter; only custom sampler vtables are affected, and there are none here.
- Grammar: a repetition bound >= 2000 now degrades to unbounded instead of
  raising (#26613), reachable from Schema/Grammar via a large `maxItems`.

Fixed: max_tokens was not an upper bound under MTP.

The verify loop emits up to `1 + n_draft` tokens per iteration but checked the
caller's budget only on iteration entry, so the final iteration could run past
it. `max_tokens: 16` returned 16, 17 or 18 tokens for the same prompt under
greedy decoding, measured across six successive generate/3 calls on one session.
The overshoot is not constant: it tracks how many drafts the target accepts in
the last iteration, and acceptance varies between runs on a reused session
(11, 10, 12, 11, 10, 12 of 15 drafted). The token sequence was deterministic
throughout; only the stopping point moved, which is why it surfaced as stream/3
and generate/3 returning different-length prefixes of the same continuation ---
misleading, since generate/3 is stream_events/3 joined. The loop now re-checks
the budget per token; breaking mid-iteration leaves positions decoded but not
emitted, which the existing partial-accept rollback already discards.

Pre-existing, not from this bump: the loop last changed in #79 (v0.8.39). It
survived because the :mtp suite had only ever run against one model, where the
boundary happened to land consistently.

Tests: the :mtp and :embeddings suites now run against real models per release.
`unsloth/Qwen3.5-0.8B-MTP-GGUF` (~0.8 GB) makes MTP cheap to exercise --- the
previously documented MTP GGUFs start at ~21 GB, which is why that suite went
unrun and the bound above stayed broken. max_tokens is now pinned as an exact
bound at 1, 4 and 16 instead of being covered only by a stream-vs-generate
comparison that could not tell a streaming bug from a budget bug.

Verified on an M1 Max, source builds with both LLAMA_BACKEND=metal and
LLAMA_BACKEND=cpu, each running generation, embedding and MTP suites against real
GGUFs: 520 passed, 11 excluded on both (:slow and the known-broken :mtp_cancel).
`mix credo --strict`, `mix dialyzer` and `mix format --check-formatted` are clean.
A source build from the Hex tarball clones 4801e3c56 and links, confirming the
Makefile pin agrees with the submodule.
@nyo16
nyo16 merged commit 0ef3fce into master Aug 12, 2026
9 checks passed
@nyo16
nyo16 deleted the bump-llama-cpp-b10362 branch August 12, 2026 03:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant