Carry ggml-org#25731 (TML Inkling) merged onto b10665 by danielhanchen · Pull Request #138 · unslothai/llama.cpp

danielhanchen · 2026-08-28T06:46:46Z

Replaces #108. The pin has to move because ggml-org#27742 (Qwen3.8-Flash-Next) was squash-merged upstream as 6c84c7d5 and is in every base tag from b10660 on, which is what broke the nightly in run 33141605830.

#108 is ggml-org#25731 merged onto b10630. Against a base that carries the qwen4exp squash it no longer merges: upstream declares filter_idx / needs_mem_idx immediately above if (arch == LLM_ARCH_FALCON_H1) { in src/llama-model.cpp, and Inkling edits that same if line. additive_merge.py refuses, correctly, because the merge base is not empty.

The old qwen4exp carry #132 used to hide this. Its carry-only commit f52a4c39 moved the filter_idx declaration off the Inkling anchor. #132 is being dropped in the same change, so the accommodation goes with it and Inkling has to face upstream's arrangement directly.

This branch is ggml-org#25731's head 4f2eb74b with b10665 merged into it and the one conflict resolved as the union:

llama_memory_hybrid::layer_filter_cb filter_idx  = nullptr;
const bool needs_mem_idx = (arch == LLM_ARCH_QWEN4EXP);
if (arch == LLM_ARCH_FALCON_H1 || arch == LLM_ARCH_INKLING) {
    // all layers have both an attention KV cache and a recurrent (conv) state

needs_mem_idx is true only for LLM_ARCH_QWEN4EXP, so Inkling still lands on the plain llama_memory_hybrid branch and filter_idx stays null there. Both architectures keep the behaviour their own side intended.

Base branch is inkling-25731-b10665-upstream-base, which is tag b10665 (ca3d5a3e) unmodified, so the diff here is only the Inkling work.

The three old carry commits are gone, on purpose

#108 carried three fixes on top of the upstream PR. All three are now in the upstream branch itself, so this carry is a plain merge with nothing extra:

60007bad, the RESIZE_ALGO_BICUBIC_PILLOW to RESIZE_ALGO_BICUBIC conversion after ggml-org#27594 deleted the enum
3223f7b2, the out-of-bounds memcpy(result->op_params + 16, ...) in ggml_flash_attn_ext_banded that wrote onto ggml_tensor::flags
27278df7, the handle_ssm_conv spacing in ggml/src/ggml-backend-meta.cpp

Verified by grep against this tree: no RESIZE_ALGO_BICUBIC_PILLOW, no op_params + 16 write, correct spacing.

Verification

Built with -DGGML_RPC=ON -DLLAMA_BUILD_EXAMPLES=ON -DLLAMA_BUILD_TOOLS=ON -DLLAMA_BUILD_SERVER=ON -DLLAMA_BUILD_TESTS=ON, which the nightly legs do not do (they set LLAMA_BUILD_TESTS=OFF):

120 targets, exit 0, zero warnings
test-llama-archs -s 1 exits 0, zero FAIL rows
qwen4exp OK (0.00e+00), deepseek4 OK (0.00e+00), so the resolution leaves the qwen4exp path intact
inkling SKIP, matching the upstream PR's own skip entry

Upstream

The same resolution is open as a PR into danielhanchen:add-inkling (danielhanchen#3). Once that merges, ggml-org#25731 goes back to mergeable and this carry can be dropped in favour of pinning the upstream commit directly.

Hybrid attention model: 55 sliding-window plus 11 global layers, banded content-dependent relative position bias instead of RoPE, per-layer short convolution state, fine-grained MoE (256 experts top-6 plus 2 shared), attention log-scaling past 128K, 1M context. Includes the GGML_OP_FLASH_ATTN_EXT_BANDED operator (CPU and CUDA, fused into the MMA flash attention kernel with an fp16 accumulator overflow guard), HF to GGUF conversion, chat template with typed content block parsing (interleaved thinking, narration and tool calls), mmproj vision and audio support, and backend op tests at production shapes.

# Conflicts: # ggml/src/ggml-cuda/mmq.cuh # src/llama-model-saver.cpp # src/llama-vocab.h

Upstream 910196f renamed common_chat_params::thinking_end_tag to thinking_end_tags and changed it from std::string to a vector, so data.thinking_end_tag = END_MESSAGE no longer compiles.

# Conflicts: # src/llama-arch.cpp

# Conflicts: # ggml/include/ggml-rpc.h # ggml/src/ggml-backend-meta.cpp # ggml/src/ggml-cuda/ggml-cuda.cu # src/llama-model-saver.cpp # tests/test-llama-archs.cpp

# Conflicts: # src/llama-model.cpp

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 2e9d6fb214

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-08-28T07:02:33Z

+add_executable(test-flash-attn-bias test-flash-attn-bias.cpp)
+target_link_libraries(test-flash-attn-bias PRIVATE ggml)
+add_test(NAME test-flash-attn-bias COMMAND test-flash-attn-bias)


Obtain approval before adding standalone test files

Unless prior maintainer approval exists, adding test-flash-attn-bias.cpp here, along with the adjacent generic-hash test, violates the repository's explicit restriction on new files under tests/* and forces two additional binaries into every LLAMA_BUILD_TESTS build. Reuse the existing backend test infrastructure or obtain maintainer approval before registering these targets.

AGENTS.md reference: AGENTS.md:L87-L87

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-08-28T07:02:34Z

+    test_cases.emplace_back(new test_flash_attn_ext_banded(128, 8, 1, 512,  8192, 1024, 1, GGML_TYPE_F16, GGML_TYPE_F32));
+    test_cases.emplace_back(new test_flash_attn_ext_banded(128, 8, 1, 512, 16384, 1024, 1, GGML_TYPE_F16, GGML_TYPE_F32));
+    test_cases.emplace_back(new test_flash_attn_ext_banded(128, 8, 1, 512, 16403, 1024, 1, GGML_TYPE_F16, GGML_TYPE_F32));
+    test_cases.emplace_back(new test_flash_attn_ext_banded(128, 8, 1, 512, 16896, 1024, 1, GGML_TYPE_F16, GGML_TYPE_F32));
+    test_cases.emplace_back(new test_flash_attn_ext_banded(128, 8, 1, 512, 17024, 1024, 1, GGML_TYPE_F16, GGML_TYPE_F32));


Move production-scale attention cases out of default CTest

The test-backend-ops CTest registration invokes this executable without filters, and its --help confirms that the default mode compares results with the CPU backend. These repeated 512-query x 8K-17K cases, plus the 32K case below, therefore execute both the tested and reference attention paths in every default run; even the first 8K case did not finish within 15 seconds on the inspected CPU. Move these production-scale stress cases behind an opt-in suite or filter and retain small default correctness cases to avoid adding minutes or timeouts to CI.

AGENTS.md reference: AGENTS.md:L87-L87

Useful? React with 👍 / 👎.

danielhanchen · 2026-08-28T07:51:18Z

Retired by #140. danielhanchen#3 landed this same resolution on add-inkling as edee0e1, so ggml-org#25731 is mergeable again and the pin now points at it directly. The swap is a provable no-op: identical trees (2feb59ed) and an identical final merged tree (eb067b53) for the whole pin chain. Branch inkling-25731-b10665 and its base branch stay in place.

danielhanchen and others added 18 commits July 18, 2026 08:09

Merge remote-tracking branch 'origin/master' into pr-25731-head

453c438

# Conflicts: # ggml/src/ggml-cuda/mmq.cuh # src/llama-model-saver.cpp # src/llama-vocab.h

Merge remote-tracking branch 'upstream/master' into add-inkling-fix

e38da9a

inkling : adapt to the renamed thinking_end_tags chat API

d8f8750

Upstream 910196f renamed common_chat_params::thinking_end_tag to thinking_end_tags and changed it from std::string to a vector, so data.thinking_end_tag = END_MESSAGE no longer compiles.

Fix merge conflicts

79ca732

Merge upstream master and fix conflicts

8a67b19

Fix mmproj decoder width for Inkling Small

d69b7e6

Fix some GitHub Issues

02142bb

Merge tag 'b10229' into add-inkling

1e6f9e4

# Conflicts: # src/llama-arch.cpp

Fix quantized V cache issue

68b15cf

Fix merge conflicts

3fd7901

Fix merge conflicts

e2404dd

Fix merge conflicts

0a9841f

Fix merge conflicts

368b24c

Fix merge conflicts

c44c9a1

Fix merge conflicts

b5baf81

# Conflicts: # ggml/include/ggml-rpc.h # ggml/src/ggml-backend-meta.cpp # ggml/src/ggml-cuda/ggml-cuda.cu # src/llama-model-saver.cpp # tests/test-llama-archs.cpp

Fix merge conflicts

4f2eb74

Merge remote-tracking branch 'up/master' into add-inkling

2e9d6fb

# Conflicts: # src/llama-model.cpp

danielhanchen requested a review from CISC as a code owner August 28, 2026 06:46

chatgpt-codex-connector Bot reviewed Aug 28, 2026

View reviewed changes

danielhanchen mentioned this pull request Aug 28, 2026

Carry ggml-org#25731 (TML Inkling) merged onto b10630 #108

Closed

danielhanchen closed this Aug 28, 2026

Navigation Menu

Uh oh!

Carry ggml-org#25731 (TML Inkling) merged onto b10665 - #138

Carry ggml-org#25731 (TML Inkling) merged onto b10665#138
danielhanchen wants to merge 18 commits into
inkling-25731-b10665-upstream-basefrom
inkling-25731-b10665

danielhanchen commented Aug 28, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

chatgpt-codex-connector Bot Aug 28, 2026

Uh oh!

chatgpt-codex-connector Bot Aug 28, 2026

Uh oh!

danielhanchen commented Aug 28, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

danielhanchen commented Aug 28, 2026

The three old carry commits are gone, on purpose

Verification

Upstream

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Aug 28, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Aug 28, 2026

Choose a reason for hiding this comment

Uh oh!

danielhanchen commented Aug 28, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants