Skip to content

qwen4exp: keep the QSA indexer cache in lockstep with the attention cache #135

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
vladimir-voinea wants to merge 1 commit into
base: qwen4exp/qwen3.8-flash-next
Choose a base branch
from
Open

qwen4exp: keep the QSA indexer cache in lockstep with the attention cache #135

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 8 additions & 1 deletion src/llama-memory-hybrid-idx.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,10 +52,17 @@ llama_memory_hybrid_idx::llama_memory_hybrid_idx(

LLAMA_LOG_INFO("%s: creating indexer KV cache, size = %u cells\n", __func__, kv_size);

// the indexer cache is a side buffer addressed cell-for-cell by the attention
// cache: it must track the attention cache's cells exactly or QSA top-k reads
// the wrong cells. sharing the attention cache's cell metadata (v_cells_impl)
// via `other` makes the two agree by construction -- the indexer keeps its own
// K/V tensors but sees the same positions/sequences/used-set as attention, so
Comment on lines +55 to +59

@chatgpt-codex-connector chatgpt-codex-connector Bot Aug 27, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Shorten the added cache comment

This new block is a seven-line explanation with sentences split across comment lines; the repo guidance asks comments to stay concise, usually 1-2 lines, and not to split a sentence just to wrap. In this hot memory code, condensing it to the invariant, metadata is shared with the attention cache while tensors are not, would reduce maintenance noise.

AGENTS.md reference: AGENTS.md:L73-L81

Useful? React with 👍 / 👎.

// used_max_p1 and hence n_kv can never drift apart. seq ops on the indexer
// become no-ops (the shared cells are managed by the attention cache).
return new llama_kv_cache(
model, hparams_idx, type_k, type_v, v_trans, offload, unified,
kv_size, n_seq_max, n_pad, n_swa, swa_type,
nullptr, filter_idx, nullptr, nullptr, "idx_");
get_mem_attn(), filter_idx, nullptr, nullptr, "idx_");

@chatgpt-codex-connector chatgpt-codex-connector Bot Aug 27, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Preserve indexer tensors when sharing cells

Passing get_mem_attn() here sets llama_kv_cache::other on mem_idx. In that mode llama_kv_cache::state_write() and state_read_sinfo() immediately return, but the idx_ cache still allocates its own K/V tensors because the share callback is null; saving/loading a Qwen4Exp session therefore restores attention cell metadata without restoring the indexer keys that QSA top-k reads, producing stale or zero scores for past tokens. Please keep cell sharing from disabling indexer tensor serialization, or add a metadata-only sharing path.

Useful? React with 👍 / 👎.

}()) {}

llama_memory_context_ptr llama_memory_hybrid_idx::init_batch(llama_batch_allocr & balloc, uint32_t n_ubatch, bool embd_all) {
Expand Down