kv-cells: keep the used-cell set as a bitmap - #28245
Closed
ServeurpersoCom wants to merge 1 commit into
Closed
Conversation
used was a std::set<uint32_t>, one tree node per cell with no locality, for a set that is dense by nature: a large fraction of the cache. llama_kv_idx_set stores it as a bitmap instead, one word per 64 cells. insert and erase become O(1), first and last scan the words and are called once per ubatch.
Contributor
Author
|
No more gain master. Closing. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
Reduce the generation slowdown as context grows. Split out of #27977 as requested, one PR per change.
Optimize the KV cache bookkeeping: track used cells in a bitmap instead of a std::set.
Additional information
llama_kv_cells tracked the used cells in a std::set<uint32_t>: one tree node per cell, no locality, for a set that is dense by nature (a large fraction of the cache).
This stores it as a bitmap, one 64-bit word per 64 cells. insert/erase become O(1), contains is a bit test, and first/last walk words with ctz/clz instead of chasing nodes; they run once per ubatch. Pure data structure change, no behavior change.
Requirements