spec : save the dynamic ngram cache file

[petersid2022]

Overview

• When we select the COMMON_SPECULATIVE_TYPE_NGRAM_CACHE speculative implementation we create a new common_speculative_state_ngram_cache state using create_state_ngram_cache, where we instantiate the new state by specifying various parameters (e.g, n_draft, save_static and save_dynamic) by hardcoding them.

• Instead we remove those options and only depend on the existence of lookup_cache_dynamic and lookup_cache_static (both user provided) for enabling the saving and updating the dynamic cache into a file.

• on object destruction check if the above functionality is supported (i.e. if the lookup_cache_dynamic path is provided by the user) then merge all of the ngram_cache_context from each sequence info and save the final common_ngram_cache to a file using common_ngram_cache_save.

• catch std::ifstream::failure the way llama-lookup does, so a permission/read failure cannot replace the file

Additional information

Add self‑speculative decoding (no draft model required)#18471

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: NO

[ggerganov]

The new parameters are never populated. Did you test this change? What is the goal of this PR?

[petersid2022]

first of all, thanks for taking the time to review my PR!

TBH, my initial scope for this PR (after coming across the TODO on line 930 of common/speculative.cpp) was to move the save_[state,dynamic] bools from inside create_state_ngram_cache to the more centralized common.h, as the TODO instructed. That would allow for those options to be configurable from the user. Though, after thinking about it some more I decided to drop the booleans and instead rely on the existence or not of path_[static,dynamic] parameter.

The way I went about testing my changes was using the below command:

./build/bin/llama-server --port 1234 -m ~/models/Qwen3.5-9B-Q8_0.gguf --spec-type ngram-cache --lookup-cache-static ~/static.bin --lookup-cache-dynamic ~/dynamic.bin

P.S: In the common_speculative_state_ngram_cache constructor, I noticed that if the common_ngram_cache_load call fails on either one of them we abort execution. Could we instead call common_ngram_cache_save() so that we create the file?

[Green-Sky]

I just ran into the same TODO, what is the state of this pr?

[petersid2022]

Hello! I intend to work on this PR very soon.

[petersid2022]

@ggerganov kind reminder for a code review of this PR, that simply saves the dynamic ngram cache file (when a path is provided) to the filesystem on object destruction. Thanks in advance!
BR,
Peter Sideris