Repository navigation
Revert "server : add --cache-disk for a persistent prompt cache" (#81) - #154
Merged
Merged
Conversation
This reverts commit 0f36c46, reversing changes made to 307c50d. --cache-dir drives runaway disk writes on ROCm: the slot state is copied straight between the GPU and a MAP_SHARED file, the runtime pins the mapped pages, and writeback keeps rewriting the file until the server exits. Users reported ~1.6 TB written to NVMe in an hour, and decode stalling under 5+ GB/s of continuous writes with a 75k-token context. Reproduced on gfx1151: one 503 MiB entry became 5.7 GB of writes within 60 s of idle. Even with that fixed (#149), an agent-style workload rewrites the whole state on every turn. Remove the feature until it can come back with bounded writes; --cache-ram is unaffected. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01G5NBdiBqRmGomhSRXBoucx
github-actions
Bot
added
documentation
dzannotti
added a commit
that referenced
this pull request
Oct 8, 2026
Brings back #81 by @storm1er (reverted in #154 after the ROCm write storm reports), so the fixes that follow can land on top of it. This reverts commit a3a9bb3. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01G5NBdiBqRmGomhSRXBoucx
rcfa
pushed a commit
to rcfa/strix-llama.cpp
that referenced
this pull request
Oct 23, 2026
…lo-box#81) Brings back halo-box#81 by @storm1er (reverted in halo-box#154 after the ROCm write storm reports), so the fixes that follow can land on top of it. This reverts commit a3a9bb3. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01G5NBdiBqRmGomhSRXBoucx
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Reverts #81 (persistent prompt cache on disk,
--cache-dir/--cache-dir-max, by @storm1er). The feature is wearing out users' SSDs, so it comes out until it can return with bounded writes.Why
Two user reports:
--cache-diron the ROCm build (b1015). On a RAM disk, a single 750 MB save never finished and kept rewriting the same file: 40 GB in 4 minutes. The CPU-only build writes it once.llama-serverexited.Cause: saves and restores copy the slot state straight between the GPU and a
MAP_SHAREDcache file. The ROCm runtime pins the mapped pages for that copy. Writeback then keeps rewriting the file for as long as the server runs. Credit for the diagnosis goes to the first reporter.Repro on a Ryzen AI Max+ 395 (gfx1151): Qwen3-1.7B Q8_0,
-c 32768 -np 1 --cache-dir, ext4 on a RAM-backed loop device. One 4.5k-token prompt was saved (503 MiB), then the server was left idle. Writes reached 5.7 GB within 60 s and were still climbing at ~208 MiB/s.#149 has a fix that stages the copy through RAM. With it, that entry is written once. Even so, an agent-style workload (a growing conversation plus one side request per turn) wrote 5.8 GB over 10 turns. Every turn saves the whole state again, and the older entries stay on disk until
--cache-dir-max. A safe version needs bounded writes, not just the fix.Effect
--cache-dirand--cache-dir-maxare gone. A config that still passes them fails at startup witherror: invalid argument: --cache-dir. Use--cache-raminstead.--cache-ram) is unchanged.Checks
--ple) touch the same files but don't depend on the feature, and no reference to it remains.--cache-ram 4096: re-sending a 4.6k-token prompt after a different one reuses 4573 of 4574 tokens, with output identical to the first run.🤖 Generated with Claude Code
https://claude.ai/code/session_01G5NBdiBqRmGomhSRXBoucx