This document records the validated baseline for the modern CacheRoute serving stack and defines the boundary between environment migration work and subsequent feature development.
| Component | Validated value |
|---|---|
| Base image | CUDA 13.0 / Ubuntu 22.04 |
| Python | 3.12.x |
| PyTorch | 2.11.0+cu130 |
| vLLM | 0.25.1 |
| LMCache | 0.5.2 |
| LMCache mode | standalone MP server |
| vLLM connector | LMCacheMPConnector |
| L2 backend | Redis RESP, DB 0 |
| Tensor parallel size | 8 |
| LMCache chunk size | 256 tokens |
| Hash algorithm | sha256_cbor |
| Model | Meta-Llama-3-70B-Instruct served as llama3-70b |
The modern startup contract is:
Redis RESP L2
-> LMCache MP :5555 / HTTP :8080
-> vLLM + LMCacheMPConnector :8000
-> Scheduler -> KDN -> Proxy -> Instance -> Client
The legacy and modern startup interfaces must not be mixed:
- legacy uses
LMCACHE_CONFIG_FILE, YAMLremote_url, and historical vLLM offloading flags; - v1 uses
lmcache server, RESP L2, andvllm serve --kv-transfer-config.
CACHEROUTE_RUNTIME_PROFILE provides three modes:
| Profile | Behavior |
|---|---|
legacy |
strict historical vllm@* Redis namespace |
v1 |
modern model-scoped LMCache Redis keys |
auto |
accepts either supported key generation |
Normal v1 registration no longer requires a manual --match. The historical
CLI default vllm@* is treated as a compatibility sentinel in v1 and auto
profiles, so the builder scans and validates the modern key layout automatically.
An explicit --match remains available for debugging and custom backends.
The 30-second build setting is a maximum first-key timeout, not a fixed sleep. After keys appear, capture completes when the key set remains unchanged for the configured quiet period.
The closeout validation used:
data/CacheRoute_dataset/knowledge_document/
668f6bd1ad2419d9dfbcda0b689311b8b6696c7ed772001bea7bd12573137b4a.txt
The document-derived KID matched the filename:
668f6bd1ad2419d9dfbcda0b689311b8b6696c7ed772001bea7bd12573137b4a
The v1 KDN builder produced:
runtime_profile: v1
key_formats: [v1]
scan_match: *
keys_before: 0
keys_after: 96
keys_new: 96
dumped_keys: 96
manifest records: 96
dump files: 96
local size: approximately 961 MiB
capture elapsed: 8.931 seconds
The registration command did not include --match or --flushdb.
After Redis DB 0 was explicitly cleared, kdn_server/kv_injector.py restored
all saved blocks:
injected: 96
missing_files: 0
Redis keys: 96
missing_keys: 0
extra_keys: 0
size_mismatches: 0
expected_payload_bytes: 1,006,632,960
actual_payload_bytes: 1,006,632,960
sample SHA256 mismatches: 0
This proves byte-for-byte preservation of the original Redis key/value pairs.
LMCache and vLLM were restarted after restore so that LMCache L1 and the vLLM
GPU cache started empty. An exact builder-style request was sent with the
normalized document as the only system message.
vLLM reported:
prompt tokens: 3293
external prefix-cache queries: 3293
external prefix-cache hits: 3072
prompt tokens cached: 3072
LMCache MP reported:
lookup requested tokens: 3072
lookup hit tokens: 3072
lookup hit rate: 100%
L2 prefetch lookup requests: 1
L2 prefetch lookup objects: 96
L2 prefetch hit chunks: 96
L2 prefetch load completed chunks: 96
L2 prefetch failure chunks: 0
L1 write chunks: 96
L1 read chunks: 96
The physical-key count is consistent with the runtime topology:
96 physical Redis keys / 8 TP ranks = 12 logical chunks
12 logical chunks * 256 tokens = 3072 cached tokens
The remaining 221 prompt tokens were not a full 256-token chunk and were recomputed by vLLM:
3293 total prompt tokens - 3072 cached tokens = 221 recomputed tokens
This closes the previous validation gap: the KDN artifact was not only saved and restored correctly; it was discovered in Redis, prefetched into LMCache L1, and consumed by vLLM as external prefix cache.
The two metrics endpoints are distinct:
| Endpoint | Owner | Important metrics |
|---|---|---|
http://127.0.0.1:8000/metrics |
vLLM | vllm:external_prefix_cache_queries_total, vllm:external_prefix_cache_hits_total, vllm:prompt_tokens_cached_total |
http://127.0.0.1:8080/metrics |
LMCache MP | lmcache_mp_lookup_requested_tokens_total, lmcache_mp_lookup_hit_tokens_total, lmcache_mp_l2_prefetch_hit_chunks_total, lmcache_mp_l2_prefetch_load_completed_chunks_total, lmcache_mp_l2_prefetch_failure_chunks_total |
LMCache metrics may not exist until the first store or lookup event. A direct KDN
Redis injection also bypasses LMCache's own store accounting, so
lmcache_mp_l2_usage_bytes must not be used as proof that Redis is empty.
Activate v1 in every terminal:
source env/docker/cu130/scripts/activate_v1.shRun the non-destructive environment check:
python3 env/docker/cu130/scripts/check_v1_environment.py --require-runningUse the staged KDN validator:
DOC=/workspace/llm-stack/CacheRoute/data/CacheRoute_dataset/knowledge_document/668f6bd1ad2419d9dfbcda0b689311b8b6696c7ed772001bea7bd12573137b4a.txt
python3 scripts/validate_v1_kdn_roundtrip.py build --document "$DOC"
python3 scripts/validate_v1_kdn_roundtrip.py inspect --document "$DOC"The restore stage is destructive only when explicitly confirmed:
python3 scripts/validate_v1_kdn_roundtrip.py inject \
--document "$DOC" \
--flush-redis \
--yesAfter injection, restart LMCache and vLLM to clear L1/GPU state, then run:
python3 scripts/validate_v1_kdn_roundtrip.py consume --document "$DOC"The validator deliberately does not combine injection and consumption into one unattended command because the service restart between the two stages is part of the correctness contract.
A reusable artifact is valid only when producer and consumer agree on at least:
- model identity and actual model path used by LMCache keys;
- served model and tokenizer;
- chat template and exact token prefix;
- tensor-parallel size and worker/rank layout;
- model/KV dtype;
- LMCache chunk size;
- LMCache hash algorithm;
- connector generation and key serialization;
- Redis database semantics.
A byte-perfect Redis restore proves transport integrity but does not override an identity mismatch. Cache consumption metrics remain the final acceptance test.
- Do not use
--flushdbduring normal KDN registration. - Do not run
FLUSHDBagainst a shared or production Redis database. - The round-trip validator requires both
--flush-redisand--yesbefore it performs destructive cleanup. redis-cliis optional; the documented tools use the Python Redis client.- The correct standalone injector is
kdn_server/kv_injector.py.
The following migration goals are complete:
- legacy environment and startup documentation remain available;
- isolated CUDA 13 / vLLM 0.25.1 / LMCache 0.5.2 image profile exists;
- LMCache MP and vLLM startup scripts exist;
- v1 profile activation is explicit and repeatable;
- KDN recognizes modern LMCache keys without manual
--match; - asynchronous writes are captured without a fixed 30-second delay;
- zero-key builds fail instead of becoming
kv_ready; - local dump/manifest integrity is verified;
- Redis restore is byte-for-byte verified;
- LMCache RESP L2 lookup/load is verified;
- vLLM external-prefix-cache consumption is verified;
- environment and round-trip checks are reproducible from repository scripts.
Future Scheduler, Proxy, Instance, predictor, and UI work can now use this v1 runtime as the modern development baseline without reopening the environment migration unless the serving-stack versions or cache identity change.