[faresobeid]

Router replay (enable_return_routed_experts) costs a lot of generation throughput on vLLM 0.32. vLLM's AuxOutput store and worker code are unchanged since 0.31, where the numbers below were measured. In the v031 defaults experiment (GLM-4.5-Air TP8, about 400 running requests), arm C vs arm B was −25% generation. Most of the cost is CPU work in vLLM's AuxOutput path on the output worker (TP0) and in EngineCore, not GPU work. This PR adds three vLLM patches in patches.py. Routing rows are unchanged.

Where the time goes

GLM-4.5-Air TP8 on H200, one node, /inference/v1/generate, closed loop with 400 concurrent requests, max 16k tokens, sampling masks on. Diagnostics: per-function timers, per-thread CPU, GC callbacks, and stack samples in worker TP0 and EngineCore.

1. Store eviction on the output worker (largest cost). vLLM sizes BlockObjectStore to the KV cache, so under load it is full and every put evicts. _evict_to_fit scans the LRU from the front and skips keys referenced by running requests. Those keys pile up at the front, so each put rescans ~300k keys: 9 ms per put, 46 puts/s, on the writer thread. It holds the GIL against the thread that launches the next step, and execute_model goes from 2.7 to 13.8 ms per call.
2. One array per request per step. The output worker returns dict[request_id, AuxRequestOutput]. Each step pickles one small array per running request to EngineCore, each as its own out-of-band buffer. EngineCore rebuilds them one object at a time. The cost grows with requests per step, and the churn drives GC.
3. Engine GC. vLLM freezes the startup heap of EngineCore and of the workers but keeps the default thresholds.

Fix

• monkey_patch_aux_output_store
  • Before upstream _evict_to_fit runs on a full store, it moves the referenced keys at the front of the LRU to the end. A referenced key cannot be evicted, and release moves it to the end anyway, so the victims are identical to upstream.
  • The output worker builds the store at 2× vLLM's size. At 100% KV the referenced rows alone can exceed a store of exactly KV size. put then raises BlockObjectStoreError in the writer thread and the engine dies. The condition does not depend on LRU order, so upstream raises on the same sequence. Before this PR the engine was too slow to fill the KV cache in this harness. With only the eviction fix, it reached 100% KV and died.
• monkey_patch_packed_aux_outputs + PackedAuxOutputs: the worker sends one object per step (id→index dict, token starts, offsets, one rows array). It pickles as three byte strings and answers the scheduler's in / [] lookups.
• monkey_patch_engine_gc: EngineCore keeps vLLM's freeze and gets set_threshold(50_000, 20, 100). The workers get gc.collect() + gc.freeze() + the same thresholds after compile_or_warm_up_model.

Results (gen tok/s, steady window 240–900 s)

Earlier runs (node A, 400 concurrent, 16k):

arm	gen tok/s	tok/s per request	KV usage
router replay off, main	9,356	23.5	0.72
router replay on, main	5,724 (−39%)	14.4	0.85
on, GC + packing (no store fix)	8,250	20.9	0.76
on, store eviction only (1× size)	crashed at 100% KV
on, all three (1× size)	9,412	23.8	0.72
off, all three	9,596	24.1	0.72

Ablation of the branch as pushed (node B, same node for every row):

arm	load	gen tok/s	tok/s per request	running	KV mean / peak
all three	400 conc, 16k	9,679	24.6	393	0.71 / 0.98
store + GC (no packing)	400 conc, 16k	9,493 (−1.9%)	24.3	390	0.72
store only	400 conc, 16k	8,309 (−14%)	21.2	392	0.73 / 1.00, no error
all three	1000 conc, 2k, max_num_seqs=1024	9,328	9.6	970	0.25
store + GC (no packing)	1000 conc, 2k, max_num_seqs=1024	9,078 (−2.7%)	9.4	971	0.25

• Store only: EngineCore gen-2 GC takes 10.7% of wall time. With GC tuning, EngineCore and the workers run 0 collections in the window.

• Packing: worker TP0 MessageQueue.enqueue takes 0.32 vs 0.94 ms per call at 400 running, and 0.46 vs 2.85 ms at 970 running (2 calls per step). EngineCore's unpickle is hidden inside dequeue, so the bench below gives its size.

• Scaling of packing (CPU bench of the per-step path; pickle + unpickle per step, ms):

  requests per step	400	1000	2000
  per-request dict	6.1	18.7	31.7
  packed	0.13	0.37	0.8

  The cost is the same for 46 or 78 MoE layers × top-8: it is per request, not per byte. Both halves sit on the step's critical path: the worker main thread, then the EngineCore loop. So packing matters more as engines run more requests per step, which large sparse MoE engines with DP/EP do. At 1000 requests it is already ~2.7% end to end on this model.

• Store headroom cost: host RAM per engine ≤ 2 × KV tokens × MoE layers × top-k × itemsize, on the output worker only. The arena is an anonymous mmap, so only used slots cost RSS.

  • Here: 4.37M tokens × 46 × 8 × 1 B = 1.6 GB, so up to 3.2 GB.
  • DeepSeek-V3-class model (58 × 8, uint8), 4M KV tokens per engine: 1.9 GB → 3.7 GB.
  • 78 × 8 at 8M tokens: 5 GB → 10 GB per engine.
  • With more than 256 experts the rows are uint16, which doubles these numbers.

Checks

• Routing rows. 24 sequential top_k=1 requests (12 prompts × 300/1000 tokens) on node B: token ids and routed_experts bytes are bitwise identical between main and this branch (24/24). Earlier, router replay off gave identical tokens (24/24).
• Store eviction. Compared with upstream on random op sequences (300 seeds × 300 puts): same stored keys, same order of unreferenced keys (the only order that affects eviction), same errors. Bench with 450k slots and 320k live keys: 27.7 → 0.40 ms per step.
• Unit tests. tests/unit/orchestrator, tests/unit/test_configs.py, tests/unit/inference with -m "not gpu": 381 passed (vLLM 0.32 venv). ruff check and ruff format --check are clean.

Notes

• Upstream-worthy: the eviction scan, store capacity at full KV, and packed per-step aux outputs.
• Upstream: [Bugfix][AuxOutput] Fix routed-experts store overflow at full KV vllm-project/vllm#62044 (store fix; closed without merge, so the store patch stays), #58328 (open; the store grows past 2x KV instead of raising), #61086 (open; per-step R3 packing), #62086 (open draft; GC thresholds).
• Open question (API server, not this PR). When ~100 length-capped 16k requests finish together, the API server loop stalls for ~25 s and clients see ReadErrors (~110 per 400-concurrent arm). Each response carries about 8 MB of routed-experts JSON. By-handle payloads (feat: pass router-replay ids and sampling masks by handle on multi-node runs #3902) avoid this path.

---

Note

Medium Risk
Monkey-patches vLLM engine, worker, and API-server internals and changes generate response shapes for logprobs and sampling masks; mitigated by unit tests and documented parity checks for routing rows and eviction.

Overview
Cuts CPU/GC cost on the router-replay path and on /inference/v1/generate under heavy RL concurrency by avoiding per-token Python objects and tuning garbage collection.

Engine / aux-output path: Registers four vLLM monkey patches in patches.py. Routed experts per scheduler step are sent as a single PackedAuxOutputs blob (replacing per-request dict pickling). The aux-output BlockObjectStore gets a 2× capacity headroom and an eviction tweak so referenced LRU keys are moved to the end before upstream eviction (avoids multi-ms rescans at full KV). EngineCore and GPU workers freeze the post-warmup heap and use higher gc thresholds; the API server does the same after startup.

Generate handler: PrimeRlServingTokens strips heavy fields early via _PackedOutputs, then returns base64-packed numpy for routed_experts, completion_logprobs, and CSR sampling_mask ({ids, counts}). Non-streaming requests use flat logprobs, and detokenize is disabled when there are no stop strings. A separate patch keeps FINAL_ONLY sampling masks as PackedSamplingMask in the output processor.

Tests / deps: Adds PackedAuxOutputs pickle and store-eviction parity tests; updates serving-token unit tests. Bumps renderers and drops pytest-cov from the lockfile.

^{Reviewed by Cursor Bugbot for commit e1afb9c. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Stale Bugbot comment from a previous run.

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

There are 2 total unresolved issues (including 1 from previous review).

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit c245572. Configure here.}

[cursor]

Worker GC freeze on every warmup

Medium Severity

compile_or_warm_up_model always runs gc.freeze() after the original warmup. A second call permanently pins every object allocated since the last freeze, including request-scoped leftovers, so worker RSS can grow without bound across sleep/wake, CUDA-graph recapture, or a later recompile.

 

^{Reviewed by Cursor Bugbot for commit c245572. Configure here.}