fix(scheduler): hybrid radix hit corruption — GDN snapshot copy-on-donate + admission barriers - #287
Conversation
…onate + admission barriers Under the default radix prefix cache, hybrid (GDN) models intermittently serve corrupted answers on cache-HIT requests that are co-batched with in-flight decode: ~10% wrong under concurrent load in ground-truth probing (empty or garbled output, special-token leaks into content; with a multimodal tower, blank visual grounding). Cold prefill is always clean, and failures cluster in sticky per-branch windows that self-heal when the branch is re-donated. Root structure: chunk-commit and finish donate the request's snapshot slot into the radix tree by OWNERSHIP TRANSFER, so one linear-state slot id is shared between the tree and the request's still-in-flight pipeline. A late write through that alias poisons every subsequent hit of the branch. Stream-level wait_stream ordering does not close the window (full-device syncs at the donate/restore points shrink the failure rate from ~10% to ~3% but not to zero); severing the alias does. Fix: - copy-on-donate: the tree receives a private clone of the snapshot slot (written once behind the donate barrier, read-only for its tree lifetime); the request keeps its own slots -- no replacement alloc at chunk commit, both ping-pong slots and the live slot freed at finish. - full-device barriers before hit-admission restores and donation bookkeeping (request-level events, sub-millisecond; they also guarantee the clone's source is quiescent). Cost: one ~MB slot copy per donation and one device sync per prefill commit / hit admission; decode throughput is unchanged. Validated on a v0.1.2-based deployment serving GLM-5.3-Flash-NVFP4 (34 KDA layers on the same hybrid_radix/linear_state pool path as qwen3.5 GDN) on an RTX PRO 6000 Blackwell: a deterministic hammer that alternates identical prompts with abandoned or saturating concurrent streams reproduced the corruption at 3/30 (images) and 2/24 (pure text) before the fix, and runs 0/60 and 48/48-clean after; an overnight ground-truth soak (~2,200 probes: exact-match arithmetic, needle retrieval, multi-turn recall, media A/B alternation with per-request content forensics) measured the pre-fix hit corruption at ~10% and confirmed payloads and cache keys correct on every failure, isolating the fault to the hybrid hit path.
|
Correction after deeper forensics — the corruption evidence in this PR's description does not implicate upstream. Slot-level checksum instrumentation overturned the ownership-aliasing theory. The true root cause of every corrupted hit we observed was in our downstream model integration: our KDA attention op (GLM-5.3 support, #270) never implemented the ×64-boundary track-snapshot write that the hybrid-radix design expects ( With the writer implemented on our side, restore-side checksums show real boundary state on every hit and 90 adversarial hammer rounds run clean — with or without the changes in this PR. What remains of this PR on its own merits, with no corruption claim attached:
Happy to re-scope the PR to the guard + optional copy-on-donate, or close it — maintainers' call. Apologies for the initial misattribution; the ground-truth probes and checksum forensics that led here are documented in the linked repository. |
1 participant
Problem
Under the default
--cache-type radix, hybrid (GDN) models intermittently serve corrupted answers on cache-HIT requests that are co-batched with in-flight decode. Measured with deterministic ground-truth probes on a production deployment: ~10% of hit requests wrong under concurrent load — empty or garbled output, special-token leaks into content (<|assistant|>mid-text), and with a multimodal tower, answers describing a blank image. Cold prefill is always clean. Failures cluster in sticky per-branch windows (a poisoned branch keeps failing for a while, then self-heals when re-donated).Every hybrid GDN model on the
hybrid_radix/linear_statepool path is exposed; nothing about the trigger is model-specific.Root structure
_cache_req_hybriddonates the request's snapshot slot into the radix tree by ownership transfer (chunk commit donates the frozen ping-pong slot; finish donates the live slot "zero-copy"). That leaves one linear-state slot id shared between the tree and the request's still-in-flight pipeline. A late write through any retained alias poisons every subsequent hit of that branch — which is exactly the observed sticky-window signature.Stream-order alone does not close the window: adding full-device syncs at the donate and restore points shrank the failure rate from ~10% to ~3% but not to zero. Severing the alias does.
Fix
_clone_slot_for_tree), written once behind the donate barrier and read-only for its tree lifetime. The request keeps its own slots — no replacement alloc at chunk commit; both ping-pong slots and the live slot are freed at finish._restore_linear_states) and donation bookkeeping (_cache_req_hybrid): request-level events, sub-millisecond, and they guarantee the clone's source is quiescent.Cost: one ~MB slot copy per donation plus one device sync per prefill commit / hit admission. Decode throughput unchanged in before/after measurement.
Validation
On a v0.1.2-based deployment serving GLM-5.3-Flash-NVFP4 (34 KDA layers riding the same
hybrid_radix/linear_statepath as qwen3.5 GDN; support PR #270) on an RTX PRO 6000 Blackwell 96 GB:The hammer scripts are small and self-contained; happy to attach them or port them into
tests/if useful (they pair naturally with the regression-harness proposal in #281).