Repository navigation
fix(maru_vllm): stop the worker when a GPU copy cannot be confirmed finished - #159
seohui-XCENA wants to merge 9 commits into
Conversation
…inished A load or store whose GPU copy raised a CUDA error, or whose wait or query on the copy's stream or event did, was handled like any other failure: the load was reported for recompute and its reads released, the store's pages freed, while the copies might still write or read vLLM's blocks, which vLLM then reused. The connector now classifies the error instead of the path. A CUDA error (torch.AcceleratorError, or a RuntimeError naming a CUDA error) from a copy, a kernel launch or a wait sets one stop state shared by loads and stores and logs it once. Every later connector hook raises _UnconfirmedCopyError, so vLLM fails the engine. Any other error, such as torch.OutOfMemoryError, a rejected argument or layout, or a miss, is handled as before, and a failed wait is never rescued by a later one. Once stopped, the helpers that give something back keep it until the process exits: page and handle frees, read lease releases, batch_store and _DONE markers, load and store reports, dropped refs, and the handler close and buffer teardown at shutdown. Background loads and stores check the stop before each batch, no read promote is sent, and shutdown issues no queued store batch. What a queued copy uses (reads, pages, staging, slot mappings, device temporaries) goes into its load's or batch's refs before the copy is issued. A load that fails after its copies were queued is reported only once they ended. By backend: - cxl: packed, layerwise and layerwise-overlap loads, parked-load admission and group loads, and the synchronous, write-behind, group, Mamba state and MLA block stores. - cpu and mixed: loads without a placement kernel queue their copies on the current stream; a CUDA error there keeps the leases, the slot mapping and the device slab. A failure before or inside the first copy that is not a CUDA error recomputes as before. - remote_cxl: leased and group loads keep their storage-session leases and staging slots; a write-behind batch keeps its pages, staging and slot mappings, and its requests do not finish. The batch takes the staging buffer once. The ctypes cudaMemcpy2DAsync failure now reads as a CUDA error. A queued store batch that fails to issue at shutdown is logged and the rest of shutdown runs.
The load failure section now says that a CUDA error from a GPU copy of a load or a store, or from waiting for it, stops the worker while other errors are handled as before, that a load failing after its copies were queued is reported once they ended, what a stopped worker keeps until the process exits, and how to recover.
On every backend (cxl, cpu, mixed, remote_cxl): - The inline packed load uploads its kernel pointer table on the compute stream, and the load stream's kernels read it. The table is now in the load's refs, so a CUDA error keeps it. The loader-thread, store and group tables are cached by the worker. - The CUDA calls no except classified now reach the classifier: the first stream wait or event record of the group, packed and layerwise loads and of the synchronous and write-behind stores, the stream wait in wait_for_layer_load, the slot mapping copy of the synchronous kernel store, every group store of wait_for_save (the MLA index and group pointer table uploads included), the layerwise activation without a common CUDA stream (CPU, or layers on several devices), and the maru_log_timing events, whose errors the trace code dropped. - Pinning a slot mapping is a host allocation before the first copy: its failure fails the load as on main instead of stopping the worker. - Admission stops the worker while it still holds the deferred lock, so vLLM's poll cannot report that load in between, and the poll checks the stop again once it holds the lock. - Once stopped, stores issue nothing from the next chunk or request on: the layerwise and packed fallback saves, the group fallback, Mamba state and MLA stores, and each request of the synchronous kernel store. Loads log as loaded only the requests they took. - The _DONE marker is stored and freed through the stop gate, and the single-group wait_for_save hook raises once stopped.
A failed load whose copies run on the forward's own stream (cpu and mixed without the placement kernel, inline maru_use_layerwise loads, layers on several devices) is reported at once, as those copies are ordered before vLLM's recompute. A call already past the stop check finishes its one issue, and only the device buffers in use at the failure are kept; earlier per-copy buffers go back to the caching allocator in stream order.
…precisely On every backend (cxl, cpu, mixed, remote_cxl): - The engine thread's kernel pointer table uploads, of the inline packed load and of the single-group packed store, now reach the classifier: a CUDA error stops the worker, any other error propagates as before. - A hook whose timing event record or query stopped the worker raises instead of returning: get_finished_loading after its trace poll, wait_for_layer_load after its trace query, and finish_forward, which also raises once stopped. - Only a timing event's record or query reaches the classifier. Setting up the timing clock (its stream) is dropped on failure as before. - A pinned slot mapping is in the load's refs before its own copy to the device is issued (cxl parked packed loads, and packed loads on every backend), so a CUDA error from that copy keeps it. - The layerwise activation keeps the entries it took when its first stream wait fails.
Kernel-less loads on remote_cxl and layerwise parked loads whose layers span several devices also report a failure at once. Errors other than CUDA errors keep each path's handling: recompute, fall back, or propagate.
…'s events With maru_log_timing on, the timing clock records an event on its own stream and waits for it, when it is first set up and when it is refreshed. A CUDA error from that record or wait may be one of earlier copies, so it now stops the worker like any other timing event's, and the hook that met it (start_load_kv, get_finished_loading) raises. Creating the clock's stream or events stays best effort. On every backend (cxl, cpu, mixed, remote_cxl).
Recording an event can also stop the worker, and an error that is not a CUDA error, from a load or a store, may also propagate out of the hook.
The stop also covers events queued after a copy, the maru_log_timing events among them, and the connector call that met the error raises before vLLM's next one would. The timing clock test checks that the clock stops calibrating at its first failed record or wait.
There was a problem hiding this comment.
LGTM. This replaces a silent KV corruption risk with an explicit stop, and the normal path is unchanged.
One thought for a follow-up, not for this PR. vLLM's OffloadingConnector faced the same problem: when one copy fails, the copies queued before it on the same stream may still be running, but the failure handling gives back the memory of the whole request. In vllm-project/vllm#52838, vLLM records an event after the failure and gives the memory back only once that event completes, that is, once the earlier copies have finished. After that it keeps serving. It brings the engine down only if it cannot even record the event.
This works because after a non-sticky error, such as running out of memory, the failed call is never queued and the GPU keeps working, so waiting confirms that the earlier copies are done. After a sticky error the wait fails too, and then stopping as this PR does is the right call, since the process needs a restart anyway.
🤔 Background & Motivation (Why)
cxl,cpu,mixedandremote_cxl.Before, on main:
sequenceDiagram participant V as vLLM participant C as Maru connector participant G as GPU copy (CUDA stream) participant P as Maru page C->>G: queue a copy between the page and vLLM's blocks C->>G: wait for it G-->>C: CUDA error Note over G: the copy may still be running C->>V: report the load as failed C->>P: free the page V->>V: recompute into the same blocks Note over V,P: the running copy can still overwrite the recomputed blocks,<br/>or touch the page after another store reuses it🏗️ Design Changes
After this PR:
sequenceDiagram participant V as vLLM participant C as Maru connector participant G as GPU copy (CUDA stream) participant P as Maru page C->>G: queue a copy between the page and vLLM's blocks C->>G: wait for it G-->>C: CUDA error Note over C: the worker stops Note over C,P: the page, lease, staging and the requests' blocks<br/>stay held until the process exits C-->>V: the connector call raises V->>V: fail the engine (restart to recover)%%{init: {"htmlLabels": false, "flowchart": {"htmlLabels": false, "wrappingWidth": 400}}}%% flowchart LR E["Error in a load or a store"] --> Q{"CUDA error from a copy,<br/>a kernel launch, or an<br/>event or stream wait?"} Q -- "yes" --> S["Stop the worker:<br/>keep what the copy may use,<br/>every hook raises"] Q -- "no: out of memory, rejected<br/>layout, retrieve failure, miss" --> M["Handled as before:<br/>recompute, fall back,<br/>skip the store, or propagate"]docs/source/integration/vllm.md.📝 Implementation Details
Three rules in
maru_vllm/connector.py:_unconfirmed(e)is true fortorch.AcceleratorErroror aRuntimeErrornaming a CUDA error, andtorch.OutOfMemoryErroris never one. Every copy, kernel launch, event record or query, and stream wait reaches it through_stop_if_unconfirmedor_checked. The first such error sets one stop state shared by loads and stores, and a later successful wait does not undo it.batch_store,_DONEmarkers), report to vLLM, drop refs and close the handler keep instead (_kept_if_stopped,_release,_store_objects,_free_handles_best_effort). Every hook raises (_raise_if_stopped), background jobs check the stop before each batch, and no read promote is sent. A call already past its check finishes its one issue, and what that issue uses is kept.For reviewers:
✅ Tests
tests/unit/test_vllm_unconfirmed_copies.py(new; fault injection oncxl,cpu,mixedandremote_cxl), and updated connector, KV group and remote_cxl transfer tests. The full unit suite passes without a GPU, and the vLLM tests pass on RTX PRO 6000 and H200 GPUs (vLLM 0.32).cxlbackend, 30 visits each of main and this PR, run side by side on two H200 GPUs. Every measured request was a full Maru hit with no load failure, and outputs at concurrency 1 were identical between main and this PR. TTFT p50 against main: +0.6% at concurrency 1, +1.0% at 8, +0.0% at 32, within run-to-run noise.🔗 Related Issues (optional)