Skip to content

fix(maru_vllm): stop the worker when a GPU copy cannot be confirmed finished - #159

Open
seohui-XCENA wants to merge 9 commits into
xcena-dev:mainfrom
seohui-XCENA:fix/vllm-copy-safety-v2
Open

seohui-XCENA wants to merge 9 commits into
xcena-dev:mainfrom
seohui-XCENA:fix/vllm-copy-safety-v2

Conversation

@seohui-XCENA

@seohui-XCENA seohui-XCENA commented Nov 20, 2026 •

Copy link
Copy Markdown
Collaborator

🤔 Background & Motivation (Why)

  • The vLLM connector copies KV between vLLM's GPU blocks and Maru storage. When CUDA reported an error during such a copy (the copy or kernel launch itself, or a wait or query on its stream or event), the connector handled it like any other failure: a load was reported for recompute and its reads released, and a store's pages were freed. The copies might still be running, so vLLM could compute into blocks a copy still writes, and Maru could reuse pages a copy still reads or writes.
  • This applies to every existing backend: cxl, cpu, mixed and remote_cxl.

Before, on main:

sequenceDiagram
    participant V as vLLM
    participant C as Maru connector
    participant G as GPU copy (CUDA stream)
    participant P as Maru page
    C->>G: queue a copy between the page and vLLM's blocks
    C->>G: wait for it
    G-->>C: CUDA error
    Note over G: the copy may still be running
    C->>V: report the load as failed
    C->>P: free the page
    V->>V: recompute into the same blocks
    Note over V,P: the running copy can still overwrite the recomputed blocks,<br/>or touch the page after another store reuses it
Loading

🏗️ Design Changes

  • Behavioral change on every backend: a CUDA error from a GPU copy of a load or a store, or from waiting for one, stops the vLLM worker. The connector call that met the error raises, or else the next one vLLM makes, and vLLM fails the engine; recovery is an engine restart. What those copies may still use is kept until the process exits, at shutdown too: pool pages, read leases, staging buffers, slot mappings, the device buffers in use, the requests' GPU blocks and the handler that maps them.

After this PR:

sequenceDiagram
    participant V as vLLM
    participant C as Maru connector
    participant G as GPU copy (CUDA stream)
    participant P as Maru page
    C->>G: queue a copy between the page and vLLM's blocks
    C->>G: wait for it
    G-->>C: CUDA error
    Note over C: the worker stops
    Note over C,P: the page, lease, staging and the requests' blocks<br/>stay held until the process exits
    C-->>V: the connector call raises
    V->>V: fail the engine (restart to recover)
Loading
  • Any other error is handled as before. Only a CUDA error from the copy path stops the worker:
%%{init: {"htmlLabels": false, "flowchart": {"htmlLabels": false, "wrappingWidth": 400}}}%%
flowchart LR
    E["Error in a load or a store"] --> Q{"CUDA error from a copy,<br/>a kernel launch, or an<br/>event or stream wait?"}
    Q -- "yes" --> S["Stop the worker:<br/>keep what the copy may use,<br/>every hook raises"]
    Q -- "no: out of memory, rejected<br/>layout, retrieve failure, miss" --> M["Handled as before:<br/>recompute, fall back,<br/>skip the store, or propagate"]
Loading
  • A load that fails after some of its copies were queued is reported only once those copies have ended, or at once when they run on the stream vLLM recomputes on.
  • Documented in the "Load failures" section of docs/source/integration/vllm.md.

📝 Implementation Details

Three rules in maru_vllm/connector.py:

  1. Classify the error, not the path. _unconfirmed(e) is true for torch.AcceleratorError or a RuntimeError naming a CUDA error, and torch.OutOfMemoryError is never one. Every copy, kernel launch, event record or query, and stream wait reaches it through _stop_if_unconfirmed or _checked. The first such error sets one stop state shared by loads and stores, and a later successful wait does not undo it.
  2. Once stopped, nothing is given back. The shared helpers that free pages, release leases, register stores (batch_store, _DONE markers), report to vLLM, drop refs and close the handler keep instead (_kept_if_stopped, _release, _store_objects, _free_handles_best_effort). Every hook raises (_raise_if_stopped), background jobs check the stop before each batch, and no read promote is sent. A call already past its check finishes its one issue, and what that issue uses is kept.
  3. Register before issue. What a queued copy reads or writes goes into its load's or batch's refs before the copy is issued. A per-copy device temporary that is allocated and used on the same stream goes back to the caching allocator after its copy is queued, which reuses it only in that stream's order.

For reviewers:

  • No new synchronize, event wait or lock runs when no CUDA error happens; the extra cost is a stop-state read and refs list appends per request or chunk.
  • Main tests whose fakes raise CUDA errors now expect a stop and kept pages instead of a free or a recompute.

✅ Tests

  • Unit tests: tests/unit/test_vllm_unconfirmed_copies.py (new; fault injection on cxl, cpu, mixed and remote_cxl), and updated connector, KV group and remote_cxl transfer tests. The full unit suite passes without a GPU, and the vLLM tests pass on RTX PRO 6000 and H200 GPUs (vLLM 0.32).
  • Integration tests
  • Manual tests: real vLLM 0.32 with Llama-3.1-8B-Instruct on the cxl backend, 30 visits each of main and this PR, run side by side on two H200 GPUs. Every measured request was a full Maru hit with no load failure, and outputs at concurrency 1 were identical between main and this PR. TTFT p50 against main: +0.6% at concurrency 1, +1.0% at 8, +0.0% at 32, within run-to-run noise.
  • No tests needed (reason: )

🔗 Related Issues (optional)

…inished

A load or store whose GPU copy raised a CUDA error, or whose wait or
query on the copy's stream or event did, was handled like any other
failure: the load was reported for recompute and its reads released,
the store's pages freed, while the copies might still write or read
vLLM's blocks, which vLLM then reused.

The connector now classifies the error instead of the path. A CUDA
error (torch.AcceleratorError, or a RuntimeError naming a CUDA error)
from a copy, a kernel launch or a wait sets one stop state shared by
loads and stores and logs it once. Every later connector hook raises
_UnconfirmedCopyError, so vLLM fails the engine. Any other error, such
as torch.OutOfMemoryError, a rejected argument or layout, or a miss, is
handled as before, and a failed wait is never rescued by a later one.

Once stopped, the helpers that give something back keep it until the
process exits: page and handle frees, read lease releases, batch_store
and _DONE markers, load and store reports, dropped refs, and the
handler close and buffer teardown at shutdown. Background loads and
stores check the stop before each batch, no read promote is sent, and
shutdown issues no queued store batch. What a queued copy uses (reads,
pages, staging, slot mappings, device temporaries) goes into its load's
or batch's refs before the copy is issued. A load that fails after its
copies were queued is reported only once they ended.

By backend:
- cxl: packed, layerwise and layerwise-overlap loads, parked-load
  admission and group loads, and the synchronous, write-behind, group,
  Mamba state and MLA block stores.
- cpu and mixed: loads without a placement kernel queue their copies on
  the current stream; a CUDA error there keeps the leases, the slot
  mapping and the device slab. A failure before or inside the first copy
  that is not a CUDA error recomputes as before.
- remote_cxl: leased and group loads keep their storage-session leases
  and staging slots; a write-behind batch keeps its pages, staging and
  slot mappings, and its requests do not finish. The batch takes the
  staging buffer once.

The ctypes cudaMemcpy2DAsync failure now reads as a CUDA error. A
queued store batch that fails to issue at shutdown is logged and the
rest of shutdown runs.
The load failure section now says that a CUDA error from a GPU copy of
a load or a store, or from waiting for it, stops the worker while other
errors are handled as before, that a load failing after its copies were
queued is reported once they ended, what a stopped worker keeps until
the process exits, and how to recover.
On every backend (cxl, cpu, mixed, remote_cxl):

- The inline packed load uploads its kernel pointer table on the
  compute stream, and the load stream's kernels read it. The table is
  now in the load's refs, so a CUDA error keeps it. The loader-thread,
  store and group tables are cached by the worker.
- The CUDA calls no except classified now reach the classifier: the
  first stream wait or event record of the group, packed and layerwise
  loads and of the synchronous and write-behind stores, the stream wait
  in wait_for_layer_load, the slot mapping copy of the synchronous
  kernel store, every group store of wait_for_save (the MLA index and
  group pointer table uploads included), the layerwise activation
  without a common CUDA stream (CPU, or layers on several devices), and
  the maru_log_timing events, whose errors the trace code dropped.
- Pinning a slot mapping is a host allocation before the first copy:
  its failure fails the load as on main instead of stopping the worker.
- Admission stops the worker while it still holds the deferred lock, so
  vLLM's poll cannot report that load in between, and the poll checks
  the stop again once it holds the lock.
- Once stopped, stores issue nothing from the next chunk or request on:
  the layerwise and packed fallback saves, the group fallback, Mamba
  state and MLA stores, and each request of the synchronous kernel
  store. Loads log as loaded only the requests they took.
- The _DONE marker is stored and freed through the stop gate, and the
  single-group wait_for_save hook raises once stopped.
A failed load whose copies run on the forward's own stream (cpu and
mixed without the placement kernel, inline maru_use_layerwise loads,
layers on several devices) is reported at once, as those copies are
ordered before vLLM's recompute. A call already past the stop check
finishes its one issue, and only the device buffers in use at the
failure are kept; earlier per-copy buffers go back to the caching
allocator in stream order.
…precisely

On every backend (cxl, cpu, mixed, remote_cxl):

- The engine thread's kernel pointer table uploads, of the inline packed
  load and of the single-group packed store, now reach the classifier:
  a CUDA error stops the worker, any other error propagates as before.
- A hook whose timing event record or query stopped the worker raises
  instead of returning: get_finished_loading after its trace poll,
  wait_for_layer_load after its trace query, and finish_forward, which
  also raises once stopped.
- Only a timing event's record or query reaches the classifier. Setting
  up the timing clock (its stream) is dropped on failure as before.
- A pinned slot mapping is in the load's refs before its own copy to the
  device is issued (cxl parked packed loads, and packed loads on every
  backend), so a CUDA error from that copy keeps it.
- The layerwise activation keeps the entries it took when its first
  stream wait fails.
Kernel-less loads on remote_cxl and layerwise parked loads whose layers
span several devices also report a failure at once. Errors other than
CUDA errors keep each path's handling: recompute, fall back, or
propagate.
…'s events

With maru_log_timing on, the timing clock records an event on its own
stream and waits for it, when it is first set up and when it is
refreshed. A CUDA error from that record or wait may be one of earlier
copies, so it now stops the worker like any other timing event's, and
the hook that met it (start_load_kv, get_finished_loading) raises.
Creating the clock's stream or events stays best effort.

On every backend (cxl, cpu, mixed, remote_cxl).
Recording an event can also stop the worker, and an error that is not
a CUDA error, from a load or a store, may also propagate out of the
hook.
The stop also covers events queued after a copy, the maru_log_timing
events among them, and the connector call that met the error raises
before vLLM's next one would. The timing clock test checks that the
clock stops calibrating at its first failed record or wait.

@kihwan-XCENA kihwan-XCENA left a comment •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. This replaces a silent KV corruption risk with an explicit stop, and the normal path is unchanged.

One thought for a follow-up, not for this PR. vLLM's OffloadingConnector faced the same problem: when one copy fails, the copies queued before it on the same stream may still be running, but the failure handling gives back the memory of the whole request. In vllm-project/vllm#52838, vLLM records an event after the failure and gives the memory back only once that event completes, that is, once the earlier copies have finished. After that it keeps serving. It brings the engine down only if it cannot even record the event.

This works because after a non-sticky error, such as running out of memory, the failed call is never queued and the GPU keeps working, so waiting confirms that the earlier copies are done. After a sticky error the wait fails too, and then stopping as this PR does is the right call, since the process needs a restart anyway.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants