Repository navigation
Conversation
Contributor
🏷️ CI GuideRuns automatically on every eligible PR before approval:
Heavy model tests:
|
PerryZhang01
pushed a commit
that referenced
this pull request
Oct 12, 2026
…ould see #2501 and #2502 report, for DeepseekV4ForCausalLM: a GPU memory access fault on any prefill step whose rows are all one token (reached by a `block*m + 1` prompt sent twice, since the block-aligned hit leaves a 1-token tail), and wrong output -- not a crash -- for prefill batches up to the largest capture size. #2502's root cause does not transfer: V4 relies on ATOM's fx split at its attention op and breakable cudagraph replaces it, while V4.1 does no fx splitting and breakable is the only way it gets graphs at all. The SHAPES still had to be checked, and this line had never sent one. Neither class reproduces. The fault: the trigger was reached, not just the shape. Six shapes produced an uncached tail of exactly one token on their second send (257, 513, 1025, 2049, 8193 and max_batched+1, all with `cached = tokens - 1` read off the hit counters), under FULL_AND_PIECEWISE with prefix caching on. Zero memory access faults, zero engine deaths. A shape that took no hit (129, shorter than a block) is reported as NOT REACHED rather than as passing. The wrong output: with caching off on both arms and concurrency 1, every shape's eager-vs-graph teacher-forced logprob deviation sits at or below that shape's own noise floor -- ratios 0.40 to 1.12 across 14 shapes, including the 1-token tails, the window-aligned shapes that are not block multiples, and a block-aligned control. Two criteria had to be discarded first, and the control is what discarded them: Sampled text. 13 of 17 shapes had the EAGER arm differing from ITSELF between two sends, and the control shape -- implicated by neither PR -- differed too. The oracle was not stable, so identity could not be the test. Logprobs across a cache hit. Two identical requests to one server with caching on agreed on 1 of 161 positions, deviating by up to 1.71. And a sizing defect worth more than either: the chat endpoint's template adds ~88 tokens, so the first attempt's `block*m + 1` shapes were never sent at all -- it measured 341 tokens while printing "257". The probes now build prompts through `/v1/completions`, which applies no template, and verify the count against `usage.prompt_tokens`; a shape that cannot be hit exactly is reported MISSIZED. One target, a literal 1-token prompt, is MISSIZED and remains untested. What this could have seen is bounded by a thing it measured on the way: the engine is not deterministic at concurrency 1 with caching off and temperature 0. Re-sending one prompt to one arm moves its logprobs by up to 7.6 nats, and the per-shape mean floor runs from 3.4 nats at two tokens to 5e-3 at 2,048. A defect smaller than that floor is invisible here. That nondeterminism is measured and unexplained; it is not a finding about the shapes. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
DeepSeek-V4 on the vLLM plugin crashes with "Memory access fault by GPU" (then EngineDeadError) on a prefill step whose rows are all one token long. It happens with --enforce-eager too; it is not a cudagraph issue. Such a step comes up whenever the last prompt chunk of a request is a single token: - a prompt of 128*m + 1 tokens sent again: vLLM's block-aligned prefix hit leaves one token to compute (897, 1025, 1153, 2049, 8193, ... tokens); - a fresh prompt of 8192*k + 1 tokens with --max-num-batched-tokens 8192: its last chunk is one token; - a fresh 1-token prompt. Root cause: the bridge correctly builds the step as a prefill (is_prefilling), but its indexer split put every leading row of decode length into the paged decode group. The dense prefill group was then empty while _populate_indexer's cu[-1] = max(cu[-1], 1) still claimed one committed row, so IndexerVllm.indexer_score_topk ran _score_topk_prefill on an empty slice and cp_gather_indexer_k_quant_cache gathered through an empty block table (out-of-bounds read). Native ATOM does not crash on this shape: its scheduler runs the 1-token chunk as its own prefill batch and the core indexer scores the row on _score_topk_prefill with its own committed K. Do the same here: _indexer_decode_group ends the paged decode group at the first row that is still prefilling, as a longer row does, so such a row takes the dense prefill path and the dense group is never empty on a prefill-classified step. indexer_score_topk keeps a fallback that skips the dense path on an empty slice. Test: tests/plugin/test_vllm_deepseek_v4_indexer_split.py (CPU). Co-authored-by: Cursor <cursoragent@cursor.com> Signed-off-by: linsun12 <linsun12@amd.com>
linsun12
force-pushed
the
linsun12/up-dsv4-1token-indexer
branch
from
October 12, 2026 20:24
db7e538 to
92a9ab6
Compare
linsun12
marked this pull request as ready for review
October 12, 2026 20:28
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
The new regression tests are excluded from automated unit-test CI and additionally skip without AITER.
1 open finding
What changed in this PR
Fixes DeepSeek-V4 vLLM indexer routing for one-token prefill rows, preventing out-of-bounds GPU memory access.
Changes:
- Routes prefilling rows through the dense indexer.
- Skips dense scoring when no prefill rows remain.
- Adds split and dispatch regression tests.
| File | Description |
|---|---|
atom/plugin/vllm/deepseek_v4_bridge.py |
Uses prefill state when splitting indexer rows. |
atom/plugin/vllm/models/deepseek_v4.py |
Avoids dense scoring on empty prefill slices. |
tests/plugin/test_vllm_deepseek_v4_indexer_split.py |
Tests one-token and mixed-batch routing. |
🧠 Review effort: Balanced
💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| import pytest | ||
| import torch | ||
|
|
||
| pytest.importorskip("aiter", reason="the V4 model module reaches AITER ops") |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This fixes a GPU memory access fault in the vLLM plugin's DeepSeek-V4 path on any prefill step whose rows are all one token long.
Problem
vLLM + ATOM plugin, DeepSeek-V4 (
DeepseekV4ForCausalLM), prefix caching on, also with--enforce-eager: the engine dies withMemory access fault by GPU node-…on all TP ranks, thenEngineDeadError.It happens whenever the last prompt chunk of a request is a single token:
128·m + 1tokens sent again: the block-aligned prefix hit leaves one token to compute (129, 897, 1025, 1153, 2049, 8193, … tokens);8192·k + 1tokens with--max-num-batched-tokens 8192: its last chunk is one token;Repro on
main(a0527c9db): send a 1025-token prompt, then the same prompt again. The first request is answered; the second returns HTTP 500 and the server log showsRoot cause
The bridge builds such a step as a prefill.
build_atom_v4_attention_metadatathen split the indexer rows by length only: the leading run of rows with the first row's length, if<= decode_q, went to the paged decode indexer. When every row is one token long, that is every row, so the dense prefill group was empty — but_populate_indexer'scu[-1] = max(cu[-1], 1)still claimed one committed row.IndexerVllm.indexer_score_topkthen ran_score_topk_prefillon the empty slice and gathered one row through an empty block table: an out-of-bounds read.Fix
_indexer_decode_group(lens, decode_q, is_prefilling): the paged decode group also ends at the first row that is still prefilling. Such a row takes the dense prefill path, as in native ATOM, so the dense group is not empty on these steps. Batches without a prefilling row split exactly as before. (is_prefillingis vLLM's CPUnum_computed < num_promptmask, so this adds no sync.)IndexerVllm.indexer_score_topk: if no prefill rows are left, return the decode result instead of running the dense path on an empty slice.Tests
tests/plugin/test_vllm_deepseek_v4_indexer_split.py: a prefilling row of decode length ends the decode group; no dense call when no prefill rows are left; mixed batches keep their split.tests/plugin(without sglang): same failures asmain(63, unrelated: rtpllm, offload, kv-cache layout, proxy arena layout), plus the new tests passing.rocm/atom-dev:vllm-v0.31.0-nightly_20261012, DeepSeek-V4-Pro-0813, TP8, fp8 KV, prefix caching on,--max-num-batched-tokens 8192:maina0527c9db--enforce-eager)EngineDeadError--enforce-eager: 120 / 120 answered, no fault; default cudagraph: 120 / 120 answered, no faultmain: CI config; this PR: with prefix caching)