fix(serve): stop hardcoding vLLM GPU memory utilization (EAI-7353) - #153
Open
rominf wants to merge 3 commits into
Open
fix(serve): stop hardcoding vLLM GPU memory utilization (EAI-7353)#153rominf wants to merge 3 commits into
rominf wants to merge 3 commits into
Conversation
vLLM sizes its KV cache as a fraction of the device's TOTAL VRAM, not of free VRAM and not scaled to the model. The adapter hardcoded 0.80 with no way to override it, so a 0.5B model reserved ~200 GB on a large card and left nothing for anything else. Drop the hardcoded value entirely: when the user asks for nothing, pass no flag and let vLLM apply its own default, rather than owning a number that silently diverges from upstream. Add `rocm serve --gpu-memory-utilization <FRACTION>` for the cases that do need a specific share, routed through the engine recipe's required_flags exactly like --tool-call-parser, so there is no protocol change and other engines warn-and-ignore. An explicit value is validated as 0 < v <= 1 and fails the command on bad input instead of surfacing later as a vLLM argparse error. Extract vllm_serve_args as a pure function so the spawned argv is assertable in tests, and document the fraction-of-total-VRAM footgun. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
Address review feedback on the utilization change: The vLLM doc claimed rocm-cli passes no `--gpu-memory-utilization` at all, which is wrong when a model's catalog recipe authors one. Name both sources and state that an explicit user value takes precedence. The E2E drain-wait comment claimed its 90%-of-total free-VRAM cap sat "deliberately at vLLM's own default utilization". That hardcodes an upstream number this change argues against owning, and is optimistic: a device holding display memory can never report 90% free, so the wait now burns its full deadline where it used to have slack. Say so plainly — the wait is best-effort, so the cost is time, not correctness. Also give the range-validation error the same `e.g. 0.5` example the parse-failure branch already carries. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
rominf
force-pushed
the
EAI-7353-fix-gpu-memory-utilization
branch
from
July 31, 2026 08:28
9881e67 to
4091ace
Compare
Assert that an existing non-vLLM engine recipe survives the override untouched, mirroring the sibling tool-call-parser test. Point the vLLM TheRock acceptance runbook at the renamed test so the documented command runs, and describe the memory reservation as vLLM's default fraction rather than a hardcoded 0.80 that goes stale when upstream moves. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
rominf
marked this pull request as ready for review
July 31, 2026 10:15
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Serving a 0.5B model on a large-VRAM machine reserved roughly 200 GB of VRAM.
The vLLM adapter always passed
--gpu-memory-utilization 0.80, and vLLM reserves that fraction of total device VRAM for its KV cache regardless of how small the model is. There was no way for a user to override it.Changes
--gpu-memory-utilizationargument is passed at all, so vLLM's own default applies. Chosen over hardcoding vLLM's current 0.9 so the value cannot drift out of sync with upstream.rocm serve --gpu-memory-utilization <FRACTION>so the value can be set explicitly. The flag is vLLM-scoped and follows the existing--tool-call-parserprecedent: same warn-and-ignore behavior on other engines, routed throughEngineRecipeHint.required_flags, so the engine protocol is unchanged.0 < value <= 1with an error that names the flag.vllm_serve_argshelper so the launch argv is unit-testable.docs/vllm.md: drop the now-stale "0.80 by default for display/WSL headroom" rationale, document the flag, and state that it is a fraction of total — not free — VRAM.docs/testing.md: point the vLLM TheRock acceptance runbook at the renamed test so the documented command runs.Why an engine-scoped flag
An engine-agnostic knob was considered and rejected: llama.cpp/lemonade has no fractional-VRAM-reservation concept, and multi-engine wrappers in the wider ecosystem (Xinference, LocalAI, RamaLama, KubeAI, Jan, llama-swap) all decline to unify this for the same reason.
gpu-memory-utilizationis also the near-universal spelling (vLLM, TGI's equivalent, Ray Serve, KServe, KubeAI, NIM).Behavior change
On machines where the old 0.80 was providing display or WSL headroom, the effective reservation now rises to vLLM's default. Passing
--gpu-memory-utilization 0.8restores the previous behavior.Related: the e2e free-VRAM floor in
tests/e2e-cucumber/tests/e2e/serving_steps.rscaps at 90% of total, which now sits exactly at vLLM's default instead of having slack. That is a tight tolerance on small cards and may deserve a deliberate margin in a follow-up.Test plan
resolve_model_surfaces_conservative_vram_default→resolve_model_omits_gpu_memory_utilization_default) acrossengines/vllmandapps/rocm: flag absent (no argument emitted), flag present (value reaches argv), and invalid input (0,1.5,-0.2,abc,NaN,inf, empty).expectations.tomlrows reference this ticket, so none needed narrowing.Risk: low for the code path; medium for users who relied on the implicit 0.80 headroom, which the docs now cover.