Skip to content

[Blog] Taking vLLM Apart: A Practical Guide to Disaggregated Serving - #347

Merged
NickLucche merged 9 commits into
vllm-project:mainfrom
hickeyma:add-disagg-serv-blog
Sep 30, 2026
Merged

[Blog] Taking vLLM Apart: A Practical Guide to Disaggregated Serving#347
NickLucche merged 9 commits into
vllm-project:mainfrom
hickeyma:add-disagg-serv-blog

Conversation

@hickeyma

@hickeyma hickeyma commented Sep 23, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Adds a hands on guide to disaggregated serving in vLLM.

It walks through splitting prefill and decode onto separate instances with a KV connector and moving tokenization, detokenization and tool/reasoning parsing onto a CPU-only frontend using the new /render and /derender
endpoints. It also shows how the two fit together, including prefill reusing conversation state from decode in multi-turn chat and covers who's running this in production and what's still missing.

Includes four figures: the serving pipeline, multi-turn flow, KV transfer cost and ITL tail latency.

cc @NickLucche @sagearc @DarkLight1337 @chaunceyjiang @njhill @vMaroon @hyeongyun0916

@hickeyma
Taking vLLM Apart: A Practical Guide to Disaggregated Serving
ae8f189 
Adds a hands on guide to running disaggregated serving in vLLM v0.31.0+.
It covers splitting prefill and decode with KV connectors, moving
tokenization and parsing off the GPU with the new /render and /derender
frontend and how the pieces fit together. It also looks at who's running
this in production and what's still neede to be done.

Includes pipeline, multi-turn, transfer cost and ITL tail figures.

Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com>
@hickeyma
hickeyma force-pushed the add-disagg-serv-blog branch from 7514cd6 to ae8f189 Compare September 23, 2026 17:08
chaunceyjiang
chaunceyjiang reviewed Sep 24, 2026
View reviewed changes

@chaunceyjiang chaunceyjiang left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks~ LGTM.

@hickeyma
Small updates after review
cc0b73a 
Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com>
@DarkLight1337

DarkLight1337 commented Sep 24, 2026

Copy link
Copy Markdown
Member

You should reword "Tail latency gets boring," it screams AI prose. Also I feel that Fig. 2 would be better expressed as a line graph.

hickeyma added a commit to hickeyma/vllm-project.github.io that referenced this pull request Sep 24, 2026
@hickeyma
Update after review
c37a0e7 
Review comment:
- vllm-project#347 (comment)
@hickeyma

hickeyma commented Sep 24, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

You should reword "Tail latency gets boring," it screams AI prose. Also I feel that Fig. 2 would be better expressed as a line graph.

Thanks @DarkLight1337 for the review, updated and ready for review again.

@hickeyma
Update after review
fb01e80 
Review comment:
- vllm-project#347 (comment)

Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com>
@hickeyma
hickeyma force-pushed the add-disagg-serv-blog branch from c37a0e7 to fb01e80 Compare September 24, 2026 11:24
NickLucche
NickLucche reviewed Sep 25, 2026
View reviewed changes

@NickLucche NickLucche left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @hickeyma ! I think we can shorted the PD part a bit and focus more on the gpu-less FE :)

Comment thread _posts/2026-09-20-disaggregated-serving-guide.md Outdated
Comment thread _posts/2026-09-20-disaggregated-serving-guide.md Outdated
Comment thread _posts/2026-09-20-disaggregated-serving-guide.md Outdated
Comment thread _posts/2026-09-20-disaggregated-serving-guide.md Outdated
Comment thread _posts/2026-09-20-disaggregated-serving-guide.md Outdated
Comment thread _posts/2026-09-20-disaggregated-serving-guide.md Outdated
Comment thread _posts/2026-09-20-disaggregated-serving-guide.md Outdated
Comment thread _posts/2026-09-20-disaggregated-serving-guide.md Outdated
Comment thread _posts/2026-09-20-disaggregated-serving-guide.md Outdated
@hickeyma
Review addressing trim P/D and point multi-turn at the bidirectional …
1be9327 
…post

  - Reframe the throughput claim around goodput instead of quoting the
    docs line that says P/D doesn't help throughput
  - Lead with the MoRI-IO and llm-d results and cut the L40S run down
    to a short caveat about slow transfers
  - Shrink the multi-turn section to a summary plus a link to the
    bidirectional KV transfer post, keeping the reasoning-model warning
  - Mention KV offloading/shared KV and fabric checks in the table
  - Drop the redundant kv_load_failure_policy from the examples
  - Note the DP=2 baseline isn't the strongest collocated setup
  - Remove the two figures that are no longer used

Updates based on review comment:

- vllm-project#347 (review)

Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com>
hickeyma added a commit to hickeyma/vllm-project.github.io that referenced this pull request Sep 28, 2026
@hickeyma
Extend the GPU-less frontend
182b474 
Explain why tokens in/tokens out matters beyond saving CPU: routing
on real token IDs, exact tokens for RL and eval and multimodal
preprocessing on the render tier. Add a streaming derender example
and a short section on deploying and sizing the render tier. The
renderer_num_workers advice moves up there from "What's still left".

Review comment:
- vllm-project#347 (review)

Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com>
@hickeyma

hickeyma commented Sep 28, 2026

Copy link
Copy Markdown
Contributor Author

Thanks @hickeyma ! I think we can shorted the PD part a bit and focus more on the gpu-less FE :)

Thanks @NickLucche for the review and feedback. Trimmed the P/D part. Multi-turn is now a short summary pointing to #345 and the L40S benchmark is just a caveat. The frontend section got updated with: why tokens in/tokens out matters beyond CPU, a streaming derender example and how to deploy and size the render tier.

Let me know what you think.

@hickeyma
hickeyma requested a review from NickLucche September 28, 2026 10:19
@hickeyma
Extend the GPU-less frontend
9b1ec97 
Explain why tokens in/tokens out matters beyond saving CPU: routing
on real token IDs, exact tokens for RL and eval and multimodal
preprocessing on the render tier. Add a streaming derender example
and a short section on deploying and sizing the render tier. The
renderer_num_workers advice moves up there from "What's still left".

Review comment:
- vllm-project#347 (review)

Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com>
@hickeyma
hickeyma force-pushed the add-disagg-serv-blog branch from 182b474 to 9b1ec97 Compare September 28, 2026 10:26
sagearc
sagearc reviewed Sep 28, 2026
View reviewed changes

@sagearc sagearc left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @hickeyma, very nice!

Comment thread _posts/2026-09-20-disaggregated-serving-guide.md Outdated
Comment thread _posts/2026-09-29-disaggregated-serving-guide.md
@hickeyma
Sharpen the intro and open up on the problem better
cf71714 
Review comments:
- vllm-project#347 (review)

Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com>
@hickeyma
hickeyma requested a review from sagearc September 28, 2026 11:08
NickLucche
NickLucche approved these changes Sep 29, 2026
View reviewed changes

@NickLucche NickLucche left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@hickeyma
Improve the opening paragraph
4cb1c63 
Removed unnecessary dividing line from first section and
updated the final sentnece in the opening to read better.

Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com>
@hickeyma
hickeyma force-pushed the add-disagg-serv-blog branch from 30568f5 to 4cb1c63 Compare September 29, 2026 08:48
@hickeyma
Update the date to todays release dat
4cdc756 
Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com>
@NickLucche
NickLucche merged commit f9792a2 into vllm-project:main Sep 30, 2026
2 checks passed
@hickeyma
hickeyma deleted the add-disagg-serv-blog branch September 30, 2026 10:55

This branch was successfully deployed

1 active deployment
Preview — be1f4f9b Deployed Sep 30, 2026 by vercel[bot]
View deployment
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@NickLucche NickLucche NickLucche approved these changes
@DarkLight1337 DarkLight1337 DarkLight1337 approved these changes
+2 more reviewers
@chaunceyjiang chaunceyjiang chaunceyjiang left review comments
@sagearc sagearc sagearc approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@hickeyma @DarkLight1337 @NickLucche @chaunceyjiang @sagearc