Skip to content

docs: add validated Jetson Thor DFlash2 serving setup - #164

Open
marcodelpin wants to merge 1 commit into
z-lab:mainfrom
marcodelpin:docs/jetson-thor-dflash2-256k
Open

docs: add validated Jetson Thor DFlash2 serving setup#164
marcodelpin wants to merge 1 commit into
z-lab:mainfrom
marcodelpin:docs/jetson-thor-dflash2-256k

Conversation

@marcodelpin

Copy link
Copy Markdown

Summary

  • document a tested Qwen3.8-27B DFlash 2 configuration for the 128 GB Jetson AGX Thor Developer Kit
  • pin the validated vLLM, NVIDIA PyTorch, FlashInfer, CUDA architecture, weight, and KV-cache formats
  • show explicit KV allocation for a 262,144-token context limit
  • clarify how --kv-cache-memory-bytes and --gpu-memory-utilization interact in the tested vLLM revision

Validation

The upstream package workflow was reproduced locally on Python 3.12.13:

  • built the sdist and wheel
  • passed twine check --strict
  • installed the generated wheel
  • imported dflash
  • smoke-tested dflash, dflash generate, and dflash benchmark

The documented configuration was also checked on a live 128 GB Jetson AGX Thor:

  • vLLM V2 model runner active
  • DFlash2Qwen3ForCausalLM draft model loaded
  • text generation and health check successful
  • FP8 E4M3 KV cache on sm_110
  • max_model_len=262144
  • 453,669 tokens of KV capacity, reported as 1.73 concurrent sequences at the 262,144-token limit

The 256K statement describes allocated KV capacity; it does not claim that a 256K-token prompt was executed.

AI assistance disclosure

This documentation change was AI-assisted and has not yet received independent human code review. The commands, package checks, runtime configuration, and reported capacity were verified against the tested hardware and final diff.

@marcodelpin
marcodelpin marked this pull request as ready for review August 27, 2026 19:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant