feat(sm70): support Volta / Tesla V100 via torch 2.10 cu128 downgrade - #253
Open
itongxiaojun wants to merge 4 commits into
Open
feat(sm70): support Volta / Tesla V100 via torch 2.10 cu128 downgrade#253itongxiaojun wants to merge 4 commits into
itongxiaojun wants to merge 4 commits into
Conversation
itongxiaojun
commented
Aug 28, 2026
added 4 commits
August 28, 2026 20:48
Torch 2.11 (all CUDA builds) dropped sm_70 kernels, so upstream FreeToken cannot run on Volta GPUs. This branch: - lowers the torch floor from 2.11 to 2.10 in [build-system] and deps; torch 2.10.0+cu128 is the last line shipping sm_70 cubins and runs on driver r570 (CUDA 12.8) with no driver upgrade or reboot - repoints the uv torch index from cu130 to cu128 - switches flashinfer extra from [cu13] to [cu12] and drops sglang-kernel from accel (no cu128 wheel for 0.4.5; both target sm_80+ anyway) - documents the sm_70 build path and the V100-16GB --memory-ratio tuning Validated on Tesla V100-SXM2-16GB + driver 570: source build (nvcc 12.9), Qwen3.6-35B-A3B-FP8 FTW conversion, and end-to-end /v1/chat/completions inference via the pure-Triton fallback path.
Document what surfaced while getting the FreeToken Desktop to run on a Tesla V100: - base package only: flashinfer/sglang fused kernels are sm80+/sm100+ and ship CUDA 13 builds, so on Volta they break the engine (common_ops, libnvrtc.so.13, no kernel image); uninstall them if present - in-place cu130 -> cu128 venv upgrades must drop the leftover nvidia/cu13 libs - applies to the Desktop engine venv too (~/.freetoken/venv, ~/.freetoken-cli/.venv)
Measured on Qwen3.6-35B-A3B-FP8 (V100-16GB, driver r570) while wiring the engine into an OpenAI-compatible agent harness (DSH llm-pi-ai): - default --memory-ratio 0.9 OOMs on the ~8K-token system-prompt prefill - --memory-ratio 0.6 fits prefill but collapses decode to ~0.1-0.4 tok/s - --memory-ratio 0.75 --moe-cache-rate 0.2 --num-tokens 16384: ~4.1 GiB free after init, 8099-token prefill at ~800 tok/s, decode ~20 tok/s - streaming is OpenAI-compatible (progressive chunks + [DONE]); a single empty chunk then a hang is queue starvation, not a wire-format issue - OpenAI-compatible gateway clients need api: openai-completions, a /v1 baseURL and an apiKeyEnv credential even when the gateway ignores the key
The prebuilt kernel-cache fatbin defaulted to 8.0+ arches, so a Volta GPU had no usable SASS image and fell back to JIT compilation on first use. Add 7.0 to the defaults (mirroring the sm_80 addition) so a cu128 kernel-cache wheel carries sm_70 cubins for the sm70 build.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.