feat(sm120): vendor DeepGEMM-sm120 v0.1.0 device layer - #10
Open
lucifer1004 wants to merge 4 commits into
Open
lucifer1004 wants to merge 4 commits into
lucifer1004 wants to merge 4 commits into
Conversation
lucifer1004
force-pushed
the
vendor/sm120-deepgemm-sm120-v0.1.0
branch
from
September 16, 2026 09:09
89c5349 to
180f7e1
Compare
Replace the in-tree SM120 device headers with byte-identical vendored copies from lucifer1004/DeepGEMM-sm120 v0.1.3 (merged superset of both lineages). Marker-free vendoring: no manifests, tooling, or header comments enter this fork; provenance and drift control live on the canonical repo's side (per-tag manifest, check_vendor.py --fork, scheduled downstream watcher). Signed-off-by: Zihua Wu <13583761+lucifer1004@users.noreply.github.com>
The vendored DeepGEMM-sm120 v0.1.0 device layer widens several kernel ABIs; adapt the SM120 host glue with neutral mappings: - sm120_bf16_gemm: pass the epilogue operator as a runtime argument (EpilogueArgs, same marshalling as SM100) and split the old int64_t stride_cd_m/stride_cd_batch into u32 stride_d_m/stride_c_m/ stride_d_batch; stride_c_m=0 keeps C sharing D's row stride. - sm120_fp8_fp4_gemm_1d1d: pass shape_cd_m (= shape_m), the runtime epilogue argument, and stride_c_m (= 0) after stride_cd_batch. - sm120_split_k_reduce: the kWithAccumulation template flag is gone; instantiate the 3-arg template and map accumulation onto D in place to the new runtime C operand (gmem_c = D with D's own strides, with_alpha = false). - Masked m-grouped GEMM (bf16 + fp8/fp4): disable the TMA-store epilogue (swizzle_cd_mode = 0) when m % BLOCK_M != 0. The vendored kernels dropped the old masked-boundary scalar fallback, so a boundary tile's full-tile TMA store would spill into the next group's rows; the scalar store path carries the masked row bounds. This mirrors the proven nv_dev-lineage launchers. Signed-off-by: Zihua Wu <13583761+lucifer1004@users.noreply.github.com>
The vendored v0.1.0 GEMM kernels have no boundary-tile scalar fallback in the TMA-store epilogue, and the k-grouped D descriptor is one flat 2D map over all groups, so TMA can only clamp at the outermost M dim: with m % BLOCK_M != 0, a group's partial tail M tile TMA-store spills into the next group's slab. Mirror the masked-launcher guard (and the nv_dev-lineage reference glue) in both k-grouped launchers: force swizzle_cd_mode = 0 (group-bounded scalar store path) for unaligned m. Repro evidence (num_sms=2, groups=2, m=48, n=128, ks=[8192, 128]; the asymmetric K makes group 0's tile store last, so the spill overwrites group 1's head rows deterministically): - before: bf16/bf16-out 5/5 iterations contaminated (d[1][0:16] = 0.0, expected 768); fp8/fp32-out 5/5 contaminated (d[1][0:16] = 0.0, expected 771) - bf16/fp32-out was already clean (0/5): the bf16 kernel gates its TMA-store epilogue on sizeof(cd_dtype_t) <= 2, so fp32 outputs take the scalar path; the fp8 kernel has no such dtype gate - after: all three configurations 0/5 contaminated Adds test_sm120_kgroup_unaligned_m_tail_tile_isolation to tests/test_sm120_bf16.py (fp32 and bf16 outputs) and tests/test_sm120_fp8_fp4.py (NT and TN layouts), asserting per-group correctness including slab head rows plus flat-storage guard regions. Full SM120 suite: 48 passed (46 baseline nodes + 2 new). Memcheck on the two affected files: 20 passed, 0 sanitizer errors. Signed-off-by: Zihua Wu <13583761+lucifer1004@users.noreply.github.com>
lucifer1004
force-pushed
the
vendor/sm120-deepgemm-sm120-v0.1.0
branch
3 times, most recently
from
September 16, 2026 23:15
2cf7301 to
0aecffa
Compare
Author
|
Force-pushed ( |
FP8 paged MQA logits gains PAGE_KV=32 (BLOCK_KV derived as min(PAGE_KV, 64), mirroring the FP4 sibling). Device-only update: this fork's host launcher still restricts paged FP8 to page 64, so page32 stays inert until host glue opts in (vllm-project#14). Validated on sm_120a: test_sm120_mqa.py + test_sm120_fp8_fp4.py 23/23 passed from a fresh JIT cache (only the pre-existing test_filter collection quirk remains, deepseek-ai#446). Signed-off-by: Zihua Wu <13583761+lucifer1004@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Replace this fork's in-tree SM120 device headers with byte-identical vendored copies of lucifer1004/DeepGEMM-sm120
v0.1.4— a standalone single-source-of-truth repository for the SM120 device layer, shared with the nv_dev-lineage SM120 port (deepseek-ai#447). Fixes and new SM120 kernels will land there once and propagate to each fork mechanically, instead of diverging per fork.tools/check_vendor.py --fork <path>(both directions), and a scheduled watcher that monitors registered downstream branches daily and files an issue when a tree matches no release tag.impls/sm120_fp8_fp4_sparse_mqa_logits.cuh,impls/sm120_padding.cuh,scheduler/sm120_gemm.cuh); the vendored files are a strict superset of this fork's versions (both lineages' fixes merged).Host adaptation details
sm120_bf16_gemm/sm120_fp8_fp4_gemm_1d1d: pass the new runtime args with neutral values (EpilogueIdentity{},stride_c_m = 0,shape_cd_m = m); template additions are trailing-defaulted.swizzle_cd_mode = 0whenm % BLOCK_M != 0. This is required with the vendored kernels: they dropped the old boundary-tile scalar fallback (superseded by the nv_dev-lineage mechanism), so a boundary tile's full-tile TMA store would spill into the next group's rows. The scalar store path carries the masked row bounds (row_is_valid). Mirrors the proven nv_dev-lineage glue.sm120_split_k_reduce: the removedkWithAccumulationtemplate flag maps to the runtime C operand (gmem_c = Dwhen accumulating, elsenullptr).BLOCK_KV→PAGE_KVrename is semantics-preserving here (the fork already passes the real page size); the vendored kernels add within-page TMA offsets enabling PAGE_KV 128/256.New in the vendored layer (available, inert until host glue opts in)
block_scale).tensor_map_replace_global_dim_in_smemrelocated intocommon/sm120_utils.cuh(main-lineageptx/tma.cuhlacks it).Follow-up folded in (k-grouped guard)
m % BLOCK_M != 0could spill a boundary tile's TMA store into the next group's output slab (reproduced 5/5: next group's head rows zeroed). Both k-grouped launchers now force the scalar store epilogue for unaligned m, mirroring the nv_dev-lineage glue; regression teststest_sm120_kgroup_unaligned_m_tail_tile_isolationadded to the bf16 and fp8/fp4 suites. This exposure predates the vendor migration (the old kernels had no k-grouped boundary fallback either).Validation (sm_120a)
tools/check_vendor.py: PASS (tree byte-identical to v0.1.4).sm_120aagainst both base lineages and asserts on SASS/PTX opcodes (HMMA.16816,QMMA.*.SF.*,block_scale).Merge-base friction
This branch merges into deepseek-ai/DeepGEMM
mainwith zero conflicts (dry-run); all SM120-exclusive content stays in dedicated files, andVENDOR-sm120.json/tools/check_vendor.pyare additive.Follow-up folded in (v0.1.4 re-vendor)
ea3b2f8): the FP8 paged MQA logits kernel now derivesBLOCK_KV = min(PAGE_KV, 64), adding PAGE_KV=32 support (DSv4.1 indexer caches mix 64- and 32-state pages; see [SM120] Support 32-state pages in FP8 paged-MQA logits #14). Device-only: this fork's host launcher still restricts paged FP8 to page 64, so page32 remains inert until host glue opts in — that is [SM120] Support 32-state pages in FP8 paged-MQA logits #14's rebased scope. Re-verified on sm_120a from a fresh JIT cache:test_sm120_mqa.py+test_sm120_fp8_fp4.py23/23 passed.