-
Notifications
You must be signed in to change notification settings - Fork 2.1k
Support AProjQ4 GGUFs: Q4_K dense attention projections; Metal, ROCm and CUDA performance improvements; SSD streaming support for speed-bench; quality-of-life improvements; bugfixes #621
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Closed
Closed
Changes from all commits
Commits
Show all changes
189 commits
Select commit
Hold shift + click to select a range
3797099
Support AProjQ4 GGUFs: Q4_K dense attention projections
GiorgioOppo f7695ea
Metal streaming: opt-in F_NOCACHE descriptor for expert preads
GiorgioOppo 8f5a745
Metal streaming: split expert slab preads to raise NVMe queue depth
GiorgioOppo cbe484b
CUDA/ROCm: dense Q4_K matmul for the AProjQ4 attention projections
GiorgioOppo 4a7d3aa
Accelerate Q4 CUDA prefill and add GGUF requantization
GiorgioOppo ef227db
Optimize Q4 CUDA prefill paths
GiorgioOppo fd1306a
Reuse Q4 activations across QKV projections
GiorgioOppo cdbcde3
Optimize Metal Q4 decode and SSD streaming
GiorgioOppo 9720179
Stabilize Q4 CUDA paths and direct GGUF requantization
GiorgioOppo 194b801
Enable DSpark SSD streaming across GPU backends
GiorgioOppo c5af879
Optimize DSpark inference across GPU backends
GiorgioOppo c14523e
Remove DSpark scheduler
GiorgioOppo b6eeda5
Optimize DSpark exact verification and Q4 GPU paths
GiorgioOppo 3a383e9
Snapshot GPU optimizations before upstream merge
GiorgioOppo da17afc
Optimize speculative decode and GB10 CUDA kernels
GiorgioOppo db331f4
cuda: optimize AProjQ4 decode on GB10
GiorgioOppo 40fa579
Optimize DSpark verification and GB10 GPU paths
GiorgioOppo d89028f
cuda: preserve canonical Q4 GB10 numerics
GiorgioOppo fc33759
cuda: harden Q4 K1024 persistent path
GiorgioOppo 0976f67
cuda: add opt-in canonical Q8_1 producer fold
GiorgioOppo 5363e0b
metal: add opt-in M1 IQ2 SSD mid-only producer
GiorgioOppo f9535a4
metal: fix direct-RHS threadgroup memory sizing
GiorgioOppo 41b8265
metal: merge IQ2 selected router and shared work
GiorgioOppo 61e35e2
metal: index live IQ2 SSD cache entries
GiorgioOppo 801be18
metal: pack routed prefill maps without losing duplicates
GiorgioOppo 753b82f
metal: group IQ2 SSD prefill address matmuls
GiorgioOppo 6ed2199
metal: isolate stream state and persistent exact-row cache
GiorgioOppo 6d39838
metal: add opt-in resident Q4 session overlap
GiorgioOppo 7bdbff7
metal: add opt-in Q4 SSD session union
GiorgioOppo 8498c81
cuda: batch selected expert SSD uploads
GiorgioOppo ed1cab3
cuda: overlap selected expert I/O with shared compute
GiorgioOppo 95c78d4
cuda: add raw fused IQ2 Q2 MoE MMQ path
GiorgioOppo 7c5b9a3
cuda: group IQ2 SSD prefill MMQ
GiorgioOppo f49cc0e
cuda: model persistent expert cache planning
GiorgioOppo d8958e1
cuda: allocate persistent expert cache arena
GiorgioOppo 34d365d
cuda: separate selected cache binding ownership
GiorgioOppo 7948478
cuda: cache streaming experts persistently
GiorgioOppo ad6ea5f
cuda/mmq: persist grouped Q8_1 scratch
GiorgioOppo db41283
cuda: add bit-exact Q8 HC split experiment
GiorgioOppo 448ce12
metal: promote IQ2 SSD grouped prefill defaults
GiorgioOppo 377b1ae
metal: skip redundant expert readahead during prefill
GiorgioOppo f34265b
rocm: add dense Q4_K projection support
GiorgioOppo 28e6a6e
rocm: add tiled Q4 prefill path
GiorgioOppo a7d41d4
rocm: guard Metal-only Q4 overlap policy
GiorgioOppo f0c70f9
rocm: default Q4 TILE8 and group attention decode
GiorgioOppo e3e0a7e
backend: add Metal Q4 prefill path and review fixes
GiorgioOppo 7e802d6
Update ENVIRONMENT_VARIABLES.md
GiorgioOppo f94e1cf
Update ENVIRONMENT_VARIABLES.md
GiorgioOppo b91bb77
Update ENVIRONMENT_VARIABLES.md
GiorgioOppo 7bd98a9
Update metal/moe.metal
GiorgioOppo 68a8ef9
docs: document all environment variables
GiorgioOppo 8dd4bf5
Update CONTRIBUTING.md
GiorgioOppo c6f02bf
Update Makefile
GiorgioOppo c3b29fd
Update Makefile
GiorgioOppo d9d1c81
Update Makefile
GiorgioOppo af13247
Update Makefile
GiorgioOppo d545a1f
Update metal/moe.metal
GiorgioOppo 1ab8a2f
Update metal/moe.metal
GiorgioOppo 28623a3
Update metal/moe.metal
GiorgioOppo 0d774a9
Update metal/moe.metal
GiorgioOppo 03b97c0
Update cuda/mmq/test/test_mmq_parity.cu
GiorgioOppo de2efc9
Update cuda/mmq/ds4_mmq.cu
GiorgioOppo 0e34a59
Update cuda/mmq/ds4_mmq.cu
GiorgioOppo 86b1f61
perf(metal): specialize fixed-shape Q8 attention output
GiorgioOppo 2262f13
perf(metal): use gathered attention for raw-only decode
GiorgioOppo 15985cb
bench: support SSD-streamed Metal decode comparisons
GiorgioOppo 758a31f
docs: record M1 raw-gathered decode results
GiorgioOppo 60770f7
perf(metal): prune unused prefill indexer queries
GiorgioOppo 39e43d9
perf(cpu): accelerate batched Q4_K matmul
GiorgioOppo 072cb04
perf(cuda): share Q8_1 activations across Q4_K prefill pairs
GiorgioOppo b82b434
perf(rocm): fuse paired Q4_K projection launches
GiorgioOppo dd9bfae
docs: analyze Q4_K complexity and optimization opportunities
GiorgioOppo 51f9b9e
docs: add detailed PR 621 review summary
GiorgioOppo 859993c
docs: refresh environment variable inventory
GiorgioOppo 5589ac1
fix(cuda): harden MMQ review edge cases
GiorgioOppo f762dba
test(cuda): harden persistent Q8 parity setup
GiorgioOppo fdfd7df
fix(cuda): reconcile Q8 oracle allocation diagnostics
GiorgioOppo dee75d4
build: preserve targets across ROCm merge
GiorgioOppo 5532f5d
docs: document ROCm MXFP4 environment controls
GiorgioOppo 8d84bda
Update metal/moe.metal
GiorgioOppo 59636df
metal: fuse router finalize during SSD decode
GiorgioOppo d24c098
chore: ignore local speed benchmark outputs
GiorgioOppo fffc469
docs: remove detailed PR 621 summary
GiorgioOppo 3aa7b2d
metal: fix batched Q8 static attention output
GiorgioOppo ec2f27c
cuda: centralize Q8 fold registry bridge ABI
GiorgioOppo fcea9f4
bench: support SSD-streamed Metal A/B runs
GiorgioOppo b878513
metal: speed up SSD-streamed IQ2 prefill
GiorgioOppo 33f9865
Update metal/moe.metal
GiorgioOppo 19c1d2c
Update metal/moe.metal
GiorgioOppo 77f41ec
Update metal/moe.metal
GiorgioOppo 31a97a9
Update metal/moe.metal
GiorgioOppo 0c81342
Update metal/moe.metal
GiorgioOppo e086800
metal: fix grouped address-MM unroll integration
GiorgioOppo db6db18
test(mmq): report mismatch magnitudes in the fused-raw parity case
adamlawi 9aec14a
metal: unroll fixed simdgroup matrix loops
GiorgioOppo 2e5db89
Merge pull request #10 from adamlawi/parity-diag-magnitudes
GiorgioOppo 94a763d
Update metal/moe.metal
GiorgioOppo 654da4c
Update metal/moe.metal
GiorgioOppo 378f718
Update metal/moe.metal
GiorgioOppo 0442c93
Update cuda/mmq/test/test_mmq_parity.cu
GiorgioOppo ea76e2a
Update metal/moe.metal
GiorgioOppo a73fe10
Update metal/moe.metal
GiorgioOppo f2cdaf2
Update metal/moe.metal
GiorgioOppo e80ba77
Update metal/moe.metal
GiorgioOppo 5ff3e2b
Update metal/moe.metal
GiorgioOppo 988f8d9
Update metal/moe.metal
GiorgioOppo 4f34ec7
Update metal/moe.metal
GiorgioOppo 00bf2d0
metal: fix invalid loop unroll integration
GiorgioOppo a1e326f
Update metal/moe.metal
GiorgioOppo 8e41f36
Update metal/moe.metal
GiorgioOppo 39bbef3
Update metal/moe.metal
GiorgioOppo 5dced7b
indexer: add Q4_K quantization and Metal support
GiorgioOppo 71c9fed
metal: centralize stable SiLU and repair MoE stores
GiorgioOppo 9d3e719
metal: speed up IQ2_XXS MoE prefill
GiorgioOppo 24ea7be
metal: share Q4_K scale metadata in exact-N prefill
GiorgioOppo f639f48
metal: optimize resident Q4 attention output
GiorgioOppo 0e80b90
metal: accelerate resident Q4 attn_q_b prefill
GiorgioOppo 3f59fcc
cuda, rocm: accelerate resident Q4 attn_q_b prefill
GiorgioOppo bd92e16
metal: add transient F16 q_b prefill acceleration
GiorgioOppo 8dbfa39
cuda, rocm: add transient F16 q_b prefill acceleration
GiorgioOppo 49acc2e
tests, docs: cover transient Q4 q_b prefill policies
GiorgioOppo 759605f
docs: remove obsolete PR621 complexity analysis
GiorgioOppo a490240
perf(gpu): optimize Q4 resident prefill across backends
GiorgioOppo 750b5fc
Optimize greedy token decode across GPU backends
GiorgioOppo 1d3fec3
bench(metal): isolate resident Q4 decode pair kernels
GiorgioOppo bdf2829
fix(metal): make IQ2 LUT staging geometry-safe
GiorgioOppo 531e5bb
perf(metal): cull inactive IQ2 prefill SIMDgroups
GiorgioOppo 86eb835
bench(metal): isolate resident IQ2 MoE prefill
GiorgioOppo 8f6718e
perf(rocm): add opt-in IQ2 prefill wave culling
GiorgioOppo 5789864
bench(gpu): isolate resident IQ2 MoE prefill
GiorgioOppo eec016c
perf(metal): cull inactive Q4 prefill SIMDgroups
GiorgioOppo 301b9e3
perf(rocm): gate SSD Q4 K1024 tile4 on residency
GiorgioOppo cd25137
bench(rocm): isolate resident Q4 prefill kernels
GiorgioOppo 06ed164
perf(rocm): add opt-in gfx1151 WMMA prefill attention
GiorgioOppo 0539d76
docs(rocm): document prefill experiment controls
GiorgioOppo 841c72f
fix(cuda): gate Q4 persistent kernel on residency
GiorgioOppo 149d1aa
bench(cuda): isolate resident Q4 prefill kernels
GiorgioOppo ab6486f
perf(rocm): add compressed Q4 WMMA prefill kernel
GiorgioOppo 53d2576
bench(rocm): cover Q4 attention output WMMA
GiorgioOppo 01ff4ee
perf(cuda): eliminate Q4 prefill output passes
GiorgioOppo bdd28e3
bench(cuda): isolate grouped Q4 attention prefill
GiorgioOppo 8c1d88f
perf(metal): stream transient q_b F16 from SSD
GiorgioOppo 6962d61
perf(metal): write small prefill attention directly
GiorgioOppo 21d5b25
perf(metal): share q4 prefill rhs across qkv projections
GiorgioOppo 5d2e080
perf(rocm): quantize q4 prefill rhs per wave32
GiorgioOppo 5a7505a
perf(cuda): submit grouped q4 prefill in one grid
GiorgioOppo 278ee38
docs: register q4 prefill kernel controls
GiorgioOppo 6ab475b
Update rocm/ds4_rocm_attention.cuh
GiorgioOppo fc76152
Update rocm/ds4_rocm_attention.cuh
GiorgioOppo 7a69780
Update rocm/ds4_rocm_attention.cuh
GiorgioOppo 38d8ec5
Update rocm/ds4_rocm_attention.cuh
GiorgioOppo 9ee2beb
Update rocm/ds4_rocm_indexer.cuh
GiorgioOppo 1ca0684
Update rocm/ds4_rocm_matmul.cuh
GiorgioOppo 4c37ac8
Update rocm/ds4_rocm_matmul.cuh
GiorgioOppo 7ab7fb9
Update rocm/ds4_rocm_norm_rope.cuh
GiorgioOppo 40f7681
Update rocm/ds4_rocm_q4.cuh
GiorgioOppo e9bf171
Update rocm/ds4_rocm_q4.cuh
GiorgioOppo 1d42ce4
Update rocm/ds4_rocm_q4.cuh
GiorgioOppo ccb2cf5
Update speed-bench/.gitignore
GiorgioOppo 1ff1e15
Update .gitignore
GiorgioOppo 3c8fbf3
Remove dead code across inference backends
GiorgioOppo 701d4d1
Update metal/moe.metal
GiorgioOppo e6ee504
Update metal/moe.metal
GiorgioOppo 441b55f
Update metal/moe.metal
GiorgioOppo a01b145
Update rocm/ds4_rocm_attention.cuh
GiorgioOppo ac319c1
Optimize ROCm DSpark verification prefill
GiorgioOppo 1f88ece
Optimize CUDA persistent expert planning
GiorgioOppo dbc4eb8
Optimize Metal IQ2 top-8 MoE prefill
GiorgioOppo 59a174f
Fix ROCm Q4 attention and MMQ build regressions
GiorgioOppo 6a51d69
Optimize ROCm Q4 prefill WMMA kernels
GiorgioOppo 5dd2e3c
Optimize CUDA Q4 prefill with 16-warp MMQ
GiorgioOppo 3504a3f
Extend CUDA Q4 16-warp MMQ with Stream-K
GiorgioOppo 58cc2f4
Enable ROCm Q4 prefill WMMA by default
GiorgioOppo 95f4c3c
Coalesce CUDA Q4 16-warp activation loads
GiorgioOppo 5968837
fix(rocm): stabilize gfx1151 MMQ routing maps
kyuz0 58c2112
server: support fixed-length greedy decode
kyuz0 d29f53b
metal: optimize Q4 attention output prefill routing
GiorgioOppo 1e0c220
rocm: restore PR887 WMMA attention scheduling
GiorgioOppo bc5cfc6
rocm: fix Q4 prefill parity blockers
GiorgioOppo 377fbe5
cuda: extend Q4 16-warp to attention output B
GiorgioOppo 9c5b5e9
Merge upstream/main into aprojq4-dense-attention
GiorgioOppo 5c5b1d5
Fix ROCm build after upstream merge
GiorgioOppo 555f68a
Default ROCm Q4 prefill WMMA to K64 staging
GiorgioOppo 75503b9
Fix ROCm Q4 attention output WMMA quality drift
GiorgioOppo 9f70fd5
Fix ROCm 10 DeepSeek Vision build
GiorgioOppo 9a448bb
Add Metal benchmark and test binaries
GiorgioOppo 8efbe8c
Optimize ROCm Q4 prefill with K128 staging
GiorgioOppo 35b87dd
Enable grouped CUDA Q4 prefill on GB10
GiorgioOppo 6a20b13
Add specialized CUDA grouped Q4 quantizer
GiorgioOppo File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Some comments aren't visible on the classic Files Changed page.
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Large diffs are not rendered by default.
Oops, something went wrong.
Large diffs are not rendered by default.
Oops, something went wrong.
Large diffs are not rendered by default.
Oops, something went wrong.
Large diffs are not rendered by default.
Oops, something went wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This actually might block speed-bench submissions. @antirez, @GiorgioOppo, are the conditions now only the SVGs generated by that one Python script in speed-bench?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
you can add manually ignoring the git ignore
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please answer that with Yes or No