Skip to content

metal : add top-k radix implementation - #28073

Merged
ggerganov merged 1 commit into
masterfrom
gg/metal-top-k
Aug 31, 2026
Merged

metal : add top-k radix implementation#28073
ggerganov merged 1 commit into
masterfrom
gg/metal-top-k

Conversation

@ggerganov

@ggerganov ggerganov commented Aug 31, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Overview

Modeled after #28032

Perf

M2 Ultra
Backend GGML op Op parameters Bandwidth (GB/s) master Bandwidth (GB/s) gg/metal-top-k Speedup
MTL0 TOP_K type=f32,ne=[1,1,1,1],k=1,ties=0 0.00 0.00 0.94
MTL0 TOP_K type=f32,ne=[1,16,1,1],k=1,ties=0 0.04 0.04 1.04
MTL0 TOP_K type=f32,ne=[10,1,1,1],k=10,ties=0 0.01 0.01 0.96
MTL0 TOP_K type=f32,ne=[10,16,1,1],k=10,ties=0 0.15 0.16 1.01
MTL0 TOP_K type=f32,ne=[1000,1,1,1],k=1,ties=0 0.12 0.12 1.00
MTL0 TOP_K type=f32,ne=[1000,1,1,1],k=10,ties=0 0.12 0.12 0.99
MTL0 TOP_K type=f32,ne=[1000,1,1,1],k=16,ties=0 0.12 0.12 1.00
MTL0 TOP_K type=f32,ne=[1000,1,1,1],k=32,ties=0 0.12 0.12 1.00
MTL0 TOP_K type=f32,ne=[1000,1,1,1],k=4,ties=0 0.12 0.12 1.00
MTL0 TOP_K type=f32,ne=[1000,1,1,1],k=40,ties=0 0.12 0.12 1.00
MTL0 TOP_K type=f32,ne=[1000,1,1,1],k=400,ties=0 0.17 0.17 1.00
MTL0 TOP_K type=f32,ne=[1000,1,1,1],k=8,ties=0 0.12 0.12 1.00
MTL0 TOP_K type=f32,ne=[1000,16,1,1],k=1,ties=0 1.91 1.91 1.00
MTL0 TOP_K type=f32,ne=[1000,16,1,1],k=10,ties=0 1.94 1.92 0.99
MTL0 TOP_K type=f32,ne=[1000,16,1,1],k=16,ties=0 1.95 1.94 0.99
MTL0 TOP_K type=f32,ne=[1000,16,1,1],k=32,ties=0 1.97 1.97 1.00
MTL0 TOP_K type=f32,ne=[1000,16,1,1],k=4,ties=0 1.92 1.91 1.00
MTL0 TOP_K type=f32,ne=[1000,16,1,1],k=40,ties=0 1.99 1.99 1.00
MTL0 TOP_K type=f32,ne=[1000,16,1,1],k=400,ties=0 2.68 2.67 1.00
MTL0 TOP_K type=f32,ne=[1000,16,1,1],k=8,ties=0 1.93 1.92 0.99
MTL0 TOP_K type=f32,ne=[12288,1,1,1],k=16,ties=0 0.87 0.85 0.98
MTL0 TOP_K type=f32,ne=[12288,16,1,1],k=16,ties=0 9.00 13.21 1.47
MTL0 TOP_K type=f32,ne=[131072,1,1,1],k=16,ties=0 4.99 4.98 1.00
MTL0 TOP_K type=f32,ne=[131072,16,1,1],k=16,ties=0 14.48 32.57 2.25
MTL0 TOP_K type=f32,ne=[16,1,1,1],k=16,ties=0 0.02 0.02 0.99
MTL0 TOP_K type=f32,ne=[16,16,1,1],k=16,ties=0 0.25 0.25 1.00
MTL0 TOP_K type=f32,ne=[16384,1,1,1],k=16,ties=0 1.13 1.13 1.00
MTL0 TOP_K type=f32,ne=[16384,16,1,1],k=16,ties=0 9.32 17.16 1.84
MTL0 TOP_K type=f32,ne=[2,1,1,1],k=1,ties=0 0.00 0.00 1.06
MTL0 TOP_K type=f32,ne=[200000,1,1,1],k=1,ties=0 7.44 7.34 0.99
MTL0 TOP_K type=f32,ne=[200000,1,1,1],k=10,ties=0 6.21 6.18 1.00
MTL0 TOP_K type=f32,ne=[200000,1,1,1],k=16,ties=0 5.57 5.55 1.00
MTL0 TOP_K type=f32,ne=[200000,1,1,1],k=32,ties=0 4.63 4.63 1.00
MTL0 TOP_K type=f32,ne=[200000,1,1,1],k=4,ties=0 6.99 6.98 1.00
MTL0 TOP_K type=f32,ne=[200000,1,1,1],k=40,ties=0 4.31 4.35 1.01
MTL0 TOP_K type=f32,ne=[200000,1,1,1],k=400,ties=0 1.18 2.73 2.31
MTL0 TOP_K type=f32,ne=[200000,1,1,1],k=8,ties=0 6.28 6.30 1.00
MTL0 TOP_K type=f32,ne=[200000,16,1,1],k=1,ties=0 15.91 26.08 1.64
MTL0 TOP_K type=f32,ne=[200000,16,1,1],k=10,ties=0 15.22 26.96 1.77
MTL0 TOP_K type=f32,ne=[200000,16,1,1],k=16,ties=0 14.84 26.42 1.78
MTL0 TOP_K type=f32,ne=[200000,16,1,1],k=32,ties=0 13.89 26.48 1.91
MTL0 TOP_K type=f32,ne=[200000,16,1,1],k=4,ties=0 15.63 26.04 1.67
MTL0 TOP_K type=f32,ne=[200000,16,1,1],k=40,ties=0 13.52 25.45 1.88
MTL0 TOP_K type=f32,ne=[200000,16,1,1],k=400,ties=0 6.77 26.64 3.93
MTL0 TOP_K type=f32,ne=[200000,16,1,1],k=8,ties=0 15.32 25.82 1.68
MTL0 TOP_K type=f32,ne=[24576,1,1,1],k=16,ties=0 1.50 1.46 0.97
MTL0 TOP_K type=f32,ne=[24576,16,1,1],k=16,ties=0 10.37 23.03 2.22
MTL0 TOP_K type=f32,ne=[32,1,1,1],k=32,ties=0 0.02 0.02 0.97
MTL0 TOP_K type=f32,ne=[32,16,1,1],k=32,ties=0 0.37 0.38 1.02
MTL0 TOP_K type=f32,ne=[32768,1,1,1],k=16,ties=0 2.01 1.97 0.98
MTL0 TOP_K type=f32,ne=[32768,16,1,1],k=16,ties=0 11.16 27.11 2.43
MTL0 TOP_K type=f32,ne=[4,1,1,1],k=4,ties=0 0.01 0.01 1.05
MTL0 TOP_K type=f32,ne=[4,16,1,1],k=4,ties=0 0.10 0.10 1.02
MTL0 TOP_K type=f32,ne=[40,1,1,1],k=40,ties=0 0.02 0.02 1.00
MTL0 TOP_K type=f32,ne=[40,16,1,1],k=40,ties=0 0.38 0.39 1.01
MTL0 TOP_K type=f32,ne=[400,1,1,1],k=400,ties=0 0.13 0.13 1.00
MTL0 TOP_K type=f32,ne=[400,16,1,1],k=400,ties=0 2.14 2.14 1.00
MTL0 TOP_K type=f32,ne=[4096,1,1,1],k=16,ties=0 0.36 0.36 1.01
MTL0 TOP_K type=f32,ne=[4096,16,1,1],k=16,ties=0 5.78 5.79 1.00
MTL0 TOP_K type=f32,ne=[65000,1,1,1],k=1,ties=0 4.05 4.04 1.00
MTL0 TOP_K type=f32,ne=[65000,1,1,1],k=10,ties=0 3.58 3.53 0.99
MTL0 TOP_K type=f32,ne=[65000,1,1,1],k=16,ties=0 3.42 3.39 0.99
MTL0 TOP_K type=f32,ne=[65000,1,1,1],k=32,ties=0 2.91 2.91 1.00
MTL0 TOP_K type=f32,ne=[65000,1,1,1],k=4,ties=0 3.75 3.76 1.00
MTL0 TOP_K type=f32,ne=[65000,1,1,1],k=40,ties=0 2.86 2.82 0.99
MTL0 TOP_K type=f32,ne=[65000,1,1,1],k=400,ties=0 1.02 2.00 1.96
MTL0 TOP_K type=f32,ne=[65000,1,1,1],k=8,ties=0 3.57 3.60 1.01
MTL0 TOP_K type=f32,ne=[65000,16,1,1],k=1,ties=0 13.87 33.60 2.42
MTL0 TOP_K type=f32,ne=[65000,16,1,1],k=10,ties=0 13.46 33.50 2.49
MTL0 TOP_K type=f32,ne=[65000,16,1,1],k=16,ties=0 13.25 33.45 2.53
MTL0 TOP_K type=f32,ne=[65000,16,1,1],k=32,ties=0 12.63 33.07 2.62
MTL0 TOP_K type=f32,ne=[65000,16,1,1],k=4,ties=0 13.70 33.58 2.45
MTL0 TOP_K type=f32,ne=[65000,16,1,1],k=40,ties=0 12.41 33.03 2.66
MTL0 TOP_K type=f32,ne=[65000,16,1,1],k=400,ties=0 7.05 31.21 4.42
MTL0 TOP_K type=f32,ne=[65000,16,1,1],k=8,ties=0 13.52 33.41 2.47
MTL0 TOP_K type=f32,ne=[65536,1,1,1],k=16,ties=0 3.40 3.38 0.99
MTL0 TOP_K type=f32,ne=[65536,16,1,1],k=16,ties=0 13.50 34.56 2.56
MTL0 TOP_K type=f32,ne=[8,1,1,1],k=8,ties=0 0.01 0.01 0.99
MTL0 TOP_K type=f32,ne=[8,16,1,1],k=8,ties=0 0.16 0.16 1.03
MTL0 TOP_K type=f32,ne=[8192,1,1,1],k=16,ties=0 0.64 0.63 0.99
MTL0 TOP_K type=f32,ne=[8192,16,1,1],k=16,ties=0 8.45 11.14 1.32
M5 Max
Backend GGML op Op parameters Bandwidth (GB/s) master Bandwidth (GB/s) gg/metal-top-k Speedup
MTL0 TOP_K type=f32,ne=[1,1,1,1],k=1,ties=0 0.00 0.00 1.01
MTL0 TOP_K type=f32,ne=[1,16,1,1],k=1,ties=0 0.05 0.05 1.01
MTL0 TOP_K type=f32,ne=[10,1,1,1],k=10,ties=0 0.02 0.02 1.01
MTL0 TOP_K type=f32,ne=[10,16,1,1],k=10,ties=0 0.31 0.31 1.01
MTL0 TOP_K type=f32,ne=[1000,1,1,1],k=1,ties=0 0.19 0.19 1.00
MTL0 TOP_K type=f32,ne=[1000,1,1,1],k=10,ties=0 0.19 0.19 1.00
MTL0 TOP_K type=f32,ne=[1000,1,1,1],k=16,ties=0 0.19 0.19 1.00
MTL0 TOP_K type=f32,ne=[1000,1,1,1],k=32,ties=0 0.20 0.20 1.00
MTL0 TOP_K type=f32,ne=[1000,1,1,1],k=4,ties=0 0.19 0.19 1.00
MTL0 TOP_K type=f32,ne=[1000,1,1,1],k=40,ties=0 0.20 0.20 1.00
MTL0 TOP_K type=f32,ne=[1000,1,1,1],k=400,ties=0 0.27 0.27 1.00
MTL0 TOP_K type=f32,ne=[1000,1,1,1],k=8,ties=0 0.19 0.19 1.00
MTL0 TOP_K type=f32,ne=[1000,16,1,1],k=1,ties=0 3.05 3.06 1.00
MTL0 TOP_K type=f32,ne=[1000,16,1,1],k=10,ties=0 3.08 3.08 1.00
MTL0 TOP_K type=f32,ne=[1000,16,1,1],k=16,ties=0 3.11 3.11 1.00
MTL0 TOP_K type=f32,ne=[1000,16,1,1],k=32,ties=0 3.16 3.16 1.00
MTL0 TOP_K type=f32,ne=[1000,16,1,1],k=4,ties=0 3.06 3.06 1.00
MTL0 TOP_K type=f32,ne=[1000,16,1,1],k=40,ties=0 3.15 3.17 1.01
MTL0 TOP_K type=f32,ne=[1000,16,1,1],k=400,ties=0 4.28 4.26 1.00
MTL0 TOP_K type=f32,ne=[1000,16,1,1],k=8,ties=0 3.07 3.08 1.00
MTL0 TOP_K type=f32,ne=[12288,1,1,1],k=16,ties=0 1.45 1.46 1.00
MTL0 TOP_K type=f32,ne=[12288,16,1,1],k=16,ties=0 8.66 21.97 2.54
MTL0 TOP_K type=f32,ne=[131072,1,1,1],k=16,ties=0 5.89 5.95 1.01
MTL0 TOP_K type=f32,ne=[131072,16,1,1],k=16,ties=0 10.26 48.94 4.77
MTL0 TOP_K type=f32,ne=[16,1,1,1],k=16,ties=0 0.03 0.03 1.01
MTL0 TOP_K type=f32,ne=[16,16,1,1],k=16,ties=0 0.49 0.50 1.00
MTL0 TOP_K type=f32,ne=[16384,1,1,1],k=16,ties=0 1.96 1.93 0.98
MTL0 TOP_K type=f32,ne=[16384,16,1,1],k=16,ties=0 8.79 28.14 3.20
MTL0 TOP_K type=f32,ne=[2,1,1,1],k=1,ties=0 0.00 0.00 1.01
MTL0 TOP_K type=f32,ne=[200000,1,1,1],k=1,ties=0 7.89 7.91 1.00
MTL0 TOP_K type=f32,ne=[200000,1,1,1],k=10,ties=0 7.50 7.50 1.00
MTL0 TOP_K type=f32,ne=[200000,1,1,1],k=16,ties=0 7.32 7.32 1.00
MTL0 TOP_K type=f32,ne=[200000,1,1,1],k=32,ties=0 6.89 6.90 1.00
MTL0 TOP_K type=f32,ne=[200000,1,1,1],k=4,ties=0 7.71 7.72 1.00
MTL0 TOP_K type=f32,ne=[200000,1,1,1],k=40,ties=0 6.66 6.68 1.00
MTL0 TOP_K type=f32,ne=[200000,1,1,1],k=400,ties=0 2.79 3.60 1.29
MTL0 TOP_K type=f32,ne=[200000,1,1,1],k=8,ties=0 7.56 7.56 1.00
MTL0 TOP_K type=f32,ne=[200000,16,1,1],k=1,ties=0 10.97 41.79 3.81
MTL0 TOP_K type=f32,ne=[200000,16,1,1],k=10,ties=0 10.56 41.47 3.93
MTL0 TOP_K type=f32,ne=[200000,16,1,1],k=16,ties=0 10.87 40.80 3.75
MTL0 TOP_K type=f32,ne=[200000,16,1,1],k=32,ties=0 10.71 40.78 3.81
MTL0 TOP_K type=f32,ne=[200000,16,1,1],k=4,ties=0 10.95 40.85 3.73
MTL0 TOP_K type=f32,ne=[200000,16,1,1],k=40,ties=0 10.45 40.77 3.90
MTL0 TOP_K type=f32,ne=[200000,16,1,1],k=400,ties=0 7.57 41.55 5.49
MTL0 TOP_K type=f32,ne=[200000,16,1,1],k=8,ties=0 10.93 40.82 3.74
MTL0 TOP_K type=f32,ne=[24576,1,1,1],k=16,ties=0 2.64 2.63 1.00
MTL0 TOP_K type=f32,ne=[24576,16,1,1],k=16,ties=0 9.46 36.44 3.85
MTL0 TOP_K type=f32,ne=[32,1,1,1],k=32,ties=0 0.05 0.05 1.00
MTL0 TOP_K type=f32,ne=[32,16,1,1],k=32,ties=0 0.83 0.83 1.00
MTL0 TOP_K type=f32,ne=[32768,1,1,1],k=16,ties=0 3.51 3.50 1.00
MTL0 TOP_K type=f32,ne=[32768,16,1,1],k=16,ties=0 9.82 41.33 4.21
MTL0 TOP_K type=f32,ne=[4,1,1,1],k=4,ties=0 0.01 0.01 1.00
MTL0 TOP_K type=f32,ne=[4,16,1,1],k=4,ties=0 0.17 0.17 1.01
MTL0 TOP_K type=f32,ne=[40,1,1,1],k=40,ties=0 0.05 0.05 1.00
MTL0 TOP_K type=f32,ne=[40,16,1,1],k=40,ties=0 0.84 0.84 1.00
MTL0 TOP_K type=f32,ne=[400,1,1,1],k=400,ties=0 0.31 0.31 1.00
MTL0 TOP_K type=f32,ne=[400,16,1,1],k=400,ties=0 4.85 4.86 1.00
MTL0 TOP_K type=f32,ne=[4096,1,1,1],k=16,ties=0 0.61 0.61 1.00
MTL0 TOP_K type=f32,ne=[4096,16,1,1],k=16,ties=0 6.67 6.65 1.00
MTL0 TOP_K type=f32,ne=[65000,1,1,1],k=1,ties=0 5.11 5.11 1.00
MTL0 TOP_K type=f32,ne=[65000,1,1,1],k=10,ties=0 4.87 4.88 1.00
MTL0 TOP_K type=f32,ne=[65000,1,1,1],k=16,ties=0 4.82 4.80 1.00
MTL0 TOP_K type=f32,ne=[65000,1,1,1],k=32,ties=0 4.66 4.66 1.00
MTL0 TOP_K type=f32,ne=[65000,1,1,1],k=4,ties=0 4.95 4.94 1.00
MTL0 TOP_K type=f32,ne=[65000,1,1,1],k=40,ties=0 4.59 4.58 1.00
MTL0 TOP_K type=f32,ne=[65000,1,1,1],k=400,ties=0 3.00 2.83 0.94
MTL0 TOP_K type=f32,ne=[65000,1,1,1],k=8,ties=0 4.89 4.87 1.00
MTL0 TOP_K type=f32,ne=[65000,16,1,1],k=1,ties=0 10.48 46.66 4.45
MTL0 TOP_K type=f32,ne=[65000,16,1,1],k=10,ties=0 10.12 46.48 4.59
MTL0 TOP_K type=f32,ne=[65000,16,1,1],k=16,ties=0 10.38 46.45 4.48
MTL0 TOP_K type=f32,ne=[65000,16,1,1],k=32,ties=0 10.30 46.07 4.47
MTL0 TOP_K type=f32,ne=[65000,16,1,1],k=4,ties=0 10.43 46.65 4.47
MTL0 TOP_K type=f32,ne=[65000,16,1,1],k=40,ties=0 10.24 45.90 4.48
MTL0 TOP_K type=f32,ne=[65000,16,1,1],k=400,ties=0 8.78 43.85 4.99
MTL0 TOP_K type=f32,ne=[65000,16,1,1],k=8,ties=0 10.40 46.52 4.47
MTL0 TOP_K type=f32,ne=[65536,1,1,1],k=16,ties=0 4.85 4.85 1.00
MTL0 TOP_K type=f32,ne=[65536,16,1,1],k=16,ties=0 10.08 46.90 4.65
MTL0 TOP_K type=f32,ne=[8,1,1,1],k=8,ties=0 0.02 0.02 1.00
MTL0 TOP_K type=f32,ne=[8,16,1,1],k=8,ties=0 0.29 0.29 1.01
MTL0 TOP_K type=f32,ne=[8192,1,1,1],k=16,ties=0 1.10 1.10 1.00
MTL0 TOP_K type=f32,ne=[8192,16,1,1],k=16,ties=0 7.25 18.54 2.56

Requirements

  • I have read and agree with the contributing guidelines
  • AI usage disclosure: YES. pi:llama.cpp/DeepSeek-v4-Flash-0731

@github-actions github-actions Bot added ggml changes relating to the ggml tensor library for machine learning Apple Metal https://en.wikipedia.org/wiki/Metal_(API) labels Aug 31, 2026
@ggerganov
ggerganov marked this pull request as ready for review August 31, 2026 12:39
@ggerganov
ggerganov requested a review from a team as a code owner August 31, 2026 12:39
@ggerganov
metal : add top-k radix implementation
58df04c 
Assisted-by: DeepSeek-v4-Flash-0731
@ggerganov
ggerganov merged commit 2a74817 into master Aug 31, 2026
30 of 34 checks passed
@ggerganov
ggerganov deleted the gg/metal-top-k branch August 31, 2026 18:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

Apple Metal https://en.wikipedia.org/wiki/Metal_(API) ggml changes relating to the ggml tensor library for machine learning

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@ggerganov