tune(ep/v2): give fp8 dispatch its own gfx1250 EP4 row

[jhchouuu]

The gfx1250 (4, 7168, 6) dispatch entry has a single row keyed None, so fp8 has been running bf16's geometry. Two things make that the wrong answer for fp8.

The round rule reaches further down than the None row's first edge admits

_tpi = 1 at topk 6 on wave32 (32 % 6 != 0, ep_intranode_1250x.hpp:279), so one round is block*warp tokens and the ~39 us rendezvous floor is held the moment total warps ≥ ct. Blocks past that are idle — at 64 tokens, 4 blocks hold the floor that 64 blocks hold. The None row spends 64 blocks everywhere below 512 because that is what bf16 wants, not because the tokens need them.

Coming up short of a round is a cliff, not a slope, which is the rule showing its edge: 4x16 is +30% at ct=128 and +96% at ct=256.

fp8 has no per-block warp penalty; bf16 does

bf16 64x16 runs 3-4 us behind 64x8 at every small ct, while fp8 measures the same either way. So fp8 can take 16 warps/block and cover a round with half the blocks. bf16 cannot — which is why this PR leaves the None row completely untouched.

  ct    4x16  8x16 16x16 32x16 | 64x8 64x16 128x16
  64    39.1  39.4  39.3  39.4 | 39.1  38.9
  128   50.7  39.4  39.1  39.5 | 39.1  39.5
  256   77.1  50.8  39.7  40.0 | 39.3  39.2
  512  124.1  76.9  51.5  40.7 | 40.7  40.8
  1024                         | 53.0  43.8
  2048                         |       61.0  57.9
  4096                         |       96.6  87.7
  8192                         |      162.2 145.4
 16384                         |             261.3

What it buys — two different kinds of gain

Unpinned A/B of this table against the shipped one, interleaved, 2 reps, every point through bench_ep.py's identity-expert gate:

tokens	latency	blocks
64	+0.0%	4/64
128	+0.8%	8/64
256	+1.0%	16/64
512	±0.0%	32/64
1024	unchanged	unchanged
2048	+4.1%	—
4096	+9.3%	—
8192, 16384	unchanged	unchanged

Below 1024 the latency does not move; what the smaller grid buys is CUs for whatever overlaps dispatch — the same trade the combine table makes for itself in #577. At 2048 and 4096 it is plain latency: the None row holds 64x16 out to 4096, but 128x16 is ahead from 2048 on.

Provenance

4× gfx1250, EP4, hidden 7168, topk 6, 384 experts, fp8 dispatch + bf16 combine, cuda graph, driven through tests/python/ops/dispatch_combine_v2/bench_ep.py. Small tiers are the mean of 3 routing draws at 600 iterations; 2048/4096 come from the unpinned A/B above.

The box was health-checked against the canary this file documents — bf16 4096/64x16 read 144.6 us, under the ~157 us healthy mark.

Measurements were taken at 63099a17. #577 has landed since and reworked the combine barrier, but EpDispatch1250xBody is byte-identical between that commit and current main, so the dispatch numbers carry over unchanged. The merged schedule was re-checked against #577's new combine buckets: fp8 picks up the new combine geometry at the union of edges, and the bf16 / fp4 dispatch columns are unchanged.

Scope

Only (4, 7168, 6) is measured. topk 8 keeps its own row on purpose: 32 % 8 == 0 puts _tpi at 4, so a round covers four times the tokens and every edge moves — it needs its own sweep. experts_per_rank stays a wildcard, as in the existing entry (only 96 was measured).

🤖 Generated with Claude Code