feat(projection): add InferaSim, a workload-driven serving simulator and projector- #137 - #138
Merged
Merged
Conversation
Add infera/projection: analytical + GPU-calibrated inference/serving projection (TTFT / ITL / throughput / KV-cache) and the LLM tuning agent for recipe search. Scope is the serving path. Performance is anchored by an in-process vLLM harness that supplies sub-scale depth override, cudagraph capture buckets, and pure decode-step timing; the anchor JSON is engine-neutral so other engine harvesters (SGLang / ATOM) can be added later. - inferasim CLI (inference suite + anchor-harvest shim) - inferasim-tune CLI (DSPy planner / RLM; projector-scored, no GPU by default) - confidence ladder for regime-aware anchor GPU selection - pyproject: [projection] / [projection-tuning] extras, console scripts, package-data for the model / preset YAMLs
- add a cli/main.py shim so the tuning-agent evaluator can spawn the projector via 'projection inference ...' -> infera.projection.cli - evaluator: put the Infera repo root on the subprocess PYTHONPATH so the shim can import infera.projection.cli - workload: resolve the model / preset configs from the packaged configs tree so tuning runs with zero env setup - ship example tuning assets (target_cluster_*.yaml + MI355X workload YAMLs)
Add a --prefix-cache-hit-rate knob (alias --prefix-hit-fraction), in [0,1), that models automatic prefix caching / shared-prefix reuse: the cached prefix (rate * input_len tokens) skips prefill compute and only the non-cached suffix is run through the network, still attending over the full context. TTFT and the prefill share of continuous-batching pollution therefore scale with (1 - rate); decode and KV sizing are unchanged. - request config: prefix_cache_hit_rate field + resolved_* (clamped to <1 so at least one token is always prefilled) - prefill_latency_ms: discount effective prefill tokens (measured + analytical + chunked paths); continuous-batching pollution sizes chunks off the suffix - CLI flag + launcher arg mapping + feature summary line - settable per workload via a YAML inference: block (tuning agent inherits it) rate=0 (default) is a cold cache and reproduces prior output exactly.
…l in the DES Extend the discrete-event simulator to a fleet of N engine replicas behind a router sharing a prefix pool, so prefix-cache hits are DERIVED from the routing policy + per-instance locality instead of the static --prefix-cache-hit-rate. Each request draws one of P shared prefixes (a system-prompt / template of L tokens); each instance keeps its own resident-prefix LRU; a request routed to an instance already holding its prefix is a hit and only its suffix is prefilled (seeded into the scheduler's num_computed, so the existing token-step packer and cost kernel model it exactly). - routing policies: prefix_aware (KV-aware: co-locate a prefix's requests on one home instance -> misses ~ P, independent of N), round_robin / random (scatter -> misses ~ P*N) - instances are independent except through the router, so each instance's sub-stream is simulated and the raw latency samples are pooled into one fleet-level DESResult (throughput sums; makespan is the slowest instance) - reports fleet size, routing, hit rate, avg cached tokens, per-instance hit spread; surfaces the cache-locality vs load-balance trade-off - CLI: --des-instances / --des-routing / --des-num-prefixes / --des-prefix-len / --des-prefix-zipf / --des-cache-slots; single-engine path unchanged when instances=1 and no prefix pool
In benchmark-calibrated mode the analytical TTFT discount was discontinuous at the first prefix-cache hit: hit=0 used the measured full-prefill anchor while hit>0 switched to the per-token rate path, and those two differ by the batch factor (a single-batch anchor holds full-prefill flat). Result: TTFT cliffed at the first hit instead of scaling smoothly with the hit rate. Discount the SAME chosen baseline proportionally to the non-cached suffix instead of switching cost methods, so benchmark-mode TTFT scales continuously with the hit rate. Analytical path unchanged (models suffix attention over full context); DES path was already correct (per-token measured prefill throughout).
Rename the inference-projection + tuning-agent package to projection_core and update its dotted imports, the tuning-agent path literals (model resolver, shim path, PYTHONPATH marker), the entry point, the configs package-data glob, and README paths. Pure rename; no behaviour change.
…a.projection.* Lay out the engine as first-class infera.projection.* packages (core, agents, configs, modules, platforms). The tuning-agent spawn shim lives in _tuning_shim/. inferasim / inferasim-tune are the console entry points (with infera-projection / infera-tuning kept as back-compat aliases). Path/depth logic in the tuning agent (configs/models resolution, shim path, PYTHONPATH repo-root discovery) and the package-data globs are wired for this layout. No behaviour change.
Standardise runtime log prefixes to [inferasim:...] and configuration env-var names to INFERASIM_* across the projection engine, DES, tuning agent, CLI help, example configs, and README. An import-time alias shim keeps legacy env-var names resolvable so existing environments keep working.
… DES Replace the prefix-id LRU with a content-addressed, paged KV block cache per engine instance (as real serving engines like vLLM / SGLang do). A prompt is an ordered sequence of block-hash ids; a hit is the longest contiguous leading run of resident blocks. The cache is finite (capacity in blocks) and LRU-evicts under pressure, so the prefix-cache hit rate is emergent from workload content + capacity + routing rather than a static number. Matched blocks seed num_computed so only the uncached suffix prefills (decode / KV unchanged). Add overlap-scored KV-aware routing (--des-routing kv): route each request to the instance holding the most of its leading blocks (ties -> least loaded), via a global prefix index. Surfaces the real cache-locality vs load-balance trade-off (hot prefixes overconcentrate, lowering fleet throughput). Add Mooncake trace replay (--des-mooncake-trace): JSONL/JSON with timestamp, input_length, output_length, hash_ids; the hash_ids drive real content-addressed matching and enable the DES without --request-rate. Block sequences otherwise synthesised from the shared-prefix pool, blockified at --des-block-size. New knobs: --des-block-size, --des-kv-blocks (per-instance block capacity; --des-cache-slots kept as legacy alias), --des-mooncake-trace, routing 'kv'. Fleet report now shows block size, capacity, evictions, and block-reuse rate.
…t models
Add three inference operation models to the performance projector:
- sampling / logits post-processing (module_profilers/sampling.py): a
memory-bound reduction over the vocab for greedy / temperature / top-k /
top-p, folded into every step; knobs sampling_enabled / sampling_top_k /
sampling_top_p / sampling_temperature (+ CLI). Enabled by default at the
greedy (cheapest) path; the top-k / top-p / temperature knobs are no-op at
their defaults and only add streaming passes when set.
- runtime activation quantization / cast (module_profilers/quantization.py):
fp8 / mxfp4 activation cast cost per dense and MoE layer, auto-detected from
the weight dtype; knob act_quant_dtype (+ CLI). Auto-enabled for fp8 / mxfp4
serving; a bf16 path resolves to no cast (there is nothing to charge).
- small-tensor kernel-launch floor: a depth-scaled lower bound on the
pure-simulate decode / mixed step so small-batch decode does not underflow
launch dispatch; disabled under CUDA-graph capture; knobs
kernel_launch_latency_us / kernels_per_layer (+ CLI). Off by default
(kernel_launch_latency_us=0) until calibrated.
The default projection stays equal to the pure kernel-compute roofline:
sampling runs at greedy, activation-quant charges only for low-precision
serving, and the launch floor stays off until calibrated.
Co-authored-by: Cursor <cursoragent@cursor.com>
…I examples - name the experiment-config class InferaSimConfig / InferaSimParser and the loaders load_config / convert_config_to_inference_config; the global accessor get_config; log tags InferaSimMaster; tuning-agent config_root paths. - drop the legacy env-var alias shim; INFERASIM_* is the only prefix. - correct module-path examples to infera.projection.agents.tuning_agent and point docstrings at the inferasim projector CLI. No behaviour change; projection output is unchanged. Co-authored-by: Cursor <cursoragent@cursor.com>
Cap the attention span and KV-cache footprint at a sliding window, so models
that use local attention (gpt-oss, Mistral, Gemma-2/3, Qwen2.5) are projected
against the window rather than the full context.
- model config: sink_sliding_window / sink_window_even_layers_only, picked up
straight from the model YAML.
- request config: sliding_window override (0 forces full attention) and
sliding_window_layer_fraction for models that interleave local and global
layers; resolvers blend the windowed and full layers into one representative
KV length (effective_attn_kv).
- performance: attention (prefill and decode) reads the blended window-capped
KV length; the sparse-attention path keeps the true context.
- kv_cache: per-sequence footprint caps at the window, blended by the
windowed-layer fraction.
- CLI: --sliding-window / --sliding-window-layer-fraction, plus a feature
summary line.
Full attention is unchanged: a window at or above the context, or no window at
all, reproduces the previous projection exactly.
Co-authored-by: Cursor <cursoragent@cursor.com>
Each ladder rung costs a real GPU benchmark run. A flat per-GPU pair at rung g
certifies targets up to 2*g, so rungs 1/2/4 already certify an 8-GPU target --
climbing past 4 buys little for the calibration time it spends.
- LADDER_MAX_GPUS = 4, resolved by ladder_max_gpus() from an explicit
max_gpus argument, INFERASIM_LADDER_MAX_GPUS, then the default.
- confidence_ladder takes the cap, returns next_gpus=None plus a new capped
flag once the top rung sits at the cap, so callers stop asking for a rung
they cannot run; larger targets are reported as extrapolated from the top
rung instead of silently certified.
- climb_anchor_ladder defaults its ceiling to the cap.
- launcher advisory points at the cap knob instead of printing a null rung.
Targets that a 4-GPU rung can certify keep HIGH confidence and are unaffected.
Co-authored-by: Cursor <cursoragent@cursor.com>
…ctives The intra-node collective floor was keyed on a message-size crossover: below it the training-scale fixed RCCL overhead was stripped in favour of the measured latency floor, at or above it the constant snapped straight back. That made a marginally larger message cost sharply more and then stay flat, because the constant -- not bandwidth -- was setting the price. On a TP=8 model the decode all-reduce jumped abruptly between two adjacent batch sizes. A graph-captured intra-node serving collective does not pay the training-scale constant at any size, so strip it throughout and price the collective as max(measured floor, bandwidth transfer). The floor governs small decode messages, the bandwidth term takes over smoothly as the message grows, and prefill-scale messages are bandwidth-dominated either way. Multi-node keeps the base model, whose NIC overheads are real. Also add a DeepSeek-R1 serving workload (MLA + MoE, TP=8) matching the measured sweep in bench/pd_mori_1p1d/results, for fidelity comparisons. Configurations whose decode messages already sat below the crossover are unchanged. Co-authored-by: Cursor <cursoragent@cursor.com>
TTFT was priced at a single prompt's prefill with no contention, but one engine prefills every concurrent request. A serving benchmark runs closed loop (--max-concurrency N --request-rate inf), so all N stay outstanding, the scheduler sweeps them FIFO within its token budget, and the average request sits half way down that sweep. The old model was concurrency-independent: DeepSeek-R1 reported the same TTFT at the narrowest and the widest concurrency alike. Against the measured gpt-oss-120b vLLM run (TP=8, MI355X) the TTFT error drops substantially; the residual is uncalibrated roofline optimism, which an anchor closes. The measured ratio of tail to median TTFT matches a sweep of several scheduling groups, which is what this models. Disaggregated serving gets the same treatment on its prefill pool, spread over the prefill replicas. Co-authored-by: Cursor <cursoragent@cursor.com>
Three defects made the inference memory report unusable for capacity planning, which is what drives sustainable concurrency and every memory-feasibility decision the tuning agent makes. * Weights were not tensor-sharded. `estimated_num_params(rank=...)` counts a rank's pipeline and expert share but not its TP slice, so every rank was charged the whole model. Its only caller is the inference memory path, so dividing there is safe and leaves training untouched. * `mxfp4` was missing from the dtype table and silently fell back to bf16, i.e. several times the real width. MXFP4 is block-scaled (4-bit elements plus a shared exponent per block), so its effective width is slightly wider than the nominal one rather than exactly it. * The activation working set multiplied the whole concurrent batch by the full prompt length. A scheduler step is capped by the engine's prefill budget across the batch, not per sequence. gpt-oss-120b at mxfp4/TP=8 was reported at orders more memory per rank than vLLM actually loads. DeepSeek-R1 at fp8/TP=8 now lands where its checkpoint divided across its ranks says it should. Co-authored-by: Cursor <cursoragent@cursor.com>
…antom shared expert Sliding-window attention was modelled but never reachable: the window lived in the primus_turbo kernel module config, and module keys shadow model keys when the two are merged, so a model preset could not declare its own window and every gpt-oss projection ran full attention. The window is a property of the architecture, not of the kernel, so it moves to the gpt-oss presets, which interleave windowed layers with full-attention layers. The kernel toggle (use_sink_attention) stays where it was. gpt-oss also inherited a shared expert from the DeepSeek-V2 base it extends. It has none, and the phantom expert added to per-token MoE FLOPs and weight traffic. At long context the KV cache per sequence halves and sustainable concurrency roughly doubles at every context length. Co-authored-by: Cursor <cursoragent@cursor.com>
A DeepSeek-R1 anchor projected a served TPOT well off the reference. The anchor was not wrong: re-measuring it on a newer vLLM reproduced the original numbers closely. It described a different deployment than the one it was scored against -- one without speculative decoding. Speculation was invisible to the whole pipeline. It was not on the regime axes, so an anchor measured without it sat at zero regime distance from a target using it and was reused silently; and benchmark_vllm.py could not enable it or record it, so no artifact could even state which case it belonged to. Measured attribution on MI355X, same anchor config with only speculation changed: MTP-1 accounts for most of the gap. The remainder is static-batch benchmarking versus continuous-batching serving, independently corroborated by an in-house disaggregated run. Expert parallelism was ruled out by measurement -- its speedup grows with batch while the residual shrinks. Speculation is regime-defining rather than transportable because per-token latency becomes step_cost(batch * (k+1)) / (1 + a + ... + a**k) and the acceptance rate a cannot be derived analytically from a non-speculative anchor. An artifact with no recorded setting is treated as unknown, and unknown is counted as a mismatch only against a speculating target: everything measured before this tracking ran without speculation, so legacy anchors keep working for non-speculative targets while the dangerous direction is closed. Since this adds a regime axis, the anchor store now fingerprints the axis set it indexed under and re-derives a stale index instead of comparing signatures built from two different definitions.
…r knobs TTFT was charged as one uncontended prompt, which ignores that a request also waits behind the prompts queued ahead of it. Replacing that with a full FIFO sweep over the resident set is the other extreme and overshoots badly whenever generation is long enough to leave the prefill stage idle (MiniMax at long OSL ran far above measured). Both limits are wrong for some Hyperloom workload, so price the wait with exact mean-value analysis of the closed network the serving harness actually is: it returns the bare prefill under light load and the sweep under saturation, and interpolates between them. That still left DeepSeek-R1 1P1D far under measured, because its TTFT is barely prefill at all. Two engine flags sit on the first-token path and cost no throughput, which is exactly why production configs turn them up: --stream-interval buffers N tokens per flush, so the client's first token arrives with the flush carrying it, and the decode scheduler polls for new work only every num-continuous-decode-steps x scheduler-recv-interval steps. Model both, with buffering moving time out of TPOT rather than adding to end-to-end latency. Measured error on the 1P1D sweep improves substantially, and now tracks the measurement's near-flat shape in concurrency rather than growing with it. The residual is decode-step optimism: with a measured anchor, leave-one-out TPOT lands close and TTFT within a usable band. Also returns the merged config from the launcher, so a caller can see what the model preset, module defaults and CLI overrides resolved to, and adds the first regression tests for the projector. Co-authored-by: Cursor <cursoragent@cursor.com>
The tuning agent optimized a single scalar, so "max_throughput" always walked to the largest batch that fits in HBM -- the config nobody would deploy for an interactive workload, and the one that looks best when latency is not a constraint. Serving is the constrained problem instead: maximize throughput subject to a latency promise. Add optimization.slo (ttft_ms / tpot_ms / request_latency_ms) and enforce it where the HBM cap is already enforced, so a miss is rejected through the same path and shows up in the trial log. The rejection names the budget and the overshoot because that string is what the planner reads back: it is the signal to trade concurrency away rather than to try another dtype. Unset keys stay unconstrained, and a metric the projection never reported cannot fail a budget. This is worth doing only now that TTFT is modelled: against a projector that reported milliseconds for a measurement in seconds, a TTFT budget would have ranked on noise. Co-authored-by: Cursor <cursoragent@cursor.com>
… runs There was no way to answer "did that change make the projector better", so every fix to it was argued from first principles. This scores it two ways that are deliberately kept apart: fidelity against the serving results Hyperloom has already measured, and a full-workload diff between two checkouts. The measured side carries the engine flags each run was launched under, because a projection that ignores --stream-interval and the decode scheduler's polling granularity is answering a different question than the benchmark asked. It also turns those measured results into a calibration anchor, so a session can calibrate from results it already has instead of booking GPUs, and leave-one-out checks that the anchor transports to concurrencies it was not fitted on. Adds a regression test for the collective discontinuity, which is the case that motivates reading these numbers carefully: the step made aggregate error look better while making the curve useless to search. Co-authored-by: Cursor <cursoragent@cursor.com>
…loors Three defects made the projector systematically optimistic about MoE decode, and each was worst exactly where the measured error was worst. 1. Tokens were divided by tensor-parallel size when sizing the expert GEMM. TP shards the hidden and FFN dimensions; every TP rank holds the whole token axis. Only context parallelism shards tokens, and expert parallelism splits routings and experts by the same factor. The per-expert row count was therefore wrong by ep/tp -- correct at EP==TP, and far too small at the EP=1,TP=8 configurations that DeepSeek-R1 and gpt-oss actually run. It was wrong in the same direction for prefill, where it also understated TTFT. 2. The expert dtype was applied as a hand-picked speedup multiplier instead of as operand bytes. Narrower weights are not a tuning constant: they are fewer bytes to stream and a wider matrix instruction. The GEMM roofline now takes the real width (mxfp4 is block-scaled, so a little wider than nominal) and the multiplier is gone. 3. Restoring a measured anchor to another TP/EP scaled the whole step by the simulator's ratio. A decode step is a fixed per-step cost plus work that shards, and only the second part responds to parallelism; re-scaling the fixed part inflated a less-sharded target severalfold. Decode now moves the anchor by the simulator's difference. Prefill, which has no such floor, keeps the ratio. Against Infera's own 1P1D DeepSeek-R1 concurrency sweep, uncalibrated TPOT and aggregate throughput error both fall sharply, and leave-one-out calibrated TPOT lands close throughout. Against our own static DeepSeek microbenchmarks the decode step's MAPE roughly halves, and anchor reuse across a TP+EP move improves by a large multiple. What remains is a batch-independent per-step cost the analytical path still does not model, which is why the low-batch end stays optimistic. The scheduler test now asserts against the decode step rather than TPOT: fixing (1) makes prefill genuinely more expensive at EP=1, so TPOT carries real mixed-step pollution and is no longer a stand-in for the step the scheduler actually waits on. Co-authored-by: Cursor <cursoragent@cursor.com>
…sing The measured TP ladder (gpt-oss-120b, real weights, TP=1,2,4,8, across batch) over-determines step(tp) = floor + compute(1)/tp and shows a floor that is TP-invariant across the low-batch range with small residual. Four terms were wrong against it. Attention sharded the token axis instead of the head axis, so every rank was charged for streaming the whole Q/K/V/O weight matrices; modelled attention barely moved from TP=1 to TP=8 while the measurement shards strongly. MLA had the same error, with the down-projections correctly replicated. SDPA had no HBM roofline. The tile model prices per-workgroup GEMMs on one CU and scales by wave count, which never bounds the result by the bandwidth of the device the KV cache lives in. At decode that is the whole cost, and the model implied a KV read rate many times the part's HBM bandwidth. The collective floor was an RCCL number applied to a vLLM deployment that runs its own one-shot all-reduce, putting a flat cost into every gpt-oss decode step at TP>1 regardless of batch or TP. The ladder bounds the real cost far below that, because TP=1 carries no collective at all. Per-kernel GPU occupancy was modelled only as host launch latency, disabled under graph capture and applied as a max. Graph replay removes the dispatch, not the execution, and the small latency-bound kernels run alongside the large data-bound ones rather than instead of them. Ladder MAPE over the measured points falls from unusable (the tp1->tp2 delta had the wrong sign at low batch) to a usable band. The occupancy constant is the ladder floor divided by an assumed kernel count, and the collective floor is bounded by measurement rather than measured. Both are marked interim; measure_kernel_floor.py measures the first directly.
…cy on every path Two defects in the previous commit, both caught by scoring the measured serving runs before and after. The SDPA HBM roofline derived KV bytes from head counts, which is right for MHA/GQA but wrong for MLA: DeepSeek caches one compressed latent (kv_lora_rank + rope dims) that every head shares and that TP replicates rather than shards. Charging it per head overstated DeepSeek's KV by a large factor and pushed TPOT from close to measurement to far above it across the concurrency sweep. The attention profiler now supplies the cache's real per-token footprint, including its own dtype, since fp8 KV with bf16 activations is common. Per-kernel occupancy was charged only in ``_decode_step_latency_ms``, which the continuous-batching path does not call -- so every vLLM workload was missing it while the disaggregated path had it. A mixed step runs the same decode kernels with a prefill chunk added, so both steps pay it. With both corrected, the occupancy constant taken from the gpt-oss ladder also holds on DeepSeek without refitting, tracking measurement across the concurrency sweep. That is the cross-model check the constant needed to be more than one model's tuning. TPOT MAPE over the serving runs falls, and minimax bf16 moves from under to near measurement on both TPOT and throughput. Still open: gpt-oss TPOT low and TTFT low everywhere, both of which look like prefill/admission modelling rather than the decode step.
…ed values The MLA regression got in because nothing asserted that a modelled byte stream has to fit through the memory system. These tests are written as inequalities against MI355X's limits, so they survive the model getting more accurate and fail as soon as a term goes unbounded again: SDPA may not imply more KV read bandwidth than the part has, MLA's latent must cost less than a per-head footprint, attention must shard with TP, occupancy must survive graph capture while the launch floor must not, and the intra-node all-reduce floor must stay inside what the ladder allows. Also adds a prefill check against the ladder, which mainly documents why that data cannot be tuned against: at high batch it reads the same at TP=1 and TP=8, so no speedup from eight times the hardware, and those points measure admission rather than FLOPs. It does establish direction -- modelled prefill sits above measurement -- which places the low serving TTFT in the queueing model rather than the prefill cost.
The high-batch TP under-prediction looked like it could come from the MoE GEMM: the roofline gives sharding near-ideal 1/etp relief, and narrow slices might stream less efficiently than wide ones. Measured on MI355X, that is not where it comes from. The grouped GEMM the decode step actually issues holds most of peak bandwidth across etp=1..8 and delivers close to the ideal relief, so the roofline is right here and an efficiency knob tuned in at this spot would only be covering for a term that lives somewhere else. Timing experts one at a time says the opposite, but that is an artifact: a lone [1 x k] x [k x n] call is latency-bound and reads a small fraction of peak. The benchmark now batches over the experts a step touches. Pins the measured relief as a bound so it is not "fixed" the wrong way, and repoints the router hook at its vLLM 0.25 path.
The TP ladder put the decode model's error in a specific place: fitting step = floor + compute/tp on tp=2,4,8 and using tp=1 -- the only rung that runs no collective -- to pin floor+compute left an implied all-reduce far larger than what was being charged for gpt-oss at high batch. Measuring vLLM's custom all-reduce directly confirms it. That is the kernel a decode step runs -- everything under its size threshold goes to it, not to RCCL -- and its cost rises gently with batch on 8 ranks, fitting floor + bytes/bw very closely. Two things were wrong: the composition is additive, not max(floor, bandwidth), which at decode message sizes drops a term of comparable size to the one it keeps; and the achieved bandwidth is far below the node link, with no clean ring or one-shot factor mapping one onto the other, so it has to be measured. Charging comm properly then exposed that the occupancy floor had been absorbing it. The per-kernel occupancy constant came from a floor fit that left comm in, so the same milliseconds were about to be billed twice. Refitting with measured comm removed gives a floor that is flat where it should be, across the low-batch range on the most floor-dominated rung, and a correspondingly smaller per-kernel constant. Ladder MAPE over all points improves, as does the static decode case. Also records, without wiring in, that a grouped GEMM only reaches peak bandwidth once it is large enough to saturate memory: measured from a small fraction of peak on one expert climbing to a high plateau, both models collapsing onto one curve against bytes moved. It belongs to the step, not to each call within it -- a decode step streams multi-GB back-to-back under one graph replay -- and applying it per call costs a large amount of MAPE.
MoE decode cost tracks how many *distinct* experts a step reads, which the model estimated from a uniform router. Real routers are not uniform, so this generalises the coupon-collector to a Zipf popularity law, reducing exactly to the old closed form at skew 0. Existing gpt-oss vLLM sweeps under forced Zipf routing turn out to be a controlled experiment on this -- same model and batch, only the distribution changes -- and they confirm the mechanism: decode time is linear in the distinct count across the swept skew range, at a near-constant cost per expert. The default stays uniform, because the same data says realistic imbalance does not move the count. At a mild skew -- already a heavier max/mean load than a balance-loss router shows -- the distinct count barely differs from uniform's. Recorded because fitting the skew to the ladder is tempting: it has a clean optimum at a much higher skew, and taking it would improve MAPE. But that skew implies an extreme max/mean expert load, so the fit is an unrelated error wearing a plausible-looking knob, and the mid-batch residual is not routing skew. Pinned as a test so it does not get adopted later.
Hyperloom spawns one process per config, so every projected config paid the import cost before doing any work. That cost dwarfed the projection itself -- nearly all of the wall time was overhead -- and most of it was torch. Nothing in an analytical projection needs torch. It arrived through module_profilers.utils, which is entirely real-GPU benchmarking helpers (benchmark_layer, CUDA-graph timing, routing patches), imported at module scope by six profilers that only call it in their measure-on-hardware branch. Moving those imports into the branches that use them cuts the projection stack's startup by roughly an order of magnitude, with the GPU path unchanged. Also adds speed_benchmark.py to quantify the trade Hyperloom is making, with the real-measurement baseline taken from this machine's ladder campaign rather than assumed: projecting a config is orders of magnitude cheaper than measuring one on 8 GPUs, and the config grid it sweeps costs a fraction of a second instead of minutes of GPU time.
A deployment search spends most of its GPU time discovering that candidates are slow or do not fit, and it only needs the ranking to decide what to measure. This projects a whole TP x EP x concurrency space in one process and returns it ranked, so real GPU time goes to the finalists. Measured on gpt-oss-120b, the whole space projects in seconds against the GPU time it would take to measure at this machine's own campaign rate. Doing the sweep in-process matters as much as the per-config cost, since the fixed setup -- now almost entirely the Origami import -- is paid once instead of per config. Feasibility is the part that needs no accuracy argument at all: a config whose weights and KV cache do not fit is unrunnable for a reason the memory model settles exactly. Infeasible points are kept and marked rather than dropped, so a caller can tell "does not fit" from "was never tried", and one bad point costs one point instead of the sweep.
A served anchor measured a decode step and left prefill to the analytical GEMM model, whose absolute level its own authors disclaim -- the origami ratio exists because only sim(target)/sim(bench) survives it. Decode escapes that bias by being anchored, and prefill had no escape in either mode: priced from the absolute number when simulating, and left simulated when not. It reaches a first token two to ten times sooner than every measured fleet in the InferenceX set. Prefill was left out for a real reason. A raw TTFT is not invertible into a prefill step -- it carries admission granularity, the streaming flush and client overhead, and inverting it overstates the step by one to two orders of magnitude. But every one of those terms is constant in prompt length, so they cancel in a difference. Probed at concurrency 1 at two prompt lengths, TTFT(L2) - TTFT(L1) is the cost of the extra tokens and nothing else. The intercept the difference throws away is kept as a direct reading of the host and admission floor, which was previously only estimable by inversion. Served path only, and on by default there. The offline LLM() entrypoint reports a prefill step of its own but resolves different attention and MoE kernels, so anchoring on it calibrates against a stack no fleet runs; asking for both is refused rather than silently resolved. The measured rate fills the whole batch curve, since prefill is linear in total prompt tokens and a lone point would be held flat across batch instead. --prefill-anchor-validate probes a third interior length so the pairwise slopes can be compared, which checks the linearity the difference assumes rather than asserting it.
Origami and the SDPA simulator both had to be told what a GB300 is before the InferenceX NVIDIA rows could be scored at all. Neither tile model has a Blackwell target, so both entries borrow the gfx950 arch enum and carry Blackwell Ultra's peaks: this reproduces GB300's roofline, not its kernels, and is marked temporary in both files for that reason. Co-authored-by: Cursor <cursoragent@cursor.com>
Prefill attention was priced as a single GEMM with the whole KV length as its N dimension, on the assumption Origami scales linearly in N. It does not: a 122k-wide N saturates and returns about the same time as an 8k one, so long-context prefill attention stayed nearly flat -- 13 ms at 8k against 21 ms at 122k -- while the FLOPs behind it grew fifteenfold. Pricing the loop the kernel actually runs, 64 KV columns per iteration, restores the growth. kv_tile_n was already declared here and already reported in the result metadata; it had simply never been used to charge anything. Two bounds come with it. A per-tile time from Origami can fall 20x below a single CU's peak, which left the loop memory-bound on a few hundred kilobytes, so tiles are floored at 70% of one-CU peak -- the compute analogue of the efficiency the HBM roofline already assumes. And causal masking now discounts only the square suffix it applies to, since scores against a cached prefix are all live and halving them understated nothing but a 9k-attends-130k step. Decode keeps the single-GEMM path: it is HBM-bound, the tests pin that bound, and a 1-row tile loop would make it a launch-bound count of tiny GEMMs. Co-authored-by: Cursor <cursoragent@cursor.com>
Kimi-K3 runs 69 of its 93 layers as Kimi Delta Attention, which keeps a d×d state per head and updates it with an O(d^2) matvec per token. The linear_attention_* keys recorded that but nothing read them, so every layer was charged as full MLA: the cache came out about 3.9x oversized and attention was priced as growing with context on all 93 layers rather than on 24. Both halves land together, and they have to. Modelling the memory alone would be worse than modelling neither -- search would fit larger batches while still charging every one of them 93-layer attention. So the token cache is stored only on the MLA layers, the KDA layers carry a fixed recurrent state per sequence, and attention reads a blended KV length whose linear part is the state extent rather than the prompt. The state is an activation in bf16, not a quantized key, and its heads shard the way GQA's do: tensor-parallel splits them, data-parallel attention keeps them whole and splits the requests instead. Co-authored-by: Cursor <cursoragent@cursor.com>
multi_latent_attention is a switch that decides which layer Megatron builds, not a description of what gets served. DeepSeek-V4 holds it off so the trainer does not take the V3 MLA builder -- V4 layers need their own CSA/HCA/SWA branches -- yet what it serves is still latent attention: one compressed K=V vector plus a decoupled RoPE head. The decode kernel counter read that switch literally and charged 4 attention kernels per layer where the model issues 9. Over 61 layers that is 1.3 ms of decode step per token nobody pays. It hides at high batch, where the step is data-bound, and it is the whole error at low batch, where it is not. The KV cache already worked around the same switch to charge V4's 576-byte latent, by testing the shape directly. Both now go through one predicate, because a decode step that skips MLA's projections while the cache charges MLA's latent describes no model that exists. Co-authored-by: Cursor <cursoragent@cursor.com>
The offline sweep repeats its prompts, so with prefix caching on, the prefill it measures is a block-lookup curve rather than cold prompt processing. Nothing recorded which of the two an artifact held, so a cache-hit curve could be loaded to price traffic that misses -- two different observables wearing one name. The benchmark now writes the mode into its metadata and the projector matches on it: a hit curve is accepted only when the target itself is configured as a full prefix hit, and otherwise prefill and TTFT stay simulated while decode calibrates as usual. When it is accepted, the suffix discount is dropped, because a hit anchor already contains the lookup and the single forward that produces the first token, and discounting it again would apply the hit twice. Co-authored-by: Cursor <cursoragent@cursor.com>
A FLOPs-only TTFT read about 0.2x of the InferenceX single-turn rows, and three terms account for the gap. Every request costs the server a fixed amount before any prompt is touched -- accept, parse, admit, look up the prefix cache, allocate pages, open the stream. Differencing TTFT across two prompt lengths at concurrency 1 separates that intercept from the slope, and it comes to ~150-190 ms on the SGLang and atom stacks and ~46-94 ms on the vLLM ones: an order of magnitude more than any prompt-length term can explain at 1k. The slope from that same fit anchors the level of prefill compute. It is not applied directly -- prefill is superlinear in the prompt and a slope fit over a short span cannot express that -- so the projector takes its own slope over the same two lengths and scales by the ratio, keeping the roofline's shape across context while moving only its level onto measurement. Both lengths are recorded, since a rate means nothing without the span it was fit on. This matters out of proportion to its size: under a closed load, TTFT is the queueing response of a shared prefill station running like S/(1-U), and the corpus sits at utilizations from 0.2 to 0.97, where a 1.3x error in S becomes a 2-4x error in TTFT. So the wait is modelled as a closed loop over the clients actually in flight rather than as a FIFO of single requests. All three default to zero. They describe a serving stack rather than the silicon or the model, so there is no architecture profile to resolve them from, and a projection that has not been told which engine it is modelling should not invent one. Co-authored-by: Cursor <cursoragent@cursor.com>
Under disaggregation the memory projector kept the global concurrency after swapping in the decode pool's parallelism, so every replica was charged the whole system's KV. Doubling the decode pool then freed no HBM at all, and the measured MiniMax GB300 4-prefill/2-decode winner at 4096 in flight was rejected for exceeding 288 GB by seven -- while silicon ran it. A replica holds C / decode_replicas sequences, which is how the decode projector has always split it, and activations follow the same local batch. TTFT gets the same correction from the other side. A prefill pool serves its arrivals as a batched station, not as a FIFO of single requests, so the wait now comes from the closed-loop response over the requests actually in flight per replica and the pool's occupancy is reported alongside it. Co-authored-by: Cursor <cursoragent@cursor.com>
Warmup sized its GPU count to the target's own TP, so a TP8 target wanted eight GPUs before it could be anchored at all, and each TP got its own artifact. Measuring at TP4 and transporting upward instead makes the anchor reachable on half a node, and the ladder it was validated on says the transport holds: TP4 to TP8 scores 0.94x bias with a Spearman of 1.000 on the gpt-oss decode points. Below four the target's own TP is kept, since there is nothing to transport from that is cheaper. Lookup follows the same rule -- ties on regime distance now break toward the TP the policy mandates, so a store holding several parallelisms returns the one the target would have been measured at rather than whichever happened to be nearest. Co-authored-by: Cursor <cursoragent@cursor.com>
Benchmark mode was pinned to vLLM. The harness already drives SGLang and ATOM through the same adapters the platform serves with, so the pin was not describing a capability -- it just meant an architecture the local vLLM build cannot construct was unmeasurable, which is DeepseekV4ForCausalLM on the ROCm image we have. --bench-serving-backend picks the engine, and the two never share an anchor cache entry, because two engines serving one config are two measurements. The client was the second half of the same pin. Load was generated by vLLM's bench serve, and an SGLang image need not contain vLLM at all, so the models worth measuring under SGLang were exactly the ones we could not drive traffic at. What a client owes this harness is two numbers, mean TTFT and mean TPOT over a closed loop, and both engines ship a script reporting them under those names. vLLM's stays the default wherever it exists, keeping already-harvested anchors comparable; SGLang's stands in when it does not, and the artifact records which one ran. Resolved before the server is launched rather than after, since a missing load generator makes the run pointless and the weights take minutes to become resident. Co-authored-by: Cursor <cursoragent@cursor.com>
The config named GLM-5 and said nothing about how its attention is sparse. DSA keeps the top-2048 tokens chosen by a lightning indexer, and 5.2 shares that indexer across every four layers while leaving the window itself at 2048 on all of them -- so the topk to set on the request is 2048, not a quarter of it. Comment only. Co-authored-by: Cursor <cursoragent@cursor.com>
Under attention-DP a mixed step carries one prefill chunk per rank, each for a different request, and the step is priced unsharded because a rank under DP holds every head of its own chunk. The steady-state window charged every request the full chunk count against that longer step, billing the same prefill once per rank. The effect was not small and it inverted the ranking. On the MI355X agentic corpus the DP-attention rows ran 3-5x over measurement -- 145 ms against 48.9 measured at batch 104 -- which made DP attention look slower than plain TP8 when MLA's shared latent makes it 2.4x faster per pure decode step, and that is why the fleets run it. Total-throughput bias moves 0.84x -> 1.03x, the TPOT tail 2.35x -> 1.53x, and the measured winner rises from 4th of 5 configurations to 1st. A request's own service time is still n_chunks of those steps, so TTFT is left alone; only the shared window it rides holds dp of them at once. Co-authored-by: Cursor <cursoragent@cursor.com>
The mixed-step fix has nothing guarding it. The defect it repaired was not a wrong constant but a wrong denominator, so it returns silently: any later change that bills the prefill chunk per request rather than per step puts long-context TPOT back above the tensor-parallel one. Written as the inequality the axis exists to satisfy rather than as the numbers it currently produces. At 32k a rank under DP reads its own sequences' latent instead of the fleet's, so its decode step is cheaper, and a blend of cheaper steps cannot come out dearer. On the pre-fix model this reads 1252 ms against 539 ms and fails. Co-authored-by: Cursor <cursoragent@cursor.com>
A checkpoint that wants --trust-remote-code, or an attention backend other than the engine's default, does not start without them, and a model that does not start is unmeasurable for a reason that has nothing to do with its performance. This is what blocked harvesting anchors for MiniMax-M3 and DeepSeek-V4-Pro. Sent in the "=" form because the value is itself a flag string, which argparse would otherwise read as the next option rather than as this option's argument. Co-authored-by: Cursor <cursoragent@cursor.com>
Store lookup already filters by model, but an explicit --load-benchmark skips the store entirely. A foreign anchor does not fail when applied: it prices one architecture's kernels onto another and returns a confident number for a machine nobody ran. Scoring MiniMax-M3 against a DeepSeek-V4-Flash anchor that way came back 1.46x off with nothing said. INFERASIM_ALLOW_FOREIGN_ANCHOR keeps the deliberate cross-model experiment available, since that is a real thing to want to measure. Co-authored-by: Cursor <cursoragent@cursor.com>
…loyed in Asked for the best way to serve a frontier MoE at agentic context on MI355X, the inference tuner returned nothing at all: forty seed trials, forty rejections, "no legal serving config found". The search space could not describe a deployable configuration, so every candidate was thrown out on memory or latency before its throughput was read. Six things were missing, and each one alone was fatal: - Attention data parallelism was absent from the search entirely, though the projector and memory model have supported it throughout. For a model whose KV is one latent that every head reads, this is the difference between storing the cache once per replica and once per rank. - Weights could only be bf16 or fp8. These checkpoints ship at 4 bits and do not fit on this part at 8, so every trial was rejected on memory. - The dense-MLP and attention projections had no precision of their own, so a 4-bit checkpoint was sized at 4 bits and streamed at 8. - Prompts were admitted to the engine unchunked. At 130k the activation working set alone is 575 GB, twice the device. - Prefix reuse could not be stated. An agentic trace resends a long transcript and reprefills almost none of it; priced as a cold prompt, TTFT came back at ninety-six seconds. - A checkpoint's own sparse-attention indexer was dropped when the architecture was read, so a model trained to read 1024 KV entries per query was served dense against 130k. That is a thirteen-minute first token. Two of these are not tuning knobs at all but properties of the traffic and the checkpoint, and they are seeded from the workload rather than searched. Also stop dropping --max-concurrency when the profile leaves it unset. The documented default is the batch size, but None was passed straight through and the flag omitted, so the projector admitted its own default instead of the batch being tuned and returned a TPOT 4.5x high. All four models now yield legal configurations that clear both the memory cap and the latency budgets. DeepSeek-V4-Pro settles on TP8 with attention DP across the group at batch 32: 112 GB/GPU, TTFT 2401 ms, TPOT 30.8 ms. Co-authored-by: Cursor <cursoragent@cursor.com>
The agent could authenticate with a bearer token and a base URL and nothing else, which is enough for a vendor endpoint and not enough for a gateway in front of one. A gateway typically wants its own subscription key and an identity for the caller, neither of which the token or the URL carries, so every completion was refused before a model saw it -- first for a missing subscription key, then for a missing caller identity. The agent stage degraded quietly when this happened: the LLM was unreachable, no trial ran, and the run still finished by re-submitting the seed incumbent as its answer. Co-authored-by: Cursor <cursoragent@cursor.com>
Acceptance rate and draft cost were free parameters with nothing coupling them, so the search set acceptance to 1.0 and draft cost to 0 and collected the throughput. It reported 354 tok/s/GPU for DeepSeek-V4-Pro against an honest 142, and eighteen of the fifty-eight legal trials in that run rested on the same corner. A draft model that is never wrong is the target model, and running it is not free. Speculation now requires an acceptance rate strictly inside (0, 1) and a nonzero draft cost. The axis stays searchable; only the corner that cannot be built is closed. Co-authored-by: Cursor <cursoragent@cursor.com>
Requiring a nonzero draft cost closed the corner where the draft was perfect and free, but left the class of problem untouched: acceptance and cost are one object described by two independent knobs, so the search went to acceptance 0.95 at a cost of 0.10 -- the same free lunch, one step less brazen -- and reported 367 tok/s/GPU against an honest 194. What a draft has to buy is the odds of acceptance, and near certainty those get superlinearly expensive because a draft that is always right is the target model. The cost floor now follows the odds: about 0.12 of a step at acceptance 0.7, 0.45 at 0.9, and past a full step just below 0.96, where there is no longer any reason to speculate at all. Co-authored-by: Cursor <cursoragent@cursor.com>
…d by The projector reports twenty-odd serving metrics. The tuner parsed nine and could rank on six, so most questions anyone asks about a serving fleet were not questions it could be asked at all. The omission that mattered most was total tokens per GPU -- prompt plus generation -- which is the figure InferenceX leads with and which the tuner never even read off the output. It is not a rescaling of decode throughput: at the agentic shape the prompt is 144x the generation, the two differ by more than a hundredfold, and they order configurations differently. Asked for the best serving configuration, the tuner was answering about generation alone. Also now readable: the prompt side per GPU rather than only as a fleet total (ranking on a fleet total rewards buying more GPUs), what a single user feels as distinct from what the fleet delivers, and how much of the decode budget prefill chunks are eating. Twenty-one objectives, each mapped to the InferenceX metric it answers. Energy is deliberately not among them: InferenceX reports average power and joules per token, the projector models no power at all, and an objective that scored None every time would read as a failed search rather than a missing model. Co-authored-by: Cursor <cursoragent@cursor.com>
DeepSeek-R1's own warmup was refused for a DeepSeek-R1 projection. R1 runs on the V3 architecture preset, so INFERASIM_MODEL says "deepseek_v3" while the artifact says "deepseek-ai/DeepSeek-R1", and identity was resolved from the preset alone -- the two names disagree because one names an architecture and the other names weights. Calibrating R1 against its own measurement was only possible by setting INFERASIM_ALLOW_FOREIGN_ANCHOR, which claims a cross-model experiment that is not happening and disables the check for real mismatches too. Widening models_match so the architecture stands in for the checkpoint would be the wrong repair, and there is a test that says so: an anchor measures expert routing, which is weights, not geometry, so a V3 anchor may legitimately fail to price R1. That test is unchanged. Instead the target may name its own checkpoint. --bench-model / INFERASIM_BENCH_MODEL already carries the real checkpoint id for the benchmark path, which needs it for the same reason -- a structural preset does not name weights -- so anchor identity now honours it and falls back to the preset. Both the explicit --load-benchmark guard and the store's model filter resolve through one helper, so they cannot disagree about which model a projection is for. A foreign anchor is still refused, and the message now points at the fix rather than only at the override. Co-authored-by: Cursor <cursoragent@cursor.com>
The seed sweep stopped at batch 64 with an unchunked prompt, which at agentic context is not a configuration that runs -- batch 64 unchunked costs a ~50 s first token -- so the entire large-batch region read as infeasible and only the LLM stage ever reached it. That made the good answer cost twenty minutes of agent time and a reachable model endpoint, for a region the plan can simply enumerate. Concurrency in those seeds is placed at attention-DP tier boundaries rather than round numbers. A rank holds ceil(concurrency / dp) sequences, so TTFT is a sawtooth with period dp: it steps up at each boundary and falls across the tier, and the best point is always at a tier top. Sampling round numbers lands mid-tier and reports a worse configuration than exists. For DeepSeek-V4-Pro at 130k/900 the deterministic plan now finds batch 64 with a chunked prompt at 183 tok/s/GPU, against 130 before and 194 for the agent stage that previously had to discover it. Co-authored-by: Cursor <cursoragent@cursor.com>
Four of the catalogued objectives scored None on every trial, and a search whose every trial is unscoreable reports "no legal serving config found" -- which reads as an infeasible model rather than a missing metric. Asked to optimize any of them, for any of the four models, the tuner returned nothing and gave no hint why. Two different faults. max_sustainable_concurrency was parsed off the projection all along but never declared on the result, so asdict dropped it before scoring ever saw it; concurrency_used was lost the same way. MFU, TFLOP/s and iteration time are training-mode figures the serving path does not emit at all, and no declaration fixes that -- they now sit in an explicit unsupported set alongside the energy metrics, so the omission is stated rather than discovered. Also read the serving metrics that were being printed and thrown away: the step time when a prefill chunk lands in it, the weight and activation footprints separately from the total, collective time split by phase (prefill and decode pay very different collective costs, and one blended number hides which phase a parallelism choice is hurting), and the GPU count a replica costs. Co-authored-by: Cursor <cursoragent@cursor.com>
…able Three settings decide whether the high-throughput region is reachable, and the plan was pinning all three. Chunking was always on in those seeds. Chunking is what makes a cold 130k prompt survivable, but under heavy prefix reuse only a small tail is ever computed -- at 92% reuse a 130k prompt is a ~10k prefill -- and then chunking it just buys more steps for no benefit. Both are now seeded and the search decides. The token budget was pinned at 8192, which splits even that ~10k tail across steps that land in decode and inflate TPOT. A budget covering the post-reuse prompt is now seeded alongside it, worth ~2% throughput and ~5% TTFT on GLM. The concurrency ladder stopped at three quarters of the batch. Admission is the knob that buys TPOT back: a large batch that misses its TPOT budget at full admission is usually not the wrong batch, it is the right batch admitting too many sequences at once, and only a lower tier shows that. The ladder now reaches an eighth of the batch, which for GLM-5.2 at 130k/900 takes the legal configuration count from 14 to 61. Co-authored-by: Cursor <cursoragent@cursor.com>
The ladder was capped at the batch, on the assumption that admitting more requests than a step runs is meaningless. It is not: concurrency is how many requests are in flight, not how many run in a step, and a scheduler holding more than it runs is ordinary continuous batching -- the surplus is what keeps the next step full. What bounds it is the memory model, not the batch. The cap hid the region where these models are actually fastest. On GLM-5.2 at 130k/900 the agent stage kept walking out of the seeded region to batch 256 admitting 312, which fits in 258.8 GB of a 288 GB card and holds both SLOs, and the deterministic plan could not express it at all. Every objective that ranks on throughput was being answered from the wrong half of the search space. The ladder now runs from an eighth of the batch to twice it, still on tier boundaries, and the memory gate rejects the rungs that do not fit. Co-authored-by: Cursor <cursoragent@cursor.com>
Two faults that cancelled into plausible-looking numbers. GLM-5.2 and MiniMax-M3 are both sparse-attention models -- a top-2048 window, stated in the first ten lines of each model file -- but neither declared it anywhere a program could read, so every projection of them ran dense. At 130k context that is sixty times the attention reads they actually perform. Meanwhile the tuner offered a 512/2048 indexer to every model as though it were a serving knob. For the two above that accidentally restored something true. For Kimi-K3, which has no indexer, it invented one, and the search took it every time because at this context nothing else comes close -- reporting a throughput that checkpoint cannot reach. Sparsity is an architectural property; a model is served with the window it was trained with, or dense. Both models now declare their window under the request-side name their own headers already told readers to use, and the tuner offers each model only its own window, or off. Kimi-K3 is served dense, because it is dense. index_topk is deliberately not the field used: _is_dsv4 falls back to that key to recognise re-exported DeepSeek checkpoints, and GLM-5.2 carrying it would hand GLM the dsv4 gfx942 kernel path. Co-authored-by: Cursor <cursoragent@cursor.com>
The multi-instance router derives each request's hit from a content-addressed block cache and hands the seeded requests over in ``prebuilt``. The single-engine path had no equivalent: it accepted --prefix-cache-hit-rate, built every request with no prefill progress, and then reprefilled the whole prompt. On an agentic shape, where almost none of a resent transcript is new, that is the difference between prefilling the uncached tail and prefilling the entire prompt. It surfaced as the simulator saturating at a small fraction of the rate the analytical path sustains for the same configuration -- a gap that reads as a dispute about physics and is really a dropped flag. Seeded with the same rule the analytical path uses, so the two agree on what a cache hit means: at least one token stays uncached, leaving a fully-cached prompt a forward pass to emit its first token from. Co-authored-by: Cursor <cursoragent@cursor.com>
Legality priced a replica at tp * pp * ep, so on a single node the full-width combination -- the one --enable-expert-parallel produces, where the same ranks hold disjoint experts -- was rejected as needing more GPUs than the cluster has. The projector it feeds disagrees: it reports replica GPUs as TP times PP and models that combination on those same ranks quite happily. The search therefore only ever saw expert parallelism with tensor parallelism shrunk to make room for it, a shape nobody deploys, and at the small bf16 batch those seeds carry, where it dies on memory before it can be scored. Expert parallelism came back as a finding when it was a constraint. Priced at TP times PP now, requiring only that the expert degree divide the replica's ranks. The seed sites that shrank TP to fit sweep at full width instead, and expert parallelism is seeded in the large fp4 region where the winners live, so it is rejected on latency or memory rather than on arithmetic. The answer does not move -- expert parallelism still loses on these models at this scale -- but it is now earned rather than assumed. Co-authored-by: Cursor <cursoragent@cursor.com>
The projector has offered per-request heterogeneity for a while -- a uniform spread around the configured lengths, or a replayed workload of arrivals and lengths -- but the tuning agent carried only the offered rate and arrival model, so every trial was priced at a single input/output point. No measured trace looks like that. The agentic ones span a wide range of prompt lengths, and a point estimate cannot express long requests holding their KV reservations while short ones cycle through: it moves the mean hardly at all while the tail roughly doubles. Since every latency budget here is written against a mean, that is the one statistic blind to the effect. Adds the spread, the replayed workload and the simulated request count as search axes, validated and emitted to the projector. Defaults leave a trial projecting exactly as it did. Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds
infera/projection/— InferaSim, a workload-driven simulator and projector for the serving stack. It answers serving-configuration questions without occupying a node to ask each one: time to first token, inter-token latency, throughput, KV and weight memory, feasibility, and cost. One honest experiment on a large model costs a full node for minutes, and these questions are asked in the thousands, so the search happens in simulation and hardware is spent only on the shortlist.It is deliberately two coupled models over one measured foundation:
The governing idea is measure sparsely, transport analytically: a small number of cheap sub-scale benchmarks are harvested into anchors, and every other configuration is projected from the nearest applicable anchor rather than measured.
The change is additive. 273 new files under
infera/projection/andtests/unit/projection/, with no existing platform code modified apart frompyproject.toml, which gains aprojectionextra and theinferasim/inferasim-tuneentry points.Type of change
Changes
README.md(task-oriented) andARCHITECTURE.md(how it works, and an explicit statement of what it does not model).Checklist:
152 unit tests under
tests/unit/projection/, all of which run without a GPU.