diff --git a/README.md b/README.md index 81b116a..e909d3d 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ # Frontier MoE on consumer Blackwell (SM120) -**Official vLLM v0.24.0 + a 7.4k‑line patch** that serves frontier Mixture‑of‑Experts models — +**Official vLLM v0.24.0 + a 14.0k-line, 70-file patch** that serves frontier Mixture-of-Experts models — **GLM‑5.2 (753B)**, **DeepSeek‑V4‑Flash (159B)** and **Kimi‑K2.7‑Code (1T)** — on consumer/workstation Blackwell (RTX PRO 6000, RTX 5090), hardware their official checkpoints cannot even fit on. Three ideas carry it: @@ -10,7 +10,8 @@ cannot even fit on. Three ideas carry it: gate) restores precision exactly where it matters. 2. **Tiered expert residency** — when even the 2‑bit base outgrows VRAM, it moves to pinned host RAM — and, one tier further, to an **NVMe pack file with a pinned‑RAM arena** — and - the GPU becomes an **expert cache** (miss → batched fetch + bit‑identical graph replay). + the GPU becomes an **expert cache** (miss -> batched fetch + graph replay, with explicit + telemetry for any accepted second-order approximation). That puts 753B on two 96 GB cards and 159B on a single RTX 5090, and the packs double as a **persistent quantization cache** (reboots skip the re‑quant). 3. **A rebuilt serving base** — vLLM v0.24.0 actually working on SM120 (the release is @@ -63,10 +64,13 @@ medians; prefill = 8k‑token prompt, uncached): | 2× RTX PRO 6000 (TP2) | 210 tok/s | 5 790 tok/s | 512K | — | | 4× RTX 5090 (TP4) | 214 tok/s | 6 100 tok/s | 16K | — | | **1× RTX 5090 (32 GB)** | **~31 tok/s** (14 GiB pool + NVMe stores) | ~400–540 tok/s | **32K** | **~30 GiB** | +| **1× RTX 5090 (32 GB), no-MTP quality config** | not measured | not measured | **128K; PASS at 120K** | **≤100 GiB eval cgroup incl. cache** | Retrieval behind the window column: needle PASS at 453K on the PRO 6000 (947K‑token KV -measured) and at 29.7K on the single 5090 (131K‑token KV). "—" in host RAM = all‑VRAM -config, no host expert store. +measured), at 29.7K on the single-5090 throughput configuration (131K-token KV), and 3/3 exact +at 120K on the canonical no-MTP 128K quality configuration (720,677-token KV). The 128K row is +a quality/context result, not a throughput measurement. "—" in host RAM = all-VRAM config, no +host expert store. **Batched serving** (aggregate decode tok/s at N concurrent streams; per‑stream in parentheses at N=32): @@ -81,6 +85,10 @@ configs. MTP also runs under **pipeline parallelism** (draft propagation + draft share across ranks): DS4 on 4× RTX 5090 **PP4** does 184 tok/s vs 93 without (~2×), and greedy decode under PP is **bit‑deterministic** (6/6 identical runs, with and without MTP). Methodology: **[docs/v024-port.md](docs/v024-port.md)**. +The guarded single-5090 W2 quality/context result, including P0/P1/P2 evidence boundaries, is +reported in **[docs/benchmarks/ds4-w2-5090-2026-07-11.md](docs/benchmarks/ds4-w2-5090-2026-07-11.md)**; +the sanitized receipt index is +**[evidence/public/ds4-w2-2026-07-11/](evidence/public/ds4-w2-2026-07-11/)**. --- @@ -128,11 +136,12 @@ routed working set). MoE routing is concentrated enough to make this practical: coverage serves ~96% of token→expert routings** on DS4, **~51% serves ~91%** on GLM — measured live, not simulated. -Misses stay correct through the gate's replay trick: the desc kernel zeroes a missing +First-order misses are restored through the gate's replay trick: the desc kernel zeroes a missing expert's contribution and bumps an in‑graph miss counter; the runner fetches **all** missing routed experts in one batched pinned‑H2D transfer (51.6 GiB/s here; a 64‑expert fetch ≈ 3 ms) -and replays the step's graph once — **bit‑identical** to a fully resident forward -(unit‑tested). A **miss‑tolerance knob** (`VLLM_MOE_W2_BASE_MISS_TOL=k`, runtime‑tunable) +and replays the step's graph once. The result is bit-identical to a fully resident forward +only when that replay creates no second-order misses (unit-tested for the zero-residue case). +A **miss-tolerance knob** (`VLLM_MOE_W2_BASE_MISS_TOL=k`, runtime‑tunable) skips the replay when ≤ k of the step's ~600 routings miss — +12% decode on GLM TP2 at tol 8 (28.3 → 31.7 tok/s) with clean quality probes (arithmetic, PL coherence, needle retrieval; quantitative eval pending). @@ -149,14 +158,20 @@ e.g. `--gpu-memory-utilization 0.95`) before touching any other knob. **Misses are restored adaptively.** A replayed step can re‑route onto experts the first pass never fetched (second‑order misses); the runner re‑checks after each replay and keeps replaying **only while the step is within `VLLM_MOE_W2_FP_THRESH` of miss‑free** (default 0 -= the mandatory first‑order restore only, the throughput‑optimal setting; raising it buys -bit‑deterministic fixed points on converged working sets at a decode cost — runtime‑tunable -via `VLLM_MOE_W2_FP_THRESH_FILE`). The accepted residue is second‑order only and -KPI‑visible (`fp-residue`). += the mandatory first‑order restore only, the throughput‑optimal setting; raising it can chase +zero-residue fixed points on converged working sets at a decode cost, but does not itself promise +deterministic output — runtime‑tunable via `VLLM_MOE_W2_FP_THRESH_FILE`). At the default +threshold, newly fetched second-order +experts are available to later steps but their current-step contributions remain zero. This +is an explicit throughput/quality approximation, not a miss-free or bit-deterministic claim; +its frequency and pair count are KPI-visible as `fp-residue` and are reported beside quality. Results: **DeepSeek‑V4‑Flash 159B on one RTX 5090** (72.7 GiB of 2‑bit planes vs 32 GB of VRAM): ~31 tok/s steady with MTP, 32K window served, coherent — and ~30 GiB of host RAM with the NVMe stores (below, RSS‑measured) instead of ~80 GiB pinned. +The canonical no-MTP quality configuration separately serves a 128K window: 120/120 +machine-exact and semantically correct, 0/120 frozen-rule sink detections, and 3/3 exact +retrieval at 120K prompt tokens. Throughput was not measured for that configuration. **GLM‑5.2 753B on two RTX PRO 6000**: 28–32 tok/s with the full three‑tier stack (NVMe 2‑bit base + pinned arena → GPU 2‑bit cache → GPU FP4) at a 128K single‑user window — see the GLM table above. Neither model can otherwise run on that hardware at any precision. @@ -214,15 +229,53 @@ Operational notes: drive on a PCIe **Gen3 x4** link (3.7 GB/s). Cold working‑set shifts and first‑touch prefills do pay drive speed. `VLLM_MOE_W2_TIER_DIRECT=1` switches misses to O_DIRECT (hard RAM budget, page cache stays flat; raw drive latency on every miss). -- **Prefill can't wipe the arena** (scan discipline: prefill working sets fill free slots - but never evict the decode hot set), and the arena's hot set persists to +- **Cold pack rebuilds fail closed instead of consuming the host.** MXFP4 expert parameters are + zero-sized at model construction and materialized only for the layer currently loading; that + layer is quantized, durably written, and released before the next one starts. Exact pack + identity skips the build entirely, and completion guards prevent a streamed layer from being + built twice. While W2 + a pack store are active, the sequential safetensors loader also + preflights every shard, requests eviction of + each released shard with `POSIX_FADV_DONTNEED`, and retries after the model consumer unwinds; + the pack writer does `fdatasync` + `DONTNEED` after + every completed layer. Explicit checkpoint prefetch is refused and a requested + multi-thread load is serialized. The final mmap-backed tensor per shard is cloned under the + same guard so the consumer cannot keep the whole file mapped. Pinned-arena allocations and + indivisible shard/layer I/O + keep a **16 GiB host MemAvailable floor** and **4 GiB hard cgroup `memory.max` + headroom** by default. `memory.high` remains an observable soft reclaim/throttle boundary; + it does not reduce the hard allocation budget. Tune the floors with + `VLLM_MOE_W2_MIN_MEM_AVAILABLE_GB` and + `VLLM_MOE_W2_MIN_CGROUP_HEADROOM_GB`; `VLLM_MOE_W2_CACHE_CONTROL=required` is the safe + default (`best-effort` or `off` are explicit unsafe overrides). Linux cgroup-v2 page cache is + accounted in `memory.current`; every check logs anon/file/file-mapped/swap/events plus its + available/headroom/transient budget for a cold-restage memory trace. In the all-43-layer + cold canary, cgroup memory.current peaked at 51.81 GB; observed anon/file maxima were + 8.23/44.43 GB, host + `MemAvailable` stayed above 73.79 GB, cgroup swap and PSI/OOM counters remained zero, + and the host recorded no swap-out; host swap-in increased by 160 pages. The full canary used + the mechanism-equivalent `e7417054a6e8` + predecessor; the integrated `41d7b2f96ca3` image retains that P0 code and passed the exact-head + baked safety suites. See the [sanitized P0 receipts](evidence/public/ds4-w2-2026-07-11/p0/). +- **Prefill is semantic, eager, and arena-safe.** The runner derives prefill from prompt + progress and propagates it explicitly through normal, replay, pipeline, gate, and DBO paths; + capture is refused for a real W2 prefill. Prefill disables draft-affinity/gate promotion, + holds exact per-layer BASE pins through the forward, and uses fill-only arena scan discipline. + Long prefills use the bulk base-only path; short prefills retain FP4 recovery without being + misclassified as decode. The arena's hot set persists to `.heat.json` and **preheats on boot** (57 GiB ≈ 35 s; `VLLM_MOE_W2_TIER_PREHEAT=0` opts out). Reader pool: `VLLM_MOE_W2_STORE_THREADS` (default 8). +- **Tier managers run only between forwards.** A target step acquires the same exclusion lock + used by background promotion/eviction passes and holds it through target execution, BASE + replay, confidence-gate re-forward, and the pipeline barrier. Only the worker's final + `finish_forward_step()` releases the lock and wakes one manager pass. LRU timestamps use a + floating-point comparison key, and force-promote plus manager ticks consume immutable + layer-local seen sets, so saturated-LRU eviction and background tiering cannot mutate a live + route or overflow an integer timestamp. - **Observability:** with `VLLM_MOE_W2_DELTA_TRACE=1` the summary carries a `[base] tiered store: arena N/M | fetch rows X ram + Y nvme (Z% ram) | … p50/p99` line — the ram‑hit % *is* the arena‑coverage curve; grow `BASE_RAM_GB` if it sags. -- Replays stay **bit‑identical** across backends (bytes are bytes; only the copy source - changes) — unit‑tested per backend × cold/warm/reboot/evict/overflow/scan/preheat in +- Expert-store rows stay **byte-identical** across backends (only the copy source changes) — + unit-tested per backend × cold/warm/reboot/evict/overflow/scan/preheat in `tools/test_store_backends.py`. --- @@ -424,8 +477,8 @@ Release **`baseline-2026-07-10`** — one row per supported recipe (`bench/recip ## Repository layout -- **`patch/vllm-moet-v0.24.0.patch`** — the delta vs official vLLM `v0.24.0` (37 files, - +7.4k lines; applies clean on the tag). Goes with the pins above. +- **`patch/vllm-moet-v0.24.0.patch`** — the single canonical delta vs official vLLM `v0.24.0` + (70 files, +14,018/-226 source lines; applies clean on the tag). Goes with the pins above. - **`Dockerfile.sm120-v024`** — the image: official `vllm/vllm-openai:v0.24.0` + patch + pins + cubins. - **`kernels/`** — SASS (`sass/`) + prebuilt SM120 cubins (`cubins-sm120/`, incl. the K=6144 diff --git a/docs/benchmarks/ds4-w2-5090-2026-07-11.md b/docs/benchmarks/ds4-w2-5090-2026-07-11.md new file mode 100644 index 0000000..3384d92 --- /dev/null +++ b/docs/benchmarks/ds4-w2-5090-2026-07-11.md @@ -0,0 +1,354 @@ +# DeepSeek-V4-Flash W2 on one RTX 5090 — 2026-07-11 + +The W2 lane now passes all three release priorities on one 32 GB RTX 5090: + +- **P0 passes:** the exact frozen release candidate completed an actual 46-shard, 43-expert-layer + cold restage under bounded memory. During the cold-restage window it retained a + 103,726,841,856-byte host-available floor, used no cgroup or host swap, recorded no memory-limit + or OOM event, and restored the pre-existing serving surface with both protected pack + fingerprints unchanged. +- **P1 passes at 32K:** the historical predecessor no-MTP configuration produced **119/120 exact-clean + answers and 0/120 sink detections under the frozen rules** across seeds 42, 43, and 44. All + **30/30 fixed prewarm prompts** passed without retry, and every seed passed both its pre-score + and post-score pool gate. +- **P2 passes at 128K:** the comprehensive `41d7b2f` artifact produced **120/120 machine-exact and + semantically correct answers with 0/120 sink detections**, then retrieved exact needles at + 120,000 prompt tokens at depths 0.1, 0.5, and 0.9. The frozen RC supplement then produced + **79/80 machine-exact, 80/80 semantically correct, and 0/80 sinks** and repeated the deepest + exact 120,000-token case. + +This is an empirical reliability result for the recorded sampler and pool policy. It is **not** +a claim of bit-deterministic output, miss-free replay, or zero second-order residue. + +## Artifact lineage + +The evidence intentionally preserves four artifact roles rather than implying that every +measurement came from one image: + +| Role | Source patch / image | +|---|---| +| P0 all-layer cold canary | mechanism-equivalent predecessor `e7417054a6e8`; measured memory belongs only to this canary | +| Historical P1 32K receipts | patch `55d30bb9cf9bef45e7130fd5afe0090c5b540be22f6de3abfc51b782f738a6f7`; image `sha256:7385d21d26b665884e97a97dc67a100db328ed7b00b634e4d18f8aedd9f29eab` | +| Comprehensive integrated P2 | official vLLM `ee0da84ab9e04ac7610e28580af62c365e898389`; patch `41d7b2f96ca3b966cac1b7ed5cff37bc03c27c616ed254aca961cc75a9ffe31d`; image `sha256:66abc2f145244e03ff0f0fcca088be813ad311f2963c62b281e4bb888ac605e9`; wrapper `5e69228e1e8e6f7345ae96657ab33b50e5805be7` | +| Frozen RC, exact-head verification, and focused acceptance | origin/main cutoff `94be3aa3d7a8b82c7fc9687990a7edb6035f69f3`; official vLLM `ee0da84ab9e04ac7610e28580af62c365e898389`; patch `241ba984b1c56f5dc7adbc8d7f519d60b5746024bf7dfeb875e3546a668e79a7`; image `sha256:fc6e1244d60855fe45ccc0236daaaa722abcb8d354200eb92aca104bd954d3f2`; wrapper `0a73c2bcf262ba0aa53560d6290ad2af350e34fc` | + +The comprehensive `41d7b2f` patch changed 70 files with 14,018 insertions and 226 deletions; the +frozen RC changes 70 files with 14,223 insertions and 226 deletions. Apply-check and +whitespace-check pass, all 70 baked source files match the freshly applied tree, and the exact RC +image passes the CUDA padded-route, store-safety, MXFP4 stream-safety, GPU store-backend, and +base/full-FP4/split-FP4 numerical forward suites. Split-FP4 over the base cache is now supported +behind its explicit flag with order-independent tier/base residency coupling. The default serving +sentinel kept split mode off and used full-nibble FP4 planes; a separate exact-image E=8 +three-tier GPU fixture validated split mode and transient-miss/eviction behavior. The sanitized +receipt is +[`exact-image-verification.json`](../../evidence/public/ds4-w2-2026-07-11/exact-image-verification.json). + +## P0 — bounded cold restage + +**Outcome: the cold-restage wedge mechanism is bounded, and the exact frozen RC repeated the +full cold-restage path.** The earlier mechanism canary completed all 43 routed-expert layers in +249.2 seconds. Its historical measurements remain below for provenance. + +| Observed metric | Historical all-layer cold canary | +|---|---:| +| Samples | 245 | +| Host `MemAvailable` minimum | 73,793,277,952 bytes | +| Cgroup `memory.current` maximum | 51,807,191,040 bytes | +| Cgroup memory peak | 51,807,522,816 bytes | +| Anonymous-memory maximum | 8,231,129,088 bytes | +| File-cache maximum | 44,430,127,104 bytes | +| Cgroup swap maximum | 0 bytes | +| Pressure-full `avg10` maximum | 0.0 | +| `memory.high`, `memory.max`, OOM, and OOM-kill event deltas | all 0 | +| Host swap-in / swap-out page deltas | 160 / 0 | +| Host OOM-kill delta | 0 | +| Production pack fingerprint after cleanup | unchanged | +| Pre-existing serving surface after cleanup | restored and healthy | + +The frozen RC then cold-loaded all 46 checkpoint shards and built all 43 expert-pack layers. Shard +loading completed in 203 seconds and the model became ready 340 seconds after container start. +The cold-restage-only trace covers the first 220.7 seconds through completion of all checkpoint +shards: + +| Observed metric | Frozen RC cold restage | +|---|---:| +| Samples | 218 | +| Host `MemAvailable` minimum | 103,726,841,856 bytes | +| Cgroup `memory.current` / peak maximum | 22,386,843,648 / 22,386,843,648 bytes | +| Anonymous-memory maximum | 12,597,563,392 bytes | +| File-cache maximum | 11,680,821,248 bytes | +| Cgroup swap maximum | 0 bytes | +| Pressure-full `avg10` maximum | 0.54 | +| `memory.high`, `memory.max`, OOM, and OOM-kill event deltas | all 0 | +| Host swap-in / swap-out page deltas | 0 / 0 | +| Protected pack fingerprints after cleanup | both unchanged | +| Pre-existing serving surfaces after cleanup | restored and healthy | + +The implementation fixes the original page-cache and staging failure mode at its source: + +1. MXFP4 expert tensors are materialized, quantized, durably written, and released one layer at + a time instead of accumulating the whole checkpoint. +2. Released checkpoint shards and completed pack layers receive cache-drop advice; checkpoint + eviction is retried after the model consumer releases its final mapping. +3. Explicit checkpoint prefetch is refused, requested parallel shard loading is serialized, and + an exact completed pack skips restaging. +4. Each indivisible shard, layer, and arena operation preflights both host-available memory and + hard cgroup headroom. Required cache control fails closed; `memory.high` remains an observable + soft boundary rather than being mistaken for hard allocation capacity. +5. The guarded cleanup path verifies that the temporary lane is gone, the original pack is + byte-identical by fingerprint, and the serving surface is healthy again. + +Evidence: + +- [Frozen RC cold-restage summary and trace](../../evidence/public/ds4-w2-2026-07-11/p0/frozen-rc-summary.json) + ([memory trace](../../evidence/public/ds4-w2-2026-07-11/p0/frozen-rc-memory-trace.tsv)) +- [All-layer cold-restage summary and trace](../../evidence/public/ds4-w2-2026-07-11/p0/cold-restage-summary.json) + ([memory trace](../../evidence/public/ds4-w2-2026-07-11/p0/cold-restage-memory-trace.tsv)) +- [Cleanup receipt](../../evidence/public/ds4-w2-2026-07-11/p0/cold-restage-cleanup-receipt.json) +- [Constrained pack-build probe](../../evidence/public/ds4-w2-2026-07-11/p0/baked-constrained-pack-build.json) +- [Checkpoint-residency and cache-drop probe](../../evidence/public/ds4-w2-2026-07-11/p0/baked-checkpoint-residency.json) +- [Next-shard cache-drop retry probe](../../evidence/public/ds4-w2-2026-07-11/p0/baked-checkpoint-retry.json) + +### P0 evidence boundary + +The 245-sample table belongs only to mechanism-equivalent predecessor patch `e7417054a6e8`. The +218-sample table belongs only to the exact frozen RC and ends after checkpoint-shard completion; +it intentionally excludes the later quality and context workload. The final RC nevertheless ran +the actual full 46-shard, 43-layer cold-restage lifecycle, passed the exact-image mechanism probes, +and completed guarded cleanup. The two traces are retained separately rather than combining their +maxima. + +## P1 — stable 32K quality and liveness + +**Outcome: P1 moves from 16/120 historical sinks to 0/120 sink detections under the frozen rules on the historical predecessor +configuration, with 119/120 exact-clean answers.** Each seed independently meets the requested +0–2 sink bar. + +The final configuration is a 32,768-token window, 8 GiB BASE cache, 6 GiB FP4 delta cache, LRU +delta policy, confidence-gate tau 0.75, FP8 KV cache, MTP disabled, and one maximum sequence. The +40-item evaluation uses fixed interleaving, temperature 0.6, top-p 0.95, a 700-token ceiling, no +retries, and eval seeds 42–44. + +| Eval seed | Historical exact-clean | Historical sinks | Final exact-clean | Final sink detections | Fixed prewarm | Pre/post pool gates | +|---:|---:|---:|---:|---:|---:|---| +| 42 | 27/40 | 5/40 | 39/40 | **0/40** | 10/10 | pass / pass | +| 43 | 22/40 | 7/40 | 40/40 | **0/40** | 10/10 | pass / pass | +| 44 | 27/40 | 4/40 | 40/40 | **0/40** | 10/10 | pass / pass | +| **Combined** | **76/120** | **16/120 (13.3%)** | **119/120** | **0/120** | **30/30** | **6/6 pass** | + +The lone final exact mismatch is seed 42 item `r17`: it ends with a mathematically equivalent +formatted `3/8`, but the frozen exact-normalization rule does not accept that rendering. It is +reported as non-exact and is not a sink; it is not silently promoted into the exact-clean total. + +The baseline is context, not a causal control arm. It is a corrected re-score of historical warm +runs at tau 0.67, without the new fixed-prewarm and pool-lifecycle comparability contract. The +final series uses tau 0.75 and predecessor provenance. The before/after table therefore establishes +the operational delta, but it does not attribute that delta to one isolated knob. Baseline details +are in the [sanitized corrected re-score](../../evidence/public/ds4-w2-2026-07-11/baseline/README.md) +and [machine-readable aggregate](../../evidence/public/ds4-w2-2026-07-11/baseline/baseline-corrected.json). + +### Prewarm and pool gates + +Every final seed first passed the same ten-prompt `ds4-w2-prewarm-v4` suite at temperature 0, +top-p 1, and fixed seed `20260711`, with no retry. The release policy was frozen before scoring at +SHA-256 `1afb9b0d96d05c9d7a374dff88c94c76a3e854c8d046fb8ef1783275ad3f5630`. +It requires full FP4 occupancy, zero unrestored BASE experts, a live FP4 tick/eviction count, and +at least 16 additional evictions over each scoring run. All pre-gates and post-gates passed: + +| Eval seed | FP4 occupancy before/after | Unrestored BASE before/after | Post-score eviction delta | Post-score residue, latest 64-step window¹ | +|---:|---:|---:|---:|---:| +| 42 | 481/481 → 481/481 | 0 → 0 | 76,911 | 63/64 | +| 43 | 481/481 → 481/481 | 0 → 0 | 80,902 | 64/64 | +| 44 | 481/481 → 481/481 | 0 → 0 | 75,113 | 63/64 | + +¹ Residue is retained telemetry, not a release-policy rejection threshold; the approximation and +its interpretation are explicit below. + +The frozen release policy is +[`pool-gate-policy.json`](../../evidence/public/ds4-w2-2026-07-11/p1/pool-gate-policy.json). +Per-seed manifests, raw rows, and prewarm receipts are published as +[seed 42](../../evidence/public/ds4-w2-2026-07-11/p1/seed42.manifest.json), +[seed 43](../../evidence/public/ds4-w2-2026-07-11/p1/seed43.manifest.json), and +[seed 44](../../evidence/public/ds4-w2-2026-07-11/p1/seed44.manifest.json). + +### Explicit second-order approximation + +The BASE miss path first fetches every expert pair missed by the routed forward, then replays the +step. Correcting early layers can change later routing and expose experts that the original pass +never requested. At the default continuation threshold (`FP_THRESH=0`), those second-order experts are +fetched for later steps, but their contribution to the current step remains zero. Consequently, +`base_unrestored_experts = 0` proves fetch/store liveness; it does not prove a miss-free forward. + +A separately frozen strict diagnostic required zero residue. It passed 10/10 prewarm prompts but +failed before scoring: 57 of the latest 64 steps carried second-order residue, despite 481/481 FP4 +occupancy and zero unrestored experts. The sanitized diagnostic is +[`strict-zero-residue-diagnostic.json`](../../evidence/public/ds4-w2-2026-07-11/p1/strict-zero-residue-diagnostic.json). +The run manifests therefore retain residue snapshots, while the final quality policy deliberately +omits a residue rejection threshold. The three-seed frozen sink-detection result is the empirical +acceptance test for that explicit approximation. + +This boundary matters: the final post-score windows still report second-order residue. The P1 +verdict is stable observed quality plus pool liveness, **not** bit-identical logits, identical token +streams across boots, or mathematical equivalence to a fully resident forward. + +### Why the implementation stopped corrupting the run + +The predecessor patch fixed the implementation-level state hazards exposed by the earlier runs; +the integrated patch retains them and adds the manager/forward exclusion described below: + +- Prefill is derived from prompt progress and propagated explicitly through normal, replay, + pipeline, gate, and microbatch paths. Real W2 prefill is eager; long prefills use the aligned + bulk path, while short prefills keep FP4 recovery instead of being misclassified as decode. +- Padded CUDA-graph routes carry an explicit token-to-slot mapping, so padding cannot enter miss, + promotion, or output accounting. Routed rows are padded to the kernel block size and invalid + alignment fails loudly. +- BASE residency uses an immutable, layer-local seen set. Prior main-stream work is drained before + reuse, only the previous layer's pins are cleared, and each current layer's hits/promotions stay + pinned through its forward. A later layer can no longer overwrite a shared snapshot or evict the + rows still in use. +- Background tier managers use the same exclusion lock as the foreground forward. The target step + acquires it before routing, and the worker holds it through BASE replay, confidence-gate + re-forward, and the pipeline barrier; `finish_forward_step()` alone releases it and wakes one + manager pass. `step_begin()` only clears pins. LRU comparison keys are floating-point before + assigning infinity, and force-promote plus manager ticks receive immutable seen sets. +- Draft-affinity prefetch and confidence-gate promotion are disabled during prefill, and real + prefill under capture is refused instead of silently taking decode semantics. + +### Near-edge context and lifecycle + +The same 32,768-token no-MTP server accepted **3/3 exact needles at 30,000 prompt tokens** at +depths 0.1, 0.5, and 0.9. Tokenizer calibration and response usage both had to report exactly +30,000 tokens, and each terminal passphrase had to match exactly. This is P1 near-edge validation +and a rehearsal of the P2 harness; it is not a 128K result. See the +[context manifest](../../evidence/public/ds4-w2-2026-07-11/p1/context-30k.manifest.json) and +[receipts](../../evidence/public/ds4-w2-2026-07-11/p1/context-30k.receipts.jsonl). + +Across scoring and context, cgroup swap, hard-limit, and OOM events stayed at zero. The 96 GiB +soft `memory.high` boundary was crossed narrowly for 13 samples and generated reclaim events; +PSI full `avg10` peaked at 0.49. The guarded lane retained at least 4,294,172,672 bytes of hard +headroom, then stopped cleanly, preserved both pack fingerprints, restored both services, and +removed the 230,317,622,095-byte disposable store. Public receipts: +[aggregate](../../evidence/public/ds4-w2-2026-07-11/p1/aggregate.json), +[runtime and memory audit](../../evidence/public/ds4-w2-2026-07-11/p1/runtime-clean.json), and +[cleanup](../../evidence/public/ds4-w2-2026-07-11/p1/cleanup.json). + +### Excluded MTP diagnostic + +The MTP-on investigation crossed noncanonical patch/image identities and used a different runtime +shape, so it contributes **zero rows** to the final 120-response series. Its private diagnostic +receipts are not used to support any public result. Combining it with the historical no-MTP seeds +would erase both the provenance and gate boundary; this report does not do so. + +### Excluded first integrated candidate + +The first upstream-integrated candidate (`ec4640f` patch generation, `5467a7ff` image prefix) +reached a healthy 128K server but failed frozen prewarm at item 3 before scoring. It contributes +zero quality rows and no context receipt. That failure exposed a background tier-manager pass +overlapping the live target/replay path. Intermediate patch `4708c9d41b50` added the shared +forward-exclusion lifecycle and passed the full gate. The comprehensive `41d7b2f96ca3` union then +integrated the newer upstream split-FP4 work, preserved that exclusion and restage safety, made +split mode fail closed over the base cache at that generation, and reran the entire gate from a +fresh namespace. The later frozen RC adds explicit split/base residency coupling and validates it +separately while leaving split mode off in the serving acceptance run. + +## P2 — 128K context + +**Outcome: the comprehensive `41d7b2f` single-5090 configuration serves a validated 131,072-token window and +retrieves 3/3 exact needles at 120,000 prompt tokens without weakening the quality gate.** This is +the largest context tested in this lane. No decode- or prefill-throughput claim is attached to it. + +The P2 runtime keeps the P1 BASE/FP4 policy—8 GiB BASE, 6 GiB delta, LRU, tau 0.75, FP8 KV—but +sets `--max-model-len 131072`, disables MTP, and permits one maximum sequence. Model loading used +22.87 GiB of device memory. The engine exposed 720,677 KV tokens and reported 5.50× capacity for +a full 131,072-token request. The exact image reached health 200, completed the controller's +30-second hold, and +entered scoring with zero start-to-ready swap, memory-high, hard-limit, or OOM events. See the +[readiness receipt](../../evidence/public/ds4-w2-2026-07-11/p2/readiness.json) and +[sanitized server identity](../../evidence/public/ds4-w2-2026-07-11/p2/server.json). + +### 128K quality + +Each seed repeated the same frozen 40-item sampler and ten-prompt prewarm used for P1: + +| Eval seed | Machine exact | Semantic equivalent | Sink detections | Fixed prewarm | Pre/post gates | Score-time evictions | +|---:|---:|---:|---:|---:|---|---:| +| 42 | 40/40 | 40/40 | **0/40** | 10/10 | pass / pass | 42,760 | +| 43 | 40/40 | 40/40 | **0/40** | 10/10 | pass / pass | 43,118 | +| 44 | 40/40 | 40/40 | **0/40** | 10/10 | pass / pass | 44,469 | +| **Combined** | **120/120** | **120/120** | **0/120** | **30/30** | **6/6 pass** | **130,347** | + +Seed 44 item `c18` returned `(2,5)` for expected `(2, 5)`. The frozen exact scorer normalizes that +tuple whitespace and counts it exact; the legacy `lenient` diagnostic does not. It is not a sink, +and the machine total remains 120/120. All 120 responses were HTTP 200 with +`finish_reason=stop`, and no retries or request errors occurred. The public +[quality aggregate](../../evidence/public/ds4-w2-2026-07-11/p2/quality-aggregate.json) and +[per-seed manifests for seed 42](../../evidence/public/ds4-w2-2026-07-11/p2/seed42.manifest.json), +[43](../../evidence/public/ds4-w2-2026-07-11/p2/seed43.manifest.json), and +[44](../../evidence/public/ds4-w2-2026-07-11/p2/seed44.manifest.json) retain the machine/semantic +boundary and pool snapshots. + +### Exact 120K retrieval + +| Needle depth | Tokenizer count | Response usage | Exact answer | Request wall time | +|---:|---:|---:|---|---:| +| 0.1 | 120,000 | 120,000 | pass | 250.041 s | +| 0.5 | 120,000 | 120,000 | pass | 248.028 s | +| 0.9 | 120,000 | 120,000 | pass | 245.272 s | + +All three admitted receipts were fresh zero-tolerance requests. Each case calibrated to exactly +120,000 tokens through `/tokenize` before its sole inference; no rejected calibration contributed +a chat response in this run. Response usage also read 120,000 and each terminal passphrase +matched exactly. The public +[composite manifest](../../evidence/public/ds4-w2-2026-07-11/p2/context-120k.manifest.json) and +[selected receipts](../../evidence/public/ds4-w2-2026-07-11/p2/context-120k.receipts.jsonl) +document the selection boundary. + +Retrieval success is not a miss-free-forward claim. The three requests emitted interim KPI +windows with 42, 15, and 13 residue steps respectively, then each closed with a zero-replay, +zero-residue latest window. The exact answers empirically validate retrieval under the declared +approximation; they do not make every internal forward miss-free. + +### Frozen RC acceptance supplement + +The exact frozen RC reran the two quality seeds and deepest context case most sensitive to the +final split/base-coupling and padded-route hardening delta. It did not relabel or duplicate the +comprehensive `41d7b2f` seed-44 or depth-0.1/0.5 receipts. + +| Eval seed | Machine exact | Semantic equivalent | Sink detections | Fixed prewarm | Pre/post gates | Score-time evictions | +|---:|---:|---:|---:|---:|---|---:| +| 42 | 40/40 | 40/40 | **0/40** | 10/10 | pass / pass | 46,560 | +| 43 | 39/40 | 40/40 | **0/40** | 10/10 | pass / pass | 44,345 | +| **Combined** | **79/80** | **80/80** | **0/80** | **20/20** | **4/4 pass** | **90,905** | + +All 80 responses were HTTP 200 with `finish_reason=stop`, with no prewarm retry and no unrestored +BASE expert. Seed 43 item `r17` returned exact mathematical content as LaTeX +`\(\frac{3}{8}\)` for expected `3/8`; the strict machine scorer correctly kept it out of the +machine-exact total while the semantic scorer accepted it. It was not a sink. + +The RC then calibrated and issued one depth-0.9 request whose tokenizer count and response usage +were both exactly 120,000 tokens. It returned the exact needle with HTTP 200 and +`finish_reason=stop` in 250.673 seconds. Two earlier zero-tolerance candidates were rejected by +tokenizer-only calibration, so neither produced a chat request. + +Across the complete 1,744.3-second RC lifecycle, cgroup and host swap stayed at zero, no +hard-limit or OOM event occurred, and hard cgroup headroom stayed at or above 4,294,537,216 bytes. +The 96 GiB soft boundary was crossed narrowly and generated 19,598 `memory.high` events; that +reclaim is reported rather than called zero. Cleanup removed the disposable payload, preserved +both protected pack fingerprints, and restored both serving surfaces at HTTP 200. See the +[frozen RC sentinel](../../evidence/public/ds4-w2-2026-07-11/p2/frozen-rc-sentinel.json). + +### Comprehensive `41d7b2f` cleanup + +After the final exact receipt, the guarded controller stopped the isolated container, found both +production and immutable-seed pack fingerprints byte-identical, restored both serving services at +HTTP 200, and removed only the 230,317,622,135-byte disposable pack copy. The 40-byte increase +over the predecessor is the two `fp4_split=false` metadata fields; both payload pack sizes are +unchanged. Cgroup swap, hard-limit, OOM, and OOM-kill events stayed at zero. Host `MemAvailable` +never fell below 103,752,298,496 bytes, and hard cgroup headroom never fell below 4,294,488,064 +bytes. File-cache growth crossed the 96 GiB soft `memory.high` boundary for two sampled instants, +produced 28,089 soft-high events, and recorded PSI full `avg10` up to 0.79. That +reclaim/pressure is reported explicitly rather than +called zero. Immediate post-restart curls briefly saw connection refusal; the controller completed +cleanup and later follow-up checks returned HTTP 200/200. See the +[runtime audit](../../evidence/public/ds4-w2-2026-07-11/p2/runtime-clean.json) and +[cleanup receipt](../../evidence/public/ds4-w2-2026-07-11/p2/cleanup.json). diff --git a/docs/quality.md b/docs/quality.md index 5da8fac..2330088 100644 --- a/docs/quality.md +++ b/docs/quality.md @@ -18,6 +18,71 @@ runs on the *same* serving stack (only `MODEL_DIR` differs). 3. **Arithmetic** — 5 multi‑step problems (the official model itself tops out at 3/5). 4. **Long context** — passphrase needle at 100K/250K/440K depths (`tools/needle_probe.py`). +## DS4-W2 reliability methodology (P1/P2 release gate) + +The release gate uses `tools/ds4_eval/`; it is stricter than a transcript spot check. Every warm +run must first pass the fixed ten-prompt `ds4-w2-prewarm-v4` suite (temperature 0, top-p 1, seed +`20260711`, independent of the eval seed). There are no hidden retries: the first bad response +aborts and its receipt remains on disk. The run manifest binds the result to immutable server +provenance, including the boot, container and image identities, complete source-diff hash, +checkpoint and pack fingerprints, launcher hash, runtime argv, W2 environment, and engine settings. + +Before looking at quality results, register one pool policy and retain its SHA-256. The same frozen +policy evaluates the configured live metrics before and after scoring. The release policy requires +full FP4 occupancy, zero unrestored BASE experts, and measured eviction progress so a stale +saturated pool cannot pass; replay and second-order residue remain recorded in every snapshot even +when they are not rejection thresholds. A failed pre- or post-gate invalidates the run regardless +of its answers. At the default continuation threshold (`FP_THRESH=0`), second-order experts are +fetched for later steps but their current-step contributions remain zero, so `UNRESTORED=0` is a +mechanism-liveness gate rather than a miss-free or bit-deterministic claim. The three-seed quality +receipts are the empirical acceptance test for that explicit throughput approximation. + +P1 scores the fixed 40-item set separately at eval seeds 42, 43, and 44 (120 responses total). +Exact clean correctness is primary: the normalized terminal answer must match and the response +must not be a sink. The answer-anywhere `lenient` score is diagnostic only and never rescues a +sink. Sink rules catch max-token non-completions, repeated 3/4-grams, repeated lines, collapsed +vocabulary, and special-token spew; duplicate IDs, missing rows, or conflicting token-count aliases +fail closed. The corrected historical tau-0.67 baseline is **16/120 sinks (13.3%)** across those +three seeds. It is context only, not a tau-0.75 control: its warmups and pool evidence do not meet +the new comparability contract. +The corrected baseline summary is published under +[`evidence/public/ds4-w2-2026-07-11/baseline/`](../evidence/public/ds4-w2-2026-07-11/baseline/). + +Use repository-relative inputs and a localhost endpoint; write each seed to a new output directory: + +```bash +PORT="${PORT:-18001}" +python3 tools/ds4_eval/eval_rig.py \ + --items tools/ds4_eval/items.json \ + --server-provenance evidence/example-p1/server.json \ + --pool-gate-policy evidence/example-p1/pool-gate.json \ + --pool-command-json '["docker","logs","ds4-w2-candidate"]' \ + --output-dir evidence/example-p1/seed-42 \ + --run-label example-p1-s42 \ + --url "http://localhost:${PORT}/v1/chat/completions" \ + --model deepseek-v4-flash-w2 --mode warm --eval-seed 42 \ + --eval-temperature 0.6 --eval-top-p 0.95 --eval-max-tokens 700 \ + --expected-count 40 +``` + +Repeat with seeds 43 and 44 and distinct labels/directories. P2 is a two-part contract on the same +ready server: first run this quality gate, then run `tools/ds4_eval/context_probe.py` with MTP off, +`--max-num-seqs 1`, a validated 131,072-token window, and 120,000-token needles at depths 0.1, +0.5, and 0.9. The probe requires exact tokenizer/usage token counts and an exact terminal +passphrase. A successful 128K boot without retrieval, or retrieval without the quality gate, is +not a P2 verdict. + +The guarded single-RTX-5090 application of this protocol is recorded in +[`ds4-w2-5090-2026-07-11.md`](benchmarks/ds4-w2-5090-2026-07-11.md), with sanitized machine +receipts under [`evidence/public/ds4-w2-2026-07-11/`](../evidence/public/ds4-w2-2026-07-11/). +The historical predecessor P1 series at 32K produced 119/120 machine-exact and 120/120 +semantically correct answers with 0/120 frozen-rule sink detections. The current integrated +artifact's independent P2 series at 128K improved to 120/120 machine-exact and semantically +correct, with 0/120 sink detections, thereby satisfying the P1 stability gate at the larger +window. P2 then passed exact 120,000-token needles at depths 0.1, 0.5, and 0.9. Each case was +calibrated with tokenizer-only requests before its sole inference; the public composite contains +only fresh zero-tolerance receipts whose tokenizer and response-usage counts both equal 120,000. + ## Bits‑vs‑quality ablation (same stack, only the expert codes change) | codebook | bits | MTP acc. length | draft accept % | arithmetic | coherence | |---|---|---|---|---|---| diff --git a/docs/v024-port.md b/docs/v024-port.md index b2079f7..b8b08e9 100644 --- a/docs/v024-port.md +++ b/docs/v024-port.md @@ -2,7 +2,7 @@ The project targets **official vLLM v0.24.0**, which ships DeepSeek‑V4 + SM120 natively (`vllm/models/deepseek_v4/`, FlashInfer SM120 sparse‑MLA, GLM‑5.x `GlmMoeDsaForCausalLM`). -Our overlay is a **7.4k‑line patch** (37 files): the 2‑bit expert planes, the FP4 delta +Our overlay is a **14.0k-line patch** (70 files, 14,018 insertions): the 2-bit expert planes, the FP4 delta cache, the confidence gate, the cubit dispatch, the expert stores — plus the SM120 fixes below. @@ -40,9 +40,10 @@ Environment pins that go with the patch (both required on SM120): ## Our hooks -- `mxfp4.py` (`Mxfp4MoEMethod`) — FP4‑checkpoint path (DeepSeek‑V4‑Flash): host‑stage experts - at `create_weights`, build 2‑bit planes at `process_weights_after_loading`, `moe_w2_forward` - in `apply`. +- `mxfp4.py` (`Mxfp4MoEMethod`) — FP4-checkpoint path (DeepSeek-V4-Flash): create zero-sized + expert parameters, materialize one guarded layer while its tensors load, build/write its + 2-bit planes once, then release it before the next layer. Exact pack identity skips the + materialization; `moe_w2_forward` runs in `apply`. - `fp8.py` (`Fp8MoEMethod`) — FP8 block‑quant checkpoint path (DS4‑Flash‑Base, **GLM‑5.2‑FP8**): same three hooks; the loader re‑quantizes fp8+f32‑block‑128 to the sign‑symmetric 2‑bit codebook at load (`build_layer_planes_fp8`, float64 math, golden‑tested @@ -61,8 +62,9 @@ Environment pins that go with the patch (both required on SM120): RAM; the GPU pool caches hot experts through the same slot‑table/manager/eviction machinery as the delta tier. Decode misses zero the pair's contribution, bump an in‑graph miss counter, and the runner fetches all missing routed experts synchronously (batched pinned - H2D, 51.6 GiB/s measured) and replays the step's graph once — replay bit‑identical to a - resident forward (unit‑tested, `internal` test_base_cache). Prefill prefetches per layer + H2D, 51.6 GiB/s measured) and replays the step's graph once. Replay is bit-identical to a + resident forward only when the replay creates no second-order misses (the zero-residue case + is unit-tested). Prefill prefetches per layer via `ensure_resident`. TP MAX‑reduces the miss decision. Under **PP** a miss is local to its stage (per‑stage counter, inputs still held in the stage's static buffers), so each stage re‑runs only its own **segment** before activations flow downstream — no cross‑stage @@ -75,7 +77,10 @@ Environment pins that go with the patch (both required on SM120): `VLLM_MOE_W2_FP_MAX` bounds ping‑pong). An unconditional loop collapsed decode at low coverage (GLM TP2 29→16 tok/s, DS4‑14 GiB 43→15 — chasing a moving target at up to 8 forwards/step); the live A/B behind the default: thresh 16 → 19.2, thresh 0 → 28.3 tok/s - at identical quality probes. Slots touched by any pass of a step are pinned against + at identical quality probes. With the default threshold, second-order experts are fetched + for later steps but their current-step contributions remain zero; `fp-residue` reports this + explicit approximation, so zero `UNRESTORED` is not a miss-free or determinism claim. + Slots touched by any pass of a step are pinned against eviction until the next step (the passes must not cannibalize each other's fetches); on tight pools an emergency eviction pass (synchronous callers only) relaxes the 2‑tick coldness bound rather than leave a miss UNRESTORED. Coexists with the FP4 need‑pool (explicit @@ -94,9 +99,10 @@ Environment pins that go with the patch (both required on SM120): | 11 GiB (util 0.90) | 15.2% | 96.5–97.7% | 32.7 / **27–28 tok/s** | | 14 GiB (util 0.95) | 19.3% | 98.7–98.9% | 43.4 / **~31 tok/s** | - The pre‑fix column is the single‑replay stack (silently kept second‑order zeros — the - nondeterminism the fixed‑point restore later closed); the shipped column is the final - adaptive‑replay + NVMe‑store stack (2026‑07‑10 evening, same box/bench idiom). The pool + The pre-fix column is the single-replay stack (silently kept second-order zeros); the + shipped column is the adaptive-replay + NVMe-store stack, which measures that residue and + can chase it when `FP_THRESH` is raised but accepts it at the throughput-oriented default + (2026-07-10 evening, same box/bench idiom). The pool slope survives the stack change (+12–15% for 3 GiB). 14 GiB does NOT fit at util 0.90 (KV needs 0.46 GiB after graphs) — raise `--gpu-memory-utilization` alongside the pool. The engine logs pool @@ -120,23 +126,60 @@ Environment pins that go with the patch (both required on SM120): pinned 33.0 / pack 25.5 / tiered+20 GiB arena **32.8 tok/s (parity)** at RSS 26–33 vs 42–44 GiB; GLM TP2 both stores on NVMe: expert‑store RAM ~568 → ~136 GiB, 28–32 tok/s (tol 0–8, adaptive replay), needle 4/4 to **121K prompt tokens** at a served 128K window - (KV 157K tokens measured). Prefill batches are scan‑flagged (fill free arena slots, - never evict the decode hot set — a caller flag, NOT a batch‑size heuristic: GLM's - 100+‑row decode replay fetches misclassified and froze the arena at −66%); the arena hot - set persists to `.heat.json` and preheats on boot (57 GiB ≈ 35 s). **Boot‑from‑pack** + (KV 157K tokens measured). Prefill is derived authoritatively from each request's + `num_computed_tokens < num_prompt_tokens` state and carried as + `ForwardContext.has_prefill` through normal, replay, pipeline, gate, and DBO execution. + A real W2 prefill is eager (capture fails loud), disables draft-affinity prefetch and + confidence-gate promotion, and uses fill-only arena reads so it cannot evict the decode + hot set. Long prefills use the aligned bulk base-only path; short prefills retain FP4 + recovery and mblock-4 instead of falling into the decode path. Padded CUDA-graph routes + carry an explicit token-slot mapping and are masked out of miss, promotion, and output + accounting. The arena hot set persists to `.heat.json` and preheats on boot + (57 GiB approximately 35 s). **Boot-from-pack** (`_try_skip_requant`, all three loaders): a layer present in every serving pack skips dequant→re‑quant entirely — GLM TP2 second boot 408 s vs ~11 min, no ~405 GiB transient; the pack is a persistent quantization cache keyed by shape/config sidecar match. Also - fixed here, exposed by long‑prefill testing but **pre‑existing**: the manager tick and - forward‑thread paths ran concurrent seen‑snapshots into one shared pinned `_seen_host` - (torch's two‑pass `nonzero` overruns its output when the input mutates mid‑call → - TensorAdvancedIndexing.cpp:3008 assert → glibc heap corruption → dead worker on ≥16K - prefills; a torn snapshot could also evict an in‑flight expert's slot). Snapshots are now - serialized under a dedicated lock. Backends unit‑tested byte‑identical - (`tools/test_store_backends.py`: 3 backends × cold/warm/reboot/evict/overflow/scan/ + fixed here, exposed by long-prefill testing but **pre-existing**: manager and forward + threads previously reused one mutable pinned seen-mask, so a later layer could overwrite + the snapshot while the current layer selected or fetched rows. Each residency operation + now drains prior main-stream work, freezes an immutable layer-local `seen_set`, clears only + the previous BASE layer's pins, and pins the current layer's hits/promotions until its + forward completes. A target step also acquires the exclusion lock shared with each background + tier-manager pass. That lock stays held through target execution, BASE replay, confidence-gate + re-forward, and the pipeline barrier; only `finish_forward_step()` releases it and wakes one + manager pass. `step_begin()` therefore clears pins without racing a wake. Saturated LRU uses a + floating-point timestamp key before assigning infinity, and both force-promote and manager + ticks receive immutable seen sets. Backends unit-tested byte-identical + (`tools/test_store_backends.py`: 3 backends x cold/warm/reboot/evict/overflow/scan/ preheat, both IO modes). Ops: packs on a bind‑mounted real FS (not overlayfs); ~1 TB NVMe for the full GLM stack; parity holds even on a Gen3‑x4 drive (3.7 GB/s) — steady‑state misses ride the page cache, cold shifts pay drive speed. + Cold rebuilds are bounded separately from steady reads: MXFP4 uses guarded one-layer + materialization/build/write/release rather than retaining every layer's expert tensors; + pack-skip and completion guards prevent unnecessary or duplicate work. Eviction is + requested for each + released safetensors shard and each durable pack layer, then retried after the model + consumer unwinds, + checkpoint prefetch is refused, requested multi-thread shard loading is serialized, and + arena/shard/layer operations preflight a default 16 GiB host + 4 GiB hard cgroup + `memory.max` reserve. `memory.high` is traced as a soft reclaim/throttle boundary and does + not reduce that hard allocation budget. The + safety controls are `VLLM_MOE_W2_CACHE_CONTROL`, + `VLLM_MOE_W2_MIN_MEM_AVAILABLE_GB`, and + `VLLM_MOE_W2_MIN_CGROUP_HEADROOM_GB`; cache control defaults to fail-closed `required`. + The all-43-layer cold canary completed with a 51,807,191,040-byte cgroup peak, + 8,231,129,088-byte anon peak, 44,430,127,104-byte file peak, and a + 73,793,277,952-byte host `MemAvailable` floor; cgroup swap, PSI, limit, and OOM events were + zero, and the host recorded no swap-out; host swap-in increased by 160 pages. + That all-layer canary used predecessor patch `e7417054a6e8`; integrated patch + `41d7b2f96ca3` retains the mechanism and passed the exact-head baked safety tests. Sanitized + trace and cleanup receipts are under + [`evidence/public/ds4-w2-2026-07-11/p0/`](../evidence/public/ds4-w2-2026-07-11/p0/). + Historical predecessor receipts establish the 32K P1 lane; the current integrated image's + guarded 128K series independently satisfies the same stability gate. Both are documented in + [`ds4-w2-5090-2026-07-11.md`](benchmarks/ds4-w2-5090-2026-07-11.md): 0/120 frozen-rule sink + detections in each series, with exact 120,000-token retrieval at depths 0.1, 0.5, and 0.9 on + the integrated image. This is a quality/context result; no 128K throughput is claimed. - **Deterministic unpermute**: the MoE output scatter used atomic `index_add_`, so identical runs wobbled (~1.6e‑2 on prefill) and greedy decode was not reproducible (surfaced by the PP determinism investigation; never PP‑specific). Valid `sorted_ids` form a permutation of diff --git a/evidence/public/ds4-w2-2026-07-11/README.md b/evidence/public/ds4-w2-2026-07-11/README.md new file mode 100644 index 0000000..d556ec7 --- /dev/null +++ b/evidence/public/ds4-w2-2026-07-11/README.md @@ -0,0 +1,100 @@ +# DS4 W2 guarded validation evidence — 2026-07-11 + +This directory contains the sanitized, publishable receipts for the single-RTX-5090 +DeepSeek-V4-Flash W2 safety, quality, and context run. The measured narrative and evidence +boundaries are in [`docs/benchmarks/ds4-w2-5090-2026-07-11.md`](../../../docs/benchmarks/ds4-w2-5090-2026-07-11.md). + +## Artifact lineage + +| Evidence role | Artifact identity | +|---|---| +| P0 all-layer cold canary | mechanism-equivalent predecessor `e7417054a6e8`; measured memory belongs only to this canary | +| Historical P1 32K receipts | patch `55d30bb9cf9bef45e7130fd5afe0090c5b540be22f6de3abfc51b782f738a6f7`; image `sha256:7385d21d26b665884e97a97dc67a100db328ed7b00b634e4d18f8aedd9f29eab` | +| Comprehensive integrated P2 predecessor | official vLLM `ee0da84ab9e04ac7610e28580af62c365e898389`; patch `41d7b2f96ca3b966cac1b7ed5cff37bc03c27c616ed254aca961cc75a9ffe31d`; image `sha256:66abc2f145244e03ff0f0fcca088be813ad311f2963c62b281e4bb888ac605e9`; wrapper `5e69228e1e8e6f7345ae96657ab33b50e5805be7` | +| Frozen release candidate and focused current-head supplement | official vLLM `ee0da84ab9e04ac7610e28580af62c365e898389`; origin/main cutoff `94be3aa3d7a8b82c7fc9687990a7edb6035f69f3`; patch `241ba984b1c56f5dc7adbc8d7f519d60b5746024bf7dfeb875e3546a668e79a7`; image `sha256:fc6e1244d60855fe45ccc0236daaaa722abcb8d354200eb92aca104bd954d3f2`; wrapper `0a73c2bcf262ba0aa53560d6290ad2af350e34fc` | + +[`exact-image-verification.json`](exact-image-verification.json) records the clean-apply shape, +baked source hashes, exact-image test matrix, and synthetic safety-probe hashes for the frozen +release candidate. The earlier `41d7b2f` artifact remains the comprehensive three-seed, +three-depth quality record; the frozen candidate adds a focused exact-head release sentinel. + +## P0 — bounded cold restage + +[`p0/`](p0/) contains the all-43-layer cold-restage memory trace and guarded cleanup receipt. +That measured canary ran on mechanism-equivalent predecessor patch `e7417054a6e8`; its memory +maxima are not represented as integrated-image measurements. The checkpoint-residency, +next-shard retry, and constrained-pack probes in the same directory were rerun on the frozen +release-candidate image and bind to its exact image ID. + +The canary kept cgroup swap, pressure, hard-limit, and OOM events at zero and recorded no host +swap-out. Host swap-in increased by 160 pages; that boundary is explicit in the summary and trace. + +The frozen candidate also received a fresh full cold restage. [`p0/frozen-rc-summary.json`](p0/frozen-rc-summary.json) +records 46/46 checkpoint shards, all 43 expert-pack layers, zero swap and OOM activity, zero +memory-limit event deltas, preserved production and immutable-seed fingerprints, restored service +health, and disposable-store removal. Its 218-sample cold-only trace is +[`p0/frozen-rc-memory-trace.tsv`](p0/frozen-rc-memory-trace.tsv). These measurements bind to the +frozen candidate and supersede the historical limitation only for that candidate; the earlier +canary files remain historical receipts. + +## P1 — 32K quality and near-edge context + +The files under [`p1/`](p1/) are retained byte-for-byte as historical predecessor evidence. +[`p1/aggregate.json`](p1/aggregate.json) reports 119/120 machine-exact, 120/120 semantically +correct, and 0/120 frozen-rule sink detections across seeds 42–44, plus 30/30 prewarm prompts and +all six pool-lifecycle gates. Per-seed manifests, raw responses, and warmup receipts are retained +beside the aggregate. Published receipt hashes are recomputed after sanitization. + +[`p1/context-30k.manifest.json`](p1/context-30k.manifest.json) and +[`p1/context-30k.receipts.jsonl`](p1/context-30k.receipts.jsonl) record 3/3 exact needles at +30,000 prompt tokens. [`p1/runtime-clean.json`](p1/runtime-clean.json) records the complete lane +through context, including soft `memory.high` reclaim and nonzero PSI; [`p1/cleanup.json`](p1/cleanup.json) +records fingerprint preservation, service restoration, and disposable-store removal. + +## P2 — comprehensive 128K series and frozen-RC sentinel + +The comprehensive P2 receipts bind to the `41d7b2f` integrated predecessor above. Its three-seed +128K series also satisfies the P1 stability requirement. +[`p2/quality-aggregate.json`](p2/quality-aggregate.json) +reports 120/120 machine-exact and semantically correct, with 0/120 frozen-rule sink detections at +a 131,072-token window. + +[`p2/context-120k.manifest.json`](p2/context-120k.manifest.json) and +[`p2/context-120k.receipts.jsonl`](p2/context-120k.receipts.jsonl) contain the admitted exact +120,000-token receipts at depths 0.1, 0.5, and 0.9. All three published receipts use zero token +tolerance. Each case calibrated to exactly 120,000 tokens with tokenizer-only requests before its +sole inference; no rejected calibration contributed a chat response in this run. +Readiness, complete runtime/memory, and guarded cleanup are recorded in +[`p2/readiness.json`](p2/readiness.json), [`p2/runtime-clean.json`](p2/runtime-clean.json), and +[`p2/cleanup.json`](p2/cleanup.json). + +[`p2/frozen-rc-sentinel.json`](p2/frozen-rc-sentinel.json) is the focused supplement for the +frozen candidate. It records two quality seeds (80 responses): 79/80 strict machine-exact, +80/80 semantically correct, 0/80 sink detections, all 20 prewarms without retry, and all four +pool gates. The sole strict mismatch is an explicitly recorded equivalent LaTeX rendering of +`3/8`. It also records one exact 120,000-token retrieval at depth 0.9 with zero token tolerance, +plus complete-run memory and cleanup results. This supplement did not repeat seed 44 or depths +0.1 and 0.5, so those broader claims remain attached only to the immediately preceding +comprehensive P2 artifact. + +The frozen source supports split-FP4 refinement over a base cache only behind the explicit split +flag, with coupled base/refinement residency. The serving sentinel kept split mode disabled and +used full-nibble FP4 delta planes. Split mode was exercised separately by the exact-image E=8 GPU +three-tier fixture; serving-sentinel results are not presented as live split-mode evidence. + +## Interpretation and sanitization + +These receipts establish bounded restaging, observed quality, pool liveness, exact selected +retrieval, and guarded cleanup for the recorded artifact and policy. They do not establish +bit-deterministic output, a miss-free forward, numerical equivalence to fully resident FP4, or +128K throughput. Second-order residue remains explicit. + +Host topology, opaque runtime IDs, host-specific paths, host- and address-specific endpoint +details, checkpoint/pack +fingerprints, launcher identity, and response IDs remain in private source receipts. Public run +labels are descriptive. The lane started on July 11 Pacific time, while UTC timestamps in the +integrated receipts fall on July 12. Verify the final public tree from this directory with: + +```bash +shasum -a 256 -c SHA256SUMS +``` diff --git a/evidence/public/ds4-w2-2026-07-11/SHA256SUMS b/evidence/public/ds4-w2-2026-07-11/SHA256SUMS new file mode 100644 index 0000000..0be7b77 --- /dev/null +++ b/evidence/public/ds4-w2-2026-07-11/SHA256SUMS @@ -0,0 +1,47 @@ +4e894b6b293a3f831cd5ce830d03f691e09a91d26b27b3a329d88fcf30f64c9f ./README.md +14e410651bc505b1bec6ec6a439be1d19600987a797fc6f615f71d8bb79e489a ./baseline/README.md +85b3eff601c3a8d1f70810fc297fa8f6a40c7e5e5b238710cc1068e86ffe7054 ./baseline/baseline-corrected.json +be456388a14da84f0a2d7f5fd30222fddeac10d3847946f6f6fc259b2c12bbd4 ./exact-image-verification.json +e7eb19fc75dc8b49d587b9f07c87c5db182cd764a2f85998aac78d85a03698b8 ./p0/baked-checkpoint-residency.json +27de4d773ac034dce3d9f6d29b70275ec195219fd69077343a47843dc764c107 ./p0/baked-checkpoint-retry.json +28d4b707ff1f49cf3b52091483a47d20ae855f001bd9937b48a69dcd0d13fe80 ./p0/baked-constrained-pack-build.json +bf514ce9a0aef75c28ac55ef6243eb1043aa67a733c1e864197b9d493d8c1171 ./p0/cold-restage-cleanup-receipt.json +9271af0f90160566729a6507f8b8e07ac7d05699e122701110393035b6135f26 ./p0/cold-restage-memory-trace.tsv +1cbbd4f5e177a16ff3de4a179fc5a8763a37c1d410266b5eb0155da731527a39 ./p0/cold-restage-summary.json +59403aa1b8d2a7ed2294a9d7b78566ef5320316e1cdaeff67e5d3d06bc031d35 ./p0/frozen-rc-memory-trace.tsv +aba59dd8e1a8051e4ea93400e0e4931e958c75d239c7dd56444afa7136eee710 ./p0/frozen-rc-summary.json +005b225341e1303db4560c782d09a6eb8c9422c12267d74712c381fb71e33add ./p1/aggregate.json +aa572d1bfa31827f949bef0e422e6593bc4889472cbf6d2ed53b70e3cd84e74a ./p1/cleanup.json +0aa4e2f999a721c0a8bd36521cc3487c5b4297958de8b02abfd63a2b992450cb ./p1/context-30k.manifest.json +cd2c7621f168683aee68e37c9008dd5faecc808b74fe93ed3b39405f841393fe ./p1/context-30k.receipts.jsonl +1afb9b0d96d05c9d7a374dff88c94c76a3e854c8d046fb8ef1783275ad3f5630 ./p1/pool-gate-policy.json +e32564fb10f3302538af4d8825f3ec95637399f1d54d7dd7aac11dfb13d902de ./p1/runtime-clean.json +e559c313cbfeb47a79743d6bc73c84f1244ac20ed36d85d07e49587bfae110fb ./p1/seed42.manifest.json +1c4978408014aff023ddb1b71eae0b9f4e8fee59e9cb4434bc38fabd05756487 ./p1/seed42.raw.jsonl +3c2bdf2e75a0967028a70ca065a0e67c8e70daf4c1ccab1d84491524155f1da0 ./p1/seed42.warmup.jsonl +b5db436fa14bc8ff3864030212e62c1e5a1ff3217f6eb8fd0ace845220f7ae54 ./p1/seed43.manifest.json +77cee2bd64cdffa4deb4ac23352cf6053e1bbb02f87bda9b24ab621358d40630 ./p1/seed43.raw.jsonl +474fb509940ff0a3c143d709538edcec81764d1ea15559f118b8da3fc3a39313 ./p1/seed43.warmup.jsonl +75c4bbf593ebc84d7d57ea778374f487a6665d2a51d90dd9669597469e0a5dd7 ./p1/seed44.manifest.json +10a93bf29ff4bd411535f2b48d3d5dfc28ff08f6b6435169b7b505e27c9569ac ./p1/seed44.raw.jsonl +f832cca3346094be00650267899a1890c5c40e961fd64d60b203238c1a2c9db8 ./p1/seed44.warmup.jsonl +5ff7e3d0d22073fdba1f0413aca5c10db573fbf235596746559fee5ec7f22c0f ./p1/server.json +5ee72db204c8e30574d426a83782474ac7ec2f67f904da0ad04ff10dab0308a3 ./p1/strict-zero-residue-diagnostic.json +cad626e05b01464f81f1bb58434e3f6bec5886f69223e8cc8d037dd17d5963ab ./p2/cleanup.json +6e16b2af587301748b96d5b24a3e06acff8b73b171b6a483918b2338d63771b3 ./p2/context-120k.manifest.json +67c2a1f834b9a6ce54d4ea4f2ce98a51c5257bfb8a78cf9a3917bbab52ea1db7 ./p2/context-120k.receipts.jsonl +db649f7008e2a0bd3a06a51ab242e884b2846eee2f9985ca06ef557ff7a5ac14 ./p2/frozen-rc-sentinel.json +1afb9b0d96d05c9d7a374dff88c94c76a3e854c8d046fb8ef1783275ad3f5630 ./p2/pool-gate-policy.json +40362d6150626552760a7843b8ea5454e0d71c0ed1af60f161b6783788f799c9 ./p2/quality-aggregate.json +fb4732bfa7700598ba8686d8d585c7508848b215bad690ba6e25a4a97c44fbac ./p2/readiness.json +1912d3b9d34aafbbbc72fb0a08212cddbbf492a6e4063567f907321b5917d0a0 ./p2/runtime-clean.json +1d5222c1195ff47766ca7b2a2ab69dd664acbe8c848f0a893170947c66c780f4 ./p2/seed42.manifest.json +db5b50a8ed30dd68f61431cae477cf65902768d400b4f7d20fc75790c5abc2b6 ./p2/seed42.raw.jsonl +c0f30b140c53006c5d2041f7e3f75f7dc977bd8980c059aec745c7ea559a909b ./p2/seed42.warmup.jsonl +17996af880a997bb5fd1967845b1da14a6af2188de43b571984d83b20b8bde33 ./p2/seed43.manifest.json +12ebd8458751bbea4d7683794a5c9e7c447784a93d2fae7dce275d59f242f5fc ./p2/seed43.raw.jsonl +c89b523c53ee899193e829eaa1b58c06cf992f1ffb587433f293221cbe9ec86d ./p2/seed43.warmup.jsonl +58834e56c51c4ef94cc591ed4eb0c1521e20111a4cc623911b2b76cd58c5b443 ./p2/seed44.manifest.json +d188e788324b3d47a8a81112e7c1e185e1297e703c78df6f30d19b5c0dae7600 ./p2/seed44.raw.jsonl +e1485a5efec3960924b722834bd04d87112bf32ad1a98373a9981455412d08ba ./p2/seed44.warmup.jsonl +f4514891b1a84f1415b3dafaac76bfa8f1261f0804561201d683faf9f84fd8ef ./p2/server.json diff --git a/evidence/public/ds4-w2-2026-07-11/exact-image-verification.json b/evidence/public/ds4-w2-2026-07-11/exact-image-verification.json new file mode 100644 index 0000000..514bd4a --- /dev/null +++ b/evidence/public/ds4-w2-2026-07-11/exact-image-verification.json @@ -0,0 +1,99 @@ +{ + "schema_version": "vllm-moet-exact-image-verification-v3", + "frozen_release_candidate": { + "official_vllm_commit": "ee0da84ab9e04ac7610e28580af62c365e898389", + "origin_main_cutoff": "94be3aa3d7a8b82c7fc9687990a7edb6035f69f3", + "wrapper_head_commit": "0a73c2bcf262ba0aa53560d6290ad2af350e34fc", + "source_patch_sha256": "241ba984b1c56f5dc7adbc8d7f519d60b5746024bf7dfeb875e3546a668e79a7", + "image_id": "sha256:fc6e1244d60855fe45ccc0236daaaa722abcb8d354200eb92aca104bd954d3f2", + "image_source_patch_label": "241ba984b1c56f5dc7adbc8d7f519d60b5746024bf7dfeb875e354200eb92aca104bd954d3f2", + "image_wrapper_commit_label": "0a73c2bcf262ba0aa53560d6290ad2af350e34fc" + }, + "immediately_preceding_comprehensive_artifact": { + "official_vllm_commit": "ee0da84ab9e04ac7610e28580af62c365e898389", + "integrated_origin_main_commit": "7d2429c109b625f23ed7926eb4e80f5d8d80f2e5", + "wrapper_head_commit": "5e69228e1e8e6f7345ae96657ab33b50e5805be7", + "source_patch_sha256": "41d7b2f96ca3b966cac1b7ed5cff37bc03c27c616ed254aca961cc75a9ffe31d", + "image_id": "sha256:66abc2f145244e03ff0f0fcca088be813ad311f2963c62b281e4bb888ac605e9", + "validation_scope": "three quality seeds and three exact 120K context depths" + }, + "applied_tree": { + "files_changed": 70, + "insertions": 14223, + "deletions": 226, + "apply_check": "pass", + "source_diff_whitespace_check": "pass", + "python_compile": "pass", + "fresh_applied_source_match": "70/70 baked source files matched" + }, + "baked_source_sha256": { + "vllm/forward_context.py": "b3b5d84c16e0f5c886186cc06fcd65e1aa577ee6e58c2746ecea4c599f778ec7", + "vllm/v1/worker/gpu_model_runner.py": "63a08d992701cfca677de0e35eab2bc00e97a4514ae6d69858d544460d35be3f", + "vllm/v1/worker/gpu_ubatch_wrapper.py": "cc9526d859ccecdeb69b42140c92ba13c6fb80a8091734ab518a819c322dd1e7", + "vllm/v1/worker/gpu_worker.py": "76331291a8cd98236a17d85ffced0a4ed2d749da7e68f163e5ffb3df922faca5", + "vllm/model_executor/layers/quantization/utils/moe_w2_cubit.py": "3fa37293b68084358bc0bf6c9630a81ee722d2c8a936f010837f71f7cde27ca0", + "vllm/model_executor/layers/quantization/utils/moe_w2_delta.py": "25235377b2e86c0a5bc84d7fdbcf7bdac0a57f8b4080a66754e795b902bbe736", + "vllm/model_executor/layers/quantization/utils/moe_w2_gate.py": "d035e371fd93a7840733ecb3c9b2b89675a5448efac0da834fd5628c1052691b", + "vllm/model_executor/layers/quantization/utils/moe_w2_planes.py": "3555b5561063ca000be91fabaf3e65a8da4b56e6c75e93c46611aa82c66e5a56", + "tests/model_executor/layers/quantization/test_moe_w2_padded_routes.py": "5452b492bcdeb64413dab659e5baa1b28571f08af1d6cb2abf67a25836a054d1", + "tests/model_executor/layers/quantization/test_moe_w2_step_pins.py": "aed09d21defaba3e524a98640005c9ba00b8b8bc3dac539ec5948250743ea1db" + }, + "canonical_synthetic_safety_probes": { + "checkpoint_residency_sha256": "e7eb19fc75dc8b49d587b9f07c87c5db182cd764a2f85998aac78d85a03698b8", + "checkpoint_next_shard_retry_sha256": "27de4d773ac034dce3d9f6d29b70275ec195219fd69077343a47843dc764c107", + "constrained_pack_build_sha256": "28d4b707ff1f49cf3b52091483a47d20ae855f001bd9937b48a69dcd0d13fe80", + "image_identity_match": true + }, + "frozen_rc_exact_image_validation": { + "source_focused_manager_window_cgroup_route_pin": "21 passed on CUDA", + "store_safety": "37 passed", + "mxfp4_stream_safety": "10 passed", + "gpu_store_backends": "all pinned, mmap, reboot, tiered, eviction, overflow, scan, and preheat checks passed", + "forward_matrix_e8": "base 2-bit, full-FP4 delta, and split-FP4 refinement passed", + "three_tier_split_e8": { + "mixed_max_relative_error": 0.02446, + "mixed_cosine_similarity": 0.999867, + "transient_miss_count": 2, + "transient_max_relative_error": 0.02661, + "eviction_mapped_experts": "4/4", + "result": "pass" + }, + "checkpoint_residency_128_mib": "pass", + "checkpoint_retry_512_mib": "pass", + "constrained_pack_48x8x4_mib": "pass below 2 GiB with zero swap" + }, + "split_fp4_base_cache_boundary": { + "support": "split-FP4 refinement over the base cache is supported behind the explicit split flag with residency coupling and the fp4s tier tag", + "serving_sentinel": "split mode was disabled, so the guarded release-candidate serving run used full-nibble FP4 delta planes", + "separate_fixture": "the exact-image E=8 three-tier fixture covered mixed residency, a transient base miss, and eviction" + }, + "serving_validation": { + "frozen_release_candidate": { + "quality_seeds": [ + 42, + 43 + ], + "responses": 80, + "strict_machine_exact": 79, + "semantically_correct": 80, + "sink_detections": 0, + "retry_free_prewarms": "20/20", + "pool_gates": "4/4", + "exact_120k_context": "1/1 at depth 0.9 with zero prompt-token tolerance" + }, + "immediately_preceding_comprehensive_artifact": { + "quality": "120/120 machine-exact, 120/120 semantically correct, 0/120 sinks across three seeds", + "retry_free_prewarms": "30/30", + "pool_gates": "6/6", + "exact_120k_context": "3/3 at depths 0.1, 0.5, and 0.9 with zero prompt-token tolerance" + } + }, + "project_validation": { + "evaluation_and_context_harness": "39 passed, 17 subtests passed", + "ruff": "changed W2 harness and targeted project files pass; broad upstream helper lint is outside this receipt", + "format": "changed W2 harness and targeted project files pass", + "benchmark_lint": "0 errors, 0 warnings", + "benchmark_render_check": "up to date" + }, + "evidence_boundary": "The frozen release candidate is the current exact image and received focused exact-image GPU, cold-restage, two-seed quality, and deepest-context sentinel validation. The immediately preceding 41d7b2f artifact remains the comprehensive three-seed and three-depth serving series; those broader counts are historical evidence and are not relabeled as frozen-RC reruns." +}