[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"project-93055":3},{"id":4,"name":5,"fullName":6,"owner":7,"repo":5,"description":8,"homepage":9,"htmlUrl":10,"language":11,"languages":10,"totalLinesOfCode":10,"stars":12,"forks":13,"watchers":14,"openIssues":15,"contributorsCount":16,"subscribersCount":16,"size":16,"stars1d":16,"stars7d":16,"stars30d":17,"stars90d":16,"forks30d":16,"starsTrendScore":16,"compositeScore":18,"rankGlobal":10,"rankLanguage":10,"license":19,"archived":20,"fork":20,"defaultBranch":21,"hasWiki":22,"hasPages":20,"topics":23,"createdAt":10,"pushedAt":10,"updatedAt":24,"readmeContent":25,"aiSummary":26,"trendingCount":16,"starSnapshotCount":16,"syncStatus":27,"lastSyncTime":28,"discoverSource":29},93055,"vLLM-Moet","kacper-daftcode\u002FvLLM-Moet","kacper-daftcode","A vLLM patch + hand‑written SM120 SASS kernels: 2‑bit MoE experts + an FP4 \"delta\" cache that recovers precision — matching the official (NV)FP4 checkpoint's quality on consumer Blackwell cards","",null,"Sass",273,18,6,1,0,38,47.64,"Other",false,"main",true,[],"2026-07-22 04:02:08","# Frontier MoE on consumer Blackwell (SM120)\n\n**Official vLLM v0.24.0 + a 7.4k‑line patch** that serves frontier Mixture‑of‑Experts models —\n**GLM‑5.2 (753B)**, **DeepSeek‑V4‑Flash (159B)** and **Kimi‑K2.7‑Code (1T)** — on\nconsumer\u002Fworkstation Blackwell (RTX PRO 6000, RTX 5090), hardware their official checkpoints\ncannot even fit on. Three ideas carry it:\n\n1. **2‑bit experts with FP4 recovery** — routed experts compress to a sign‑symmetric 2‑bit\n   codebook on hand‑written SM120 SASS kernels; a runtime FP4 tier (delta cache + confidence\n   gate) restores precision exactly where it matters.\n2. **Tiered expert residency** — when even the 2‑bit base outgrows VRAM, it moves to pinned\n   host RAM — and, one tier further, to an **NVMe pack file with a pinned‑RAM arena** — and\n   the GPU becomes an **expert cache** (miss → batched fetch + bit‑identical graph replay).\n   That puts 753B on two 96 GB cards and 159B on a single RTX 5090, and the packs double as\n   a **persistent quantization cache** (reboots skip the re‑quant).\n3. **A rebuilt serving base** — vLLM v0.24.0 actually working on SM120 (the release is\n   broken‑as‑shipped), plus MTP speculative decoding (incl. under pipeline parallelism,\n   bit‑deterministic), an **NVFP4 KV cache** (352 B\u002Ftoken), and agent‑ready tool\u002Freasoning\n   parsing.\n\n---\n\n## GLM‑5.2 (753B) — the headline model\n\nServed from the official [nvidia\u002FGLM-5.2-NVFP4](https:\u002F\u002Fhuggingface.co\u002Fnvidia\u002FGLM-5.2-NVFP4)\ncheckpoint (433 GB): the loader re‑quantizes modelopt NVFP4 experts (e2m1 × e4m3 block‑16 ×\nper‑tensor scale_2) to the sign‑symmetric 2‑bit planes at load — f64‑exact vs the reference\npipeline on real shards. Single‑stream, greedy, CUDA graphs:\n\n| hardware | config | decode | max context window (served, needle‑validated) | host RAM |\n|---|---|---:|---:|---:|\n| **4× RTX PRO 6000 (TP4)** | 2‑bit + **MTP k=2** | **105 tok\u002Fs** | **256K** | — |\n| 4× RTX PRO 6000 (TP4) | 2‑bit + MTP k=2 + **FP4 delta + confidence gate** | **83–85 tok\u002Fs** | 128K | — |\n| **2× RTX PRO 6000 (TP2)** | **three‑tier + NVMe stores** + MTP + gate | **28–32 tok\u002Fs** | **128K** | **~140 GiB** |\n\n- **4 cards:** prefill ~2.5k tok\u002Fs; MTP acceptance 2.3–2.8; needle retrieval **PASS to 126K**\n  on the nvfp4 KV cache and **to 276K** on fp8 (331K window fits at util 0.95). GLM's nominal\n  1M window is KV‑bound on 4 cards. Tool calling (`glm47`) + reasoning (`glm45`) parsers work\n  out of the box — the endpoint drives coding agents (opencode) directly.\n- **2 cards — a model that doesn't fit, running anyway:** the 2‑bit planes alone (~190 GiB)\n  match the entire 2‑GPU VRAM budget. **Three tiers** make it work: NVMe pack + 57 GiB\u002Frank\n  pinned arena for the 2‑bit base → 46 GiB\u002Frank GPU expert cache → a small gate‑filled FP4\n  pool for precision (expert stores need ~136 GiB host RAM instead of ~568 pinned). The\n  single‑user window is real: `--max-model-len 131072` with 8 GiB\u002Frank KV = **157K tokens of\n  KV measured**, needle retrieval **4\u002F4 PASS at 36K \u002F 86K \u002F 121K prompt tokens** (fp8 KV,\n  tol=0). Decode (MTP k=2, acceptance ~2.9): **28.3 tok\u002Fs** strict (tol=0), **31.7** at\n  miss‑tolerance 8 — arithmetic and retrieval probes clean at both. Bare‑2‑bit quality\n  artifacts (\"capital of Poland: Krakow\", garbled Polish) are corrected by the FP4 tier.\n  Booting from existing quantization packs takes **~7 min** (vs ~11 for a full\n  re‑quantizing load, which also stages ~405 GiB of transients).\n- **NVFP4 KV cache** (`--kv-cache-dtype nvfp4`): packed 352 B\u002Ftoken vs 656 B `fp8_ds_mla` —\n  **+38% KV pool** (415K → 571K tokens at equal settings) at decode parity, or the freed VRAM\n  goes to the FP4 pool (the standing 4‑card config runs a 19.6 GiB\u002FGPU pool + 175K‑token KV).\n\n## DeepSeek‑V4‑Flash (159B)\n\nOfficial checkpoint, 2‑bit experts + FP4 delta cache, MTP k=2, CUDA graphs (single‑stream\nmedians; prefill = 8k‑token prompt, uncached):\n\n| hardware | decode | prefill 8k | max context window (served, needle‑validated) | host RAM |\n|---|---:|---:|---:|---:|\n| **1× RTX PRO 6000 (96 GB)** | **161 tok\u002Fs** | **5 340 tok\u002Fs** | **512K** | — |\n| 2× RTX PRO 6000 (TP2) | 210 tok\u002Fs | 5 790 tok\u002Fs | 512K | — |\n| 4× RTX 5090 (TP4) | 214 tok\u002Fs | 6 100 tok\u002Fs | 16K | — |\n| **1× RTX 5090 (32 GB)** | **~31 tok\u002Fs** (14 GiB pool + NVMe stores) | ~400–540 tok\u002Fs | **32K** | **~30 GiB** |\n\nRetrieval behind the window column: needle PASS at 453K on the PRO 6000 (947K‑token KV\nmeasured) and at 29.7K on the single 5090 (131K‑token KV). \"—\" in host RAM = all‑VRAM\nconfig, no host expert store.\n\n**Batched serving** (aggregate decode tok\u002Fs at N concurrent streams; per‑stream in\nparentheses at N=32):\n\n| concurrency | 1 | 4 | 8 | 16 | 32 |\n|---|---:|---:|---:|---:|---:|\n| 1× RTX PRO 6000 | 156 | 290 | 493 | 659 | **933** (29\u002Fstream) |\n| 4× RTX 5090 (TP4) | 198 | 460 | 762 | 1 006 | **1 560** (49\u002Fstream) |\n\nFour consumer 5090s match two PRO 6000s on decode. MTP acceptance ~2.6 tok\u002Fstep across\nconfigs. MTP also runs under **pipeline parallelism** (draft propagation + drafter embedding\nshare across ranks): DS4 on 4× RTX 5090 **PP4** does 184 tok\u002Fs vs 93 without (~2×), and greedy\ndecode under PP is **bit‑deterministic** (6\u002F6 identical runs, with and without MTP).\nMethodology: **[docs\u002Fv024-port.md](docs\u002Fv024-port.md)**.\n\n---\n\n## How it fits — 2‑bit experts at FP4 quality\n\nWe compress **only the routed experts** to 2 bits (the dense stack keeps the checkpoint's\nprecision — FP8 on DS4, NVFP4 on GLM) and recover FP4 precision adaptively:\n\n- **2‑bit expert planes — the sign‑bias finding.** Naive 2‑bit *destroys* these models\n  (degenerate loops). The cause is **sign asymmetry**, not error magnitude — the optimal‑L2\n  codebook drops one sign's tail and the per‑expert bias compounds over dozens of layers.\n  Forcing a **sign‑symmetric** `{−4,−1,1,4}` codebook at the same L2 error fixes it entirely\n  (33,023 of 33,024 DS4 tensors pick it), landing MTP acceptance **at\u002Fabove** the FP4 experts\n  (2.73 ≥ 2.68 in the QUANT_PROBE study). The finding reproduces on **GLM‑5.2** (180‑tensor\n  sweep: asym bias −0.042, 99% negative; symmetric 392× smaller at equal rel‑RMS).\n- **FP4 recovery — used surgically.** Decode is HBM‑bound and an FP4 read is 2× the bytes, so\n  2‑bit is the *fast* default: a **delta cache** keeps the hot experts at FP4 (background\n  promote\u002Fevict, CUDA‑graph‑safe, `VLLM_MOE_W2_DELTA_GB=auto` sizes it from post‑KV VRAM),\n  and a **confidence gate** (`VLLM_MOE_W2_GATE=1`) re‑runs low‑confidence steps at FP4 —\n  force‑promote the step's routed experts, replay the graph once, re‑decide. Works inline on\n  TP\u002Fsingle‑GPU (incl. MTP verify steps) and as a full‑pipeline replay under PP; τ tunable at\n  runtime.\n- **The kernels.** `moe_w2_mm` (2‑bit MoE GEMM: PRMT‑LUT in‑register decode → `QMMA.SF`\n  block‑scaled tensor cores, 4 CTA\u002FSM) and `moe_w4_mm` (FP4 delta GEMM) — hand‑written SASS,\n  shipped as sources + prebuilt cubins for every sharding (K = 6144\u002F4096\u002F2048\u002F1024\u002F512), so\n  TP2\u002FTP4 work out of the box. Op‑validated (rel ~1–3e‑3, deterministic), graph‑capture‑exact.\n  Prefill runs the **AFRAG** variant (fragment‑major activations → one `LDG.128` per QMMA\n  A‑fragment; the prefill GEMM is load‑issue‑bound, not DRAM‑bound): bit‑identical outputs,\n  1.3× on the GEMM, **+12% e2e prefill** on one card — default on (`VLLM_MOE_W2_AFRAG=0`\n  opts out).\n\nAll three checkpoint flavors load: **FP4 experts** (DeepSeek‑V4‑Flash — codes remap),\n**FP8 block‑quant** (Flash‑Base, GLM‑5.2‑FP8) and **modelopt NVFP4** (GLM‑5.2‑NVFP4) — the\nlatter two re‑quantized to the sign‑symmetric codebook at load, float64‑exact vs the\nreference pipeline.\n\n---\n\n## When the model doesn't fit at all — the GPU as an expert cache\n\n`VLLM_MOE_W2_BASE_CACHE_GB=N` inverts residency: the **whole 2‑bit base lives in pinned host\nRAM**, and the GPU holds only the dense stack, KV, and an N‑GiB **cache of hot experts** (the\ndelta‑tier slot machinery, read inside CUDA graphs; background prefetch converges it to the\nrouted working set). MoE routing is concentrated enough to make this practical: **~19%\ncoverage serves ~96% of token→expert routings** on DS4, **~51% serves ~91%** on GLM —\nmeasured live, not simulated.\n\nMisses stay correct through the gate's replay trick: the desc kernel zeroes a missing\nexpert's contribution and bumps an in‑graph miss counter; the runner fetches **all** missing\nrouted experts in one batched pinned‑H2D transfer (51.6 GiB\u002Fs here; a 64‑expert fetch ≈ 3 ms)\nand replays the step's graph once — **bit‑identical** to a fully resident forward\n(unit‑tested). A **miss‑tolerance knob** (`VLLM_MOE_W2_BASE_MISS_TOL=k`, runtime‑tunable)\nskips the replay when ≤ k of the step's ~600 routings miss — +12% decode on GLM TP2 at\ntol 8 (28.3 → 31.7 tok\u002Fs) with clean quality probes (arithmetic, PL coherence, needle\nretrieval; quantitative eval pending).\n\n**Pool size is the dominant knob — treat it as a config KPI.** The mandatory replay is paid\n*per step*, so decode tracks the fraction of **zero‑miss steps**, which falls off a cliff\nwith coverage while the token hit‑rate barely moves. On DS4 (1× 5090, NVMe‑store stack,\nsame box, same bench): 11 GiB pool \u002F util 0.90 = **27–28 tok\u002Fs**, 14 GiB \u002F util 0.95 =\n**~31 tok\u002Fs**. The engine reports the KPI directly: pool coverage at startup and a periodic\n**`[base] KPI: replay X% of last N steps…`** line (cadence `VLLM_MOE_W2_KPI_EVERY`, default\n500 steps). If replay % runs high, grow `VLLM_MOE_W2_BASE_CACHE_GB` (and free VRAM for it,\ne.g. `--gpu-memory-utilization 0.95`) before touching any other knob.\n\n**Misses are restored adaptively.** A replayed step can re‑route onto experts the first\npass never fetched (second‑order misses); the runner re‑checks after each replay and keeps\nreplaying **only while the step is within `VLLM_MOE_W2_FP_THRESH` of miss‑free** (default 0\n= the mandatory first‑order restore only, the throughput‑optimal setting; raising it buys\nbit‑deterministic fixed points on converged working sets at a decode cost — runtime‑tunable\nvia `VLLM_MOE_W2_FP_THRESH_FILE`). The accepted residue is second‑order only and\nKPI‑visible (`fp-residue`).\n\nResults: **DeepSeek‑V4‑Flash 159B on one RTX 5090** (72.7 GiB of 2‑bit planes vs 32 GB of\nVRAM): ~31 tok\u002Fs steady with MTP, 32K window served, coherent — and ~30 GiB of host RAM\nwith the NVMe stores (below, RSS‑measured) instead of ~80 GiB pinned.\n**GLM‑5.2 753B on two RTX PRO 6000**: 28–32 tok\u002Fs with the full three‑tier stack (NVMe 2‑bit\nbase + pinned arena → GPU 2‑bit cache → GPU FP4) at a 128K single‑user window — see the GLM\ntable above. Neither model can otherwise run on that hardware at any precision.\n\n---\n\n## One tier further — expert stores on NVMe\n\n`VLLM_MOE_W2_STORE_DIR=\u002Fpath\u002Fon\u002Freal\u002Ffs` moves the host expert stores — the 73–190 GiB of\n2‑bit base planes, and the FP4 need‑pool sections when that tier is enabled — out of RAM\ninto per‑rank **pack files**: raw rows at `(layer·E + expert) · stride`, 4 KiB‑aligned, JSON\nsidecar with shapes and the layers written. Three things fall out:\n\n- **The RAM wall falls.** Single‑5090 DS4 no longer needs ~80 GiB of free host RAM\n  (measured host RAM of the serving process: 42–44 → 26–33 GiB), and GLM TP2's expert\n  stores go **~568 → ~136 GiB** host RAM — the config fits hosts that could never hold the\n  pinned stores.\n- **The pack is a persistent quantization cache.** The first boot writes it while\n  quantizing; every later boot **skips the dequant→re‑quant entirely** and serves experts\n  straight from the pack — GLM TP2 boots in **~7 min instead of ~11** and skips the\n  ~405 GiB staging transient. Stale packs (shape\u002Fconfig mismatch) rebuild automatically;\n  layers absent from a pack (e.g. the MTP drafter) quantize as before.\n- **Decode stays at pinned parity — give it an arena.** `VLLM_MOE_W2_BASE_RAM_GB=\u003CGiB|auto>`\n  pins an MRU **arena** over the base pack: arena hits are zero‑copy pinned views (H2D DMAs\n  straight from the arena — no syscall, no memcpy), misses read through the page cache into\n  the arena slot. The arena behaves as a victim cache of the GPU pool (85% of fetches served\n  from RAM at 27% arena coverage on DS4).\n\nThe three selectable host‑store backends, benched head‑to‑head (DS4 1× 5090, 11 GiB GPU\npool, MTP k=2, same box and bench):\n\n| host store | decode | host RAM (process RSS) |\n|---|---:|---:|\n| pinned — all 73 GiB in RAM (default) | 33.0 tok\u002Fs | 42–44 GiB |\n| pack only — page cache as the RAM tier | 25.5 tok\u002Fs | 15 GiB |\n| pack + **20 GiB pinned arena** | **32.8 tok\u002Fs (parity)** | 26–33 GiB |\n\nGLM‑5.2 TP2 three‑tier with both stores on NVMe (57 GiB\u002Frank arena): 28–32 tok\u002Fs steady,\nneedle retrieval 4\u002F4 to **121K prompt tokens** at a served 128K window. Enabling it is two\nenv lines on any base‑cache config (DS4 shown; same two lines serve GLM TP2):\n\n```bash\n  -e VLLM_MOE_W2_STORE_DIR=\u002Fserve\u002Fpacks \\  # pack dir on a bind-mounted ext4\u002Fxfs\n  -e VLLM_MOE_W2_BASE_RAM_GB=20 \\          # pinned MRU arena; \"auto\" = 25% of the pack\n```\n\nOperational notes:\n\n- **Disk budget:** DS4 pack 75 GB; GLM TP2 2×100 GB (base) + 2×189 GB (fp4) — plan ~1 TB of\n  NVMe for the full GLM stack including the checkpoint. Packs are read‑only after the first\n  boot (SSD wear is a non‑issue). The dir must be a real filesystem via bind mount — **not\n  overlayfs** (the container filesystem).\n- **You don't need a fast drive for steady decode.** Misses are buffered reads, so the page\n  cache acts as an opportunistic L3 under the arena — the parity numbers above come from a\n  drive on a PCIe **Gen3 x4** link (3.7 GB\u002Fs). Cold working‑set shifts and first‑touch\n  prefills do pay drive speed. `VLLM_MOE_W2_TIER_DIRECT=1` switches misses to O_DIRECT\n  (hard RAM budget, page cache stays flat; raw drive latency on every miss).\n- **Prefill can't wipe the arena** (scan discipline: prefill working sets fill free slots\n  but never evict the decode hot set), and the arena's hot set persists to\n  `\u003Cpack>.heat.json` and **preheats on boot** (57 GiB ≈ 35 s; `VLLM_MOE_W2_TIER_PREHEAT=0`\n  opts out). Reader pool: `VLLM_MOE_W2_STORE_THREADS` (default 8).\n- **Observability:** with `VLLM_MOE_W2_DELTA_TRACE=1` the summary carries a\n  `[base] tiered store: arena N\u002FM | fetch rows X ram + Y nvme (Z% ram) | … p50\u002Fp99` line —\n  the ram‑hit % *is* the arena‑coverage curve; grow `BASE_RAM_GB` if it sags.\n- Replays stay **bit‑identical** across backends (bytes are bytes; only the copy source\n  changes) — unit‑tested per backend × cold\u002Fwarm\u002Freboot\u002Fevict\u002Foverflow\u002Fscan\u002Fpreheat in\n  `tools\u002Ftest_store_backends.py`.\n\n---\n\n## The base: vLLM v0.24.0 on SM120\n\nUpstream v0.24.0 ships DeepSeek‑V4 + GLM‑5.x + SM120 natively — but the release cannot\nactually serve them on SM120. The patch carries the fixes (details in\n[docs\u002Fv024-port.md](docs\u002Fv024-port.md)):\n\n- **DeepGEMM**: release pin has no family‑120 host paths (\"Unknown SF transformation\",\n  einsum\u002Findexer asserts) → pin **nv‑dev `a6b593d2`** (as vLLM main did).\n- **flashinfer**: official 0.6.12 pin predates the SM120 DS4 attention API → **0.6.14**.\n- `cooperative_topk` uses thread‑block **cluster launch** (SM90\u002F100‑only) → gated off on SM12x.\n- o_proj fp8 einsum: SM100 packed scale layout NaNs on SM120 → SM90‑style raw f32 scales.\n- CUDA‑graph capture: `thread_local` error mode on **all four** capture paths (the expert\n  caches' background threads must not invalidate capture).\n\nWith the 2‑bit knobs off, the patch is exactly these base fixes — stock behaviour otherwise.\n\n## Quickstart\n\n```bash\ngit clone https:\u002F\u002Fgithub.com\u002Fkacper-daftcode\u002FvLLM-Moet && cd vLLM-Moet\n\n# official vllm-openai:v0.24.0 image + patch + pins + SM120 cubins\nDOCKER_BUILDKIT=1 docker build -f Dockerfile.sm120-v024 -t vllm-moet-sm120:v024 .\n```\n\n**Easiest path — run a benchmarked recipe.** The recipes image downloads the\ncheckpoint from HuggingFace on first run and starts the exact configuration\nthe benchmark table below was measured with (one recipe per supported\nmodel×hardware combo, see `bench\u002Frecipes\u002F`):\n\n```bash\nDOCKER_BUILDKIT=1 docker build -f Dockerfile.recipes -t vllm-moet-recipes:v024 .\n\ndocker run --rm vllm-moet-recipes:v024 --list          # supported configs\ndocker run --rm --gpus all --network host --ipc host --shm-size 64g \\\n  -v \u002Fsrv\u002Fmodels:\u002Fmodels -e HF_TOKEN=... \\\n  vllm-moet-recipes:v024  glm-5.2-nvfp4\u002Fpro6000x4-tp4-mtp\n```\n\n(`-e KNOB=...` overrides any recipe knob, `\u003Crecipe> --print` shows what would\nrun without serving, args after `--` go to vllm serve; details in\n[bench\u002FREADME.md](bench\u002FREADME.md).)\n\nOr hand‑roll the serve. **GLM‑5.2 on 4× PRO 6000** (the standing agent‑serving config:\n128K window, MTP, FP4 pool + gate, tool\u002Freasoning parsers):\n\n```bash\ndocker run --rm --gpus '\"device=0,1,2,3\"' --network host --ipc host --shm-size 64g \\\n  -v \u002Fpath\u002Fto\u002FGLM-5.2-NVFP4:\u002Fmodel:ro \\\n  -e VLLM_MOE_W2=1 -e VLLM_MOE_W2_DELTA_GB=auto -e VLLM_MOE_W2_GATE=1 \\\n  vllm-moet-sm120:v024 \\\n  --model \u002Fmodel --served-model-name glm-5.2 --trust-remote-code \\\n  --tensor-parallel-size 4 --disable-custom-all-reduce \\\n  --kv-cache-dtype fp8 --max-model-len 131072 \\\n  --gpu-memory-utilization 0.90 --max-num-batched-tokens 2048 --max-num-seqs 4 \\\n  --speculative-config '{\"method\":\"mtp\",\"num_speculative_tokens\":2}' \\\n  --tool-call-parser glm47 --enable-auto-tool-choice --reasoning-parser glm45 \\\n  --compilation-config '{\"cudagraph_mode\":\"FULL_AND_PIECEWISE\",\"custom_ops\":[\"all\"]}' \\\n  --port 8000\n```\n\n(`--kv-cache-dtype nvfp4` enables the 352 B\u002Ftoken KV cache; the image ships it ready — the\nFlashInfer JIT sources are patched and precompiled at build (`tools\u002Fnvfp4_flashinfer_sm120\u002F`,\nadded to the vLLM tree by the patch) and the packed-write kernel is prebuilt at\n`\u002Fopt\u002Fnvfp4-ds-mla`. Currently validated to 128K windows.)\n\n**GLM‑5.2 on 2× PRO 6000** (three‑tier + NVMe stores, the 128K\u002Fneedle‑121K config from the\ntable; ~140 GiB host RAM + ~580 GB NVMe for the packs, first boot writes them):\n\n```bash\ndocker run --rm --gpus '\"device=0,1\"' --network host --ipc host --shm-size 64g \\\n  -v \u002Fpath\u002Fto\u002FGLM-5.2-NVFP4:\u002Fmodel:ro -v \u002Fnvme\u002Fpacks-glm:\u002Fpacks \\\n  -e VLLM_MOE_W2=1 -e VLLM_MOE_W2_BASE_CACHE_GB=46 -e VLLM_MOE_W2_DELTA_GB=2 \\\n  -e VLLM_MOE_W2_GATE=1 -e VLLM_MOE_W2_BASE_MISS_TOL=8 \\\n  -e VLLM_MOE_W2_STORE_DIR=\u002Fpacks -e VLLM_MOE_W2_BASE_RAM_GB=57 \\\n  vllm-moet-sm120:v024 \\\n  --model \u002Fmodel --served-model-name glm-5.2 --trust-remote-code \\\n  --tensor-parallel-size 2 --disable-custom-all-reduce \\\n  --kv-cache-dtype fp8 --max-model-len 131072 --kv-cache-memory-bytes 8589934592 \\\n  --gpu-memory-utilization 0.90 --max-num-batched-tokens 2048 --max-num-seqs 2 \\\n  --speculative-config '{\"method\":\"mtp\",\"num_speculative_tokens\":2}' \\\n  --compilation-config '{\"cudagraph_mode\":\"FULL_AND_PIECEWISE\",\"custom_ops\":[\"all\"]}' \\\n  --port 8000\n```\n\n(31.7 tok\u002Fs at `MISS_TOL=8` as shown, 28.3 at strict `0` — both probe‑clean on this stack;\ndrop the two `STORE_DIR`\u002F`BASE_RAM_GB` lines for the all‑RAM variant, which then needs\n~200 GiB free host RAM and re‑quantizes on every boot.)\n\n**DeepSeek‑V4‑Flash on one PRO 6000** (161 tok\u002Fs, 512K window):\n\n```bash\ndocker run --rm --gpus '\"device=0\"' --network host --ipc host --shm-size 64g \\\n  -v \u002Fpath\u002Fto\u002FDeepSeek-V4-Flash:\u002Fmodel:ro \\\n  -e VLLM_MOE_W2=1 -e VLLM_MOE_W2_DELTA_GB=1 \\\n  vllm-moet-sm120:v024 \\\n  --model \u002Fmodel --served-model-name deepseek-v4-flash --trust-remote-code \\\n  --kv-cache-dtype fp8 --block-size 256 --max-model-len 24576 \\\n  --gpu-memory-utilization 0.95 --max-num-batched-tokens 1024 --max-num-seqs 4 \\\n  --tokenizer-mode deepseek_v4 --no-scheduler-reserve-full-isl \\\n  --speculative-config '{\"method\": \"deepseek_mtp\", \"num_speculative_tokens\": 2}' \\\n  --compilation-config '{\"cudagraph_mode\":\"FULL_AND_PIECEWISE\",\"custom_ops\":[\"all\"]}' \\\n  --port 8000\n```\n\n`VLLM_MOE_W2=0` = stock FP4 path (needs ≥2 cards for DS4; GLM's stock NVFP4 does not fit this\nbox at all). TP: `--tensor-parallel-size 2|4` + `--disable-custom-all-reduce`. Single‑5090\nDS4 (host‑resident base): `-e VLLM_MOE_W2_BASE_CACHE_GB=14 -e VLLM_MOE_W2_DELTA_GB=0` with\n`--max-model-len 8192 --gpu-memory-utilization 0.95 --max-num-seqs 2` (~80 GiB free host\nRAM — **or add the two NVMe‑store lines from the section above and run in ~30 GiB**; MTP\nworks). Do not shrink the pool below 14 GiB to \"play it safe\" — pool size is the dominant\nperf knob (see the KPI note above; 11 GiB costs ~12% decode and doubles the missing pairs\nper step) and 14 GiB needs util 0.95 to leave room for KV.\n\n## Quality\n\nMethod: baseline is the untouched official checkpoint; our variant changes only the expert\ncodes (same stack, byte‑identical dense\u002Fscales\u002Fheaders), so any delta is the quantization\nalone — see [docs\u002Fquality.md](docs\u002Fquality.md). The QUANT_PROBE study (identical quant scheme\nand cubins): MTP acceptance 2.73 vs 2.68 FP4 reference, draft accept 86.3% vs 84.1%, 12\u002F12\ncoherent greedy outputs; bare 2‑bit agrees with FP4 on 89% of next‑token picks — the delta\ncache + gate close that gap. Live serving reproduces the acceptance (~2.6 tok\u002Fstep on DS4,\n2.3–3.0 on GLM).\n\n## The SM120 toolchain we built\n\nThese kernels exist only because we first built the assembler and the ISA data they need.\nConsumer Blackwell (sm_120) has **no public SASS toolchain**. Current CUDA does expose the\nblock‑scaled MMA *instruction* itself (PTX `kind::mxf8f6f4` compiles to `QMMA.SF` — DeepGEMM's\nSM120 port uses it), but everything these kernels are actually made of — hand scheduling\nagainst measured latencies and control words, the PRMT‑LUT decode interleaved into the QMMA\nstream, register‑bank and occupancy shaping (regcount 64 → 4 CTA\u002FSM) — is decided by ptxas\nand unreachable from CUDA\u002FPTX. So the stack underneath this repo is end‑to‑end ours:\n\n- **[`blackwell-isa`](https:\u002F\u002Fgithub.com\u002Fkacper-daftcode\u002Fblackwell-isa)** — a machine‑readable\n  **SM120 SASS ISA database**: 1,994 instruction forms, 128‑bit encoding templates + operand\u002F\n  bitfield maps, and per‑opcode scheduling metadata (pipeline\u002Flatency\u002Fthroughput, control‑word\n  classes). Reverse‑engineered and hardware‑validated on RTX 5090 (47,244 instructions decoded\n  across 178 cubins at 100% coverage; 5,014\u002F5,014 roundtrip‑fuzz). It documents what the CUDA\n  toolchain hides — e.g. `QMMA.SF` block‑scaled FP4 MMA and an undocumented `E3M4` type code.\n  Ships a [searchable HTML reference](https:\u002F\u002Fkacper-daftcode.github.io\u002Fblackwell-isa\u002FSM120_ISA_REFERENCE.html).\n- **[`cubit`](https:\u002F\u002Fgithub.com\u002Fkacper-daftcode\u002Fcubit)** — an **SM120 SASS assembler\u002Fdisassembler**\n  built on that database. It turns the hand‑written `.sass` sources in `kernels\u002Fsass\u002F` into the\n  cubins this server loads, and is the only tool needed to rebuild or audit them.\n\n**ISA ([`blackwell-isa`](https:\u002F\u002Fgithub.com\u002Fkacper-daftcode\u002Fblackwell-isa)) → assembler\n([`cubit`](https:\u002F\u002Fgithub.com\u002Fkacper-daftcode\u002Fcubit)) → SASS kernels → this vLLM.** None of the\nkernels here can be *written* through stock CUDA on sm_120 — the instructions compile, the\nkernels don't; this toolchain is what makes them possible.\n\n## Kimi-K2.7-Code (1T MoE) on 4× RTX PRO 6000\n\n**[nvidia\u002FKimi-K2.7-Code-NVFP4](https:\u002F\u002Fhuggingface.co\u002Fnvidia\u002FKimi-K2.7-Code-NVFP4)\n(1T params, 384 experts top‑8, H=7168, dense MLA, 256K) serves on 4× RTX PRO 6000 (TP4)**\n— a checkpoint whose 595 GB of weights cannot even load on this box (384 GB total VRAM):\nthe 2‑bit planes total **265.8 GiB** (~66 GiB\u002Frank + ~8 GiB BF16 dense\u002Fvision), leaving\nroom for a 337K‑token fp8 KV pool and the FP4 delta tier in 96 GB\u002Fcard. The loader path\nis the same modelopt‑NVFP4 requant as GLM‑5.2‑NVFP4 (f64‑exact dequant → sign‑symmetric\n2‑bit), through a new **K=7168** cubin family (`kernels\u002F`, generated + op‑validated like\nthe rest).\n\nMeasured (TP4, 131072‑token window, greedy, CUDA graphs, no MTP — the checkpoint ships\nno drafter head; 2026‑07‑10): **51 tok\u002Fs** single‑stream decode (**222 tok\u002Fs** aggregate\nat 8 streams), **2 448 tok\u002Fs** 8K‑unique prefill, needle retrieval **PASS at\n8K\u002F32K\u002F80K\u002F128K**, arithmetic 5\u002F5, generated code executes, `kimi_k2` tool‑calling\nround‑trip works. Serve recipe and\nthe bring‑up findings (a checkpoint‑specific **zero‑sign balancing** fix the sweep gate\ncaught — the INT4→NVFP4 export writes all exact zeros as +0, which would inject 3× the\nbias that degenerates GLM; an SM12x smem fix for dense‑MLA triton decode; a\n`merge_attn_states` stride fix for >64K chunked prefill):\n**[docs\u002Fkimi-k27-code.md](docs\u002Fkimi-k27-code.md)**.\n\n## Benchmark results\n\n\u003C!-- bench:table:begin (generated by bench\u002Frunner\u002Frender.py - do not edit) -->\n\nRelease **`baseline-2026-07-10`** — one row per supported recipe (`bench\u002Frecipes\u002F`), measured by `bench\u002Frunner\u002Fbench.py`; full report: [`docs\u002Fbenchmarks\u002Fbaseline-2026-07-10.md`](docs\u002Fbenchmarks\u002Fbaseline-2026-07-10.md). Single-stream decode and prefill are medians; batch is aggregate tok\u002Fs at the noted concurrency.\n\n| model | hardware | config | ctx | decode tok\u002Fs | batch | prefill 8K | needle | notes |\n|---|---|---|---:|---:|---:|---:|---|---|\n| deepseek-v4-flash | 1x RTX 5090 (32 GB) | host-resident 2-bit base, GPU as expert cache | 8K | **38** | — | — | — | acc 2.83 † |\n| deepseek-v4-flash | 4x RTX 5090 TP4 | consumer-card throughput | 16K | **214.4** | 1 560 @32 | 6 101 | — | acc 2.6 † |\n| deepseek-v4-flash | 1x RTX PRO 6000 | throughput (24K ctx, FP4 delta auto, MTP k=2) | 24K | **161.2** | 933 @32 | 5 340 | — | acc 2.6 † |\n| deepseek-v4-flash | 1x RTX PRO 6000 | 512K window (delta pool traded for KV) | 512K | — | — | — | PASS ≤453K tok | † |\n| deepseek-v4-flash | 2x RTX PRO 6000 TP2 | throughput | 24K | **209.6** | 380 @3 | 5 791 | — | acc 2.6 † |\n| glm-5.2-nvfp4 | 2x RTX PRO 6000 TP2 | host-resident base, 44 GiB\u002Frank expert cache | 32K | **33** | — | — | PASS ≤27K tok | acc 3 † |\n| glm-5.2-nvfp4 | 4x RTX PRO 6000 TP4 | 2-bit base + MTP k=2, 128K window | 128K | **105** | — | 2 500 | PASS ≤276K tok | † |\n| glm-5.2-nvfp4 | 4x RTX PRO 6000 TP4 | + FP4 delta (auto) + confidence gate tau=0.60 | 128K | **84** | — | — | — | † |\n| kimi-k2.7-code-nvfp4 | 2x RTX PRO 6000 TP2 | host-resident base, 52 GiB\u002Frank cache (~39% coverage) | 16K | **14.4** | — | — | PASS ≤8K tok | † |\n| kimi-k2.7-code-nvfp4 | 4x RTX PRO 6000 TP4 | GPU-resident 2-bit + FP4 delta, 256K window | 256K | **51** | 222 @8 | 2 448 | PASS ≤248K tok | † |\n| kimi-k2.7-code-nvfp4 | 4x RTX PRO 6000 TP4 | + Eagle3 drafter (k=3, drafter TP4) | 256K | **57** (±44%) | — | — | — | acc 3.5 † |\n\n† imported from pre-harness measurements (README\u002Fdocs history) — re-measured on the next release.\n\n\u003C!-- bench:table:end -->\n\n## Repository layout\n- **`patch\u002Fvllm-moet-v0.24.0.patch`** — the delta vs official vLLM `v0.24.0` (37 files,\n  +7.4k lines; applies clean on the tag). Goes with the pins above.\n- **`Dockerfile.sm120-v024`** — the image: official `vllm\u002Fvllm-openai:v0.24.0` + patch + pins +\n  cubins.\n- **`kernels\u002F`** — SASS (`sass\u002F`) + prebuilt SM120 cubins (`cubins-sm120\u002F`, incl. the K=6144\n  GLM‑5.x family) + generators (`gen\u002F`) + `MANIFEST.md`.\n- **`docs\u002Fv024-port.md`** — the port: pins, SM120 fixes, apply recipe, benchmark methodology.\n- **`docs\u002Fquality.md`** — quality methodology.\n- **`bench\u002F`** — the release benchmark system: recipes (the tested serve configs, one YAML per\n  supported model×hardware combo), the runner that stands up each config and measures it, and\n  committed results per release. The table above is rendered from `bench\u002Fresults\u002F` by\n  `bench\u002Frunner\u002Frender.py` (CI keeps them in sync); process: **[bench\u002FREADME.md](bench\u002FREADME.md)**,\n  per‑release detail: `docs\u002Fbenchmarks\u002F`.\n- **`Dockerfile.recipes`** — the user‑facing image: picks a recipe by id, downloads the\n  checkpoint into the `\u002Fmodels` volume, and serves the exact benchmarked configuration\n  (`docker\u002Fserve_recipe.py` is the entrypoint; the bench's docker runtime measures this same\n  image).\n","这是一个针对消费级Blackwell显卡（如RTX 5090、RTX PRO 6000）优化的大规模MoE模型推理补丁，基于vLLM v0.24.0实现。核心功能包括：2-bit专家权重压缩与FP4增量缓存协同恢复精度；三级专家驻留机制（GPU内存→主机内存→NVMe存储），支持753B级GLM-5.2等前沿MoE模型在有限VRAM下运行；并集成NVFP4 KV缓存、MTP推测解码及工具调用解析能力。适用于资源受限环境下的超大规模MoE模型部署与推理服务场景。",2,"2026-07-11 02:30:47","CREATED_QUERY"]