[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"project-92280":3},{"id":4,"name":5,"fullName":6,"owner":7,"repo":5,"description":8,"homepage":9,"htmlUrl":9,"language":10,"languages":9,"totalLinesOfCode":9,"stars":11,"forks":12,"watchers":13,"openIssues":14,"contributorsCount":14,"subscribersCount":14,"size":14,"stars1d":14,"stars7d":14,"stars30d":15,"stars90d":14,"forks30d":14,"starsTrendScore":14,"compositeScore":16,"rankGlobal":9,"rankLanguage":9,"license":17,"archived":18,"fork":18,"defaultBranch":19,"hasWiki":20,"hasPages":18,"topics":21,"createdAt":9,"pushedAt":9,"updatedAt":22,"readmeContent":23,"aiSummary":24,"trendingCount":14,"starSnapshotCount":14,"syncStatus":25,"lastSyncTime":26,"discoverSource":27},92280,"bw24","avifenesh\u002Fbw24","avifenesh","From-scratch Rust+CUDA LLM inference engine for RTX 5090 (sm_120a) — NVFP4, MoE, MTP speculative decoding, tuned against measured hardware limits",null,"Rust",259,31,1,0,4,44.92,"MIT License",false,"main",true,[],"2026-07-22 04:02:05","# bw24\n\n[![License: MIT](https:\u002F\u002Fimg.shields.io\u002Fbadge\u002Flicense-MIT-blue.svg)](LICENSE)\n![Rust](https:\u002F\u002Fimg.shields.io\u002Fbadge\u002Frust-edition%202024-orange.svg)\n![CUDA](https:\u002F\u002Fimg.shields.io\u002Fbadge\u002FCUDA-12.8%20%2F%2013.1-76B900.svg)\n![arch](https:\u002F\u002Fimg.shields.io\u002Fbadge\u002Farch-sm__120a%20(Blackwell)-black.svg)\n\n![bw24 vs llama.cpp perf board](docs\u002Fperf-card.svg)\n\nFrom-scratch LLM inference engine in Rust + CUDA, built for one machine: an RTX 5090 Laptop (Blackwell sm_120a, 24 GB, 175 W with dynamic boost). No frameworks, no ggml — every kernel written and tuned against measured hardware limits, with llama.cpp as the benchmark to beat on the same rig.\n\nThe [Performance](#performance) section and card above show same-session measurements against llama.cpp's best config. It also loads HF safetensors checkpoints directly (no GGUF conversion) and runs a 121 GB MoE on this 24 GB card.\n\n## Why this project\n\n- Real, running inference on sm_120a (consumer Blackwell), with every shipped optimization backed by its measured win\u002Floss record.\n- Gated on bit-exactness — argmax match and speculative self-consistency, verified on every kernel change.\n- Runs Qwen3.5\u002F3.6 dense and MoE checkpoints on 24 GB — from GGUF or straight from HF safetensors, no conversion — including models far larger than VRAM+RAM.\n\n## Requirements\n\n- NVIDIA Blackwell consumer GPU (sm_120a). Primary target: RTX 5090 Laptop.\n- CUDA toolkit 13.1 (the build default — `crates\u002Fbw24-engine\u002Fbuild.rs` uses `\u002Fusr\u002Flocal\u002Fcuda-13.1\u002Fbin\u002Fnvcc`; override with `BW24_NVCC=\u002Fpath\u002Fto\u002Fnvcc`).\n- Rust (edition 2024), [cudarc](https:\u002F\u002Fgithub.com\u002Fcoreylowman\u002Fcudarc) 0.19 with dynamic loading.\n- A model — GGUF or an HF safetensors directory (pass either path). Tested: Qwen3.5-9B, Qwen3.6-27B, Qwen3.6-35B-A3B MoE (common quants), nvidia\u002FQwen3.6-27B-NVFP4, MiniMax-M3 REAP50, Hy3-REAP50.\n\n## Quick start\n\n```bash\n# build\ncargo build --release\n\n# verify all kernels against the CPU reference\n.\u002Ftarget\u002Frelease\u002Fkernel-check\n\n# generate text (the fast path is the default — no flags needed)\nBW24_CHAT=1 .\u002Ftarget\u002Frelease\u002Frun-gen \u002Fpath\u002Fto\u002Fmodel.gguf --prompt \"Explain KV caches in one paragraph.\"\n\n# speculative decoding with the embedded MTP draft head (Qwen3.6)\nBW24_SPEC_K=3 .\u002Ftarget\u002Frelease\u002Frun-spec \u002Fpath\u002Fto\u002Fqwen36-27b.gguf\n\n# OpenAI-compatible server\n.\u002Ftarget\u002Frelease\u002Fbw24-server\n```\n\nEvery tuned kernel path is the default; environment flags exist only for runtime parameters (prompt, draft depth, trims), machine-specific configuration, and rollback seams (`BW24_FAST=0` drops to the f32 oracle path). The catalog lives in `docs\u002FFLAGS.md`.\n\n`run-gen` prints a prefill\u002Fdecode correctness gate (prefill argmax must match decode argmax) before timing anything — if that line says MISMATCH, the numbers after it don't count.\n\n## Workspace layout\n\n| Crate | What it does |\n|---|---|\n| `bw24-engine` | Core: CUDA kernels (`cu\u002F`), forward passes, speculative decoding, MoE cache, CUDA-graph decode |\n| `bw24-gguf` | GGUF parser + tensor loading (memory-mapped) |\n| `bw24-tokenizer` | BPE tokenizer + chat templates from GGUF metadata |\n| `bw24-runtime` | CUDA device\u002Fstream\u002Fmemory primitives over cudarc |\n| `bw24-server` | HTTP server (axum), OpenAI-compatible `\u002Fv1` endpoints |\n| `bw24-probe` | Standalone hardware microbenches (`probe\u002F*.cu`: bandwidth, tensor-core peaks, layout experiments) |\n\n## What's inside\n\n- **NVFP4 (W4) decode path** — block-scaled FP4 matvec with split-plane repack, warp-level dp4a, and an int8 W4A8 tensor-core GEMM for prefill. Auto-dispatches per matrix shape.\n- **MTP speculative decoding** — draft with the model's embedded multi-token-prediction head, verify K+1 tokens in one batched target forward. The whole draft chain runs inside a captured CUDA graph; exactness is enforced by a K=1..8 self-consistency gate (all K must emit identical tokens).\n- **MoE on 24 GB** — expert-major CSR batching, decode-once dequant kernels, int8 tensor-core expert GEMM, and an SLRU expert-residency cache with VRAM → host → disk spill.\n- **FlashAttention-style kernels** — fused prefill and decode attention with quantized KV (q8_0 K \u002F q5_1 V default; FP8 and q4_0 arms exist behind flags — q4_0 V measured quality-taxed and stays off), register-resident dequant, split-K for long context.\n- **CUDA-graph decode** — the full per-token decode is one graph replay; per-step host round-trip is 4 bytes.\n- **Hybrid architectures** — full-attention + gated-delta-net (SSM) layer mixes, as in Qwen3.6.\n- **Safetensors loader** — HF checkpoints load directly (no GGUF conversion): modelopt NVFP4 repacks byte-exact into the GGUF block layout, FP8-E4M3 and large-BF16 tensors re-encode to Q8_0\u002FNVFP4 at load, V-head permutations apply on packed bytes, MoE experts stream through a disk-tier repack cache for models far bigger than VRAM+RAM.\n- **Sigmoid-router MoE** (MiniMax\u002FDeepSeek-style) — e_score_correction_bias selection, swigluoai activation, gate-optional attention; with the measured law that cross-kernel-family FP-order differences are architectural on discontinuous top-k routing (exactness binds within a config).\n\n## Correctness discipline\n\nEvery kernel change must pass, in order:\n1. `kernel-check` — every quant kernel vs a CPU reference.\n2. `run-gen` argmax gate — prefill and decode paths must agree on the next token.\n3. `run-spec` self-consistency — speculative output at K=1..8 must be token-identical to plain decode.\n\nFloating-point summation order is part of the contract: two mathematically equal kernels that reduce in different orders can flip an argmax at tight logit margins. Several \"faster\" kernels were rejected for exactly this (`research\u002Ftune-data\u002F`).\n\n## Performance\n\n\u003C!-- PERF-DATE:START (generated by tools\u002Fupdate-perf-board.py — do not hand-edit; edit research\u002Ftune-data\u002Fcurrent-board.json instead) -->\nMeasured 2026-07-09 on the target rig (RTX 5090 Laptop, N≥2 medians, both engines measured in the SAME session and thermal regime, no flags (tuned paths are defaults)) against llama.cpp built on the same machine, same exact prompts, both engines re-baselined the same day. Boards move with the tuning campaign — `research\u002Ftune-data\u002Frig5090.jsonl` is the running record; the README is refreshed with every board-moving merge.\n\u003C!-- PERF-DATE:END -->\n\n**Plain decode first** (no speculation, tg128 at 512-token context — the honest floor comparison):\n\n\u003C!-- PERF-PLAIN:START (generated by tools\u002Fupdate-perf-board.py — do not hand-edit; edit research\u002Ftune-data\u002Fcurrent-board.json instead) -->\n| Model | bw24 plain | llama.cpp plain | Ratio |\n|---|---|---|---|\n| Qwen3.6-27B NVFP4 | 44.8 | 40.3 | **1.11x** |\n| Qwen3.5-9B NVFP4 | 128.1 | 113.9 | **1.12x** |\n| Qwen3.6-35B-A3B MoE | 170.7 | 159.6 | **1.07x** |\n| Qwen3.5-9B NVFP4 ST (safetensors, modelopt) | 129.1 | 123.7 | 1.04x |\n| Qwen3.6-27B NVFP4 ST (NVIDIA official) | 45.2 | 41.2 | **1.10x** |\n\u003C!-- PERF-PLAIN:END -->\n\nDepth is part of the contract: at 6.3k-token context the leads hold (1.04-1.07x) except the 35B at 0.99x. Every attention\u002Fsplit change validates across the depth axis, not just one point.\n\n**Speculative decoding** (MTP head, both engines at their measured best config) as the bonus layer on top:\n\n\u003C!-- PERF-SPEC:START (generated by tools\u002Fupdate-perf-board.py — do not hand-edit; edit research\u002Ftune-data\u002Fcurrent-board.json instead) -->\n| Model | bw24 spec | llama.cpp spec-best | Ratio |\n|---|---|---|---|\n| Qwen3.5-9B (K=3 + native trim) | 246 \u002F 206 \u002F 172 | 186 \u002F 158 \u002F 155 | **1.32x** \u002F **1.30x** \u002F **1.11x** |\n| Qwen3.6-27B (K=3 + generic trim) | 109 \u002F 100 \u002F 78.8 | 86.4 \u002F 89.9 \u002F 73.2 | **1.26x** \u002F **1.11x** \u002F **1.08x** |\n| Qwen3.6-35B-A3B (K=3 + trim + zero-draft) | 251 \u002F 232 \u002F 193 | 215 \u002F 208.5 \u002F 201.7 | **1.17x** \u002F **1.11x** \u002F 0.96x |\n| Qwen3.5-9B ST (K=2-3 per-content + native trim) | 203.9 \u002F 192.5 \u002F 256.0 | 122.9 \u002F 122.2 \u002F 118.2 | **1.66x** \u002F **1.58x** \u002F **2.17x** |\n| Qwen3.6-27B ST (K=3 + HPOST + own-head trim, per-content pmin) | 92.9 \u002F 88.1 \u002F 84.6 | 79.7 \u002F 84.7 \u002F 71.3 | **1.17x** \u002F 1.04x \u002F **1.19x** |\n\u003C!-- PERF-SPEC:END -->\n\nThe three columns are short-code \u002F medium-code \u002F long-agentic prompts, raw-continuation protocol, llama.cpp at serve-best config. Optimal config is content-class dependent. Caveat on the long-agentic column: a text audit found the greedy continuations there partially degenerate into repetition on both engines symmetrically — ratios stand, absolutes overstate real agentic throughput (`rig5090.jsonl` tag `p3-degeneration-audit`; prompt replacement pending). The speculative edge comes from three mechanisms — FR-Spec vocabulary trims, whole-round confidence gating, and per-content-class draft depth — detailed in [`HANDOVER.md`](HANDOVER.md).\n\n**Reproducing:** every artifact is public — trimmed draft-head GGUFs, exact prompts, and full configs at [huggingface.co\u002FAvifenesh\u002Fbw24-bench](https:\u002F\u002Fhuggingface.co\u002FAvifenesh\u002Fbw24-bench). llama.cpp build\u002Fserve flags are in [docs\u002FCOMPETITOR-SETUP.md](docs\u002FCOMPETITOR-SETUP.md); the harness is `research\u002Fe2e\u002Frun-e2e.sh`.\n\n**Known gaps** (tracked and diagnosed in `research\u002Ftune-data\u002F`): prefill trails llama.cpp at 0.55-0.76x (decomposed — the residual is int8-vs-native-FP4 tensor-core ceiling under the exactness contract), the 35B loses two cells narrowly, and vLLM's batched MTP wins on big-VRAM boxes.\n\nBeyond llama.cpp: the safetensors path runs checkpoints llama.cpp cannot — NVIDIA's official Qwen3.6-27B-NVFP4 (2.3x the tuned local vLLM reference) and giant spilled MoEs: MiniMax-M3 REAP50 (121 GB) and Hy3-REAP50 (82 GB) both generate correct text on this 24 GB \u002F 60 GB machine through VRAM→RAM→NVMe expert tiers. Details in [`HANDOVER.md`](HANDOVER.md).\n\n## Limitations\n\n- Built for sm_120a only; tuning choices assume this exact memory\u002Fcompute ratio.\n- Model coverage: Qwen3.5\u002F3.6 dense and MoE, MiniMax-M3, Hy3 — not a general GGUF runner.\n- Single GPU, single stream. No tensor parallelism, no continuous batching.\n- APIs and flags change without notice; moving research codebase.\n\n## Docs\n\n- [`ARCHITECTURE.md`](ARCHITECTURE.md) — tech stack + sm_120a feasibility ledger (what the silicon can and cannot do, measured).\n- [`HANDOVER.md`](HANDOVER.md) — the living state-of-work doc (current standings, laws, open lanes); internal but readable.\n- [`docs\u002Fdecisions\u002F`](docs\u002Fdecisions\u002F) — design decision records: internal weight format, quant\u002FGEMM policy, safetensors import, hybrid-architecture plan.\n- [`docs\u002FCOMPETITOR-SETUP.md`](docs\u002FCOMPETITOR-SETUP.md) — how each competitor engine is built and tuned to its peak on this box (the \"beat them at their best\" contract).\n- [`research\u002Ftune-data\u002Fcurrent-board.json`](research\u002Ftune-data\u002Fcurrent-board.json) — the numbers behind the Performance section and `docs\u002Fperf-card.svg`, both regenerated by [`tools\u002Fupdate-perf-board.py`](tools\u002Fupdate-perf-board.py) — edit the JSON, never the generated regions directly.\n- [`research\u002Fsm120-empirical-capabilities.md`](research\u002Fsm120-empirical-capabilities.md) — microbenched silicon peaks for this GPU.\n- [`research\u002Fbenchmarks.md`](research\u002Fbenchmarks.md) — the A\u002FB measurement protocol.\n- [`research\u002Ftune-data\u002F`](research\u002Ftune-data\u002F) — every tuning experiment as JSONL: config → measured result, wins and losses both. ~215 records and counting; treat it as a labeled corpus of what sm_120a actually rewards.\n\n## Contributing\n\nIssues and PRs welcome — see [CONTRIBUTING.md](CONTRIBUTING.md).\n\n## License\n\nMIT — see [LICENSE](LICENSE).\n","bw24 是一个从零构建的 Rust + CUDA 大语言模型（LLM）推理引擎，专为 NVIDIA Blackwell 架构 RTX 5090 笔记本 GPU（sm_120a）优化。支持 NVFP4 低比特量化、MoE 模型高效加载与执行、MTP 驱动的多跳推测解码，并全部基于实测硬件极限手工调优 CUDA 内核，无需 ggml 或其他框架依赖。可直接加载 Hugging Face safetensors 或 GGUF 格式模型（如 Qwen3.6 MoE），在 24 GB 显存上运行远超显存容量的 121 GB MoE 模型。适用于追求极致硬件利用率、低延迟本地部署的 Blackwell 消费级 GPU 场景，尤其适合科研验证、边缘侧高性能 LLM 推理及定制化推理栈开发。",2,"2026-07-08 04:29:57","CREATED_QUERY"]