[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"project-95917":3},{"id":4,"name":5,"fullName":6,"owner":7,"repo":5,"description":8,"homepage":9,"htmlUrl":10,"language":11,"languages":10,"totalLinesOfCode":10,"stars":12,"forks":13,"watchers":14,"openIssues":15,"contributorsCount":16,"subscribersCount":16,"size":16,"stars1d":16,"stars7d":16,"stars30d":17,"stars90d":16,"forks30d":16,"starsTrendScore":16,"compositeScore":18,"rankGlobal":10,"rankLanguage":10,"license":19,"archived":20,"fork":20,"defaultBranch":21,"hasWiki":20,"hasPages":20,"topics":22,"createdAt":10,"pushedAt":10,"updatedAt":23,"readmeContent":24,"aiSummary":25,"trendingCount":16,"starSnapshotCount":16,"syncStatus":26,"lastSyncTime":27,"discoverSource":28},95917,"Qwen3.8-Flash-Next-Single-DGX-Spark","MiaAI-Lab\u002FQwen3.8-Flash-Next-Single-DGX-Spark","MiaAI-Lab","Qwen3.8-Flash-Next on ONE DGX Spark (TP=1)","https:\u002F\u002Fx.com\u002FMiaAI_lab",null,"Python",302,42,1,21,0,166,4.9,"GNU Affero General Public License v3.0",false,"main",[],"2026-09-21 02:04:29","\u003Ch1 align=\"center\">Qwen3.8-Flash-Next on ONE DGX Spark (TP=1)\u003C\u002Fh1>\n\n\u003Cp align=\"center\">\n  \u003Csub>by \u003Ca href=\"https:\u002F\u002Fx.com\u002FMiaAI_lab\">Mia'a AI Lab\u003C\u002Fa>\u003C\u002Fsub>\n  \u003Cbr>\u003Cbr>\n  \u003Ca href=\"https:\u002F\u002Fgithub.com\u002Fsponsors\u002FMiaAI-Lab\" target=\"_blank\" rel=\"noopener noreferrer\" style=\"display:inline-block;margin:0 8px;vertical-align:middle;\">\u003Cimg src=\"https:\u002F\u002Fimg.shields.io\u002Fbadge\u002FSponsor%20me%20on%20GitHub-181717?style=for-the-badge&logo=githubsponsors&logoColor=white\" alt=\"Sponsor me on GitHub\" height=\"28\" style=\"height:28px;width:auto;vertical-align:middle;border:0;\" \u002F>\u003C\u002Fa>\n  \u003Ca href=\"https:\u002F\u002Fx.com\u002FMiaAI_lab\" target=\"_blank\" rel=\"noopener noreferrer\" style=\"display:inline-block;margin:0 8px;vertical-align:middle;\">\u003Cimg src=\"https:\u002F\u002Fimg.shields.io\u002Fbadge\u002FFollow%20me%20on%20X-000000?style=for-the-badge&logo=x&logoColor=white\" alt=\"Follow Mia on X\" height=\"28\" style=\"height:28px;width:auto;vertical-align:middle;border:0;\" \u002F>\u003C\u002Fa>\n\u003C\u002Fp>\n\nSelf-contained recipe for serving the `Mia-AiLab\u002FQwen3.8-Flash-Next-NVFP4`\ncheckpoint (99 GB) from a single DGX Spark's 121 GiB unified memory, via vLLM\nwith the PLE table offloaded and memory-mapped. This is a **vision-language**\nmodel: text, images and video all work out of the box (see below). Nothing here depends on the\n2-node files it was derived from.\n\n```\ncp .env.sample .env        # edit IMAGE \u002F HF_TOKEN if needed\n.\u002Fdownload.sh              # fetch the ~99 GB checkpoint (resumable)\n.\u002Fstart.sh                 # ~10-12 min to \u002Fhealth; serves on :8888\n.\u002Fstop.sh                  # container + watchdog, graceful\n```\n\n`start.sh` never downloads anything — it resolves the checkpoint from the local\nHugging Face cache and fails fast if it is absent. Budget ~130 GiB of free disk:\n99 GB for the checkpoint plus the ~27 GB packed PLE table built on first launch.\n\n`.\u002Fstart.sh --no-launch` prints the derived memory budget and the docker\ncommand without running anything. `.\u002Fstop.sh` sends SIGTERM and waits up to\n`STOP_TIMEOUT` (default 30 s) so vLLM can unlink its POSIX shared memory —\nthe container runs with `--ipc host`, so segments it leaves behind leak onto\nthe host's `\u002Fdev\u002Fshm` until reboot. `.\u002Fstop.sh --force` skips the wait.\n\n## Measured profile\n\n`.env.sample` ships **262,144 context (YaRN off), MTP 3, `HOST_RESERVE_GIB=26`,\n`KV_TARGET_GIB=20`, `KV_CACHE_DTYPE=fp8`, `MAMBA_SSM_CACHE_DTYPE=bfloat16`,\n`MAX_NUM_SEQS=4`, `MAX_NUM_BATCHED_TOKENS=2048`**, with the V2 model runner\npinned through `EXTRA_DOCKER_ARGS`.\nEverything below was measured on this host on 2026-09-04; each row names the\nconfiguration it came from, because the numbers move a lot between them.\nDecode numbers are not in this table: they predate the 2026-09-05 optimisation\npass and are superseded by the [sparkDash sweep](#prefill-and-decode-measured-with-sparkdash)\nbelow (48.7 tok\u002Fs single-stream prose, 162.9 aggregate at 8 streams).\n\n| Configuration | KV pool | Prefill @400k | Needles 5\u002F50\u002F95% |\n|---|---|---|---|\n| 262k, `KV_TARGET_GIB=20`, BF16 | 21.28 GiB = 736,837 tok (2.81x a 262k req) | — | — |\n| 512k YaRN, `KV_TARGET_GIB=20`, BF16 | 19.2 GiB = 704,558 tok (1.34x a 512k req) | 1,537 tok\u002Fs (TTFT 260.3 s) | 3\u002F3 PASS |\n| 512k YaRN, `KV_TARGET_GIB=22`, BF16 | 796,196 tok (1.52x a 512k req) | 1,883 tok\u002Fs @32k | 12\u002F14 (see FP8 section) |\n| 512k YaRN, `KV_TARGET_GIB=22`, FP8 | 22.2 GiB = 1,431,164 tok (2.73x a 512k req) | 1,495 tok\u002Fs (TTFT 267.7 s); 1,769 tok\u002Fs @32k | 15\u002F20 (see FP8 section) |\n\nThe same profile at `KV_TARGET_GIB=16` and `MAX_NUM_SEQS=5` (262k,\n`HOST_RESERVE_GIB=26`, FP8) was measured on 2026-09-05. The shipped wish is now\n20, which this host clips to 16.67 GiB — see [Configuration](#configuration):\n\n| | |\n|---|---|\n| GPU budget | GMU 0.780 = 94.87 GiB |\n| Available KV cache | 16.46 GiB = **992,584 tokens** (3.79x a 262k request) |\n| Time to `\u002Fhealth` | 10 min 51 s (checkpoint read from NVMe) |\n| Host MemAvailable, 2 min after `\u002Fhealth` | 15.7 GiB (MemFree 5.1 GiB) |\n| Host MemAvailable, 40 idle minutes | 15.5–16.4 GiB (MemFree ≥ 4.4 GiB) |\n| Host MemAvailable after two ~90k prompts | 16.2 → 15.05 GiB after the first, 15.2 GiB 60 s after the second (min 14.9 during prefill; MemFree ≥ 3.5 GiB) |\n| Host MemAvailable, five concurrent ~60k prompts | 14.9 → 14.57 GiB at +60 s (min 14.26 during; MemFree ≥ 3.24 GiB); 5\u002F5 completed, no watchdog event |\n| 2.5 h under the qwen-code harness (~38 requests, 19 of them 50–100k tokens, up to 3 concurrent) | 14.2–14.9 GiB between turns, min 12.8 GiB at 3 concurrent; driver 96.6 → 97.5 GiB in one step |\n| `NV_ERR_NO_MEMORY` in `journalctl -k` | 0 across launch and all of the above |\n\nThe idle figure used to be quoted here as ~12.9 GiB; that came from a\n2026-09-04 run — before the host-side cap existed, so `KV_TARGET_GIB=20` then\nmeant 20 GiB of KV rather than today's clipped 16.67 — and before the day's\nco-tenants were on the box. See the safety\nrules for why the number matters.\n\nOne honest gap: `KV_TARGET_GIB=20` is now measured end to end — ten launches,\nthe 1\u002F2\u002F4\u002F8-stream decode sweep, needles 15\u002F15 at 32k and a 45-minute soak on\n2026-09-06 — but every one of those ran at **512k YaRN**, not at the shipped\n262k native rope. The 262k row above is at BF16 and predates the `MADV_RANDOM`\nmmap change. Short-context (32k) prefill is measured for both dtypes; see the\nFP8 section.\n\n`KV_TARGET_GIB` shipped as 22, then 20, until 2026-09-05. Both lost servers:\nthree on 2026-09-04. The rows above at 22 are real measurements, but the host\nthey were taken on had 6.9–8.8 GiB of `MemAvailable` left, against a 6 GiB\nwatchdog floor and a GPU driver that refuses allocations before that. The\nbudget is now capped from the host side (`HOST_RESERVE_GIB`, see\n[Safety rules](#safety-rules)); the KV pool is whatever the cap leaves. The\nshipped wish is 20 and this host clips it to 16.67 GiB, so 20 and 22 of actual\nKV are no longer reachable through `KV_TARGET_GIB` alone; a pinned\n`GPU_MEMORY_UTILIZATION` still gets you there, with a warning.\n\n### Prefill and decode, measured with sparkDash\n\nBoth sweeps below were measured with\n[sparkDash](https:\u002F\u002Fgithub.com\u002FMiaAI-Lab\u002FsparkDash) against this server.\n`bench\u002Fsweep.py` drives it one concurrency level at a time and reads vLLM's\nown counters around each level, which is what makes the ms-per-step and\ntokens-per-step columns below comparable across launches; `bench\u002Fmixed.py`\ncovers the decode-under-prefill case sparkDash has no mode for. The two\nprefill columns are **not** a clean A\u002FB — they differ in rope config and KV\ntarget as well as chunk width — so each is labelled with what it was measured\nat.\n\n#### 2026-09-06: BF16 recurrent state and every verify width on a graph\n\nMeasured on 512k YaRN, MTP 3, FP8 KV, 2,048 chunks, `CUDAGRAPH_CAPTURE_SIZES=auto`,\n`MAMBA_SSM_CACHE_DTYPE=bfloat16`, the V2 model runner pinned, and\n**`MAX_NUM_SEQS=8`** — which is the one part of this row that `.env.sample`\ndoes not ship, because every sweep here was short-context. Prose, 600 tokens,\nthree repeats per level, `bench\u002Fsweep.py`:\n\n| streams | ms\u002Fengine step | tokens\u002Fstep | aggregate | per stream |\n|---|---|---|---|---|\n| 1 | 61.5 | 3.00 | **48.7 tok\u002Fs** | 48.7 tok\u002Fs |\n| 2 | 74.3 | 2.83 | **74.6 tok\u002Fs** | 37.3 tok\u002Fs |\n| 4 | 96.2 | 2.84 | **113.7 tok\u002Fs** | 28.4 tok\u002Fs |\n| 8 | 131.0 | 2.81 | **162.9 tok\u002Fs** | 20.4 tok\u002Fs |\n\nTwo changes separate this from the 2026-09-05 row. `MAX_NUM_SEQS=8` with a FULL\ndecode graph at all eight verify widths (4 through 32) is what makes the\n8-stream column reachable at all. BF16 recurrent state is worth **+8.5% at 8\nstreams** on its own, in a matched pair on the same launch config: 151.6 →\n164.5 tok\u002Fs, step 141.4 → 130.4 ms, with per-position draft acceptance\nunchanged within rounding (0.80\u002F0.59\u002F0.41 against 0.79\u002F0.58\u002F0.42) and needle\nretrieval unchanged at 15\u002F15.\n\n**Speculative depth was swept properly here for the first time** (K=0\u002F1\u002F2\u002F3 at\nevery stream count, FULL graphs throughout). K=3 wins at every concurrency,\nK=2 ties it, K=1 loses 8–14% and K=0 loses 32–46%. There is no crossover, so\n`MTP_K_SCHEDULE` has nothing to schedule. The full table is in the CHANGELOG.\n\n**Decode under a concurrent prefill** is the one place the shipped chunk width\nhurts. With two streams decoding and one 64k prompt arriving, the gap between\ntheir streamed chunks for the 34.5 s of that prefill is p50 1,057 ms \u002F\np95 1,111 ms \u002F p99 1,400 ms, against 78 ms on the quiet server. **Those are per\nengine step, not per token**: vLLM emits one streamed chunk per step carrying\nthat step's accepted tokens (~2.7 here, measured), so per-token latency is\nroughly a third of the figures above. The two streams together delivered 74\nsteps' worth of output inside the window — ~5.8 tok\u002Fs if acceptance holds at\nits quiet-server value. That is one engine step per 2,048-token chunk, by\nconstruction. Halving the chunk to 1,024 takes p95 to 666 ms and p99 to 674 ms\nand raises in-window delivery to 139 steps (~9.3 tok\u002Fs), for 5.5% of prefill\nthroughput at 64k. It is\nnot the shipped default (see the CHANGELOG for why the bar was not met), but if\nyour traffic is long prompts arriving against live streams, measure it.\n\n**Prefill on this configuration**, three sparkDash ladders on 2026-09-06 (one\nat 256k), server warm:\n\n| context | TTFT | prefill | 2026-09-05 | change |\n|---|---|---|---|---|\n| 8k | 3.74 s | **2,200 tok\u002Fs** (2,155–2,238) | 1,764 | +24.7% |\n| 16k | 7.13 s | **2,304 tok\u002Fs** (2,293–2,311) | 2,265 | +1.7% |\n| 32k | 14.18 s | **2,314 tok\u002Fs** (2,305–2,324) | 2,265 | +2.2% |\n| 64k | 29.05 s | **2,257 tok\u002Fs** (2,257–2,258) | 2,222 | +1.6% |\n| 128k | 61.09 s | **2,146 tok\u002Fs** (2,144–2,148) | 2,110 | +1.7% |\n| 256k | 134.89 s | **1,944 tok\u002Fs** | 1,913 | +1.6% |\n\n**Prefill did not change; only the 8k row looks like it did.** Fit\n`TTFT = tokens \u002F rate + overhead` across 16k–128k and the per-token rate is\n2,125 tok\u002Fs here against 2,089 on 2026-09-05 — **+1.7%**, with the same\n−0.60 s intercept in both. Every row above 8k agrees with that. The 8k point\nsits above the fit in *both* ladders, by 1.34 s in the 2026-09-05 run and\n0.47 s here, because 8k runs first and pays the PLE page-cache warm-up; a\nserver that has been up for hours has mostly already paid it. Read the 8k row\nas a statement about cache state, not about kernels.\n\nBF16 recurrent state is the only serving change since 2026-09-05, and prefill\nis within ~2% either way, so its effect here is not separable from run-to-run\nvariation — unlike decode, where it is worth +8.5% at 8 streams. Note the\n64k and 128k rows reproduce to ±0.05% across the three ladders, so that ±2% is\na between-launch figure, not measurement noise.\n\n#### 2026-09-05: after the decode optimisation pass\n\nMeasured on the shipped profile (512k YaRN, 2,048 chunks, `MAX_NUM_SEQS=4`,\nMTP 3, FP8 KV, `KV_TARGET_GIB=20` → 16.18 GiB = 974,768 tokens) with\n`CUDAGRAPH_CAPTURE_SIZES=auto`, the PLE gather prefetch, and reduced-vocabulary\ndrafting (`MTP_DRAFT_VOCAB`, 65,536 tokens) all active. The \"before\" columns are\nthe same rope config and chunk width, so decode is a matched pair; prefill\ndiffers only in `KV_TARGET_GIB` (22 → 20), which does not affect prefill rate.\n\n**Decode on prose**, by concurrent stream count:\n\n| streams | TTFT | aggregate | per stream | before | change |\n|---|---|---|---|---|---|\n| 1 | 270 ms | **46.3 tok\u002Fs** | 46.3 tok\u002Fs | 36.9 | **+25.5%** |\n| 2 | 466 ms | **73.0 tok\u002Fs** | 36.5 tok\u002Fs | 57.4 | +27.2% |\n| 3 | 355 ms | **91.9 tok\u002Fs** | 31.2 tok\u002Fs | — | — |\n| 4 | 346 ms | **108.1 tok\u002Fs** | 27.7 tok\u002Fs | 85.9 | +25.8% |\n\nAlmost all of that is the reduced draft vocabulary. The MTP drafter reads its\nown 1.18 GiB BF16 `lm_head` once per draft step, three of the four `lm_head`\nreads in an MTP-3 engine step; slicing it to 65,536 rows saves 2.61 GiB per\nstep, and decode here is close enough to the memory-bandwidth wall that bytes\nremoved convert almost one-for-one into time. Accuracy is unchanged — the\ntarget model verifies every drafted token — measured at 250 MGSM problems per\nlanguage, English 94.8% vs 93.6% and Chinese 86.4% vs 86.4%. See the CHANGELOG\nentry for the full method.\n\n**Prefill**, same chunk width as the shipped column below:\n\n| context | TTFT | prefill | before | change |\n|---|---|---|---|---|\n| 8k | 4.67 s | **1,764 tok\u002Fs** | 1,646 | +7.2% |\n| 16k | 7.25 s | **2,265 tok\u002Fs** | 2,052 | +10.4% |\n| 32k | 14.49 s | **2,265 tok\u002Fs** | 2,073 | +9.3% |\n| 64k | 29.52 s | **2,222 tok\u002Fs** | 2,037 | +9.1% |\n| 128k | 62.15 s | **2,110 tok\u002Fs** | 1,945 | +8.5% |\n| 256k | 137.03 s | **1,913 tok\u002Fs** | 1,791 | +6.8% |\n\nThe prefill gain is most likely the PLE page-fault prefetch rather than the\ndraft vocabulary, which does not touch prefill: the PLE row gather runs for\nevery prefilled token, so a 2,048-token chunk gathers 16 rows per token —\n~32,768 of them — against the ~256 a 4-stream MTP-3 decode step gathers. That\ngather is single-threaded and was taking each\nmissing 4 KiB page fault on its own; batching the reads with\n`posix_fadvise(WILLNEED)` measured 13x on a cold 280-row gather in isolation and\nonly ~3% on decode, where there are too few faults per step for it to matter.\nThis was not isolated with an A\u002FB, so read the attribution as inference from\nthe mechanism, not as a measurement.\n\nThese numbers came from one run each. The decode figures are content-dependent\nfor the reason given at the end of this section, and the `x3` TTFT below `x2`\nis run-to-run noise.\n\n**Prefill.** At the shipped 2,048-token chunk width throughput peaks around\n32-64k and falls away with context. Raising `MAX_NUM_BATCHED_TOKENS` to 8,192\nflattens it from 16k out to 128k, because per-chunk overhead is amortised over\n4x fewer chunks:\n\n| context | shipped: 2,048 chunks (512k YaRN, `KV_TARGET_GIB=22`) | opt-in: 8,192 chunks (262k native, `KV_TARGET_GIB=20`) |\n|---|---|---|\n| 8k | 5.00 s · 1,646 tok\u002Fs | **3.69 s · 2,228 tok\u002Fs** |\n| 16k | 8.00 s · 2,052 tok\u002Fs | **7.16 s · 2,293 tok\u002Fs** |\n| 32k | 15.83 s · 2,073 tok\u002Fs | **13.87 s · 2,366 tok\u002Fs** |\n| 64k | 32.20 s · 2,037 tok\u002Fs | **28.31 s · 2,316 tok\u002Fs** |\n| 128k | 67.41 s · 1,945 tok\u002Fs | **58.88 s · 2,227 tok\u002Fs** |\n| 256k | 146.40 s · 1,791 tok\u002Fs | not re-measured |\n\nThe one matched pair — same server, same kernel, only the chunk width changed —\nis 32k: **2,133 → 2,366 tok\u002Fs (+10.9%), TTFT 15.38 → 13.87 s (−9.8%)**. The\nrest of the right-hand column is one run each and should be read as indicative.\n\n### Raising the prefill chunk width (opt-in)\n\n**The 8,192 column is not the default.** If you want the faster prefill and\nTTFT above, set it yourself:\n\n```\nMAX_NUM_BATCHED_TOKENS=8192 .\u002Fstart.sh     # one launch\n```\n\nor edit the line in `.env` to make it stick.\n\nIt is paid for out of the KV pool, not the GPU budget. 8,192 chunks raise peak\nactivation to 1.27 GiB, and vLLM profiles that *before* it sizes the KV cache,\nso the pool absorbs it: 1,145,289 tokens measured at `KV_TARGET_GIB=20`,\nstill 4.37x a full 262k request. The best matched evidence for the size of that\ntrade is PR #2's own pair at `KV_TARGET_GIB=22`, 1,282,724 → 1,249,637 tokens —\nabout **−2.6%** of pool for **+11%** prefill.\n\nHost `MemAvailable` sat at 8.1-8.3 GiB idle and low-watered at 7.4 GiB across\nthe 8,192 sweep. It is offered as a knob rather than a default because the\nsupporting observation is minutes, not hours: if you serve long sessions near\nthe memory floor, measure it on your own workload before committing to it.\n\n**Decode on prose** (2,048 chunks, 512k YaRN), by concurrent stream count.\nThese are the pre-2026-09-05 figures, kept because the tables above are stated\nas deltas against them:\n\n| streams | TTFT | aggregate | per stream |\n|---|---|---|---|\n| 1 | 418 ms | 36.9 tok\u002Fs | 36.9 tok\u002Fs |\n| 2 | 445 ms | 57.4 tok\u002Fs | 29.7 tok\u002Fs |\n| 4 | 550 ms | 85.9 tok\u002Fs | 23.4 tok\u002Fs |\n\nDecode speed on this model is **strongly content-dependent**, because MTP\nspeculative decoding accepts more drafts on predictable text. Measured\n2026-09-06 on sparkDash prose: **2.80 tokens of a possible 4 per engine step**,\nper-position acceptance **0.80 \u002F 0.59 \u002F 0.41** — 60% of drafted tokens\naccepted. Quoting text back out of the context goes higher still; dense\ntechnical prose sits lower. Treat single-stream decode as a range rather than\none number.\n\nThe figures quoted here until 2026-09-06 — acceptance length 2.1 of 4, per\nposition 0.65 \u002F 0.33 \u002F 0.14, and \"~41 tok\u002Fs\" as the *best* case — were taken\nunder PIECEWISE CUDA graphs with the full 248,320-token draft vocabulary. They\nare superseded in both directions: acceptance is much higher, and ordinary\nprose now measures 48.7 tok\u002Fs single-stream.\n\n\n## Multimodal (images and video)\n\nThe checkpoint is multimodal (`is_multimodal: true`, `language_model_only:\nfalse`, a 27-layer vision tower) and the launcher enables it by default —\nnothing extra to configure. The vision tower is already counted in the\n\"weights on GPU\" figure, so images and video cost no additional GPU budget.\n\nVerified on this host 2026-09-04 against the running server:\n\n| Modality | Test | Result |\n|---|---|---|\n| Image | 336x336 PNG, three colour bands | named all three in order; 179 prompt tokens |\n| Video | 4 s clip, 16 frames, one colour per second | named all four **in temporal order**; 376 prompt tokens |\n\nUse the standard OpenAI content-part shapes — `image_url` and `video_url`,\neither an `http(s):\u002F\u002F` URL or a `data:` URI:\n\n```\ncurl -s localhost:8888\u002Fv1\u002Fchat\u002Fcompletions -H 'Content-Type: application\u002Fjson' -d '{\n \"model\":\"qwen3.8-flash-next\",\"max_tokens\":600,\"temperature\":0,\n \"messages\":[{\"role\":\"user\",\"content\":[\n   {\"type\":\"image_url\",\"image_url\":{\"url\":\"https:\u002F\u002Fexample.com\u002Fphoto.jpg\"}},\n   {\"type\":\"text\",\"text\":\"Describe this image.\"}]}]}'\n```\n\nThree things to know before leaning on it:\n\n- **MTP speculative decoding degrades on multimodal requests.** The draft model\n  cannot take multimodal embeddings, so vLLM logs `using text-only draft inputs\n  instead` and falls back for those requests. The answer is still correct — the\n  target model sees the image — but decode runs closer to the non-speculative\n  speed. Text-only requests are unaffected.\n- **Video is token-hungry.** Frame count and resolution drive prompt length\n  fast. At `YARN=1` with the shipped FP8 KV you have **2.16x** a full-length\n  request in KV (measured 2026-09-06 on the running server), so two concurrent\n  long video requests already contend; the 262k profile has far more headroom.\n  The 1.34x quoted here before was a BF16-KV measurement.\n- **Long video at 512k is untested here.** The tests above were long-text *or*\n  short-multimodal, never both at once.\n\n## Configuration\n\nPrecedence is **environment > `.env` > built-in default in `start.sh`**, so any\nknob can be overridden per launch:\n\n```\nMAX_MODEL_LEN=65536 MTP_NUM_SPECULATIVE_TOKENS=0 .\u002Fstart.sh\n```\n\nThe safety-relevant knob is `HOST_RESERVE_GIB` (default 26): the GPU budget\nis capped at `MemTotal − HOST_RESERVE_GIB` no matter what `KV_TARGET_GIB`\nasks for, and `start.sh` prints \"KV target X reduced to Y\" when the cap binds.\n`KV_TARGET_GIB` is a wish under that cap: the shipped 20 is clipped to\n16.67 GiB here, ~1.13M FP8 tokens.\n`HOST_SLACK_GIB` sizes the container cgroup cap (GPU budget + this); it bounds\nhost-side memory only and does not protect the host from the GPU side.\n\n### Long context beyond 262k (YaRN)\n\nThe model's native context is 262,144. Going past it needs YaRN rope scaling,\nwhich is off by default. The two lengths live side by side in `.env` and the\n`YARN` flag alone picks which one is served:\n\n```\nYARN=0                     # 0 = native rope, 1 = YaRN\nMAX_MODEL_LEN=262144       # served at YARN=0; cannot exceed native 262144\nYARN_MAX_MODEL_LEN=524288  # served at YARN=1; ignored entirely at YARN=0\n```\n\nSo `YARN=1` is the only edit needed to go to 512k, and flipping it back to `0`\nreturns to 262k without touching anything else. For a single launch:\n`YARN=1 .\u002Fstart.sh`.\n\n`start.sh` derives the scaling factor itself (`YARN_MAX_MODEL_LEN \u002F 262144`,\nrounded up — 2.0 for 512k) and passes it to vLLM as a `--hf-overrides`\ndeep-merge into `text_config.rope_parameters`, which is the field this model\nactually reads. The existing `mrope_section`, `rope_theta` and\n`partial_rotary_factor` are preserved, so the attention path keeps the same\n`MRotaryEmbedding` and mrope stays enabled.\n\n512k fits with **no other change**: it needs 14.4 GiB of KV, well inside what\n`KV_TARGET_GIB` provides at the shipped 20 or at 22, and the GPU budget and\ncgroup cap are unchanged from 262k. Measured at `YARN=1`, BF16,\n`KV_TARGET_GIB=20` (2026-09-04):\n\n| | |\n|---|---|\n| Available KV cache | 19.2 GiB = **704,558 tokens** (1.34x a full 524,288 request) |\n| Host MemAvailable idle | ~11.3 GiB |\n| Output | coherent; MTP 3 and YaRN run together without incident |\n\nAt `KV_TARGET_GIB=22` with FP8 the same context gets 2.73x headroom instead of\n1.34x — see [FP8 KV cache](#fp8-kv-cache-default). That 22 predates the\nhost-side cap; the shipped profile measures **2.16x** at 524k today.\n\n400k prefill stress test (salted to defeat prefix caching, needles planted at\n5% \u002F 50% \u002F 95% depth):\n\n| | |\n|---|---|\n| Prompt | 400,062 tokens |\n| TTFT (prefill) | 260.3 s = **1,537 tok\u002Fs** |\n| Needle retrieval | **3\u002F3 PASS**, including 95% depth |\n| Host MemAvailable low-water | **10.97 GiB** (watchdog floor is 6 GiB) |\n| Peak container RSS | 18.7 GiB of the 103 GiB cap |\n\nDecode measured 40 tok\u002Fs on that run, but the answer is three codes copied out\nof the context — MTP's best case, not typical decode speed.\n\n| Setting | Result |\n|---|---|\n| `YARN=1` | serves `YARN_MAX_MODEL_LEN`; `MAX_MODEL_LEN` is ignored (logged) |\n| `YARN=0` with `MAX_MODEL_LEN` > 262144 | refused: tells you to set `YARN=1` |\n| `YARN_MAX_MODEL_LEN` > `YARN_CEILING_MODEL_LEN` (524288) | refused: above the validated ceiling |\n| `YARN=1` with `YARN_MAX_MODEL_LEN` at or below 262144 | warns, serves that length with native rope |\n| 1M even with the ceiling raised | refused by the Step 2 budget check (cap 112 GiB vs 105 GiB ceiling) |\n\nYaRN trades some short-context accuracy for the longer window, so leave it off\nunless you need more than 262k. The 512k path is what every sparkDash sweep\nabove was measured on — decode, prefill and the 45-minute soak all ran at\n`YARN=1`. It is the shipped **262k native-rope** profile that has not been\nbenchmarked end to end.\n\n### Reasoning is on by default\n\nThis build reasons before answering, and `start.sh` passes\n`--reasoning-parser qwen3`, so the thinking block arrives in a separate\n`reasoning` field rather than inside `content`. The chat template enables it\nwhenever the flag is unset:\n\n```jinja\n{%- if enable_thinking is undefined or enable_thinking is true %}\n```\n\nTurn it off **per request** — no restart, so reasoning and non-reasoning\ntraffic can share one server:\n\n```json\n{\"model\":\"qwen3.8-flash-next\",\n \"chat_template_kwargs\":{\"enable_thinking\":false},\n \"messages\":[{\"role\":\"user\",\"content\":\"What is 17*23? One line.\"}]}\n```\n\nMeasured on this host:\n\n| | default | `enable_thinking: false` |\n|---|---|---|\n| reasoning tokens | 41 | **0** |\n| completion tokens | 47 | **12** |\n| `content` | `\"\\n\\n391\"` | `\"17 * 23 = 391\"` |\n\nTwo consequences worth knowing:\n\n- **It is why `content` can come back empty.** With a small `max_tokens` the\n  reply is often still inside its reasoning. Budget ~400+ tokens, or disable\n  thinking. This is not a bug — see the sanity test above.\n- **It dominates latency on simple work.** Reasoning ran to 4,841 tokens on the\n  hardest task in our suite. For extraction, classification or short factual\n  answers, disabling it is a large win; leave it on for anything that needs\n  actual multi-step reasoning.\n\n### FP8 KV cache (default)\n\n`KV_CACHE_DTYPE=fp8` roughly doubles the KV pool by storing the main KV in\nfp8-e4m3. The QSA Triton kernels cast FP8 tiles to BF16 for the tensor-core\ndots and apply the per-tensor K\u002FV scales once, to the score and the output\naccumulator. **This is the shipped default**, on the strength of the\nmeasurements below.\n\n```\nKV_CACHE_DTYPE=fp8    # ~2x KV pool, enables a 1M context (default)\nKV_CACHE_DTYPE=auto   # BF16 KV, if you would rather not take the trade\n```\n\nMeasured on this host 2026-09-04, identical prompts, `YARN=1`,\n`KV_TARGET_GIB=22`, idle server:\n\nAll rows below are at matched settings (`KV_TARGET_GIB=22`, 512k YaRN) unless\nnoted. KV pool varies a little between restarts, so a range is given.\n\n| | BF16 | FP8 | Δ |\n|---|---|---|---|\n| KV pool | 779,671–796,196 tok | **1,431,164–1,502,014 tok** | **~1.8–1.9x** |\n| Concurrency @ 524,288 | 1.49–1.52x | **2.73–2.86x** | ~+85% |\n| Prefill @400k | 1,537 tok\u002Fs | 1,495 tok\u002Fs | −2.7% |\n| Prefill @32k (2 runs each) | 1,883 tok\u002Fs | 1,769 tok\u002Fs | −6.1% |\n| Reasoning suite (11 tasks) | **11\u002F11** | **11\u002F11** | same |\n| Needle miss rate @32k | 2\u002F14 (14%) | 5\u002F20 (25%) | p=0.67, **n.s.** |\n\nOnly the 12 full-attention layers shrink (~84% of bytes\u002Ftoken); the QSA\nside\u002Fcompressor caches stay BF16, which is why the gain is ~1.85x rather than\n2x, and why `KV_MULT` in `start.sh` is 0.58 rather than 0.5.\n\n**The short-context penalty has since been removed.** The original patch\ndequantised each tile with vLLM's `_cast_kv_tile`, which materialises an FP32\ntile (`(data.to(tl.float32) * scale).to(Q.dtype)`), and halved `block_n` to\nkeep that inside GB10's shared-memory budget. That cost more on a short kernel\nthan a long one: −6.1% at 32k versus −2.7% at 400k, in the rows above.\n\nHoisting the per-tensor scales outside the dots removes the FP32 tile, so FP8\nruns at the same `block_n` as BF16. The scales are scalars, so this is exact\nbefore rounding: `(Q·K)·k_scale` for the score, and `v_scale` on the normalised\noutput, which factors cleanly through the split-K LSE merge. It is also\nslightly *more* accurate than dequantising first — FP8→BF16 is exact, whereas\nrounding `scale × fp8` into BF16's 8-bit mantissa is not.\n\nMeasured by [@lidaiqing](https:\u002F\u002Fgithub.com\u002Flidaiqing) on this host (#2), FP8\nbefore vs after the hoist, at matched settings:\n\n| | before | after | Δ |\n|---|---|---|---|\n| Prefill @32k | 1,827 tok\u002Fs | 1,942 tok\u002Fs | **+6.3%** |\n| Decode, 1 stream | 24.06 tok\u002Fs | 23.36 tok\u002Fs | −2.9% |\n| Decode, 4 streams | 56.44 tok\u002Fs | 60.91 tok\u002Fs | +7.9% |\n| Decode, 8 streams | 60.16 tok\u002Fs | 60.43 tok\u002Fs | +0.4% |\n| Sparse QSA kernel, 512 rows | 2.984 ms | 1.772 ms | **−40.6%** |\n| Block selector kernel | 0.1392 ms | 0.1008 ms | −27.6% |\n\nThe kernel is 40% faster in isolation but attention is not the bottleneck at\nthese settings, so end-to-end decode barely moves; the win that survives is\nshort-context prefill. Single-stream decode is within this model's\ncontent-dependent MTP variance. On identical tensors the maximum\nsparse-attention error was 1.53e-5 (one BF16 ULP) and the BF16 path was\nbit-identical, as the algebra predicts.\n\n**On quality.** Both dtypes score 11\u002F11 on the reasoning suite\n(4 multi-step short tasks, 4 long chain-of-thought up to ~4,800 reasoning\ntokens, 3 tasks combining three facts from a ~100k-token context). Both max it\nout, so the honest reading is *no gross regression at n=11* — enough to rule\nout the 6\u002F6 → 2\u002F6 collapse the reference measured, not enough to detect finer\ndrift. A suite everything passes cannot rank anything.\n\n**A caution about needle tests on this model.** The 95%-depth needle at 32k is\nflaky *regardless of KV dtype*: BF16 missed it 2\u002F14 times, FP8 5\u002F20, which\nFisher's exact test cannot distinguish (p=0.67). The two shallower needles were\nfound 34\u002F34 times in both. So a single needle run is weak evidence here — an\nisolated PASS or FAIL at 95% depth says little, and comparisons need matched\nsample counts on both sides. The 3\u002F3 results quoted elsewhere in this README\nare single samples and should be read with that in mind.\n\n**A caveat before trusting these numbers: quality is not settled.** Needle retrieval passing at 5\u002F50\u002F95% depth shows the\nscales and dequantisation are broadly right, and short factual\u002Farithmetic\nanswers were correct. It does **not** clear the failure mode that matters: the\nreference measured a long-reasoning benchmark falling from **6\u002F6 to 2\u002F6** with\nFP8 KV. This is sparse attention — quantised keys perturb which blocks the\nindexer selects, not merely the attention output — so degradation can appear\nas fluent, plausible, wrong reasoning while needles still pass. No\nlong-reasoning A\u002FB has been run on this host, and the scale hoist has not\nchanged that — its one-BF16-ULP bound is a numerical result, not a quality\none. Treat FP8 as a capacity trade for workloads you have validated\nyourself.\n\n### BF16 recurrent state (default)\n\n`MAMBA_SSM_CACHE_DTYPE=bfloat16` overrides the checkpoint's\n`mamba_ssm_dtype = float32` for the GDN recurrent state. The fused kernel\naccepts it (`FUSED_GDN_STATE_DTYPES = (float32, bfloat16)`), and vLLM says so\nat startup:\n\n```\nconfig.py:799 WARNING  Qwen3.5 model specifies mamba_ssm_dtype='float32' in its config,\n              but --mamba-ssm-cache-dtype='bfloat16' was passed. Using the user-specified value.\ninterface.py:915  Setting attention block size to 1664 tokens   (3200 at float32)\n```\n\nThe state is pure per-step traffic — roughly 0.23 GB per sequence read and\nwritten every engine step — so halving it converts almost directly into step\ntime on a machine this close to the bandwidth wall. Halving the mamba page also\nlets vLLM pick a 1,664-token attention block instead of 3,200, which doubles\nprefix-cache granularity for multi-turn traffic.\n\nMeasured 2026-09-06 (512k YaRN, MTP 3, FP8 KV, `MAX_NUM_SEQS=8`, prose, three\nrepeats):\n\n| | float32 (checkpoint) | bfloat16 | Δ |\n|---|---|---|---|\n| decode @ 1 stream | 44.6 tok\u002Fs | 47.6 tok\u002Fs | +6.8% |\n| decode @ 8 streams | 151.6 tok\u002Fs | **164.5 tok\u002Fs** | **+8.5%** |\n| engine step @ 8 streams | 141.4 ms | 130.4 ms | −7.8% |\n| tokens per step @ 8 streams | 2.80 | 2.80 | unchanged |\n| attention block | 3,200 tok | 1,664 tok | halved |\n| needles @32k, 5 runs | 15\u002F15 | **15\u002F15** | unchanged |\n\nOnly the 8-stream row clears the ±5%-in-all-three-repeats bar this host uses;\nthe others are positive but inside the noise floor.\n\n**This is a precision change, so read the quality evidence before trusting it.**\nNeedle retrieval at 32k is identical to float32 across five runs at 5%, 50% and\n95% depth, and a 4-turn continuation — the case that would expose a recurrence\ndegrading as it is carried forward — produces a final summary that recalls every\nelement of the conversation in both dtypes. That is the same bar the FP8 KV\ndefault was held to, and it is one night's evidence rather than a graded task\neval. Set `MAMBA_SSM_CACHE_DTYPE=` empty to go back to the checkpoint's\nfloat32.\n\n### PLE mmap access pattern\n\nThe packed PLE table is advised `MADV_RANDOM` (in `patch_ple_offload.py`).\nWithout it the kernel faults in a ~64 KiB window to serve each 90-byte row\nlookup. Measured on this host:\n\n| | default mmap | `MADV_RANDOM` |\n|---|---|---|\n| Disk read per decoded token | ~1,366 KiB | **57 KiB** (−24x) |\n| Host MemAvailable | ~10.9 GiB | **~12.95 GiB** |\n\nDecode speed did not change measurably — decode was never disk-*throughput*\nbound (1.4 MiB\u002Ftoken at ~26 tok\u002Fs, the rate at the time, is only ~36 MB\u002Fs). The real win is the\n~2 GiB of unified memory no longer wasted on readahead that is thrown away,\nwhich is what funds the KV pool `KV_TARGET_GIB` asks for.\n\n## Safety rules\n\nEach of these cost a hard host hang or a dead server during bring-up.\n\n- **Budget the GPU from the host side.** vLLM detects this GPU as integrated\n  and treats host `MemAvailable` — page cache included — as free GPU memory,\n  then fills the GPU side to exactly `GMU × MemTotal`. Nothing in vLLM keeps\n  anything back for the host. `start.sh` therefore caps the budget at\n  `MemTotal − HOST_RESERVE_GIB` (26 GiB by default) and derives the KV pool\n  from the remainder. What the reserve has to hold, measured here: other\n  containers and sessions ~7 GiB (`start.sh` prints the live figure as \"host\n  footprint now\" and warns above 9), vLLM's own host-side processes ~6, the PLE\n  page cache that keeps decode off NVMe ≥6, free pages the NVIDIA driver needs\n  to allocate at all ≥3, and 2–3 GiB of per-request growth (below). The page\n  cache is not spare memory.\n- **Keep host `MemAvailable` at or above ~10 GiB under load.** Exhausting the\n  unified pool hangs the kernel with no OOM kill and no logs; the driver starts\n  refusing allocations (`NV_ERR_NO_MEMORY` in `journalctl -k`, which works\n  without sudo) well before that, at `MemFree` ~3 GiB.\n- **`comfy-h3.service` must stay disabled.** It polls `127.0.0.1:8888` and\n  launches ComfyUI (a GPU co-tenant) as soon as anything answers there.\n  `start.sh` refuses port 8888 while that service is active.\n- **Never set `PLE_OFFLOAD=false` at TP=1** — 99 GB through UVM hangs the host.\n- **The stock QSA backend refuses FP8 KV** (`supported_kv_cache_dtypes =\n  [\"auto\",\"bfloat16\"]`). `patch_qsa_fp8_kv.py` in this repo adds it; without\n  that patch `KV_CACHE_DTYPE=fp8` cannot work, and reading a quantised cache\n  as BF16 would produce silent garbage rather than an error.\n- **Do not raise `YARN_CEILING_MODEL_LEN` past 524288 at BF16.** A 1M context\n  needs ~28.8 GiB of KV, driving the container cap to 112 GiB against a 105 GiB\n  hard ceiling; `start.sh` refuses it at two independent checks. With\n  `KV_CACHE_DTYPE=fp8` a 1M request needs only ~16.7 GiB and the budget does\n  fit — but 1M has **never been run on this host**, at either dtype. Raising\n  the ceiling means you are the one testing it.\n- `docker --memory` does not bound GPU allocations on GB10, only host-side\n  memory. vLLM's `--gpu-memory-utilization` is what bounds the GPU.\n- **Kernel VM tunables.** The box ships with `vm.min_free_kbytes=45155` and\n  `vm.watermark_scale_factor=10`: a 44 MB free-page floor and reclaim that\n  starts at 0.1 %. `files\u002Fsysctl-spark3.conf` holds the values a sibling Spark\n  measured six crash-free bring-ups with; `start.sh` warns when the box is at\n  the defaults. They are **not applied** by anything in this repo, and the\n  file's header explains why the watchdog floor must be re-derived before\n  they are: at those values the same physical state reads roughly 11–15 GiB\n  lower in `MemAvailable` (computed from the kernel's watermark formula, not\n  measured here).\n\n### What happened on 2026-09-04\n\nThree servers died in one evening at `KV_TARGET_GIB=22`, all under a qwen-code\nagent harness (up to five agents, 370 requests averaging 72k input tokens over\nfive hours, pointed at `127.0.0.1:8888`). The budget arithmetic left 20.7 GiB\nof the 121.6 GiB pool for everything that is not the GPU, against the ≥22 GiB\nlisted above. `sar` shows the first server spending its last hour at 6.3–6.6\nGiB of `MemAvailable`; the kernel log shows the driver refusing four\nallocations in the eight seconds before the second death; the watchdog's own\nlog shows the third at `MemFree` 2.6 GiB. The earlier reading of the first two\ndeaths as watchdog noise was wrong: the debounce added that day is a good\nchange and does not touch the cause.\n\nThe growth is real and permanent. Each new largest request (70–95k tokens)\ngrows driver-side memory by ~2 GiB — workspaces the startup profile never\ntouched, held by PyTorch's caching allocator, which this build only releases\nunder pressure inside the model loader, never while serving. In the watchdog\nlog it appears as the container cgroup going *down* (PLE page cache evicted)\nwhile `MemAvailable` goes down and `MemFree` stays flat; the new `driver`\ncolumn makes it visible directly. The reserve is sized to absorb it.\n\n### Watchdog\n\n`files\u002Fmemwatch.sh` runs alongside the container, polls `\u002Fproc\u002Fmeminfo` every\nsecond, and stops the container on either of two floors, each debounced over\n**5 consecutive** samples (a lone excursion logs `recovered after N sub-floor\nsample(s)` and resets the counter — `MemAvailable` moves ~107 MiB between\nsamples here, with excursions past 1 GiB):\n\n- `MemAvailable \u003C MEMWATCH_MIN_GIB` (default 6): the page cache is gone.\n- `MemFree \u003C MEMWATCH_MIN_FREE_GIB` (default 2) **while** `MemAvailable \u003C\n  MEMWATCH_FREE_GATE_GIB` (default 10): the driver's failure point. The gate\n  is not optional. With the stock watermarks `MemFree` legitimately sits near\n  zero whenever the page cache is full of reclaimable data — measured during\n  weight loading: `MemFree` 0.9 GiB, `MemAvailable` 32 GiB, zero driver\n  errors — and an ungated version of this trigger killed a healthy launch.\n\nEvery 10 s it counts `NV_ERR_NO_MEMORY` lines in `journalctl -k` and logs any\nnon-zero count. Read it together with `MemAvailable`: a handful during\nstartup, when the driver takes the weights and then the KV pool in two large\nbursts while `MemFree` is transiently ~1 GiB under the page cache from the\ncheckpoint read, is the driver bouncing off free pages and retrying (measured\n2026-09-05 08:15–08:16: five of them at `MemAvailable` 17–34 GiB, launch\nsucceeded; the launch seven hours earlier had none — it depends on where\nkswapd is when the burst lands). The fatal pattern is the same line with\n`MemAvailable` under ~10 GiB, when there is no cache left to reclaim. The\ntimeline\n(every 5 s, every sample once within 1 GiB of a floor) carries `avail`,\n`free`, `swapfree`, the container cgroup, `cached`, `anon`, `shmem`, `mapped`,\n`sunreclaim` and the derived `driver` figure (`MemTotal − MemFree − Buffers −\nCached − AnonPages − Slab − PageTables − KernelStack`: memory outside page\ncache, anon and cgroup accounting, i.e. taken through the NVIDIA driver;\n95.5 GiB at idle here against a 94.87 GiB budget).\n\nBefore stopping it archives `docker logs --tail 3000` and a copy of its own\nlog to `logs\u002Farchive\u002F\u003Ccontainer>-\u003Ctimestamp>-{container,memwatch}.log`, then\nsends SIGTERM with a 30 s grace period (`MEMWATCH_GRACE`) and falls back to\nSIGKILL, so vLLM can unlink its POSIX shared memory — a hard kill leaks those\nsegments onto the host's `\u002Fdev\u002Fshm` until reboot, because the container runs\nwith `--ipc host`. vLLM does not honour SIGTERM while still loading weights;\na stop in that phase ends in the SIGKILL. `start.sh` archives the previous\ncontainer and watchdog logs the same way before it relaunches.\n\n## Sanity test\n\n```\ncurl -s localhost:8888\u002Fv1\u002Fchat\u002Fcompletions -H 'Content-Type: application\u002Fjson' -d '{\n \"model\":\"qwen3.8-flash-next\",\"temperature\":0,\"max_tokens\":400,\n \"messages\":[{\"role\":\"user\",\"content\":\"In one sentence, what is a DGX Spark?\"}]}' \\\n | python3 -c \"\nimport json,sys\nm=json.load(sys.stdin)['choices'][0]['message']\nprint('reasoning:', (m.get('reasoning') or '')[:200])\nprint('content  :', m.get('content'))\"\n```\n\nThis build emits reasoning **before** the answer, in a `reasoning` field rather\nthan `content`. Budget at least ~400 `max_tokens`: at 200 the reply is still\ninside its reasoning, so `content` comes back empty on a perfectly healthy\nserver. Gibberish in either field means the PLE path has regressed (bf16 IPC\nbuffer or missing quant scales) — see the patch notes below.\n\n## Layout\n\n- `download.sh` — fetches the checkpoint into the Hugging Face cache\n  (resumable; honours `HF_TOKEN` for gated repos). Uses the host's\n  `huggingface_hub` if present, otherwise the container image.\n- `start.sh` — launcher: derives the GPU budget from live memory under the\n  `HOST_RESERVE_GIB` cap, builds the packed PLE table on first run,\n  regenerates the patched vLLM files, archives the previous run's logs, starts\n  the container and `files\u002Fmemwatch.sh`.\n- `stop.sh` — stops the watchdog, then the container (gracefully by default);\n  reports leftover `\u002Fdev\u002Fshm` segments without deleting them.\n- `files\u002Fpatch_ple_layer.py`, `files\u002Fpatch_modelopt_mxfp8.py`,\n  `files\u002Fpatch_ple_offload.py` — generators that rewrite the patched vLLM\n  files from pristine `*.orig` \u002F `orig\u002F` copies on **every** launch. Those\n  copies are not in the repo — `start.sh` extracts them from the image on\n  first run. Edit the generators; edits to the generated files are overwritten.\n- `files\u002Fbuild_ple_packed_table.py` — one-time packed PLE table builder\n  (27 GB output under `~\u002F.cache\u002Fvllm\u002Fple_cache\u002F`, memory-mapped at runtime).\n- `files\u002Fsysctl-spark3.conf` — recommended kernel VM tunables, not applied by\n  anything here; read its header first.\n\n- `bench\u002Fsweep.py` — decode sweep. Submits one\n  [sparkDash](https:\u002F\u002Fgithub.com\u002FMiaAI-Lab\u002FsparkDash) job per concurrency level\n  and snapshots `\u002Fmetrics` around each, so every level also yields ms per\n  engine step, tokens per step and per-position draft acceptance, plus host\n  memory minima and the `NV_ERR_NO_MEMORY` count for that level.\n- `bench\u002Fmixed.py` — decode under a concurrent prefill: two streams decoding\n  when a ~64k prompt arrives, reporting the p95\u002Fp99 gap between their streamed\n  chunks (one per engine step, ~2.7 tokens each) inside the prefill window. sparkDash has no mode for this shape.\n\nThe published prefill and decode numbers were measured with sparkDash, driven\nby those two scripts. Both need an idle server: the counter deltas and\nsparkDash's own figures include any other traffic on the port.\n\n## What is patched and why\n\n- **PLE layer** (`patch_ple_layer.py`): NVFP4\u002FFP8 dispatch for the PLE table;\n  offloaded rows carry codes *and* scales (90 B\u002Fhead); the GPU-side placeholder\n  learns its quant method from config because its constructor is skipped under\n  offload; tolerates multi-call `load_weights`; slices the 2560-wide IPC buffer\n  to the 1440 valid bytes.\n- **ModelOpt** (`patch_modelopt_mxfp8.py`): BF16 fallback for MXFP8 shapes that\n  FlashInfer rejects.\n- **PLE offload** (`patch_ple_offload.py`): GB10 has no CUDA stream memory ops\n  (`CAN_USE_STREAM_MEM_OPS=0`, measured), and vLLM's offload semaphore used them\n  and deadlocked after graph capture. Replaced with a host-side handshake — the\n  GPU worker posts a request, the CPU worker copies and writes a sequence number\n  to shared memory, the GPU worker proceeds. It also attaches the memory-mapped\n  packed table instead of loading 27 GB into RAM. The mmap is advised\n  `MADV_RANDOM`: without it the kernel faults in a ~64 KiB window to serve each\n  90-byte row lookup, and measurements here showed **24x** more disk read per\n  decoded token (1,366 -> 57 KiB\u002Ftoken) plus ~2 GiB of page cache wasted on\n  readahead that is never used.\n- **FP8 KV cache** (`patch_qsa_fp8_kv.py`, via `KV_CACHE_DTYPE=fp8`): casts\n  FP8 K\u002FV tiles to BF16 for the tensor-core dots and applies the per-tensor\n  scales once to the score and the normalised output, plumbs `k_scale`\u002F\n  `v_scale` into the kernels, and relaxes the four BF16-only guards and the\n  inherited FlashAttention rejection. Avoiding an FP32 dequantisation tile lets\n  FP8 keep the BF16 `block_n`. Raises the KV pool from ~800k to **1,132,586\n  tokens** on the shipped profile (measured 2026-09-06; 1.43-1.50M at the\n  pre-cap `KV_TARGET_GIB=22`), which is what makes a 1M context arithmetically\n  possible on one Spark. **On by default** and still a real\n  quality trade — see the warning `start.sh` prints.\n  The FP8-KV approach is credited to\n  [lancelind\u002Fqwen3.8-Flash-DGX](https:\u002F\u002Fgithub.com\u002Flancelind\u002Fqwen3.8-Flash-DGX)\n  (Apache-2.0), reimplemented here against this image's own sources. That\n  credit applies to this one patch; nothing else in this repository derives\n  from that project.\n\n## License\n\nCopyright (C) 2026 MiaAI Lab (https:\u002F\u002Fx.com\u002FMiaAI_lab)\n\nLicensed under the **GNU Affero General Public License v3.0 or later**\n(AGPL-3.0-or-later). See `LICENSE`. Every source file carries an\n`SPDX-License-Identifier: AGPL-3.0-or-later` header.\n\nThis program is free software: you can redistribute it and\u002For modify it under\nthe terms of the GNU Affero General Public License as published by the Free\nSoftware Foundation, either version 3 of the License, or (at your option) any\nlater version. It is distributed in the hope that it will be useful, but\nWITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or\nFITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License\nfor more details.\n\nBecause this is AGPL and this repository exists to run a **network server**:\nif you modify these scripts and offer the resulting service to users over a\nnetwork, section 13 requires you to offer those users the corresponding\nsource of your modified version.\n\n### What the license does and does not cover\n\nIt covers the files in this repository — the launcher, the patch generators,\nthe packed-table builder and the watchdog. It does **not** relicense anything\nthey operate on, each of which carries its own terms:\n\n- **vLLM** (Apache-2.0) — not redistributed here. `start.sh` extracts the\n  pristine `*.orig` sources from the container image at runtime, and the patch\n  generators emit modified copies onto your machine only. Those generated files\n  keep vLLM's own Apache-2.0 headers and remain Apache-2.0 works.\n- **The container image** `vllm\u002Fvllm-openai:qwen38-flash-next` and its\n  dependencies — upstream terms apply.\n- **The model checkpoint** `Mia-AiLab\u002FQwen3.8-Flash-Next-NVFP4` — weights are\n  governed by the checkpoint's own license, not by this repository's.\n","该项目提供在单台NVIDIA DGX Spark服务器上高效部署Qwen3.8-Flash-Next视觉语言模型（支持文本、图像与视频多模态输入）的端到端服务方案。基于vLLM推理框架，采用NVFP4量化权重、内存映射PLE表卸载及FP8 KV缓存等技术，在121 GiB统一内存约束下实现超长上下文（262k tokens）低延迟推理。适用于科研机构或企业需在单节点DGX硬件上快速验证多模态大模型能力、进行轻量级API服务或私有化部署的场景。",2,"2026-09-06 02:30:05","CREATED_QUERY"]