[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"project-94989":3},{"id":4,"name":5,"fullName":6,"owner":7,"repo":5,"description":8,"homepage":9,"htmlUrl":10,"language":11,"languages":10,"totalLinesOfCode":10,"stars":12,"forks":13,"watchers":14,"openIssues":15,"contributorsCount":16,"subscribersCount":16,"size":16,"stars1d":16,"stars7d":17,"stars30d":17,"stars90d":16,"forks30d":16,"starsTrendScore":18,"compositeScore":19,"rankGlobal":10,"rankLanguage":10,"license":20,"archived":21,"fork":21,"defaultBranch":22,"hasWiki":21,"hasPages":21,"topics":23,"createdAt":10,"pushedAt":10,"updatedAt":24,"readmeContent":25,"aiSummary":26,"trendingCount":16,"starSnapshotCount":16,"syncStatus":27,"lastSyncTime":28,"discoverSource":29},94989,"Qwen3.8-27B-SGLang-DGX-Spark","MiaAI-Lab\u002FQwen3.8-27B-SGLang-DGX-Spark","MiaAI-Lab","Qwen3.8 27B on SGLang for DGX Spark","https:\u002F\u002Fx.com\u002FMiaAI_lab",null,"Shell",226,19,101,1,0,109,22,73.9,"MIT License",false,"main",[],"2026-08-24 04:01:23","# Qwen3.8 27B on SGLang for DGX Spark\n\n[![SGLang](https:\u002F\u002Fimg.shields.io\u002Fbadge\u002FSGLang-cookbook-blue)](https:\u002F\u002Fdocs.sglang.io\u002Fcookbook\u002Fautoregressive\u002FQwen\u002FQwen3.8-27B)\n[![Model](https:\u002F\u002Fimg.shields.io\u002Fbadge\u002Fmodel-Qwen3.8--27B-informational)](https:\u002F\u002Fhuggingface.co\u002FRadixArk\u002FQwen3.8-27B-NVFP4)\n[![arch](https:\u002F\u002Fimg.shields.io\u002Fbadge\u002Farch-arm64%20%2F%20GB10-lightgrey)](#)\n\n\u003Cp align=\"center\">\n  \u003Csub>by \u003Ca href=\"https:\u002F\u002Fx.com\u002FMiaAI_lab\">Mia's AI Lab\u003C\u002Fa>\u003C\u002Fsub>\n  \u003Cbr>\u003Cbr>\n  \u003Ca href=\"https:\u002F\u002Fko-fi.com\u002FZ8Z3SPLOD\" target=\"_blank\" rel=\"noopener noreferrer\" style=\"display:inline-block;margin:0 8px;vertical-align:middle;\">\u003Cimg src=\"https:\u002F\u002Fstorage.ko-fi.com\u002Fcdn\u002Fkofi6.png?v=6\" alt=\"Buy Me a Coffee at ko-fi.com\" height=\"28\" style=\"height:28px;width:auto;vertical-align:middle;border:0;\" \u002F>\u003C\u002Fa>\n  \u003Ca href=\"https:\u002F\u002Fx.com\u002FMiaAI_lab\" target=\"_blank\" rel=\"noopener noreferrer\" style=\"display:inline-block;margin:0 8px;vertical-align:middle;\">\u003Cimg src=\"https:\u002F\u002Fimg.shields.io\u002Fbadge\u002FFollow%20me%20on%20X-000000?style=for-the-badge&logo=x&logoColor=white\" alt=\"Follow Mia on X\" height=\"28\" style=\"height:28px;width:auto;vertical-align:middle;border:0;\" \u002F>\u003C\u002Fa>\n\u003C\u002Fp>\n\nOpinionated, ready-to-run scripts to serve **[Qwen3.8-27B](https:\u002F\u002Fhuggingface.co\u002FRadixArk\u002FQwen3.8-27B-NVFP4)** with **[SGLang](https:\u002F\u002Fdocs.sglang.io)** in Docker on an NVIDIA DGX Spark (GB10, aarch64). Three swap-in serving modes — EAGLE\u002FMTP, DSpark, or DFlash2 — with every tuning choice measured on-device instead of guessed.\n\n**DSpark and DFlash2 are faster on code.** Versus MTP, DSpark gives the essay back; DFlash2 does not. Everyday chat on the same streamed probe comes out a DFlash2 win once tokens are counted right, and the long-essay probe is on the MTP side of the table. **All measured numbers, ranges, counting notes and caveats live in [Measured on this box](#measured-on-this-box) — one place, nothing repeated.**\n\nThe launch flags start from the **[SGLang cookbook's DGX Spark cell](https:\u002F\u002Fdocs.sglang.io\u002Fcookbook\u002Fautoregressive\u002FQwen\u002FQwen3.8-27B)** (NVFP4 + DSpark), then pin choices measured on this box: GDN **bf16** (cookbook float32 was −3%), `extra_buffer_lazy`, mem **0.90**, chunk **8192**, DSpark **block 7 \u002F 8 draft tokens**, torch.compile + decode graphs, X5 cpuset.\n\n- **NVFP4 W4A4** checkpoint (default; BF16 and FP8 available via `QUANT=…`)\n- **native 262K context, YaRN off, and 10 concurrent requests by default** — optionally extend to a validated 1M via YaRN (see “[Long context & concurrency](#long-context--concurrency-up-to-1m-10-concurrent)”)\n- **FP8 KV cache** (`fp8_e4m3`, ~2× KV memory savings; uses the NVFP4 checkpoint's calibration scales)\n- **GDN state pool** sized correctly from `MAX_CONCURRENT_REQUESTS` (concurrency × 4 state slots; the spec verify window is a **separate** engine-side buffer — verified in the build's `kv_cache_configurator`)\n- **Pinned to GB10's ten 3.9 GHz Cortex-X5 cores** (`--cpuset-cpus 5-9,15-19`) — the scheduler\u002Ftokenizer never land on the 2.8 GHz A725 efficiency cores (measured +2–7% decode)\n- **Thinking mode on by default** (`--reasoning-parser qwen3` → `reasoning_content`) and **tool calling** (`qwen3_coder` parser)\n\n---\n\n## Requirements\n\n| Component | Detail |\n|---|---|\n| Hardware | NVIDIA DGX Spark \u002F GB10 (aarch64, SM121; 128 GB unified memory) |\n| Docker | With NVIDIA Container Toolkit \u002F GPU passthrough working (`docker run --gpus all`) |\n| SGLang image | `lmsysorg\u002Fsglang:qwen38-27b` (model-specific build from the cookbook; multi-arch incl. arm64) |\n| CLI tools | `docker`, `curl` |\n| Hugging Face token | `HF_TOKEN` defined in `~\u002F.bashrc` (picked up automatically; higher rate limits) |\n\nThere is no separate download step: the container pulls the checkpoint into `.\u002F.cache\u002Fhuggingface` on first start (~22 GB for the NVFP4 repo; the cookbook cites ~16.5 GB for the NVFP4 LM weights alone, before the MTP head).\n\n## Quick start (ships as native 262K, 10 concurrent)\n\n```bash\n# 1. Copy the sample config once (creates .\u002F.env if you don't have one)\ncp .env.sample .env\n\n# 2. Start the server\n.\u002Fstart-dspark.sh    # DSpark — code ~51.5; default chat ~23; long essay ~18\n# .\u002Fstart.sh         # MTP — code ~34.5; default chat ~21; long essay ~24\n# .\u002Fstart-dflash.sh  # DFlash2, NVFP4 target — code ~50.9; essay ~25.4; chat ~29–67 (streamed)\n#                    #   (bf16 base: DF_TARGET=bf16 — unbenched on this box)\n\n# 3. Use it\ncurl http:\u002F\u002F127.0.0.1:8888\u002Fv1\u002Fmodels\n\n# 4. Stop it\n.\u002Fstop.sh\n```\n\n`.env.sample` ships with `YARN=0`, `CONTEXT_LENGTH=262144` (native) and `MAX_CONCURRENT_REQUESTS=10` — so a fresh clone serves **262K context, YaRN off, 10 concurrent** after just the `cp` above. `.env` is the live config (plain `VAR=value` lines read by `start.sh`): shell exports of the same names win, `.env` fills the gaps, start.sh defaults apply last. Changes only take effect on the **next** launch — `.\u002Fstop.sh && .\u002Fstart-dspark.sh` (or `.\u002Fstart.sh` for MTP). For anything above native context (e.g. 1M) or a different concurrency, see [Long context & concurrency](#long-context--concurrency-up-to-1m-10-concurrent). Note: DSpark cannot use YaRN \u002F context > 262144 on this build (ditto DFlash2 — same draft-config leak).\n\n`start-dflash.sh` is self-contained but different: no released SGLang image has DFlash2 support yet (it merged upstream 2026-08-19, after every published tag including the pinned `qwen38-27b`), so on a machine without the local `lmsysorg\u002Fsglang:qwen38-27b-dflash2` image the script **builds it automatically** (needs git + network once) via `patch\u002Fbuild-dflash2-image.sh`, which overlays the mainline python tree at that commit plus `patch\u002Fdflash2_nvfp4_head.patch` (quantized-head selector support for the NVFP4 target via `lm_head.quant_method` — no dense dequant; a dequant-once approach hard-rebooted this box at graph capture) onto the pinned GB10 image. First boot then pulls the ~2.7 GB draft if missing.\n\nAll start scripts are idempotent: if the container is already running they say so and exit; if a stopped container exists they remove it first. `.\u002Fstop.sh` stops whichever engine is up.\n\n## Scripts\n\n| Script | What it does |\n|---|---|\n| `start.sh` | Launches the SGLang container (`docker run -d`, host network, `--shm-size 32g`), streams logs to `.sglang.log`, records the container ID in `.sglang.pid`, and polls `http:\u002F\u002F127.0.0.1:8888\u002Fv1\u002Fmodels` until the server is ready. **EAGLE\u002FMTP speculative decoding** (`SPEC_STEPS\u002FSPEC_TOPK\u002FSPEC_DRAFT = 3\u002F1\u002F4`). Monitoring on by default: Prometheus `--enable-metrics` + `--enable-cache-report` (per issue #3). |\n| `start-dspark.sh` | Same service, **DSpark** instead of EAGLE: block-7 \u002F unquant draft, torch.compile + decode-graph caps, `--num-continuous-decode-steps 2`, mem 0.90. Thin wrapper (`EXTRA_ARGS` → `start.sh`). **Code 51.5 vs MTP 34.5 — see [Measured on this box](#measured-on-this-box) for all numbers.** |\n| `start-dflash.sh` | Same service, **DFlash2** block-diffusion draft (default `z-lab\u002FQwen3.8-27B-DFlash2@50307d4`, pinned — same draft as the `incoai\u002F…` mirror; `DRAFT_MODEL`\u002F`DRAFT_REVISION` env-overridable). **Default target: NVFP4** `RadixArk\u002FQwen3.8-27B-NVFP4` with `--mem-fraction-static 0.90` and quantized-head selector support baked into the image (in-place `lm_head.quant_method.apply` — no dense dequant; a dequant-once approach hard-rebooted this box); `DF_TARGET=bf16` selects `Qwen\u002FQwen3.8-27B` instead. Builds the image automatically on first run from `patch\u002F` if missing. Measured numbers: [Measured on this box](#measured-on-this-box). |\n| `patch\u002Fbuild-dflash2-image.sh` | Builds the DFlash2 image (`--full` default: pinned mainline python tree + `patch\u002Fdflash2_nvfp4_head.patch`, needs git+network first time; `--minimal`: only the 5 DFlash2 modules from `patch\u002Foverlay-dflash2\u002F`, sha256-verified, no network). Auto-invoked by `start-dflash.sh` when the image is missing (`--minimal` iff `IMAGE=…-minoverlay`). |\n| `bench\u002Fndec.py` | Two-call net-decode A\u002FB (LRUCache + essay, thinking off). How the DSpark vs MTP numbers above were measured. Run twice; trust the second; treat code deltas <15% as noise. |\n| `bench\u002Fbench.sh` | Essay \u002F tool-call **wall-time** bench + 16K TTFT probe (includes prefill). Different clock from `ndec.py`. |\n| `stop.sh` | Stops the serving engine (idempotent; also cleans up any experiment processes still alive). Leaves the stopped container in place for `docker logs` post-mortem. |\n\nRuntime artifacts: `.sglang.log` (server log), `.sglang.pid` (container ID), `.cache\u002F` (HF + Triton caches). All are git-ignored.\n\n> Whitelisted for tracking: `start.sh`, `start-dspark.sh`, `stop.sh`, `start-dflash.sh`, `patch\u002F`, `bench\u002F`, `README.md`, `CHANGELOG.md`, `.env.sample`, `LICENSE`, `.gitignore`. Experiment scripts and analysis docs stay untracked by design.\n\n## Which engine to use\n\n**EAGLE\u002FMTP and DSpark** are the same NVFP4 27B on the same `lmsysorg\u002Fsglang:qwen38-27b` image — only the speculative decoder changes. **DFlash2 (`start-dflash.sh`)** defaults to the same NVFP4 weights, on a derived image (see [Scripts](#scripts) \u002F Configuration below) that exists only because DFlash2 support is newer than any released image. DFlash2 can also target the bf16 base (`DF_TARGET=bf16`) — not benched.\n\nPerformance numbers for all three engines live in the **single table in [Measured on this box](#measured-on-this-box)** — below is engine selection only (no repeated tok\u002Fs):\n\n| | `.\u002Fstart-dspark.sh` (DSpark block-7) | `.\u002Fstart.sh` (EAGLE\u002FMTP 3\u002F1\u002F4) | `.\u002Fstart-dflash.sh` (DFlash2, NVFP4) |\n|---|---|---|---|\n| Best for | agents, code, tools, **normal chat** (**default here**) | long-form writing | code AND long-form writing (essay holds; chat is a win too) |\n| Memory | 22 GB target + ~2.7 GB draft, mem 0.90 | 22 GB target (in-checkpoint MTP), mem 0.95 | NVFP4: 22 GB target + ~2.6 GB draft, **mem 0.90**; bf16: 52 GB weights |\n\nDSpark\u002FMTP columns are live 2026-08-18 evening; DFlash2 on 2026-08-19 (n=5, same single boot, same probes) — numbers and caveats are in [Measured on this box](#measured-on-this-box). **Take-aways, with that caveat firmly in mind:** DFlash2 on NVFP4 ties DSpark on code (inside the \u003C15% noise band), *beats MTP* on the long essay, and *beats both on every short-chat condition* once counted from `completion_tokens` (the earlier ~9.2 “tok\u002Fs” reading was an SSE event-counting artifact — see the counting note in [Measured on this box](#measured-on-this-box)). Watchpoints: `mem-fraction-static 0.95` + DFlash2 wedged the box once (hard reboot; see [Logs & troubleshooting](#logs--troubleshooting)) — root-caused and fixed (see the DFlash2 bullet); the current `0.90` + 16-concurrent profile boot-**validated** 2026-08-19 with no reboot (concurrency ladder in [Measured on this box](#measured-on-this-box)). bf16 target is entirely unmeasured. Do not compare these to sparkDash fill-to-max streams.\n\nTuning history worth knowing: every Tier A (kernel-path) and Tier B (config\u002Fhost) experiment measured **zero net gain** — these configs are the local optimum on this box. DSpark block-7 is the code peak; block-5 trades −16% code for +8% prose if you want it (`DSPARK_EXTRA`, see Configuration). Local-only write-ups: `TIER_A_RESULTS.md`, `TIER_B_RESULTS.md`, `TIER_C_RESULTS.md`, `DS4F.md`, `KIMI.md`, `GROK.md`, `HANDOFF.md`.\n\n## Configuration\n\nDefaults live at the top of `start.sh`:\n\n| Variable | Default | Notes |\n|---|---|---|\n| `YARN` | `0` | `0` off \u002F `1` on for `CONTEXT_LENGTH` > 262144; implicitly on at exactly `1000000`. Factor = round(`CONTEXT_LENGTH`\u002F262144) |\n| `CONTEXT_LENGTH` | `262144` | Range `262144`..`1000000` (native..1M). Combined with `YARN=1` for values above native; `1M` auto-enables YaRN even with `YARN=0`. Invalid values abort at startup |\n| `MAX_CONCURRENT_REQUESTS` | `10` | Sizes `--max-mamba-cache-size` = concurrency × 4 slots and passes `--max-running-requests` |\n| `SPEC_STEPS` \u002F `SPEC_TOPK` \u002F `SPEC_DRAFT` | `3` \u002F `1` \u002F `4` | MTP chain drafting; topk=1 requires `SPEC_DRAFT = SPEC_STEPS + 1` (validated at launch). Sweep the pair on your box and pin the winner — 3\u002F1\u002F4 is the measured peak here |\n| `CHUNKED_PREFILL` | `8192` | Prefill chunk tokens. Cookbook DGX Spark cell uses `2048`; we keep `8192` (prefill\u002FTTFT, not decode tok\u002Fs). |\n| `CPUSET` | `5-9,15-19` | Docker `--cpuset-cpus` pin to GB10's Cortex-X5 cores (A725s are 0-4, 10-14). Empty = no pinning |\n| `MAMBA_SKIP_DECODE_LOCK` | `0` | `1` sets `SGLANG_OPT_MAMBA_SKIP_DECODE_LOCK` in the container — frees one GDN state slot per request (S 4→3) |\n| `PREFILL_CUDA_GRAPH` | `0` | `1` drops `--disable-prefill-cuda-graph`. Info: this build auto-disables prefill graphs on this model anyway (GDN layers ≠ standard GQA) |\n| `EXTRA_ARGS` | — | Free-form extra SGLang flags, appended **last** (argparse last-wins, so they can override built-ins). The experiment hatch: `EXTRA_ARGS=\"--fp4-gemm-runner-backend triton\" .\u002Fstart.sh` |\n| `QUANT` | `nvfp4` | `nvfp4` → `RadixArk\u002FQwen3.8-27B-NVFP4`, `fp8` → `Qwen\u002FQwen3.8-27B-FP8`, `bf16` → `Qwen\u002FQwen3.8-27B`. All three fit in the Spark's 128 GB. |\n| (shell overrides) | — | Any variable above can also be set as a shell env var, or put in `.env` |\n| `SERVED_MODEL_NAME` | `qwen3.8-27b-sglang` | Name clients use in API requests |\n| `IMAGE` | `lmsysorg\u002Fsglang:qwen38-27b` | Cookbook-pinned image for this model |\n| `CONTAINER_NAME` | `qwen3.8-27b-sglang` | Also used by `stop.sh` |\n| `PORT` | `8888` | Listens on `0.0.0.0` via host networking |\n\n> The shipped `.env`\u002F`.env.sample` match the `start.sh` defaults above, so a fresh clone serves **native 262K context, YaRN off, 10 concurrent** out of the box. Raise context above 262K with `YARN=1` + `CONTEXT_LENGTH` (see below).\n\n`start-dspark.sh` adds a couple of knobs (shell-env or `.env`, optional):\n\n| Variable | Default | Notes |\n|---|---|---|\n| `DSPARK_EXTRA` | — | Extra SGLang flags appended AFTER the base DSpark stack, for per-boot experiments without editing the script. E.g. `DSPARK_EXTRA=\"--speculative-dspark-block-size 5 --speculative-num-draft-tokens 6\" .\u002Fstart-dspark.sh` (prose-tuned block; see below) |\n| `IMAGE` | `lmsysorg\u002Fsglang:qwen38-27b` | env override (`IMAGE=tag .\u002Fstart-dspark.sh`) to run a patched derivative image; roll back by not setting it. |\n\nNote: `--cuda-graph-max-bs` is a deprecated alias in this build; the DSpark stack uses `--cuda-graph-max-bs-decode 4`.\n\n`start-dflash.sh` knobs (shell-env; it doesn't otherwise change `start.sh`\u002F.env behavior except the model-path override):\n\n| Variable | Default | Notes |\n|---|---|---|\n| `DF_TARGET` | `nvfp4` | `nvfp4` (default) → `RadixArk\u002FQwen3.8-27B-NVFP4` (+`--mem-fraction-static 0.90`); `bf16` → `Qwen\u002FQwen3.8-27B`. The NVFP4 path requires the quantized-head selector support baked into the derived image; without it, DFLASH dies at the first request (see [Scripts](#scripts)). |\n| `DF_EXTRA` | — | Extra SGLang flags appended AFTER the base DFlash2 stack (last-wins). E.g. `DF_EXTRA=\"--mem-fraction-static 0.90\" .\u002Fstart-dflash.sh` |\n| `IMAGE` | `lmsysorg\u002Fsglang:qwen38-27b-dflash2` (local) | The derived image; the script builds it if missing (auto-invokes `patch\u002Fbuild-dflash2-image.sh`). Override with `IMAGE=tag` — e.g. `IMAGE=lmsysorg\u002Fsglang:qwen38-27b-dflash2-minoverlay` (built via `patch\u002Fbuild-dflash2-image.sh --minimal`). |\n| `DRAFT_MODEL` \u002F `DRAFT_REVISION` | `z-lab\u002FQwen3.8-27B-DFlash2` \u002F `50307d4…` | Pinned DFlash2 draft (same weights as the `incoai\u002F…` mirror). `DRAFT_MODEL=incoai\u002FQwen3.8-27B-DFlash2 DRAFT_REVISION=dedf8df…` reproduces the original n=5 baseline; `DRAFT_REVISION=''` follows a branch head. |\n\nDFlash2-specific stack facts (so nobody re-learns them on a crash): `extra_buffer_lazy` is rejected by DFLASH (AssertionError) → the script forces `--mamba-radix-cache-strategy extra_buffer`; DFLASH only supports `speculative_num_steps == 1` (engine auto-overrides start.sh's MTP 3); `--enable-dp-attention` and the overlap scheduler are off in this path; the draft (`incoai`\u002F`z-lab\u002F…-DFlash2`) is a block-diffusion drafter, not a token LLM, so EAGLE knobs (`topk`, `num_steps`) don't apply.\n\n### Long context & concurrency (up to 1M, 10 concurrent)\n\nAll long-context and concurrency handling is driven by **three variables** in `.env` (or as shell exports):\n\n| Variable | Meaning |\n|---|---|\n| `YARN` | `1` = enable YaRN rope scaling — **required** for any `CONTEXT_LENGTH` > 262144; `0` = off (sensible only at\u002Fbelow 262144) |\n| `CONTEXT_LENGTH` | desired context in tokens, range `262144`..`1000000` |\n| `MAX_CONCURRENT_REQUESTS` | parallel requests; also sets `--max-running-requests`, and sizes the GDN pool = value × 4 slots |\n\n**Step by step — 1M context with 10 concurrent (goes above the shipped default):**\n\n```bash\ncp .env.sample .env                      # once, if you have no .env yet\nnano .env                                # make sure these are set:\n#  YARN=1\n#  CONTEXT_LENGTH=1000000\n#  MAX_CONCURRENT_REQUESTS=10\n.\u002Fstop.sh && .\u002Fstart.sh                  # relaunch so new values apply\n# verify after boot:\ngrep -E \"context_len|max_running_requests\" .sglang.log\nexpect: context_len=1000000, max_running_requests=10, mamba pool 40 slots\n```\n\n| You want | `YARN` | `CONTEXT_LENGTH` | `MAX_CONCURRENT_REQUESTS` | YaRN factor |\n|---|---|---|---|---|\n| 1M + 10 concurrent | 1 | 1000000 | 10 | 4.0 (also auto-on) |\n| 512K + 10 concurrent | 1 | 524288 | 10 | 2.0 |\n| 768K + 10 concurrent | 1 | 786432 | 10 | 3.0 |\n| native 262K + 10 concurrent | 0 | 262144 | 10 | — |\n| 1M + 2 concurrent | 1 | 1000000 | 2 | 4.0 |\n\nRules of thumb:\n\n- **Above 262144 you must set `YARN=1`** (1M is the one exception — it auto-enables YaRN even with `YARN=0`, so 1M works no matter what). `YARN=0` at e.g. 524288 is allowed but produces a warning and the server stays at 262K.\n- The YaRN factor is computed for you: `round(CONTEXT_LENGTH \u002F 262144)` → 524288 gives 2.0, 786432 gives 3.0, 1000000 gives 4.0 (2.0 and 4.0 are the model card's validated points).\n- SGLang otherwise fails closed at 262K with \"User-specified context_length (...) is greater than the derived context_length\"; `start.sh` auto-sets the required `SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1` env var, so you never touch it.\n- Hardware bound (not a config knob): one KV token ≈ 32.8 KB, a full 1M sequence ≈ 33 GB, pool ≈ 75 GB → ~**2 full 1M requests** run at once regardless of `MAX_CONCURRENT_REQUESTS`; extra concurrent requests queue until KV frees.\n- **DSpark caveat**: if you switch the speculative algorithm to DSpark, keep `YARN=0` \u002F `CONTEXT_LENGTH=262144` — the YaRN override leaks into the DSpark draft config and crashes at boot.\n\n### Notable serving choices\n\n- **Recipe vs this repo:** the cookbook's DGX Spark + NVFP4 + DSpark cell uses mem 0.85, chunk 2048, GDN float32, radix `extra_buffer`; we measured those (float32 −3%, FP8 ~30% slower) and pin instead: mem 0.90, chunk 8192, GDN bf16, `extra_buffer_lazy`, block 7 \u002F 8 draft tokens, torch.compile + decode graphs, `--num-continuous-decode-steps 2`, prefill graphs off, flashinfer, FP8 KV. MTP (`start.sh`) keeps mem 0.95, EAGLE 3\u002F1\u002F4.\n- **Speculative decoding:** MTP = in-checkpoint head (no download); DSpark fetches `RadixArk\u002FQwen3.8-27B-DSpark` (~2.7 GB) once. Numbers: [Measured on this box](#measured-on-this-box), not the older 16–21 tok\u002Fs wall-time figures. MTP 3\u002F1\u002F4 is measured-optimal here (steps sweep 2→12.8, 3→17.2, 4→16.8, 5→16.3, 6→15.8); NGRAM (~30% under MTP) and prefill CUDA graphs were rejected. DSpark's draft was trained on FP8; `QUANT=fp8` did not lift acceptance. If spec decode errors at boot: `--attention-backend triton`.\n- **DFlash2 (`start-dflash.sh`):** requires a derived image (DFlash2 merged upstream 2026-08-19, after every released tag) — built automatically from `patch\u002F` on first run. Draft: block-diffusion drafter, pinned `z-lab\u002FQwen3.8-27B-DFlash2@50307d4` (mirror of `incoai\u002F…`; override `DRAFT_MODEL`\u002F`DRAFT_REVISION`), ~2.6 GB. **Crash history (fixed):** the original head handling dequantized the whole NVFP4 lm_head (~2.5 GB) at draft-graph capture and hard-rebooted the box (0.95, and 0.80 at concurrency ≥ 8–10); the image now runs the quantized head in place (`lm_head.quant_method.apply`) — no big allocation, no capture spike. The 0.90\u002F16 profile boot-**verified** 2026-08-19 (no reboot; ladder in [Measured on this box](#measured-on-this-box)). Operational: NVFP4 at `--mem-fraction-static 0.90` + `MAX_CONCURRENT_REQUESTS=16`; keep `YARN=0`\u002F`CONTEXT_LENGTH=262144` (same draft-config leak as DSpark); unmeasured: bf16 target, long context. 2026-08-19 A\u002FB (minimal 5-file overlay vs whole-tree, same base digest): 61.1\u002F28.4 vs 50.9\u002F25.4 (+20%\u002F+12%), confounded (fresh boot, conc 10 vs 4); details: `bench\u002F_ab-dflash2\u002FSUMMARY.md` (local).\n- **CPU pinning (GB10 is big.LITTLE):** container pinned to the ten 3.9 GHz Cortex-X5 cores (`5-9,15-19`); the ten 2.8 GHz A725 cores (`0-4,10-14`) stay free. Without pinning, scheduler\u002Ftokenizer processes land on little cores ~half the time. Measured +2–7% decode. Override with `CPUSET` (empty = off).\n- **GDN state pool (throughput):** `--max-mamba-cache-size` = concurrency × S; S=4 for `extra_buffer_lazy` + overlap scheduler (no accuracy cost; verified in this build's `kv_cache_configurator.py` — the verify window is a separate buffer, so folding draft tokens in (×8) over-provisions 2×). Default 10 → 40 slots (~3.1 GB at BF16, 78.4 MB\u002Fslot). `--max-running-requests` pins the scheduler cap (spec decode otherwise resets it to 48); verify after boot. `MAMBA_SKIP_DECODE_LOCK=1` drops S to 3. The stock `--mamba-full-memory-ratio 0.9` over-provisions KV and clamps concurrency; pinned at 4.21 instead.\n- **Context (up to 1M with YaRN):** `.env` `YARN` \u002F `CONTEXT_LENGTH` (262144..1000000). YaRN is applied above 262K (auto-on at exactly 1M), factor = round(len\u002F262144) — 524288→2.0, 1000000→4.0 (the card's validated points). `start.sh` passes the `rope_parameters` override + `--context-length` + `SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1` (this build requires it); verify with `grep context_len .sglang.log`. **Not compatible with DSpark\u002FDFlash2** (the override leaks into the draft config and crashes the rope validator: `AttributeError: … max_position_embeddings`) — keep `YARN=0`\u002F262K for both; YaRN is MTP-only.\n- **KV cache:** explicit `--kv-cache-dtype fp8_e4m3` (the NVFP4 checkpoint declares FP8 KV anyway; the flag keeps FP8 KV if you switch quant). ~32.8 KB\u002Ftoken, so a full 1M sequence ≈ 33 GB. Measured pool 2.48M tokens ≈ 81 GB: two 1M requests fit simultaneously; a third is admitted as KV frees.\n- **Vision:** the model is a native VLM; SGLang serves the vision tower live (image + video input supported out of the box).\n\n### Measured on this box\n\n**Probe × engine — the one canonical table** (DSpark\u002FMTP measured 2026-08-18, DFlash2 2026-08-19):\n\n| Probe | DSpark (`.\u002Fstart-dspark.sh`, block-7) | MTP (`.\u002Fstart.sh`, EAGLE 3\u002F1\u002F4) | DFlash2 (`.\u002Fstart-dflash.sh`, NVFP4 target) |\n|---|---|---|---|\n| Code — LRUCache + small test (`bench\u002Fndec.py`, n=5 all) | **51.5 tok\u002Fs** (51.4–51.7; `c2` always 518) | **34.5 tok\u002Fs** (34.5–34.6; `c2` always 508) | **50.9 tok\u002Fs** (50.8–51.1; `c2` always 600) |\n| Short chat — “what is a hash map…” (stream) | 22.0 \u002F 21.3 \u002F **23.2** (T=0 off · T=1 off · **T=1 thinking on**) | 24.6 \u002F 23.4 \u002F **21.0** | **31.7 \u002F 28.9 \u002F 66.6** (T=0 off · T=1 off · **T=1 thinking on**) |\n| Long essay — Babbage → GPUs (`bench\u002Fndec.py`, n=5 all) | **18.3 tok\u002Fs** (18.2–18.3) | **24.1 tok\u002Fs** (24.1–24.1) | **25.4 tok\u002Fs** (25.3–25.4) |\n\n**Counting + methodology (read before comparing):** DFlash2 column is n=5, 2026-08-19 (same single boot: code 50.77–51.07 median 50.90; essay 25.34–25.39 median 25.39; `c2` always 600 — the earlier n=2 run sits inside these ranges and is superseded); DSpark\u002FMTP are n=5 from the original 2026-08-18 session — same probes, same box, different day, so cross-column deltas are indicative, not a race. Within a day, code deltas \u003C15% are still noise, so call it “ties DSpark”. Short-chat counting: the DFlash2 numbers are taken from the server's own `completion_tokens` (`stream_options.include_usage`), post-first-token — this DFlash2 image batches several tokens per SSE event (~3.75 on average at a fixed ~8 events\u002Fs cadence), so a client that counts events as tokens reads ~9 “tok\u002Fs”; the DSpark\u002FMTP cells were recorded with that same event-counting script on the older stock image and stand as recorded (events ≈ single tokens there). Non-streamed same-prompt sanity check agrees (28.4 tok\u002Fs overall incl. TTFT at T=0).\n\nEngine notes that do **not** repeat the table: DSpark **increases coding speed (~1.5× on LRUCache)**; the MTP prose win is the **long essay**; DSpark slightly beat MTP on default chat (thinking on) with no degrade; thinking-off chat was a small MTP edge. Block sweep: block-7 is the code peak; block-5 is **+8% prose \u002F −16% code**. `--speculative-accept-threshold-acc \u003C1` hurt — leave at 1.0.\n\n**MTP-era wall-time** (`.\u002Fbench\u002Fbench.sh`, includes prefill; not comparable to the table above): thinking 17.2–20.5 tok\u002Fs, non-thinking 21.6–22.7, tool-call 26–28. TTFT on a fresh ~16K prompt ~8.3 s warm \u002F ~13 s first boot (Triton warmup). MTP step sweep peaked at 3\u002F1\u002F4 (see above).\n\n**DFlash2 detail (2026-08-19, same single boot — raw runs behind the table above):** ndec code 50.77–51.07, essay 25.34–25.39 (`c2` always 600; DSpark\u002FMTP code `c2` 518\u002F508). Chat raw: 31.6–31.8 \u002F 26.4–34.3 \u002F 45.6–80.2 (T0 off · T1 off · T1 think). Wall-time (non-streamed, incl. prefill — a different clock): thinking 18.8–19.7, non-thinking 22.8–24.1, tool-call 27.3–30.8; TTFT on a fresh ~16K prompt 8.2 s warm. Not measured: bf16 target, long context. Same caveats as the counting note above — different-day vs DSpark\u002FMTP and pre-fix probes (stable at 0.80\u002F4, reboots at 0.80\u002F8–10) mean one replication boot before switching the default remains the honest recommendation.\n\n**Concurrency ladder — DFLASH2 NVFP4 (2026-08-19, the post-fix 0.90\u002F16 boot, synthetic structural-decode fixture; aggregate is total across streams, stream is per-client):**\n\n| Streams | TTFT | Aggregate tok\u002Fs | Per-stream tok\u002Fs |\n|---|---|---|---|\n| 1 | 127 ms | 56.6 | 56.6 |\n| 2 | 202 ms | 58.4 | 42.4 |\n| 4 | 224 ms | 111.6 | 33.4 |\n| 8 | 280 ms | 184.9 | 30.8 |\n| 16 | 4.18 s | 227.6 | 28.2 |\n\nThis run is the first post-fix boot at the 0.90 \u002F 16 profile and it completed all 16 streams **without a reboot** — the draft-capture crash is gone. Per-stream throughput degrades gracefully as concurrency rises (56.6 → 28.2 tok\u002Fs); aggregate scales to **227.6 tok\u002Fs at 16 concurrent**. TTFT holds at 0.13–0.28 s through ×8 and jumps to **4.18 s at ×16** (16-way admission on this box). One boot, one fixture — indicative, not a guarantee; replicate before relying on it. Another clock again: not comparable to the ndec\u002Fstream\u002Ftable rows.\n\nRun-to-run variance is ~±1.5 tok\u002Fs (±7%) on those wall-time numbers. The box **drifts** (essay 19.5 → 18 tok\u002Fs over ~an hour of heavy benching — power-cap). The LRUCache two-call is window-dependent (same boot 44–51 by cap); treat code deltas **<15% as noise**. The essay probe (±1% within a boot) is the A\u002FB discriminator. Re-baseline in-session; do not compare across hours. The next step-change needs a newer `lmsysorg\u002Fsglang:qwen38-27b` image.\n\n## Thinking & tool calling\n\n- **Thinking mode is ON by default** — the chat template defaults `enable_thinking=true` and `preserve_thinking=true` (the full reasoning trace is retained across turns; good for agents and KV reuse). `--reasoning-parser qwen3` surfaces `\u003Cthink>…\u003C\u002Fthink>` as `reasoning_content` instead of inline text. Depth is tunable per request with `reasoning_effort=xhigh|medium|low` (xhigh default).\n- **Sampling defaults** come from the checkpoint's `generation_config.json` (`--sampling-defaults model`): thinking mode wants `temperature=1.0, top_p=0.95, top_k=20, min_p=0.0, presence_penalty=0.0`.\n- **Tool calling** needs no extra SGLang flag (unlike vLLM's `--enable-auto-tool-choice`): `--tool-call-parser qwen3_coder` decodes the template's `\u003Ctool_call>\u003Cfunction=…>\u002F\u003Cparameter=…>` payload into structured `tool_calls`. Just send `tools` in the request. (The hermes parser expects a different payload and would never parse.)\n\n## Using the API\n\nOpenAI-compatible base URL: `http:\u002F\u002F127.0.0.1:8888\u002Fv1` (model name: `qwen3.8-27b-sglang`).\n\n```bash\ncurl http:\u002F\u002F127.0.0.1:8888\u002Fv1\u002Fchat\u002Fcompletions \\\n  -H \"Content-Type: application\u002Fjson\" \\\n  -d '{\n    \"model\": \"qwen3.8-27b-sglang\",\n    \"messages\": [{\"role\": \"user\", \"content\": \"Explain YaRN in two sentences.\"}]\n  }'\n```\n\nNon-thinking \u002F instruct request (per the model card):\n\n```json\n{\n  \"model\": \"qwen3.8-27b-sglang\",\n  \"messages\": [{\"role\": \"user\", \"content\": \"Write a haiku about GB10.\"}],\n  \"temperature\": 0.7,\n  \"top_p\": 0.8,\n  \"top_k\": 20,\n  \"presence_penalty\": 1.5,\n  \"chat_template_kwargs\": { \"enable_thinking\": false }\n}\n```\n\nSGLang also serves an **Anthropic-compatible** endpoint at `http:\u002F\u002F127.0.0.1:8888\u002Fv1\u002Fmessages` — for Claude Code, set `ANTHROPIC_BASE_URL=http:\u002F\u002F127.0.0.1:8888` (no `\u002Fv1` suffix; Claude Code appends it). The same parser flags apply there. Coding agents that speak plain OpenAI (OpenCode, Pi, …) point at `\u002Fv1` and use the served model name.\n\n## Logs & troubleshooting\n\n- Tail the server log: `tail -f .sglang.log` (or `docker logs -f qwen3.8-27b-sglang`)\n- After a DSpark boot: `grep -oE \"speculative_algorithm='[^']+'|speculative_dspark_block_size=[0-9]+|context_len=[0-9]+\" .sglang.log | tail -3` — expect `DSPARK`, block `7`, `262144`\n- After a DFlash2 boot: `grep -oE \"speculative_algorithm='[^']+'|speculative_draft_model_path='[^']+'|speculative_num_draft_tokens=[0-9]+\" .sglang.log | tail -3` — expect `DFLASH`, `incoai\u002F…-DFlash2`, `8`. Also grep for `Initialized DFLASH draft runner` and `DFLASH selector decode … folded into the draft cuda graph` (if you see `kept eager (reason=quantized lm_head)` on a boot, the dequant patch is not in the image — rebuild)\n- If the GB10 hard-reboots (kernel log: `task sglang::schedul … blocked …` \u002F `journald … Under memory pressure`) right after `Capture target verify CUDA graph end`, it was DFlash2 + a too-high `--mem-fraction-static` (0.95) at draft-graph capture; relaunch DFlash2 at 0.90. **Root cause found 2026-08-19:** the crashes (0.95, and 0.80 at concurrency ≥ 8–10) were caused by the old dequant-once head handling materializing the full dense NVFP4 lm_head (~2.5–5 GB) during graph capture — fixed in the image (in-place `lm_head.quant_method.apply` selector; no dense dequant). If you still see this signature, it's not the mem fraction per se. If you need mixed-chat benchmarks, run them when nothing else is loaded.\n- DFlash2 SSE streams emit ~8 events\u002Fs regardless of throughput (measured 2026-08-19: median 126 ms between events) and, on this newer image, each event carries several tokens (~3.75 on average) — so event-counting a DFlash2 stream under-reads tok\u002Fs ~4×. Always count `completion_tokens` (`stream_options.include_usage`) for real rates; the DSpark\u002FMTP-era short-chat cells used an event-counting script on the older stock image, where events ≈ single tokens\n- If a DFlash2 request dies with `DFlash2 selector requires a dense FP16\u002FBF16\u002FFP32 target lm_head` on an NVFP4 target, the image lacks the quantized-head selector support — rebuild it with `patch\u002Fbuild-dflash2-image.sh` and run that image\n- `start.sh` \u002F `start-dspark.sh` print the last 200 log lines and exit if the container dies before becoming ready\n- Terminal output filters the harmless per-layer “Enabled fused SiLU+mul+FP4-quant…” notices; `.sglang.log` keeps everything\n- Concurrency check: `grep max_running_requests .sglang.log` — should equal your `MAX_CONCURRENT_REQUESTS` (default 10), not a lower clamped value\n- Mamba pool check: `grep max_mamba_cache_size .sglang.log` — expect `MAX_CONCURRENT_REQUESTS × 4`\n- First long prefill after a cold boot is slow (~13 s for a fresh 16K prompt vs ~8 s warm) — that's Triton kernel warmup, not a regression; the `.cache\u002Ftriton` volume persists it across restarts\n- If startup dies with `AttributeError: 'PreTrainedConfig' object has no attribute 'max_position_embeddings'`, you're using DSpark with `YARN=1` \u002F `CONTEXT_LENGTH=1000000` — the YaRN override leaks into the draft config. Keep `YARN=0` and `CONTEXT_LENGTH=262144` for DSpark (see Context note)\n- First start downloads ~22 GB of weights (plus ~2.7 GB DSpark draft model if you switch to DSpark); subsequent starts reuse `.\u002F.cache\u002Fhuggingface`\n\n## Repository layout\n\n```\n.\n├── start.sh          # EAGLE\u002FMTP engine (port 8888); tracked\n├── stop.sh           # stops whichever engine is up; tracked\n├── start-dspark.sh   # DSpark engine (port 8888); whitelisted for versioning\n├── start-dflash.sh   # DFlash2 engine (port 8888, bf16 or NVFP4 target); auto-builds the image from patch\u002F on first run; tracked\n├── patch\u002F             # DFlash2 image build machinery (build-dflash2-image.sh, dflash2_nvfp4_head.patch, overlay-dflash2\u002F); tracked\n├── bench\u002F\n│   ├── bench.sh     # essay \u002F tool-call wall-time bench + TTFT probe\n│   └── ndec.py      # two-call net-decode (LRUCache + essay); engine A\u002FB (any engine)\n├── .env             # live config (context \u002F concurrency \u002F quant \u002F tuning); not tracked by git\n├── .env.sample      # tracked template — copy to .env to configure\n├── .gitignore       # whitelist: start scripts, bench\u002F, README, CHANGELOG, .env.sample, LICENSE\n├── LICENSE          # MIT\n└── README.md        # tracked\n```\n\nExperiment write-ups are local-only (untracked): `DS4F.md`, `KIMI.md`, `GROK.md`, `TIER_A_RESULTS.md`, `TIER_B_RESULTS.md`, `TIER_C_RESULTS.md`, `HANDOFF.md`.\n\n## Notes\n\n- `QUANT` values: `nvfp4` → `RadixArk\u002FQwen3.8-27B-NVFP4`, `fp8` → `Qwen\u002FQwen3.8-27B-FP8`, `bf16` → `Qwen\u002FQwen3.8-27B` (all fit in the Spark's 128 GB).\n- `SERVED_MODEL_NAME`, `IMAGE`, `CONTAINER_NAME`, `PORT` are set inline in `start.sh` (not `.env`).\n\n## Credits\n\n- [SGLang cookbook — Qwen3.8-27B](https:\u002F\u002Fdocs.sglang.io\u002Fcookbook\u002Fautoregressive\u002FQwen\u002FQwen3.8-27B) — the DGX Spark serving recipe, MTP and GDN state-pool guidance\n- [Qwen3.8-27B model card](https:\u002F\u002Fhuggingface.co\u002FQwen\u002FQwen3.8-27B) — YaRN 1M-context SGLang recipe and sampling recommendations\n- [RadixArk\u002FQwen3.8-27B-NVFP4](https:\u002F\u002Fhuggingface.co\u002FRadixArk\u002FQwen3.8-27B-NVFP4) — NVFP4 W4A4 checkpoint (FP8 KV calibration scales)\n- [RadixArk\u002FQwen3.8-27B-DSpark](https:\u002F\u002Fhuggingface.co\u002FRadixArk\u002FQwen3.8-27B-DSpark) — the DSpark draft model used by `start-dspark.sh`\n- [incoai\u002FQwen3.8-27B-DFlash2](https:\u002F\u002Fhuggingface.co\u002Fincoai\u002FQwen3.8-27B-DFlash2) \u002F [z-lab mirror](https:\u002F\u002Fhuggingface.co\u002Fz-lab\u002FQwen3.8-27B-DFlash2) — the DFlash2 block-diffusion drafter used by `start-dflash.sh` (trained against the bf16 `Qwen\u002FQwen3.8-27B`)\n- [inco.ai\u002Fblog\u002Fdflash2](https:\u002F\u002Finco.ai\u002Fblog\u002Fdflash2\u002F) — DFlash2 write-up; its SGLang serving recipe is what `start-dflash.sh` pins\n- [SGLang DFLASH2 commit](https:\u002F\u002Fgithub.com\u002Fsgl-project\u002Fsglang\u002Fcommit\u002Fc14312a66) — upstream mainline DFlash2 support, merged after every released image; this repo's derived-image build tracks it\n- [hasso5703\u002Fdgx-spark-qwen38](https:\u002F\u002Fgithub.com\u002Fhasso5703\u002Fdgx-spark-qwen38) — the published DSpark-on-GB10 config (same pinned image) that the DSpark flag stack builds on\n- [SGLang](https:\u002F\u002Fgithub.com\u002Fsgl-project\u002Fsglang) — inference engine and OpenAI\u002FAnthropic-compatible server\n","该项目提供一套开箱即用的Shell脚本，用于在NVIDIA DGX Spark（GB10，aarch64架构）上高效部署和推理通义千问Qwen3.8-27B大语言模型，基于SGLang推理框架。核心功能包括支持NVFP4\u002FW4A4量化、FP8 KV缓存、原生262K上下文（可扩展至1M）、GDN状态池自动配置，以及三种实测优化的推理模式（EAGLE\u002FMTP、DSpark、DFlash2），所有参数均基于DGX Spark硬件实测调优。适用于需要在ARM架构DGX设备上低延迟、高并发运行大模型的科研与工程部署场景。",2,"2026-08-19 02:30:05","CREATED_QUERY"]