[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"project-95015":3},{"id":4,"name":5,"fullName":6,"owner":7,"repo":5,"description":8,"homepage":9,"htmlUrl":9,"language":10,"languages":9,"totalLinesOfCode":9,"stars":11,"forks":12,"watchers":13,"openIssues":13,"contributorsCount":14,"subscribersCount":14,"size":14,"stars1d":14,"stars7d":15,"stars30d":15,"stars90d":14,"forks30d":14,"starsTrendScore":16,"compositeScore":17,"rankGlobal":9,"rankLanguage":9,"license":18,"archived":19,"fork":19,"defaultBranch":20,"hasWiki":21,"hasPages":19,"topics":22,"createdAt":9,"pushedAt":9,"updatedAt":32,"readmeContent":33,"aiSummary":34,"trendingCount":14,"starSnapshotCount":14,"syncStatus":35,"lastSyncTime":36,"discoverSource":37},95015,"qwen38-27b-rtx3090","syv-ai\u002Fqwen38-27b-rtx3090","syv-ai","Qwen3.8-27B on a single RTX 3090 with vLLM: ~1,000 tok\u002Fs at 64 concurrent (int8 tensor-core GEMMs, fp16 DeltaNet state), ~114 tok\u002Fs single-user at default sampling \u002F ~124 greedy (MTP drafts, own-output draft vocab, calibrated int4 lm_head, split-KV verify attention), 150k-262k context; patches, requant scripts, benchmarks",null,"Python",293,39,4,0,136,77,74.81,"Apache License 2.0",false,"main",true,[23,24,25,26,27,28,29,30,31],"kv-cache","llm-inference","local-llm","quantization","qwen","qwen3","rtx-3090","speculative-decoding","vllm","2026-08-24 04:01:23","# Qwen3.8-27B on one RTX 3090\n\nServing setup for [Qwen3.8-27B](https:\u002F\u002Fhuggingface.co\u002FQwen\u002FQwen3.8-27B) on a\nsingle 24 GB consumer GPU with vLLM. 150k token context, OpenAI-compatible\nAPI with key auth, and two ready-made configs depending on what you're doing:\n\n| | [batch\u002F](batch\u002F) | [single-user\u002F](single-user\u002F) |\n|---|---|---|\n| for | API backends, pipelines, many concurrent requests | one or a few people chatting |\n| aggregate, 64 concurrent (128 in \u002F 512 out) | **~1,094 tok\u002Fs** steady-state decode, 942 end-to-end (~1,222 \u002F 1,042 with all layers int8) | n\u002Fa (8 slots) |\n| single-stream (C1) decode rate, realistic prompts | 46 tok\u002Fs | MTP: **114** tok\u002Fs at default sampling, **118** greedy (`CTX=fast`, 64k; 85 \u002F 89 with `CTX=long`, 150k). DFlash2 (`SPEC=dflash2`): **122** default, **132** greedy |\n| reproducing its own context (quoting a document, applying an edit) | 46 tok\u002Fs | **381 tok\u002Fs** at 25k context — 15.0 tokens per verify step, drafted straight from the prompt (`SPEC=dflash2` + `DFLASH_TOKENS=15`) |\n| trick | 16-bit recurrent state + int8 tensor-core GEMMs | MTP speculation with 4 cheap drafts, a draft vocabulary that covers what the model says, calibrated int4 lm_head\u002Fdrafter, split-KV verify attention; optionally DFlash2 (7 drafts in one pass, int4-requantized, vLLM PR #52816 backported) with a verify block the context fills |\n\nBoth modes share one install — the mode is just which launch script you run.\nSpeculation wins below ~8 concurrent users, plain batching above. Numbers are\n`vllm bench serve` on an RTX 3090 at a 250 W power limit. If the card is yours\nalone, the fastest configuration is three environment variables away:\n[If you are the only user](#if-you-are-the-only-user-do-this).\n\nPrefill is a separate budget from either: ~1,810 tok\u002Fs at 1k inputs in batch\nmode (~1,210 single-user), ~1,000 tok\u002Fs at 100k, so a 100k prompt costs ~100 s\nof TTFT ([full matrix](batch\u002FREADME.md#prefill)). How each number was won:\n[docs\u002Foptimizations.md](docs\u002Foptimizations.md).\n\n## Quick start\n\nDocker (recommended — image build, model download and requantization, then\nthe server; the API is OpenAI-compatible on port 18020):\n\n```bash\ngit clone https:\u002F\u002Fgithub.com\u002Fsyv-ai\u002Fqwen38-27b-rtx3090 && cd qwen38-27b-rtx3090\necho \"VLLM_API_KEY=$(openssl rand -hex 24)\" > .env\ndocker compose --profile single up -d      # one or a few users; or --profile batch\n```\n\nOr by hand in a venv (same steps: model download, requantization, vLLM\npatches, `verify.sh`) — see [Setup](#setup). Then pick a mode:\n[batch\u002F](batch\u002F) for throughput, [single-user\u002F](single-user\u002F) for latency.\n\n### If you are the only user, do this\n\nThe command above starts the conservative default — MTP speculation, 8 request\nslots, 64k context, 118 tok\u002Fs greedy at C1. Three settings are worth more than\nevery other knob in this repo put together:\n\n```bash\nprintf 'SPEC=dflash2\\nDFLASH_TOKENS=15\\nPREFIX_CACHE=1\\n' >> .env\ndocker compose --profile single up -d\n```\n\nor, in the venv install:\n\n```bash\nvenv\u002Fbin\u002Fpython fetch_dflash2.py     # once, 1.2 GB (Docker's prepare step does it for you)\nSPEC=dflash2 DFLASH_TOKENS=15 PREFIX_CACHE=1 bash single-user\u002Fstart_qwen.sh\n```\n\n`SPEC=dflash2` swaps Qwen's MTP head for the DFlash2 block drafter: 7 tokens\nproposed in one pass instead of 4 chained ones. `DFLASH_TOKENS=15` then lets the\ntarget verify 16 tokens per step — the drafter still proposes the 7 it was\ntrained for, and the remaining positions are filled from the request's own\ncontext, which costs nothing to draft and is exactly right whenever the answer\nquotes the prompt. `PREFIX_CACHE=1` keeps the document you already sent, both\nits attention KV and its recurrent state. One request at a time, greedy, RTX\n3090 at 250 W:\n\n| decode | MTP (default) | `SPEC=dflash2` | `+ DFLASH_TOKENS=15` |\n|---|---|---|---|\n| 8 real chat prompts | 118 tok\u002Fs | 132 | **133** |\n| reproducing a 25k-token document | n\u002Fa* | 260 | **382** |\n| request slots \u002F context | 8 \u002F 64k | 8 \u002F 64k | 4 \u002F 56k |\n\n\u003Csub>\\* drafting from the context only exists in `SPEC=dflash2`. The two right\ncolumns are one server session, where run-to-run greedy divergence is ±3-5%;\nreproduce them with `venv\u002Fbin\u002Fpython bench\u002Flabd_bench.py \u003Ctag> --ctx 20000`.\u003C\u002Fsub>\n\n`PREFIX_CACHE=1` is orthogonal to the other two and worth as much again in a\nchat client: a second turn against that same 25k-token document takes 0.56 s to\nfirst token instead of 22.4 s, with the answers unchanged token for token.\n\nAll of it is lossless: speculative decoding samples the same distribution as no\nspeculation at all, the prefix cache resumes recurrent state rather than\napproximating it, and GSM8K reads 96.0-96.5% across the three columns. What\n`DFLASH_TOKENS=15` costs is half the request slots and 8k of context — that is\nthe whole reason it is opt-in, and why the default stays where it is for anyone\nserving more than a few people. Every other knob: [single-user\u002F](single-user\u002F).\n\n## Benchmarks\n\nFull tables per mode in [batch\u002FREADME.md](batch\u002FREADME.md) and\n[single-user\u002FREADME.md](single-user\u002FREADME.md); quality in\n[docs\u002Fquality.md](docs\u002Fquality.md). Reproduce any of it with\n`bash bench\u002Frun_benchmarks.sh batch|single` against your own server.\n\n### vs. ninfer-3090\n\n[ninfer-3090](https:\u002F\u002Fgithub.com\u002FDon-Chad\u002Fninfer-3090) publishes cohort benchmarks\nfor this model on this card, using C concurrent requests of random tokens. Random\ntokens are a bad yardstick for speculative decoding — acceptance swings between 80%\nand near zero with the sample — so ours are 8 realistic chat prompts (English,\nDanish, code), 1,024-token answers, model-default sampling:\n\n| Cohort | ninfer-3090 (MTP3, random tokens) | this repo, batch | single-user, MTP | single-user, DFlash2 |\n|---|---|---|---|---|\n| C1 | 70.19 tok\u002Fs | 45.4 | 113.6 | **122.1** |\n| C2 | 89.43 tok\u002Fs | 82.6 | **194.0** | 191.4 |\n| C4 | 97.89 tok\u002Fs | 165.6 | 258.6 (289.2 with `CTX=long`) | **286.7** |\n| C8 | 161.28 tok\u002Fs | 298.5 | **379.9** (409.0 with `CTX=long`) | 373.7 |\n| C64 (128 in \u002F 512 out) | not supported | **~1,094** | — | — |\n\nDecode rate (C × 1000 \u002F mean TPOT), best of the runs we have, from\n`bench\u002Frun_benchmarks.sh`; greedy instead of default sampling reads\n131.9 \u002F 209.6 \u002F 309.6 \u002F 390.6 for DFlash2. The C64 and DFlash2 columns are from the\ncurrent stack, the other two from earlier runs; ninfer's published figures are their\nown protocol. Peak VRAM is comparable to theirs (23.0 vs 22.1 GiB at C8) — the gap is\nmostly vLLM's continuous batching plus the memory this repo's requantization frees up.\n\n### Quality\n\nThe whole stack is quantized, so the honest question is what it costs. Short\nversion: **IFBench 78.3** prompt-level strict vs 79.5 for the unquantized model\n(one point), **perplexity 8.09** on 33k held-out tokens, **GSM8K 96.5%** (200\nquestions, greedy). Speculative decoding — MTP, DFlash2 and the lookup drafter —\nis exact by construction and changes none of it; the int8-activation steps in\nbatch mode are the only knobs that trade accuracy for speed, and they cost\n0.9-3.7% perplexity depending on how far you push them. Per-configuration\ntables: [docs\u002Fquality.md](docs\u002Fquality.md).\n\n### Why this isn't just `vllm serve`\n\nNine things, from requantizing both embedding matrices to drafting straight out\nof the prompt — one line each, then the reasoning and measurements, in\n[docs\u002Foptimizations.md](docs\u002Foptimizations.md).\n\n### What each step buys\n\nMeasured cumulatively on the 3090, 64 concurrent, 128 in \u002F 512 out, `vllm bench\nserve` random dataset:\n\n| step | what it does | e2e output tok\u002Fs | steady-state decode |\n|---|---|---|---|\n| W4A16 AutoRound body (as published) + fp8 KV | int4 Marlin kernels, 66.7k-token pool | 370 (48 conc, 256\u002F256) | — |\n| + lm_head \u002F embed_tokens int8 | 2.6 GB of cache pages back | 516 | ~585 (37 requests resident) |\n| + fp16 recurrent state | 64 requests resident, half the state traffic | 707 | ~830 |\n| + int8 activations, MLP (default) | int8 tensor cores on 74% of the FLOPs | 942 | ~1,094 |\n| + int8 activations, everything (`INT8_LAYERS=.`, needs `GPU_UTIL=0.95`) | | 1,042 | ~1,222 |\n\nAnd single-stream on realistic prompts (single-user mode, T = model default \u002F\ngreedy):\n\n| step | tok\u002Fs | tokens per step | draft acceptance, position 0 |\n|---|---|---|---|\n| no speculation | 46 \u002F 46 | 1.0 | — |\n| MTP-2 as shipped (bf16 drafter, full head, fp32 state) | 66 \u002F 79 | 2.1 \u002F 2.4 | 65% \u002F 80% |\n| MTP-4, int8 drafter, 40k draft head, fp16 state | 78 \u002F 99 | 2.2 \u002F 2.7 | 58% \u002F 70% |\n| + probabilistic draft sampling (`CTX=fast`, k=4) | 90 \u002F 98 | 2.6 \u002F 2.7 | 69% \u002F 70% |\n| same with 3 drafts on FlashInfer\u002Ffp8 KV (`CTX=long`, 150k) | 84 \u002F 89 | 2.5 \u002F 2.4 | 69% \u002F 71% |\n| + sampler patch, split-KV verify attention | 93 \u002F 99 | 2.6 \u002F 2.6 | 69% \u002F 70% |\n| + draft vocab counted over the model's own outputs | 107 \u002F 109 | 2.9 \u002F 2.9 | 74% \u002F 74% |\n| + GPTQ-int4 lm_head (calibrated) | 109 \u002F 112 | 2.8 \u002F 2.8 | 73% \u002F 73% |\n| + GPTQ-int4 MTP module (**fast variant, shipped**) | **~114 \u002F 118-124** | 2.8 \u002F 2.9-3.0 | 74% \u002F 77% |\n| DFlash2 block drafter instead of MTP (`SPEC=dflash2`, int4-requantized) | **118 \u002F 126** | 3.14 \u002F 3.34 | ~75% \u002F ~78% |\n| + drafting from the context (`LOOKUP=1`, on by default) | **130** at C1, up to **259** where the model reproduces its context | 3.3-7.8 | |\n| + a 16-token verify block the context fills (`DFLASH_TOKENS=15`) | **133** at C1, up to **381** reproducing context | 3.4-15.0 | |\n\n(Steps 4-6 are the same 8-prompt protocol; greedy is deterministic for a\ngiven server and request order but differs between configs and even with\nprefix-cache hits, so single runs carry ±3-5% on tokens\u002Fstep —\n`bench\u002Frun_benchmarks.sh single` reproduces 113.6 \u002F 118.3 tok\u002Fs decode at C1,\nthe best repeats read 115 \u002F 124.)\nGoing deeper (k=5) loses again: 106 \u002F 105. k=4 is the knee, but on vLLM\n0.27.1's FlashInfer backend (needed for fp8 KV, i.e. for 150k context) four\ndrafts crash the engine with an illegal memory access as soon as one request\nfinishes while another is mid-generation — club-3090 reports the same \"n=4\neventually dies, n=3 stable\" pattern — so `CTX=long` drafts 3 and gives up\n~7%; `CTX=fast` (FlashAttention, bf16 KV, ~64k context, the default) keeps k=4\nand is also the only backend the split-KV attention patch applies to.\n\nTwo things that did *not* help, measured rather than assumed: fine-tuning the\nMTP head on the model's own outputs (KL halves, greedy top-1 on response\ntokens unchanged; `drafter\u002FREADME.md`), and retuning Marlin's tile\nconfiguration for M ≤ 16 on sm86 (3-7% per GEMM in isolation,\nnothing measurable end to end — the remaining gap to peak bandwidth is the\nmemory system's ramp on 16-92 MB reads, not the kernel).\n\n## Setup\n\nYou need: a 24 GB Ampere or newer NVIDIA card, a recent driver, Python 3.12,\n~40 GB disk. Everything below is CPU-safe to run while the GPU does other\nthings. (Or skip the venv and use the container: [docs\u002Fdocker.md](docs\u002Fdocker.md).)\n\n```bash\ngit clone https:\u002F\u002Fgithub.com\u002Fsyv-ai\u002Fqwen38-27b-rtx3090 ~\u002Fqwen-serving\ncd ~\u002Fqwen-serving\n\npython3 -m venv venv\nvenv\u002Fbin\u002Fpip install vllm huggingface_hub hf_transfer ninja\n\n# model, ~19.5 GB\nHF_HUB_ENABLE_HF_TRANSFER=1 venv\u002Fbin\u002Fhf download \\\n  dbirks\u002FQwen3.8-27B-W4A16-AutoRound \\\n  --local-dir models\u002FQwen3.8-27B-W4A16-AutoRound\n\n# requantize lm_head + embeddings + the MTP draft module (CPU only, a few minutes)\nvenv\u002Fbin\u002Fpython quant_lm_head.py models\u002FQwen3.8-27B-W4A16-AutoRound\nvenv\u002Fbin\u002Fpython quant_embed.py   models\u002FQwen3.8-27B-W4A16-AutoRound\nvenv\u002Fbin\u002Fpython quant_mtp.py     models\u002FQwen3.8-27B-W4A16-AutoRound\n# 40k-token draft head for single-user mode (uses the shipped id list)\nvenv\u002Fbin\u002Fpython build_draft_vocab.py models\u002FQwen3.8-27B-W4A16-AutoRound --ids draft_vocab_ids.json\n# single-user \"fast\" variant (~1 GB from the Hub, hardlinks the rest): int4-GPTQ\n# lm_head + drafter; single-user\u002Fstart_qwen.sh picks it up automatically\nvenv\u002Fbin\u002Fpython fetch_fast_variant.py\n# optional: the W4A16 DFlash2 block drafter (1.2 GB) for SPEC=dflash2 single-user mode\nvenv\u002Fbin\u002Fpython fetch_dflash2.py\n\n# patch vllm (all written against 0.27.1; reapply after upgrades)\nfor p in patches\u002F*.patch; do\n  patch -p1 -d venv\u002Flib\u002Fpython3.12\u002Fsite-packages\u002Fvllm \u003C $p\ndone\n# optional: the KVarN 4\u002F2-bit KV cache for 262k context (docs\u002Flong-context.md)\nbash kvarn\u002Finstall.sh\n\n# api key\nopenssl rand -hex 24 > api_key.txt\n```\n\nThen `bash verify.sh --no-server` — it checks the venv and vLLM version, that\nevery patch in `patches\u002F` is actually applied, and that the model has been\nrequantized (lm_head, embeddings, MTP module, draft head). Then pick a mode\nand follow its README:\n\n- **[batch\u002F](batch\u002F)** — throughput. `bash batch\u002Fstart_qwen.sh`\n- **[single-user\u002F](single-user\u002F)** — latency. `bash single-user\u002Fstart_qwen.sh`\n\nFirst start takes a few minutes (torch.compile, CUDA graph capture, flashinfer\nJIT). Test it:\n\n```bash\ncurl http:\u002F\u002Flocalhost:18020\u002Fv1\u002Fchat\u002Fcompletions \\\n  -H \"Authorization: Bearer $(cat api_key.txt)\" \\\n  -H \"Content-Type: application\u002Fjson\" \\\n  -d '{\"model\": \"qwen3.8-27b\",\n       \"messages\": [{\"role\": \"user\", \"content\": \"hej\"}],\n       \"chat_template_kwargs\": {\"enable_thinking\": false}}'\n```\n\nQwen recommends temperature 0.7 \u002F top_p 0.8 for instruct mode, and 1.0 \u002F 0.95\nwith thinking enabled (the default).\n\nTo check the numbers on your own card: `bash verify.sh` (also probes the live\nserver and prints which attention backend and KV pool it came up with), then\n`bash bench\u002Frun_benchmarks.sh batch` or `... single` reproduces the tables\nabove against the running server (`--prefill` and `--long` add the prefill\nmatrix and the long-context rows), `bash bench\u002Freal_rep.sh \u003Ctag> 3 0` repeats\nthe single-stream row, and `python bench\u002Fquality_battery.py \u003Ctag>` the\nperplexity \u002F GSM8K rows.\n\n## The rest\n\n| | |\n|---|---|\n| [docs\u002Foptimizations.md](docs\u002Foptimizations.md) | Every optimization in full: why it was needed, what it measured, which patch implements it. Includes the two speculative-decoding modes (MTP and DFlash2) and the lookup drafter. |\n| [docs\u002Fgotchas.md](docs\u002Fgotchas.md) | 18 things that each cost us hours — read before debugging something that looks like a vLLM bug. |\n| [docs\u002Fquality.md](docs\u002Fquality.md) | IFBench, perplexity and GSM8K per configuration. |\n| [docs\u002Fdocker.md](docs\u002Fdocker.md) | The container image, and an independent WSL2 reproduction. |\n| [docs\u002Flong-context.md](docs\u002Flong-context.md) | 262k context with the KVarN 4\u002F2-bit KV cache, and what vLLM's own per-token-head KV modes are worth here. |\n| [batch\u002F](batch\u002F) · [single-user\u002F](single-user\u002F) | The two serving modes: full benchmark tables, every env knob, systemd units. |\n| [drafter\u002F](drafter\u002F) | How the draft vocabulary, the int4 drafters and the DFlash2 requantization were built — including what did not work. |\n| [kvarn\u002F](kvarn\u002F) | The KVarN 4\u002F2-bit KV cache port. |\n\n## License\n\nApache-2.0, same as the model.\n","该项目提供在单张RTX 3090显卡（24GB显存）上高效部署Qwen3.8-27B大语言模型的完整推理方案，基于vLLM框架并集成多项深度优化。核心功能包括：支持150k–262k超长上下文、OpenAI兼容API、双模式运行（高吞吐batch模式与低延迟single-user模式）、int8张量核GEMM加速、MTP\u002FDFLASH2推测解码、校准int4 lm_head、split-KV验证注意力及KV缓存优化。适用于本地私有化部署场景，如个人开发者调试、小团队内部AI服务、文档问答与长文本编辑等对延迟或上下文长度敏感的轻量级应用。",2,"2026-08-20 02:30:03","CREATED_QUERY"]