[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"project-93152":3},{"id":4,"name":5,"fullName":6,"owner":7,"repo":5,"description":8,"homepage":9,"htmlUrl":10,"language":11,"languages":10,"totalLinesOfCode":10,"stars":12,"forks":13,"watchers":14,"openIssues":15,"contributorsCount":15,"subscribersCount":15,"size":15,"stars1d":15,"stars7d":15,"stars30d":16,"stars90d":15,"forks30d":15,"starsTrendScore":15,"compositeScore":17,"rankGlobal":10,"rankLanguage":10,"license":10,"archived":18,"fork":18,"defaultBranch":19,"hasWiki":20,"hasPages":18,"topics":21,"createdAt":10,"pushedAt":10,"updatedAt":22,"readmeContent":23,"aiSummary":24,"trendingCount":15,"starSnapshotCount":15,"syncStatus":16,"lastSyncTime":25,"discoverSource":26},93152,"wkvm","xiaol\u002Fwkvm","xiaol","Inference for hybrid LLMs: Gemma, RWKV, and all kinds of hybrids.","",null,"Python",55,4,3,0,2,39.3,false,"main",true,[],"2026-07-22 04:02:08","# wkvm\n\n**A hypervisor for model state.** State-native inference engine for RWKV-7 \u002F GDN \u002F Mamba2 and hybrid-linear models — where the primary allocation object is a fixed-size per-request **state slot**, not a paged KV block chain.\n\n> WKV + KVM: states are the VMs — create, snapshot, fork, hibernate, resume, live-migrate. The engine is the hypervisor.\n\n## Routed-Span Demo\n\n[![Gemma routed-span recurrent-mode demo](experiments\u002Fresults\u002Fgemma_routed_span_demo.gif)](experiments\u002Fresults\u002Fgemma_routed_span_demo.mp4)\n\nFull-quality MP4: [`experiments\u002Fresults\u002Fgemma_routed_span_demo.mp4`](experiments\u002Fresults\u002Fgemma_routed_span_demo.mp4)\n\nPrevious ring\u002Fconcurrency demo: [`experiments\u002Fresults\u002Fgemma_wkvm_style_demo.mp4`](experiments\u002Fresults\u002Fgemma_wkvm_style_demo.mp4)\n\n## Routed-span vs vLLM\u002FSGLang\n\nGemma-4-E4B-it on one RTX 4090. vLLM and SGLang are full-KV transformer engines; wkvm routed-span is approximate recurrent mode (`sink16 + ring1024 + routed span bank m64`), so this is a memory\u002Fthroughput comparison, not a same-semantics quality claim. Rows labelled `wkvm-native` use the native wkvm scheduler\u002Farena\u002Frunner\u002Fserver boundary. Older native rows reused HF Gemma module math; current checkpoint-native rows in [`experiments\u002Fresults\u002Fgemma_native_vllm_sglang_current_compare_20260708.md`](experiments\u002Fresults\u002Fgemma_native_vllm_sglang_current_compare_20260708.md) report `uses_hf_transformer_forward=false` and `uses_hf_model_construction=false`. Older rows labelled PoC are retained only as historical context.\n\n**Single long prompt + long output**: 13,824-token prompt + 512-token output, greedy decode, `ignore_eos=True`.\n\n| engine | semantics | facts recovered | prefill+1st | full wall | decode tok\u002Fs | e2e output tok\u002Fs | memory observed | raw result |\n|---|---|---:|---:|---:|---:|---:|---|---|\n| wkvm routed-span m64 | approximate recurrent | yes | 1.380s | 11.237s | 51.8 | 45.6 | 14.67 GiB reserved; 52.9 MiB cache | [`json`](experiments\u002Fresults\u002Flong_gen_13824_512_wkvm_routed_span_m64.json) |\n| vLLM 0.24.0 | full KV | yes | 1.813s | 8.251s | 79.4 | 62.1 | 22.54 GiB device used; 18.42 GiB alloc | [`json`](experiments\u002Fresults\u002Flong_gen_13824_512_vllm.json) |\n| SGLang 0.5.14 | full KV | yes | 1.257s | 8.515s | 70.4 | 60.1 | 21.79 GiB peak device | [`json`](experiments\u002Fresults\u002Flong_gen_13824_512_sglang.json) |\n\n**Distinct long-prompt concurrency**: the `wkvm-native` row is the fresh native engine run at 13,824 context tokens\u002Fsession and 128 decode tokens\u002Fsession. The older PoC row is faster because it used a specialized resident-decode harness, not the native scheduler\u002Fserver contract. vLLM\u002FSGLang rows are the nearest tracked full-KV engine capacity runs at 16,384 context tokens\u002Fsession and 128 decode tokens\u002Fsession, included to anchor the incumbent memory shape.\n\n| engine | workload | resident sessions | aggregate decode | memory\u002Fcapacity note | latency note | source |\n|---|---|---:|---:|---|---|---|\n| wkvm-native routed-span m64 | 13,824 ctx, distinct prompts | **32 green** | **57.9 tok\u002Fs** at B=32 | 17.96 GiB reserved; green means 19 GiB cap with 1 GiB headroom; byte-capped padded decode | p50 72.935s, p95 74.122s at B=32 | [`json`](experiments\u002Fresults\u002Fnative_gemma_bytecap_ctx13824_out128_b32_370m.json), [`frontier`](experiments\u002Fresults\u002Fnative_gemma_throughput_frontier.md) |\n| HF Transformers full-KV | 13,824 ctx, distinct prompts | **2 green**; B=4 over headroom | **52.6 tok\u002Fs** green at B=2; 86.2 tok\u002Fs over headroom at B=4 | 16.88 GiB reserved at B=2; 20.31 GiB at B=4 | p50\u002Fp95 7.457s at B=2; 12.088s at B=4 | [`json`](experiments\u002Fresults\u002Fhf_gemma_batched_chunked_ctx13824_out128_ladder.json), [`frontier`](experiments\u002Fresults\u002Fnative_gemma_throughput_frontier.md) |\n| wkvm routed-span m64 PoC | 13,824 ctx, distinct prompts | **16 green**; 32\u002F48 completed over headroom | **643.9 tok\u002Fs** green; 1039.4 tok\u002Fs over headroom | 15.97 GiB reserved, 913 MiB routed cache; specialized resident-decode harness | p50=p95 3.181s decode | [`json`](experiments\u002Fresults\u002Fgemma_routed_span_distinct_concurrency.json) |\n| vLLM 0.24.0 | nearest 16,384 ctx full-KV run | 9 cap; N=8 measured | 285.6 tok\u002Fs | 21.81 GiB device used; 18.26 GiB alloc | wall 13.42s at N=8; p50\u002Fp95 not recorded | [`bench`](experiments\u002Fresults\u002Fbench_vllm_gemma4e4b.md) |\n| SGLang 0.5.14 | nearest 16,384 ctx full-KV run | 1 true concurrent; N=8 queue-limited | 68 tok\u002Fs | 25,360-token KV pool on this stack | queue-limited; p50\u002Fp95 not recorded | [`bench`](experiments\u002Fresults\u002Fbench_sglang_gemma4e4b.md) |\n\nReadout: routed-span is slower for one exact long generation than vLLM\u002FSGLang full-KV, and the shown routed-span native row is only modestly faster than green HF Transformers on aggregate decode (**57.9 vs 52.6 tok\u002Fs**) while supporting many more resident long sessions (**32 vs 2**). The measured advantage is bounded-memory long-context concurrency when approximate recurrent semantics are acceptable; it is not a replacement for full-KV serving when exact transformer behavior is required.\n\n### Serving-path benchmark\n\nThe direct native benchmark is useful for engine throughput, but vLLM\u002FSGLang production comparisons should use the server path. `wkvm.gemma_server` exposes OpenAI-compatible token-id `\u002Fv1\u002Fcompletions`, token-id `\u002Fv1\u002Fstream` SSE events, blocking `\u002Fv1\u002Fgenerate`, async-style `\u002Fv1\u002Fsubmit` + `\u002Fv1\u002Fstatus\u002F\u003Cid>`, `\u002Fv1\u002Fcancel`, `\u002Fhealth`, and `\u002Fmetrics`. The server now bounds retained completed-request metadata, marks model-step failures as `FINISHED_ERROR`, and supports request timeout cancellation.\n\n```bash\npython -m wkvm.gemma_server --model \u002Fpath\u002Fto\u002Fgemma-4-E4B-it --slots 32 --port 8000 \\\n  --max-queue 128 --request-timeout-s 600 --max-completed-requests 4096\n\npython experiments\u002Fwkvm_serving_bench.py --backend openai-completions \\\n  --engine wkvm-native-openai-completions --url http:\u002F\u002F127.0.0.1:8000 \\\n  --served-model gemma-4-E4B-it --ctx 13824 --out 128 \\\n  --concurrency 1,2,4,8,16,32 \\\n  --json experiments\u002Fresults\u002Fwkvm_serving_ctx13824_out128.json\n\npython experiments\u002Fwkvm_serving_bench.py --backend openai-completions \\\n  --engine vllm-http-stream --url http:\u002F\u002F127.0.0.1:8001 \\\n  --served-model gemma-4-E4B-it --ctx 13824 --out 128 \\\n  --concurrency 1,2,4,8,16,32 \\\n  --json experiments\u002Fresults\u002Fvllm_serving_ctx13824_out128.json\n\npython experiments\u002Fwkvm_serving_bench.py --backend openai-completions \\\n  --engine sglang-http-stream --url http:\u002F\u002F127.0.0.1:8002 \\\n  --served-model gemma-4-E4B-it --ctx 13824 --out 128 \\\n  --concurrency 1,2,4,8,16,32 \\\n  --json experiments\u002Fresults\u002Fsglang_serving_ctx13824_out128.json\n\npython experiments\u002Fgemma_bench_report.py experiments\u002Fresults\u002Fwkvm_serving_ctx13824_out128.json \\\n  experiments\u002Fresults\u002Fvllm_serving_ctx13824_out128.json \\\n  experiments\u002Fresults\u002Fsglang_serving_ctx13824_out128.json \\\n  --out experiments\u002Fresults\u002Fserving_compare_ctx13824_out128.md\n```\n\nThis records TTFT, ITL, end-to-end latency, success\u002Ferror counts, and HTTP-stream output throughput. Fair rows must keep the same `ctx`, `out`, prompt-length mode, greedy\u002Fignore-eos behavior, and concurrency ladder. The OpenAI-compatible path sends token-id prompts to `\u002Fv1\u002Fcompletions`; wkvm and vLLM can return streamed `token_ids`, while SGLang currently relies on usage\u002Ftext\u002Flogprob fields unless its server exposes token IDs. This is the correct next comparison shape for vLLM\u002FSGLang serving benchmarks; it still does not remove the deeper gaps of full non-HF Gemma kernel ownership, CUDA graphs\u002Fstatic buffers for the Gemma hot path, and broader OpenAI compatibility.\n\n## Why\n\nFor linear\u002Fhybrid models, per-request memory is **constant and tiny** (an RWKV-7 7B state is ~20MB — ~1000× smaller than long-context KV). Built around that physics instead of paged KV, an engine gets:\n\n- **Exact admission** — scheduling is counting free slots; no fragmentation, no watermark math.\n- **Uniform decode batches** — whole-step CUDA graphs, Albatross-class throughput scaling.\n- **Sessions as objects** — hibernate\u002Fresume in one transfer, fork in one slot copy, migrate in one RDMA write.\n- **The Durable State API** — named, versioned, forkable, exportable, **mutable** state handles (`\u002Fv1\u002Fstates`). Mutable state violates the `state ≡ f(token-prefix)` invariant that paged-KV cache indexes are built on; it is the capability incumbent engines structurally cannot follow.\n\nFull-attention layers in hybrid models (Qwen3-Next \u002F Kimi-Linear class) run in a deliberately simple paged **guest pool**. Pure transformers are supported as guests for parity — and, later, at constant footprint via **recurrent mode** (sink + KV ring + a segmented bank of RWKV-7 states over evicted context; see `docs\u002FRECURRENT_MODE.md`).\n\n## Design documents\n\n- [`docs\u002FANGLE.md`](docs\u002FANGLE.md) — the full vLLM\u002FSGLang architecture map (724k\u002F626k LOC audited), what to steal, what to refuse, the nine candidate angles and why this one survived adversarial review.\n- [`docs\u002FRECURRENT_MODE.md`](docs\u002FRECURRENT_MODE.md) — constant-footprint transformer serving via multi-state (layer-wise and context-length-wise) memory banks.\n- [`ROADMAP.md`](ROADMAP.md) — milestones.\n\n## Status\n\n**M3 — the Durable State API is real and measured** (`experiments\u002Fresults\u002Fm3_results.md`): named\u002Fversioned\u002Fforkable\u002F**mutable** state handles over a tiered StateStore (GPU slot \u002F pinned host \u002F NVMe safetensors) with `\u002Fv1\u002Fstates`. Demos on one 4090: 2000 hibernated sessions at 2.29 MiB each resuming in **p50 8.2ms \u002F p99 9.5ms** (16\u002F16 exactness); an agent session surviving a **real process restart bit-exactly**, forked 64× at 8.5ms\u002Ffork, and mutated (`decay`) with recorded provenance while its parent stays intact — the operation prefix-keyed caches cannot represent. Trainer\u002Fserver logprob drift quantified honestly (mean 2e-2 at bf16 across fused-vs-chunked paths; trainer-identical chunked rescoring available by construction).\n\n**M2**: the engine serves RWKV-7 end to end — no-phases scheduler driving FLA-kernel decode from arena state slots, continuous batching (batch-vs-sequential token-identical), per-batch-bucket CUDA graphs, 8.1k tok\u002Fs at B=256 on a 4090 (1.5B). Plus a measured PoC of **recurrent mode** for transformers on gemma-4-E4B (constant footprint + segmented state bank; needle recall to 32k). See [`docs\u002FCOMPARISON.md`](docs\u002FCOMPARISON.md) for head-to-head numbers vs vLLM, SGLang, and Albatross on the same GPU, and `experiments\u002Fresults\u002F` for all raw tables.\n\n```bash\npython -m unittest discover -s tests -v\n```\n\n## Layout\n\n```\nwkvm\u002Fcore\u002Fconfig.py     # typed specs: state families, slot layouts, engine limits\nwkvm\u002Fcore\u002Frequest.py    # Request lifecycle, the num_computed_tokens invariant\nwkvm\u002Fcore\u002Farena.py      # StateArena: per-family slot allocator, exact admission\nwkvm\u002Fcore\u002Fscheduler.py  # no-phases continuous-batching scheduler\ntests\u002F                  # CPU-only invariant tests\n```\n\n## License\n\nApache-2.0 (LICENSE file pending).\n","wkvm 是一个面向状态的大型语言模型推理引擎，专为 RWKV、Gemma 及各类混合线性架构（如 GDN、Mamba2）设计。其核心特点是将模型状态（state slot）作为一级抽象单元，支持状态创建、快照、分叉、休眠与迁移等类虚拟机操作，采用 routed-span 等近似循环模式替代传统 KV 缓存，在长上下文场景下显著降低显存占用。适用于需高并发、低内存开销的长文本生成、状态敏感推理及边缘\u002F单卡部署等场景。","2026-07-12 02:30:05","CREATED_QUERY"]