[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"project-96052":3},{"id":4,"name":5,"fullName":6,"owner":7,"repo":5,"description":8,"homepage":9,"htmlUrl":9,"language":10,"languages":9,"totalLinesOfCode":9,"stars":11,"forks":12,"watchers":13,"openIssues":14,"contributorsCount":15,"subscribersCount":15,"size":15,"stars1d":15,"stars7d":15,"stars30d":15,"stars90d":15,"forks30d":15,"starsTrendScore":15,"compositeScore":16,"rankGlobal":9,"rankLanguage":9,"license":17,"archived":18,"fork":18,"defaultBranch":19,"hasWiki":18,"hasPages":18,"topics":20,"createdAt":9,"pushedAt":9,"updatedAt":21,"readmeContent":22,"aiSummary":23,"trendingCount":15,"starSnapshotCount":15,"syncStatus":24,"lastSyncTime":25,"discoverSource":26},96052,"FlashRec","sohu-mptc\u002FFlashRec","sohu-mptc","FlashRec is a CUDA-graph engine for generative recommendation: wide beam search (3–5 SID steps, n=50–512+) over a trie-constrained catalog, in-process FP8 serving, and ranked beams on \u002Fv1\u002Fchat\u002Fcompletions.",null,"Python",102,4,1,6,0,42.1,"Apache License 2.0",false,"main",[],"2026-09-20 04:01:32","![FlashRec](assets\u002Fbanner.svg)\n\n![License](https:\u002F\u002Fimg.shields.io\u002Fbadge\u002Flicense-Apache--2.0-blue.svg) ![Python](https:\u002F\u002Fimg.shields.io\u002Fbadge\u002Fpython-3.10%2B-3776AB.svg) ![Platform](https:\u002F\u002Fimg.shields.io\u002Fbadge\u002Fplatform-Linux%20%7C%20CUDA-76B900.svg) ![CI](https:\u002F\u002Fgithub.com\u002Fsohu-mptc\u002FFlashRec\u002Factions\u002Fworkflows\u002Fci.yml\u002Fbadge.svg)\n\n**[Features](#features)** | **[Quickstart](#quickstart)** | **Documentation** | **Examples** | **[Architecture](#architecture)** | **API** | **Evaluation** | **FAQ** | **[简体中文](README.zh-CN.md)**\n\n**An inference engine for generative recommendation, based on mini-sglang:**\n**wide beam search over a semantic-ID catalog, executed inside CUDA graphs.**\n\n---\n\nGenerative recommendation (GenRec) formulates item retrieval as the generation\nof semantic IDs (SIDs): short, fixed-depth token sequences that index an item\ncatalog. A typical request decodes 3–5 steps at a beam width of 50–512 or more,\nwith every continuation restricted to the valid-SID catalog. General-purpose LLM\nengines are optimized for long-sequence, single-path decoding; a wide-beam\nrequest occupies the engine, and throughput does not scale with concurrency.\nFlashRec is designed for this workload.\n\n![SoHuRec-1.7B throughput vs SGLang 0801 and SGLang-master at concurrency 32, FP8](docs\u002Ffigures\u002Fperf-serving-throughput-en.svg)\n\n![OneRec-1.7B throughput vs SGLang 0801, SGLang-master, TensorRT-LLM, and vLLM at n=50 saturation](docs\u002Ffigures\u002Fperf-onerec-qps-en.svg)\n\n## Features\n\nThe engine targets short SID depth, wide beam, and catalog-constrained decoding.\nThroughput scales with beam width `n` and concurrency; the illegal-SID rate is\n0 under the trie constraint. Measurements are in [Evaluation](#evaluation).\n\n\n|                  | FlashRec                                                        | General LLM engines                                                |\n| ---------------- | --------------------------------------------------------------- | ------------------------------------------------------------------ |\n| Depth \u002F width    | 3–5 steps × 50–512+ beams                                       | hundreds–thousands of steps × 1 sequence                           |\n| Vocabulary       | valid-SID continuations (trie)                                  | full, unconstrained                                                |\n| Wide-beam graphs | including expansion, captured at multiples of `n`, one replay   | capture often sized for decode batch; wide beam runs eager         |\n| Illegal SIDs     | **0**                                                           | ~17–27%, filtered after the fact                                   |\n| Concurrency      | beam-row slot budget; rows from different requests share a step | one wide-beam request occupies the engine; throughput roughly flat |\n\n\n**Decoding**\n\n- **CUDA graphs.** Beam widths from 50 to 512+ run in captured graphs, including\nthe beam-expansion step. Capture sizes extend to multiples of the configured\nwidth, so a wide beam is a single graph replay.\n- **SID constraint.** A fused CUDA kernel (dense or CSR-sparse trie) restricts\ndecoding to a catalog of valid semantic IDs. `lm_head` is evaluated only over\nthe SID token range. Illegal SIDs are never candidates, so `invalid_rate` is\n0; open-vocabulary baselines leave ~17–27% of beams illegal.\n- **Scheduling.** Requests are admitted into decode waves between steps under a\nbeam-row slot budget, so beam rows from different requests share a step. A\nradix prefix KV cache with longest-prefix-match scheduling reduces the cost of\nshared prompts; aging prevents starvation.\n\n**Serving**\n\n- **In-process serving.** HTTP, scheduling, weights, and the KV pool share one\nprocess and one address space.\n- **FP8 by default.** W8A8 per-channel weights and `fp8_e4m3` KV, with fused\nRMSNorm→FP8, SiLU→FP8, and QK-RoPE+KV-write. On 1.7B-class checkpoints at\n`n ≤ 128` the main gain is weight and KV-cache footprint; at `n = 512` the two\nprecisions converge on throughput, where the step is bound by bookkeeping.\n- **API.** Ranked beams on `\u002Fv1\u002Fchat\u002Fcompletions`. Deterministic top-*k* at\n`temperature = 0`; Gumbel top-*k* without replacement above it.\n- **Profiler.** `\u002Fstart_profile` \u002F `\u002Fstop_profile` compatible with\n`sglang.bench_serving --profile`.\n\n\n\n## Quickstart\n\n```bash\npip install -e .\n\n# Public GenRec checkpoint used in the documentation.\nhf download OpenOneRec\u002FOneRec-1.7B --local-dir .\u002FOneRec-1.7B\n\n# Build a SID catalog from OpenOneRec RecIF-Bench benchmark_data.\nflashrec --catalog \u002Fpath\u002Fto\u002FOpenOneRec-RecIF\u002Fbenchmark_data\n\n# Serve, constrained to that catalog. Layout is inferred from the tokenizer.\nflashrec --serve --model-path .\u002FOneRec-1.7B --port 8000 --host 0.0.0.0 \\\n  --beam-width 512 --max-tokens 5 \\\n  --sid-vocab-file data\u002Fcatalogs\u002Fsid2pid_beamrec_l4.json\n```\n\n```bash\ncurl -s http:\u002F\u002F127.0.0.1:8000\u002Fv1\u002Fchat\u002Fcompletions \\\n  -H 'Content-Type: application\u002Fjson' \\\n  -d '{\"messages\":[{\"role\":\"user\",\"content\":\"...\"}],\n       \"n\":32,\"max_tokens\":5,\"temperature\":0}'\n```\n\nEach beam is returned as one `choices[]` entry, ranked best-first, with its\nscore under `sglext.sequence_score`. With `--sid-vocab-file` the SID layout is\ninferred from the checkpoint tokenizer. If the catalog is unset, the engine\ndecodes over the full vocabulary (connectivity check only). Wide beam\n(`n ≥ 512`) also requires `--cuda-graph-max-bs 4096 --batch-slots 4096`.\n\nThe server binds `127.0.0.1` and has no authentication. Bind a public address\nonly on a trusted network or behind an authenticating proxy.\n\nRunnable offline and HTTP clients: [Examples](examples\u002F). Flags:\n[Configuration](docs\u002Fconfiguration.md).\n\n## Evaluation\n\nThroughput, retrieval metrics, and HuggingFace codebook overlap were measured on\nan **NVIDIA RTX 5090**. See\n[Evaluation](docs\u002Fbaselines.md). Unless noted otherwise, FlashRec runs FP8 with\na SID trie; the baselines are open-vocabulary. **SGLang-master**\n([PR #31626](https:\u002F\u002Fgithub.com\u002Fsgl-project\u002Fsglang\u002Fpull\u002F31626)) and\n**SGLang 0801**\n(`cswuyg\u002Fsglang` [](https:\u002F\u002Fgithub.com\u002Fcswuyg\u002Fsglang\u002Ftree\u002Ffeature\u002Fbeam_search_update_0801)`feature\u002Fbeam_search_update_0801`)\nare separate engines; do not collapse them into one row.\n\nEvaluation covers:\n\n- **OneRec:** the [OpenOneRec](https:\u002F\u002Fgithub.com\u002FKuaishou-OneRec\u002FOpenOneRec)\nRecIF-Bench video task with\n[OneRec-1.7B](https:\u002F\u002Fhuggingface.co\u002FOpenOneRec\u002FOneRec-1.7B)\n- **SoHuRec-1.7B \u002F SoHuRec-0.6B:** Sohu internal generative-recommendation serving traffic on the corresponding models\n\nSpeed-ups are FlashRec relative to that baseline. Do not divide OneRec QPS by\nSoHuRec QPS (prompt length differs by about 8×).\n\n\n| Baseline          | Setting                             | Relative throughput                                  |\n| ----------------- | ----------------------------------- | ---------------------------------------------------- |\n| **SGLang 0801**   | OneRec, `n=50` saturation           | **1.54×** (recall@32 0.034 both; invalid 0 vs 0.270) |\n| **SGLang-master** | OneRec, `n=50` saturation           | **2.02×** (recall@32 0.034 both; invalid 0 vs 0.260) |\n| TensorRT-LLM      | OneRec, `n=50` saturation           | **2.21×**                                            |\n| vLLM              | OneRec, `n=50` saturation           | **7.2×**                                             |\n| **SGLang 0801**   | SoHuRec-1.7B, `n=50–512` saturation | **2.3–3.0×**                                         |\n| **SGLang-master** | SoHuRec-1.7B, `n=50–512` saturation | **2.5–2.9×**                                         |\n| **SGLang 0801**   | SoHuRec, `n=1000`, concurrency 1    | **2.1–2.2×** (dies at conc ≥ 8)                      |\n| **SGLang-master** | SoHuRec, `n=1000`, concurrency 1    | **2.1–2.2×** (dies at conc ≥ 8)                      |\n\n\nOn OneRec, FlashRec has `invalid_rate = 0`; open-vocabulary engines leave about\n17–27% of beams illegal. The quality gap is **whether SID constraint is on**,\nnot the engine. **Do not cite FlashRec concurrency-1 recall@32** (unique-beam\ncollapse). When citing, state the device, trie vs open-vocabulary, concurrency,\nand sample count.\n\n### OneRec\n\n[OpenOneRec](https:\u002F\u002Fgithub.com\u002FKuaishou-OneRec\u002FOpenOneRec) RecIF-Bench video,\nmodel [OneRec-1.7B](https:\u002F\u002Fhuggingface.co\u002FOpenOneRec\u002FOneRec-1.7B). 5,000\nsamples, `n=50` saturation (highest completed concurrency per engine).\n\n\n| Engine        | Constraint | QPS       | conc | recall@32 (conc=8) | invalid |\n| ------------- | ---------- | --------- | ---- | ------------------ | ------- |\n| **FlashRec**  | SID trie   | **28.08** | 32   | 0.034              | **0**   |\n| SGLang 0801   | open-vocab | 18.20     | 32   | 0.034              | 0.270   |\n| SGLang-master | open-vocab | 13.88     | 16   | 0.034              | 0.260   |\n| TensorRT-LLM  | open-vocab | 12.71     | 8    | 0.034              | 0.259   |\n| vLLM          | open-vocab | 3.89      | 16   | 0.034              | 0.260   |\n\n\n**1.54×** vs SGLang 0801, **2.02×** vs SGLang-master, **2.21×** vs\nTensorRT-LLM, **7.2×** vs vLLM. At `n=1000` FlashRec saturates at **4.62 QPS**;\nSGLang-master and SGLang 0801 only hold concurrency 1 (2.5–3.0). Full matrices:\n[Evaluation](docs\u002Fbaselines.md).\n\nOn SoHuRec serving prompts: at `n=50`, 1.7B FlashRec saturates at **220 QPS**\nand 0.6B at **303 QPS**; at `n=512`, 1.7B FlashRec is **~48 QPS** vs 0801 \u002F\nSGLang-master **~16 QPS**; at `n=1000` FlashRec saturates at **24.6 QPS**\n(1.7B) and **32 QPS** (0.6B), and SGLang-master and SGLang 0801 only hold concurrency 1.\n\n### Numerical match vs HuggingFace\n\nCodebook-constrained beam search (no SID trie) against `transformers`, not\nretrieval recall. HuggingFace is a BF16 reference; FlashRec is measured in\nBF16 and in FP8. See [Evaluation](docs\u002Fbaselines.md); commands in\n[Examples](examples\u002FREADME.md).\n\n- **OneRec-1.7B (RTX 5090):** BF16 matches HuggingFace's best SID at every width\n(`n = 1–512`); beam-set overlap **86–92%**. FP8 swaps top-1 at `n = 1` on a\n0.125-nat near-tie; overlap is **80%** at `n = 20` and **70–76%** at `n ≥ 50`.\nHuggingFace's best sequence is inside the FlashRec beam from `n = 20`.\nThe public checkpoint is not FP8-trained; that drop is on-load quantization,\n**not a framework bug**.\n- **SoHuRec-1.7B \u002F SoHuRec-0.6B (FP8-trained, RTX 5090, FP8 decoder GEMM on both sides):**\nbeam-set overlap **88–96%** \u002F **88–93%**; prefill top-1 is 100% on SoHuRec-1.7B and\n94% on SoHuRec-0.6B (noisy tail, rank corr 0.641; set overlap still about 90%).\n`n = 512` is the mean of the first 3 prompts (HuggingFace OOMs on a longer\nremaining prompt). Prefer an FP8-trained checkpoint for production FP8 serving.\n\n\n\n## Supported models\n\n**Qwen3 dense**: Qwen3-0.6B \u002F 1.7B \u002F 4B \u002F 8B \u002F 14B, OneRec-1.7B, SoHuRec-1.7B \u002F\nSoHuRec-0.6B, and Qwen3-based GenRec checkpoints. Architectural parameters (GQA, `head_dim`, qk-norm, tied\nembeddings) are read from `config.json`; validated at the 1.7B scale. Broader\ndense and MoE coverage is on the [Roadmap](#roadmap).\n\nBoth BF16 checkpoints (quantized on load to W8A8 per-channel under the default\n`--quantization fp8`) and pre-quantized FP8 checkpoints carrying `weight_scale`\nare supported; any other `--quantization` value runs in BF16.\n\nModel size is bound by device memory — roughly up to 14B in FP8 on 32 GB.\nSequence length is capped by `--max-seq-len` (default 4096) and stays inside the\ncheckpoint's native positional range. The engine serves beam search over a\nsemantic-ID catalog.\n\n## Architecture\n\nA request is served in the same process that owns the weights and the KV pool:\nHTTP, scheduling, and the model share one address space.\n\n![Request path: admission, tokenization, radix prefix hit, batched prefill, SID trie expansion, CUDA-graph decode loop, ranked OpenAI response](assets\u002Farchitecture-pipeline-en.png)\n\nThe pipeline admits requests by longest-prefix-match, tokenizes off the GPU\npath, reuses radix prefix KV, prefills only the miss suffix, scores valid SID\nedges through a fused trie kernel, and decodes inside a CUDA-graph replay.\nModule layout follows\n[mini-sglang](https:\u002F\u002Fgithub.com\u002Fsgl-project\u002Fmini-sglang); at runtime the engine\ndepends on `sgl-kernel`, `flashinfer_python`, and `triton`. Request path, design\nnotes, and module map: [Architecture](docs\u002Farchitecture.md).\n\n## Documentation\n\n[Documentation](docs\u002FREADME.md) · [Architecture](docs\u002Farchitecture.md) ·\n[API](docs\u002Fapi.md) · [Configuration](docs\u002Fconfiguration.md) ·\n[Evaluation](docs\u002Fbaselines.md) · [FAQ](docs\u002Ffaq.md) · [Examples](examples\u002F) ·\n[Changelog](CHANGELOG.md)\n\n## Development\n\nInstall the hooks once per clone; they then run on every commit and match what CI\nenforces:\n\n```bash\npip install pre-commit && pre-commit install\npre-commit run --all-files     # check the whole tree\n```\n\nUnit tests run on CPU:\n\n```bash\npython -m pytest\n```\n\nOptional integration checks — live parity against an SGLang beam server, and an\naccuracy comparison against HuggingFace `transformers` — are documented in\n[Configuration](docs\u002Fconfiguration.md), along with the\nprofiling\u002Ftrace interface.\n\n```bash\n# Codebook-constrained vs HuggingFace (needs CUDA + OneRec-1.7B)\nFLASHREC_DIFF_MODEL=.\u002FOneRec-1.7B \\\n  PYTHONPATH=python python -m unittest tests.test_beam_search_diff -v\n\nFLASHREC_DIFF_MODEL=.\u002FOneRec-1.7B \\\nFLASHREC_DIFF_BEAMS=1,20,50,128,512 \\\nFLASHREC_DIFF_QUANT=bf16 \\\n  PYTHONPATH=python python -m unittest tests.test_beam_search_diff.TestBeamSearchDiff -v\n```\n\nNumbers: [Numerical match vs HuggingFace](#numerical-match-vs-huggingface).\n`FLASHREC_DIFF_QUANT=fp8` (default) is the serving path.\n\n## Roadmap\n\n- [ ] Broader GenRec model coverage: Llama \u002F Qwen \u002F GLM dense backbones and sparse models such as Qwen3-MoE\n- [ ] Tensor and expert parallelism for 30B–200B-class MoE and larger dense recommendation models\n- [ ] NVFP4 and native Blackwell storage\n- [ ] Reproducible RecIF coverage across video \u002F ad \u002F product, and published PyPI wheels\n\n\n\n## Contributing\n\nContributions are welcome. See [Contributing](CONTRIBUTING.md) for development\nsetup, tests, and the pull-request process. Participation is governed by the\n[Code of Conduct](CODE_OF_CONDUCT.md). Vulnerabilities go through the private\ndisclosure process in [Security](SECURITY.md).\n\n## Acknowledgements\n\n- [SGLang](https:\u002F\u002Fgithub.com\u002Fsgl-project\u002Fsglang)\n- [mini-sglang](https:\u002F\u002Fgithub.com\u002Fsgl-project\u002Fmini-sglang) — layout reference\n- [OpenOneRec](https:\u002F\u002Fgithub.com\u002FKuaishou-OneRec\u002FOpenOneRec) (Kuaishou) — OneRec-1.7B and RecIF-Bench\n- [cswuyg\u002Fsglang](https:\u002F\u002Fgithub.com\u002Fcswuyg\u002Fsglang\u002Ftree\u002Ffeature\u002Fbeam_search_update_0801) `feature\u002Fbeam_search_update_0801` — beam semantics alignment and accuracy tests\n\n\n\n## Citation\n\n```bibtex\n@software{flashrec,\n  title  = {FlashRec: A Wide-Beam Inference Engine for Generative Recommendation},\n  author = {Wang, Chongyang and {sohu-mptc}},\n  year   = {2026},\n  url    = {https:\u002F\u002Fgithub.com\u002Fsohu-mptc\u002FFlashRec}\n}\n```\n\nGitHub also exposes this via [CITATION.cff](CITATION.cff).\n\n## License\n\nApache-2.0, see [LICENSE](LICENSE).\n","FlashRec 是一个专为生成式推荐（GenRec）设计的 CUDA 图推理引擎，将商品检索建模为语义 ID（SID）序列的生成过程。其核心功能包括：在 Trie 结构约束的商品目录上执行宽束搜索（beam width 50–512+，深度 3–5 步）、全流程 FP8 推理、端到端 CUDA Graph 捕获与复用，以及零非法 SID 率的严格约束解码。相比通用大模型推理框架，它显著提升高并发、宽束、短序列场景下的吞吐量与合规性。适用于电商、内容平台等需低延迟、高精度、目录受限的生成式推荐服务部署。",2,"2026-09-09 02:30:05","CREATED_QUERY"]