[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"project-2671":3},{"id":4,"name":5,"fullName":6,"owner":7,"repo":5,"description":8,"homepage":9,"htmlUrl":10,"language":11,"languages":10,"totalLinesOfCode":10,"stars":12,"forks":13,"watchers":14,"openIssues":15,"contributorsCount":16,"subscribersCount":16,"size":16,"stars1d":17,"stars7d":18,"stars30d":19,"stars90d":16,"forks30d":16,"starsTrendScore":20,"compositeScore":21,"rankGlobal":10,"rankLanguage":10,"license":22,"archived":23,"fork":23,"defaultBranch":24,"hasWiki":23,"hasPages":25,"topics":26,"createdAt":10,"pushedAt":10,"updatedAt":27,"readmeContent":28,"aiSummary":29,"trendingCount":16,"starSnapshotCount":16,"syncStatus":17,"lastSyncTime":30,"discoverSource":31},2671,"ktransformers","kvcache-ai\u002Fktransformers","kvcache-ai","A Flexible Framework for Experiencing Heterogeneous LLM Inference\u002FFine-tune Optimizations","https:\u002F\u002Fkvcache-ai.github.io\u002Fktransformers\u002F",null,"Python",17263,1312,109,442,0,2,17,115,11,44.35,"Apache License 2.0",false,"main",true,[],"2026-06-12 02:00:42","\u003Cdiv align=\"center\">\n  \u003Cp align=\"center\">\n\n\u003Cpicture>\n    \u003Cimg alt=\"KTransformers\" src=\"https:\u002F\u002Fgithub.com\u002Fuser-attachments\u002Fassets\u002Fd5a2492f-a415-4456-af99-4ab102f13f8b\" width=50%>\n\n\u003C\u002Fpicture>\n\n\u003C\u002Fp>\n  \u003Ch3>A Flexible Framework for Experiencing Cutting-edge LLM Inference\u002FFine-tune Optimizations\u003C\u002Fh3>\n  \u003Cstrong>\u003Ca href=\"#-overview\">🎯 Overview\u003C\u002Fa> | \u003Ca href=\"#-inference---high-performance-kt-kernel-serving\">🚀 Inference\u003C\u002Fa> | \u003Ca href=\"#-sft---fine-tuning-with-llama-factory\">🎓 SFT\u003C\u002Fa> | \u003Ca href=\"#-citation\">🔥 Citation\u003C\u002Fa> | \u003Ca href=\"https:\u002F\u002Fgithub.com\u002Fkvcache-ai\u002Fktransformers\u002Fissues\u002F1921\">🚀 Roadmap(2026Q2)\u003C\u002Fa>  \u003C\u002Fstrong>\n\u003C\u002Fdiv>\n\n## 🎯 Overview\n\nKTransformers is a research project focused on efficient inference and fine-tuning of large language models through CPU-GPU heterogeneous computing. The project now exposes two user-facing capabilities from the kt-kernel source tree: [Inference](.\u002Fkt-kernel\u002FREADME.md) and [SFT](.\u002Fdoc\u002Fen\u002FSFT\u002FKTransformers-Fine-Tuning_Quick-Start.md).\n\n## 🔥 Updates\n* **May 6, 2026**: KTransformers at [GOSIM Paris 2026](https:\u002F\u002Fparis2026.gosim.org\u002Fzh\u002Fschedule\u002F) — \"Agentic AI on Edge\" track. We'll present KT's inference performance on consumer hardware.\n* **May 02, 2026**: DeepSeek-V4-Flash Support! ([Tutorial](.\u002Fdoc\u002Fen\u002FDeepSeek-V4-Flash.md))\n* **Apr 30, 2026**: KTransformers v0.6.1 refreshes kt-kernel inference and SFT docs with separate [Inference](.\u002Fkt-kernel\u002FREADME.md) and [SFT Quick Start](.\u002Fdoc\u002Fen\u002FSFT\u002FKTransformers-Fine-Tuning_Quick-Start.md) entry points.\n* **Mar 26, 2026**: Support AVX2-only CPU backend for KT-Kernel inference. ([Tutorial](.\u002Fdoc\u002Fen\u002Fkt-kernel\u002FAVX2-Tutorial.md))\n* **Feb 13, 2026**: MiniMax-M2.5 Day0 Support! ([Tutorial](.\u002Fdoc\u002Fen\u002FMiniMax-M2.5.md))\n* **Feb 12, 2026**: GLM-5 Day0 Support! ([Tutorial](.\u002Fdoc\u002Fen\u002Fkt-kernel\u002FGLM-5-Tutorial.md))\n* **Jan 27, 2026**: Kimi-K2.5 Day0 Support! ([Tutorial](.\u002Fdoc\u002Fen\u002FKimi-K2.5.md)) ([SFT Tutorial](.\u002Fdoc\u002Fen\u002FSFT_Installation_Guide_KimiK2.5.md))\n* **Jan 22, 2026**: Support [CPU-GPU Expert Scheduling](.\u002Fdoc\u002Fen\u002Fkt-kernel\u002Fexperts-sched-Tutorial.md), [Native BF16 and FP8 per channel Precision](.\u002Fdoc\u002Fen\u002Fkt-kernel\u002FNative-Precision-Tutorial.md) and [AutoDL unified fine-tuning and inference](.\u002Fdoc\u002Fzh\u002F【云端低价训推】%20KTransformers%2BAutoDL%2BLlamaFactory：随用随租的低成本超大模型「微调%2B推理」一体化流程.pdf)\n* **Dec 24, 2025**: Support Native MiniMax-M2.1 inference. ([Tutorial](.\u002Fdoc\u002Fen\u002Fkt-kernel\u002FMiniMax-M2.1-Tutorial.md))\n* **Dec 22, 2025**: Support RL-DPO fine-tuning with LLaMA-Factory. ([Tutorial](.\u002Fdoc\u002Fen\u002FSFT\u002FDPO_tutorial.md))\n* **Dec 5, 2025**: Support Native Kimi-K2-Thinking inference ([Tutorial](.\u002Fdoc\u002Fen\u002Fkt-kernel\u002FKimi-K2-Thinking-Native.md))\n* **Nov 6, 2025**: Support Kimi-K2-Thinking inference ([Tutorial](.\u002Fdoc\u002Fen\u002FKimi-K2-Thinking.md)) and fine-tune ([Tutorial](.\u002Fdoc\u002Fen\u002FSFT_Installation_Guide_KimiK2.md))\n* **Nov 4, 2025**: KTransformers Fine-Tuning × LLaMA-Factory Integration. ([Tutorial](.\u002Fdoc\u002Fen\u002FSFT\u002FKTransformers-Fine-Tuning_User-Guide.md))\n* **Oct 27, 2025**: Support Ascend NPU. ([Tutorial](.\u002Fdoc\u002Fzh\u002FDeepseekR1_V3_tutorial_zh_for_Ascend_NPU.md))\n* **Oct 10, 2025**: Integrating into SGLang. ([Roadmap](https:\u002F\u002Fgithub.com\u002Fsgl-project\u002Fsglang\u002Fissues\u002F11425), [Blog](https:\u002F\u002Flmsys.org\u002Fblog\u002F2025-10-22-KTransformers\u002F))\n* **Sept 11, 2025**: Support Qwen3-Next. ([Tutorial](.\u002Fdoc\u002Fen\u002FQwen3-Next.md))\n* **Sept 05, 2025**: Support Kimi-K2-0905. ([Tutorial](.\u002Fdoc\u002Fen\u002FKimi-K2.md))\n* **July 26, 2025**: Support SmallThinker and GLM4-MoE. ([Tutorial](.\u002Fdoc\u002Fen\u002FSmallThinker_and_Glm4moe.md))\n* **July 11, 2025**: Support Kimi-K2. ([Tutorial](.\u002Fdoc\u002Fen\u002FKimi-K2.md))\n* **June 30, 2025**: Support 3-layer (GPU-CPU-Disk) [prefix cache](.\u002Fdoc\u002Fen\u002Fprefix_cache.md) reuse.\n* **May 14, 2025**: Support Intel Arc GPU ([Tutorial](.\u002Fdoc\u002Fen\u002Fxpu.md)).\n* **Apr 29, 2025**: Support AMX-Int8、 AMX-BF16 and Qwen3MoE ([Tutorial](.\u002Fdoc\u002Fen\u002FAMX.md))\n* **Apr 9, 2025**: Experimental support for LLaMA 4 models ([Tutorial](.\u002Fdoc\u002Fen\u002Fllama4.md)).\n* **Apr 2, 2025**: Support Multi-concurrency. ([Tutorial](.\u002Fdoc\u002Fen\u002Fbalance-serve.md)).\n* **Mar 15, 2025**: Support ROCm on AMD GPU ([Tutorial](.\u002Fdoc\u002Fen\u002FROCm.md)).\n* **Mar 5, 2025**: Support unsloth 1.58\u002F2.51 bits weights and [IQ1_S\u002FFP8 hybrid](.\u002Fdoc\u002Fen\u002Ffp8_kernel.md) weights. Support 139K [Longer Context](.\u002Fdoc\u002Fen\u002FDeepseekR1_V3_tutorial.md#v022--v023-longer-context--fp8-kernel) for DeepSeek-V3 and R1 in 24GB VRAM.\n* **Feb 25, 2025**: Support [FP8 GPU kernel](.\u002Fdoc\u002Fen\u002Ffp8_kernel.md) for DeepSeek-V3 and R1; [Longer Context](.\u002Fdoc\u002Fen\u002FDeepseekR1_V3_tutorial.md#v022-longer-context).\n* **Feb 15, 2025**: Longer Context (from 4K to 8K for 24GB VRAM) & Slightly Faster Speed （+15%, up to 16 Tokens\u002Fs), update [docs](.\u002Fdoc\u002Fen\u002FDeepseekR1_V3_tutorial.md) and [online books](https:\u002F\u002Fkvcache-ai.github.io\u002Fktransformers\u002F).\n* **Feb 10, 2025**: Support Deepseek-R1 and V3 on single (24GB VRAM)\u002Fmulti gpu and 382G DRAM, up to 3~28x speedup. For detailed show case and reproduction tutorial, see [here](.\u002Fdoc\u002Fen\u002FDeepseekR1_V3_tutorial.md).\n* **Aug 28, 2024**: Decrease DeepseekV2's required VRAM from 21G to 11G.\n* **Aug 15, 2024**: Update detailed [tutorial](doc\u002Fen\u002Finjection_tutorial.md) for injection and multi-GPU.\n* **Aug 14, 2024**: Support llamfile as linear backend.\n* **Aug 12, 2024**: Support multiple GPU; Support new model: mixtral 8\\*7B  and 8\\*22B; Support q2k, q3k, q5k dequant on gpu.\n* **Aug 9, 2024**: Support windows native.\n\n---\n\n## 📦 Capabilities\n\n### 🚀 [Inference](.\u002Fkt-kernel\u002FREADME.md) - High-Performance kt-kernel Serving\n\nCPU-optimized kernel operations for heterogeneous LLM inference.\n\n\u003Cimg width=\"1049\" height=\"593\" alt=\"image\" src=\"https:\u002F\u002Fgithub.com\u002Fuser-attachments\u002Fassets\u002F68f423da-3f55-4025-bdc9-9ceaa554f00b\" \u002F>\n\n\n**Key Features:**\n- **AMX\u002FAVX Acceleration**: Intel AMX and AVX512\u002FAVX2 optimized kernels for INT4\u002FINT8 quantized inference\n- **MoE Optimization**: Efficient Mixture-of-Experts inference with NUMA-aware memory management\n- **Quantization Support**: CPU-side INT4\u002FINT8 quantized weights, GPU-side GPTQ support\n- **Easy Integration**: Clean Python API for SGLang and other frameworks\n\n**Quick Start:**\n```bash\ncd kt-kernel\npip install .\n```\n\n**Use Cases:**\n\n- CPU-GPU hybrid inference for large MoE models\n- Integration with SGLang for production serving\n- Heterogeneous expert placement (hot experts on GPU, cold experts on CPU)\n\n**Performance Examples:**\n| Model | Hardware Configuration | Total Throughput | Output Throughput |\n|-------|------------------------|------------------|-------------------|\n| DeepSeek-R1-0528 (FP8) | 8×L20 GPU + Xeon Gold 6454S | 227.85 tokens\u002Fs | 87.58 tokens\u002Fs (8-way concurrency) |\n\n👉 **[Full Documentation →](.\u002Fkt-kernel\u002FREADME.md)**\n\n---\n\n### 🎓 [SFT](.\u002Fdoc\u002Fen\u002FSFT\u002FKTransformers-Fine-Tuning_Quick-Start.md) - Fine-Tuning with LLaMA-Factory\n\nKTransformers × LLaMA-Factory integration for ultra-large MoE model fine-tuning.\n\n![KTransformers SFT](https:\u002F\u002Fraw.githubusercontent.com\u002Fkvcache-ai\u002Fktransformers\u002Fmain\u002Fdoc\u002Fassets\u002Fimage-20251011010558909.png)\n\n**Key Features:**\n- **Multi-Backend Support**: CPU\u002FGPU hybrid fine-tuning with INT8\u002FINT4 quantization\n- **Ultra-Large MoE Support**: Fine-tune models like DeepSeek-V3\u002FR1 on limited GPU memory\n- **Faster than ZeRO-Offload**: 6-12x training speedup in benchmarked MoE SFT workloads\n- **Lower CPU Memory**: About half the CPU memory of the previous KT SFT path in the benchmarked setup\n- **LLaMA-Factory Integration**: Seamless integration with popular fine-tuning framework\n\n| Model | GPU Memory | Training Speed | Hardware |\n|-------|------------|----------------|----------|\n| DeepSeek-V3 | ~80GB total | 3.7 it\u002Fs | 4x RTX 4090 |\n| DeepSeek-R1 | ~80GB total | 3.7 it\u002Fs | 4x RTX 4090 |\n| Qwen3-30B-A3B | ~24GB total | 8+ it\u002Fs | 1x RTX 4090 |\n\n**Quick Start:**\n```bash\ncd \u002Fpath\u002Fto\u002FLLaMA-Factory\npip install -e .\npip install -r requirements\u002Fktransformers.txt\nCUDA_VISIBLE_DEVICES=0,1,2,3 accelerate launch \\\n  --config_file examples\u002Fktransformers\u002Faccelerate\u002Ffsdp2_kt_int8.yaml \\\n  src\u002Ftrain.py \\\n  examples\u002Fktransformers\u002Ftrain_lora\u002Fqwen3_5moe_lora_sft_kt.yaml\n```\n\n👉 **[Quick Start →](.\u002Fdoc\u002Fen\u002FSFT\u002FKTransformers-Fine-Tuning_Quick-Start.md)**\n👉 **[Full Documentation →](.\u002Fdoc\u002Fen\u002FSFT\u002FKTransformers-Fine-Tuning_User-Guide.md)**\n\n---\n\n## 🔥 Citation\n\nIf you use KTransformers in your research, please cite our paper:\n\n```bibtex\n@inproceedings{10.1145\u002F3731569.3764843,\n  title = {KTransformers: Unleashing the Full Potential of CPU\u002FGPU Hybrid Inference for MoE Models},\n  author = {Chen, Hongtao and Xie, Weiyu and Zhang, Boxin and Tang, Jingqi and Wang, Jiahao and Dong, Jianwei and Chen, Shaoyuan and Yuan, Ziwei and Lin, Chen and Qiu, Chengyu and Zhu, Yuening and Ou, Qingliang and Liao, Jiaqi and Chen, Xianglin and Ai, Zhiyuan and Wu, Yongwei and Zhang, Mingxing},\n  booktitle = {Proceedings of the ACM SIGOPS 31st Symposium on Operating Systems Principles},\n  year = {2025}\n}\n```\n\n## 👥 Contributors & Team\n\nDeveloped and maintained by:\n- [MADSys Lab](https:\u002F\u002Fmadsys.cs.tsinghua.edu.cn\u002F) @ Tsinghua University\n- [Approaching.AI](http:\u002F\u002Fapproaching.ai\u002F)\n- [9#AISoft](https:\u002F\u002Fgithub.com\u002Faisoft9)\n- Community contributors\n\nWe welcome contributions! Please feel free to submit issues and pull requests.\n\n## 💬 Community & Support\n\n- **GitHub Issues**: [Report bugs or request features](https:\u002F\u002Fgithub.com\u002Fkvcache-ai\u002Fktransformers\u002Fissues)\n- **WeChat Group**: See [archive\u002FWeChatGroup.png](.\u002Farchive\u002FWeChatGroup.png)\n\n## 📦 KT original Code\n\nThe original integrated KTransformers framework has been archived to the [`archive\u002F`](.\u002Farchive\u002F) directory for reference. The project now organizes the two capabilities above from the kt-kernel source tree for clearer documentation and maintenance.\n\nFor the original documentation with full quick-start guides and examples, see:\n- [archive\u002FREADME.md](.\u002Farchive\u002FREADME.md) (English)\n- [archive\u002FREADME_ZH.md](.\u002Farchive\u002FREADME_ZH.md) (中文)\n","kvcache-ai\u002Fktransformers 是一个用于体验异构大语言模型推理和微调优化的灵活框架。该项目的核心功能包括高效的推理服务（支持CPU-GPU异构计算）和基于LLaMA-Factory的微调，同时提供了对多种最新模型的支持如DeepSeek-V4-Flash、MiniMax-M2.5等，并且能够利用AVX2指令集进行加速。此外，它还引入了专家调度机制以优化资源使用效率，并支持BF16与FP8精度格式来提高计算性能。ktransformers适用于需要在边缘设备上运行复杂AI应用或希望降低成本同时保持高性能推理及微调能力的研究人员和开发者。","2026-06-11 02:50:41","top_language"]