[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"project-74050":3},{"id":4,"name":5,"fullName":6,"owner":7,"repo":5,"description":8,"homepage":9,"htmlUrl":10,"language":11,"languages":10,"totalLinesOfCode":10,"stars":12,"forks":13,"watchers":14,"openIssues":15,"contributorsCount":16,"subscribersCount":16,"size":16,"stars1d":17,"stars7d":18,"stars30d":19,"stars90d":16,"forks30d":16,"starsTrendScore":20,"compositeScore":21,"rankGlobal":10,"rankLanguage":10,"license":22,"archived":23,"fork":23,"defaultBranch":24,"hasWiki":23,"hasPages":23,"topics":25,"createdAt":10,"pushedAt":10,"updatedAt":26,"readmeContent":27,"aiSummary":28,"trendingCount":16,"starSnapshotCount":16,"syncStatus":29,"lastSyncTime":30,"discoverSource":31},74050,"MiniCPM","OpenBMB\u002FMiniCPM","OpenBMB","MiniCPM5-1B: A SOTA 1B on-device LLM, small yet powerful.","",null,"Jupyter Notebook",9419,619,83,12,0,23,115,561,69,114.38,"Apache License 2.0",false,"main",[],"2026-06-12 04:01:12","\u003Cdiv align=\"center\">\n\u003Cimg src=\".\u002Fassets\u002Fminicpm_logo.png\" width=\"500em\" >\u003C\u002Fimg> \n\u003C\u002Fdiv>\n\n\u003Ch4 align=\"center\">\n    \u003Cp>\n        \u003Ca href=\"https:\u002F\u002Fgithub.com\u002FOpenBMB\u002FMiniCPM\u002Fblob\u002Fmain\u002FREADME-cn.md\">中文\u003C\u002Fa> | \u003Cb>English\u003C\u002Fb>\n    \u003Cp>\n\u003C\u002Fh4>\n\n\u003Cp align=\"center\">\n\u003Ca href=\"https:\u002F\u002Farxiv.org\u002Fpdf\u002F2506.07900\" target=\"_blank\">MiniCPM Paper\u003C\u002Fa> |\n\u003Ca href=\"https:\u002F\u002Fmodelbest.feishu.cn\u002Fwiki\u002FD2tFw8Pcsi5CIzkaHNacLK64npg\" target=\"_blank\">MiniCPM Wiki (in Chinese)\u003C\u002Fa> |\n\u003Ca href=\"https:\u002F\u002Fgithub.com\u002FOpenBMB\u002FMiniCPM-V\u002F\" target=\"_blank\">MiniCPM-V Repo\u003C\u002Fa> |\nJoin our \u003Ca href=\"https:\u002F\u002Fdiscord.gg\u002F3cGQn9b3YM\" target=\"_blank\">discord\u003C\u002Fa> and \u003Ca href=\"https:\u002F\u002Fgithub.com\u002FOpenBMB\u002FMiniCPM\u002Fblob\u002Fmain\u002Fassets\u002Fwechat.jpg\" target=\"_blank\">WeChat\u003C\u002Fa> |\n\u003Ca href=\"https:\u002F\u002Fmp.weixin.qq.com\u002Fs\u002FKIhH2nCURBXuFXAtYRpuXg?poc_token=HBIsUWijxino8oJ5s6HcjcfXFRi0Xj2LJlxPYD9c\">Join Us\u003C\u002Fa>\n\u003C\u002Fp>\n\n> [!NOTE]\n> ### 🏆 2026 Sparse Operator Acceleration & Race (SOAR) is Now Live!\n>\n> **The MiniCPM-SALA architecture is just the beginning. Realizing its full potential requires deep system-level synergy and cross-layer compilation optimization.**\n>\n> OpenBMB, in collaboration with **SGLang** and **NVIDIA**, invites global geeks to tackle the limits of **9B-scale, 1M-token inference** on a dedicated **NVIDIA 6000D** environment.\n>\n> * 💰 **Prize Pool:** >$100,000 USD (Top Prize: **$89,000**)\n> * 🚀 **Goal:** Optimize single and multi-batch performance via cross-layer compilation.\n>\n> 👉 **[Learn more and Register](https:\u002F\u002Fsoar.openbmb.cn\u002F)**\n\n## Changelog🔥\n- [2026.02.11] **[MiniCPM-SALA](https:\u002F\u002Fhuggingface.co\u002Fopenbmb\u002FMiniCPM-SALA)** is released! This is the first large-scale hybrid model effectively integrating sparse and linear attention for million-token context modeling. 🔥🔥🔥\n- [2025.09.29] **[InfLLM-V2 paper](https:\u002F\u002Farxiv.org\u002Fabs\u002F2509.24663) is released!** We can train a sparse attention model with only 5B long-text tokens. 🔥🔥🔥\n- [2025.09.05] **[MiniCPM4.1 series](https:\u002F\u002Fhuggingface.co\u002Fcollections\u002Fopenbmb\u002Fminicpm-4-6841ab29d180257e940baa9b)** are released! This series is a hybrid reasoning model with trainable sparse attention, which can be used in both deep reasoning mode and non-reasoning mode. 🔥🔥🔥\n- [2025.06.06] Released [**MiniCPM4**](https:\u002F\u002Fhuggingface.co\u002Fcollections\u002Fopenbmb\u002Fminicpm-4-6841ab29d180257e940baa9b)! This model achieves ultimate efficiency improvements while maintaining optimal performance at the same scale! It can achieve over 5x generation acceleration on typical end-side chips!\n- [2024.09.05] We release [**MiniCPM3-4B**](https:\u002F\u002Fhuggingface.co\u002Fopenbmb\u002FMiniCPM3-4B)! This model outperforms Phi-3.5-mini-instruct and GPT-3.5-Turbo-0125 and is comparable to several models with 7B-9B parameters like Llama3.1-8B-Instruct, Qwen2-7B-Instruct, and GLM-4-9B-Chat.\n- [2024.07.05] Released [**MiniCPM-S-1B**](https:\u002F\u002Fhuggingface.co\u002Fopenbmb\u002FMiniCPM-S-1B-sft)! This model achieves an average sparsity of 87.89% in the FFN layer, reducing FFN FLOPs by 84%, while maintaining downstream task performance.\n- [2024.04.11] Released [**MiniCPM-2B-128k**](https:\u002F\u002Fhuggingface.co\u002Fopenbmb\u002FMiniCPM-2B-128k), [**MiniCPM-MoE-8x2B**](https:\u002F\u002Fhuggingface.co\u002Fopenbmb\u002FMiniCPM-MoE-8x2B) and [**MiniCPM-1B**](https:\u002F\u002Fhuggingface.co\u002Fopenbmb\u002FMiniCPM-1B-sft-bf16)! Click [here](https:\u002F\u002Fopenbmb.vercel.app\u002F) to read our technical blog.\n- [2024.02.01] Released [**MiniCPM-2B**](https:\u002F\u002Fhuggingface.co\u002Fopenbmb\u002FMiniCPM-2B-sft-bf16)! This model performs similarly to Mistral-7B on public benchmarks (with better performance in Chinese, math, and code abilities) and overall outperforms models like Llama2-13B, MPT-30B, and Falcon-40B.\n\n## Quick Links\n\n- [Changelog🔥](#changelog)\n- [Quick Links](#quick-links)\n- [Model Downloads](#model-downloads)\n- [MiniCPM-SALA](#minicpm-sala)\n- [MiniCPM4 and MiniCPM4.1 Series](#minicpm4-and-minicpm41-series)\n    - [Highlights](#highlights)\n    - [Introduction](#introduction)\n  - [Evaluation Results](#evaluation-results)\n    - [Efficiency Evaluation](#efficiency-evaluation)\n    - [Comprehensive Evaluation](#comprehensive-evaluation)\n    - [Long Text Evaluation](#long-text-evaluation)\n  - [Inference](#inference)\n    - [Hybird Reasoning Mode](#hybird-reasoning-mode)\n    - [HuggingFace](#huggingface)\n    - [vLLM](#vllm)\n      - [Speculative Decoding](#speculative-decoding)\n        - [1. Download MiniCPM4.1 Draft Model](#1-download-minicpm41-draft-model)\n        - [2. Install EAGLE3-Compatible vLLM](#2-install-eagle3-compatible-vllm)\n        - [3. Launch vLLM Server with Speculative Decoding](#3-launch-vllm-server-with-speculative-decoding)\n        - [4. Client Usage Example](#4-client-usage-example)\n        - [vLLM Configuration Parameters](#vllm-configuration-parameters)\n      - [Standard Inference (Without Speculative Decoding)](#standard-inference-without-speculative-decoding)\n    - [SGLang](#sglang)\n      - [Speculative Decoding](#speculative-decoding-1)\n        - [1. Download MiniCPM4.1 Draft Model](#1-download-minicpm41-draft-model-1)\n        - [2. Install EAGLE3-Compatible SGLang](#2-install-eagle3-compatible-sglang)\n        - [3. Launch SGLang Server with Speculative Decoding](#3-launch-sglang-server-with-speculative-decoding)\n        - [4. Client Usage](#4-client-usage)\n        - [Configuration Parameters](#configuration-parameters)\n      - [Standard Inference (Without Speculative Decoding)](#standard-inference-without-speculative-decoding-1)\n    - [CPM.cu](#cpmcu)\n    - [llama.cpp and Ollama](#llamacpp-and-ollama)\n      - [llama.cpp](#llamacpp)\n      - [Ollama](#ollama)\n  - [BitCPM4: Quantization](#bitcpm4-quantization)\n    - [BitCPM4 Evaluation](#bitcpm4-evaluation)\n    - [BitCPM4 Inference](#bitcpm4-inference)\n  - [MiniCPM4 Application](#minicpm4-application)\n    - [MiniCPM4-Survey: Trustworthy Survey Generation](#minicpm4-survey-trustworthy-survey-generation)\n      - [Demo and Quick Start](#demo-and-quick-start)\n      - [Performance Evaluation](#performance-evaluation)\n    - [MiniCPM4-MCP: Tool Use with Model Context Protocol](#minicpm4-mcp-tool-use-with-model-context-protocol)\n      - [Demo](#demo)\n      - [Performance Evaluation](#performance-evaluation-1)\n    - [MiniCPM Intel AIPC Client: A New Edge Large Model Powerhouse](#minicpm-intel-aipc-client-a-new-edge-large-model-powerhouse)\n      - [Key Features](#key-features)\n      - [System Requirements](#system-requirements)\n      - [Download](#download)\n- [LICENSE](#license)\n    - [Model LICENSE](#model-license)\n    - [Statement](#statement)\n- [Institutions](#institutions)\n- [Citation](#citation)\n\n\n## Model Downloads\n\n  | HuggingFace | ModelScope |\n  |-------------|------------|\n  | [MiniCPM-SALA](https:\u002F\u002Fhuggingface.co\u002Fopenbmb\u002FMiniCPM-SALA) | [MiniCPM-SALA](https:\u002F\u002Fwww.modelscope.cn\u002Fmodels\u002FOpenBMB\u002FMiniCPM-SALA) |\n  | [MiniCPM4.1-8B](https:\u002F\u002Fhuggingface.co\u002Fopenbmb\u002FMiniCPM4.1-8B) | [MiniCPM4.1-8B](https:\u002F\u002Fwww.modelscope.cn\u002Fmodels\u002FOpenBMB\u002FMiniCPM4.1-8B) |\n  | [MiniCPM4.1-8B-GPTQ](https:\u002F\u002Fhuggingface.co\u002Fopenbmb\u002FMiniCPM4.1-8B-GPTQ) | [MiniCPM4.1-8B-GPTQ](https:\u002F\u002Fwww.modelscope.cn\u002Fopenbmb\u002FMiniCPM4.1-8B-GPTQ) | \n  | [MiniCPM4.1-8B-AutoAWQ](https:\u002F\u002Fhuggingface.co\u002Fopenbmb\u002FMiniCPM4.1-8B-AutoAWQ) | [MiniCPM4.1-8B-AutoAWQ](https:\u002F\u002Fwww.modelscope.cn\u002Fopenbmb\u002FMiniCPM4.1-8B-AutoAWQ) | \n  | [MiniCPM-4.1-8B-Marlin](https:\u002F\u002Fhuggingface.co\u002Fopenbmb\u002FMiniCPM-4.1-8B-Marlin) | [MiniCPM-4.1-8B-Marlin](https:\u002F\u002Fwww.modelscope.cn\u002Fopenbmb\u002FMiniCPM-4.1-8B-Marlin) | \n  | [MiniCPM4.1-8B-GGUF](https:\u002F\u002Fhuggingface.co\u002Fopenbmb\u002FMiniCPM4.1-8B-GGUF) | [MiniCPM4.1-8B-GGUF](https:\u002F\u002Fwww.modelscope.cn\u002Fopenbmb\u002FMiniCPM4.1-8B-GGUF) | \n  | [MiniCPM4.1-8B-MLX](https:\u002F\u002Fhuggingface.co\u002Fopenbmb\u002FMiniCPM4.1-8B-MLX) | [MiniCPM4.1-8B-MLX](https:\u002F\u002Fwww.modelscope.cn\u002Fopenbmb\u002FMiniCPM4.1-8B-MLX) | \n  | [MiniCPM4.1-8B-Eagle3](https:\u002F\u002Fhuggingface.co\u002Fopenbmb\u002FMiniCPM4.1-8B-Eagle3) | [MiniCPM4.1-8B-Eagle3](https:\u002F\u002Fwww.modelscope.cn\u002Fopenbmb\u002FMiniCPM4.1-8B-Eagle3) | \n  | [MiniCPM4-8B](https:\u002F\u002Fhuggingface.co\u002Fopenbmb\u002FMiniCPM4-8B)    | [MiniCPM4-8B](https:\u002F\u002Fwww.modelscope.cn\u002Fmodels\u002FOpenBMB\u002FMiniCPM4-8B) |\n  | [MiniCPM4-0.5B](https:\u002F\u002Fhuggingface.co\u002Fopenbmb\u002FMiniCPM4-0.5B) | [MiniCPM4-0.5B](https:\u002F\u002Fwww.modelscope.cn\u002Fmodels\u002FOpenBMB\u002FMiniCPM4-0.5B) |\n  | [BitCPM4-1B](https:\u002F\u002Fhuggingface.co\u002Fopenbmb\u002FBitCPM4-1B)        | [BitCPM4-1B](https:\u002F\u002Fwww.modelscope.cn\u002Fmodels\u002FOpenBMB\u002FBitCPM4-1B) |\n  | [BitCPM4-0.5B](https:\u002F\u002Fhuggingface.co\u002Fopenbmb\u002FBitCPM4-0.5B)    | [BitCPM4-0.5B](https:\u002F\u002Fwww.modelscope.cn\u002Fmodels\u002FOpenBMB\u002FBitCPM4-0.5B) |\n  | [MiniCPM4-Survey](https:\u002F\u002Fhuggingface.co\u002Fopenbmb\u002FMiniCPM4-Survey) | [MiniCPM4-Survey](https:\u002F\u002Fwww.modelscope.cn\u002Fmodels\u002FOpenBMB\u002FMiniCPM4-Survey) |\n  | [MiniCPM4-MCP](https:\u002F\u002Fhuggingface.co\u002Fopenbmb\u002FMiniCPM4-MCP)  | [MiniCPM4-MCP](https:\u002F\u002Fwww.modelscope.cn\u002Fmodels\u002FOpenBMB\u002FMiniCPM4-MCP) |\n\n\n\u003Cdetails>\n\u003Csummary>📋 Click to view all MiniCPM series models\u003C\u002Fsummary>\n\n  | HuggingFace | ModelScope |\n  |-------------|------------|\n  | [MiniCPM4-8B-Eagle-FRSpec](https:\u002F\u002Fhuggingface.co\u002Fopenbmb\u002FMiniCPM4-8B-Eagle-FRSpec) | [MiniCPM4-8B-Eagle-FRSpec](https:\u002F\u002Fwww.modelscope.cn\u002Fmodels\u002FOpenBMB\u002FMiniCPM4-8B-Eagle-FRSpec) |\n  | [MiniCPM4-8B-Eagle-FRSpec-QAT](https:\u002F\u002Fhuggingface.co\u002Fopenbmb\u002FMiniCPM4-8B-Eagle-FRSpec-QAT) | [MiniCPM4-8B-Eagle-FRSpec-QAT](https:\u002F\u002Fwww.modelscope.cn\u002Fmodels\u002FOpenBMB\u002FMiniCPM4-8B-Eagle-FRSpec-QAT) |\n  | [MiniCPM4-8B-Eagle-vLLM](https:\u002F\u002Fhuggingface.co\u002Fopenbmb\u002FMiniCPM4-8B-Eagle-vLLM) | [MiniCPM4-8B-Eagle-vLLM](https:\u002F\u002Fwww.modelscope.cn\u002Fmodels\u002FOpenBMB\u002FMiniCPM4-8B-Eagle-vLLM) |\n  | [MiniCPM4-8B-marlin-Eagle-vLLM](https:\u002F\u002Fhuggingface.co\u002Fopenbmb\u002FMiniCPM4-8B-marlin-Eagle-vLLM) | [MiniCPM4-8B-marlin-Eagle-vLLM](https:\u002F\u002Fwww.modelscope.cn\u002Fmodels\u002FOpenBMB\u002FMiniCPM4-8B-marlin-Eagle-vLLM) |\n  | [MiniCPM4-0.5B-QAT-Int4-unquantized](https:\u002F\u002Fhuggingface.co\u002Fopenbmb\u002FMiniCPM4-0.5B-QAT-Int4-unquantized) | [MiniCPM4-0.5B-QAT-Int4-unquantized](https:\u002F\u002Fmodelscope.cn\u002Fmodels\u002FOpenBMB\u002FMiniCPM4-0.5B-QAT-Int4-unquantized) |\n  | [MiniCPM4-0.5B-QAT-Int4-GPTQ-format](https:\u002F\u002Fhuggingface.co\u002Fopenbmb\u002FMiniCPM4-0.5B-QAT-Int4-GPTQ-format) | [MiniCPM4-0.5B-QAT-Int4-GPTQ-format](https:\u002F\u002Fmodelscope.cn\u002Fmodels\u002FOpenBMB\u002FMiniCPM4-0.5B-QAT-Int4-GPTQ-format) |\n  | [MiniCPM3-4B](https:\u002F\u002Fhuggingface.co\u002Fopenbmb\u002FMiniCPM3-4B) | [MiniCPM3-4B](https:\u002F\u002Fwww.modelscope.cn\u002Fmodels\u002FOpenBMB\u002FMiniCPM3-4B) |\n  | [MiniCPM-2B-sft](https:\u002F\u002Fhuggingface.co\u002Fopenbmb\u002FMiniCPM-2B-sft-bf16) | [MiniCPM-2B-sft](https:\u002F\u002Fmodelscope.cn\u002Fmodels\u002FOpenBMB\u002FminiCPM-bf16)|\n  | [MiniCPM-2B-dpo](https:\u002F\u002Fhuggingface.co\u002Fopenbmb\u002FMiniCPM-2B-dpo-bf16) | [MiniCPM-2B-dpo](https:\u002F\u002Fmodelscope.cn\u002Fmodels\u002FOpenBMB\u002FMiniCPM-2B-dpo-bf16\u002Fsummary) |\n  | [MiniCPM-2B-128k](https:\u002F\u002Fhuggingface.co\u002Fopenbmb\u002FMiniCPM-2B-128k) | [MiniCPM-2B-128k](https:\u002F\u002Fmodelscope.cn\u002Fmodels\u002Fopenbmb\u002FMiniCPM-2B-128k\u002Fsummary) |\n  | [MiniCPM-MoE-8x2B](https:\u002F\u002Fhuggingface.co\u002Fopenbmb\u002FMiniCPM-MoE-8x2B) | [MiniCPM-MoE-8x2B](https:\u002F\u002Fmodelscope.cn\u002Fmodels\u002FOpenBMB\u002FMiniCPM-MoE-8x2B) |\n  | [MiniCPM-1B](https:\u002F\u002Fhuggingface.co\u002Fopenbmb\u002FMiniCPM-1B-sft-bf16) | [MiniCPM-1B](https:\u002F\u002Fmodelscope.cn\u002Fmodels\u002FOpenBMB\u002FMiniCPM-1B-sft-bf16) |\n  | [MiniCPM-S-1B](https:\u002F\u002Fhuggingface.co\u002Fopenbmb\u002FMiniCPM-S-1B-sft) | [MiniCPM-S-1B](https:\u002F\u002Fmodelscope.cn\u002Fmodels\u002FOpenBMB\u002FMiniCPM-S-1B-sft) |\n\u003C\u002Fdetails>\n\n## MiniCPM-SALA\n#### Highlights\n\nMiniCPM-SALA (Sparse Attention and Linear Attention) is the first large-scale hybrid model effectively integrating sparse and linear attention for million-token context modeling\n\n✅ Innovative Hybrid Architecture: Synergizes 25% Sparse Attention (InfLLM-v2) for high-fidelity long context modeling with 75% Linear Attention (Lightning Attention) for global efficiency.\n\n✅ Shattering Efficiency Walls: Breaks the \"Compute Wall\" and the \"Memory Wall,\" achieving 3.5× inference speed and significantly lower KV-cache overhead compared to dense baselines. \n\n✅ Million-Token Context: Empowered by HyPE (Hybrid Positional Embedding), it scales to 1M+ tokens while maintaining strong length generalization. \n\n✅ HALO Adaptation: Utilizes Hybrid Attention via Layer Optimization (HALO), a novel distillation recipe that effectively transfers dense attention capabilities to the hybrid architecture, avoiding the severe performance degradation typical of pure linear models.\n\n#### Introduction\n\nMiniCPM-SALA is an efficient hybrid model in which 25% of the layers adopt [InfLLM-V2](https:\u002F\u002Farxiv.org\u002Fabs\u002F2509.24663) and the remaining 75% utilize Lightning Attention. This architecture enables inference of one million tokens on consumer GPUs such as the NVIDIA RTX 5090.\n\n- **SALA Hybrid Attention Mechanism**\n  - Integrates 25% InfLLM-V2 and 75% Lightning Attention, effectively leveraging the granular focus of sparse attention for local details and the high efficiency of linear attention for broad context.\n\n- **Transformer-to-Hybrid Continue Training**\n  - Circumvents the inefficiencies of cold-start training by performing an architectural transformation on the pre-trained weights, thereby reducing the total training budget to approximately 25% relative to training a comparable model from scratch.\n\n- **[HyPE](https:\u002F\u002Farxiv.org\u002Fabs\u002F2601.22156) (Hybrid Positional Encoding)**\n  - Harmonizes the performance across both short and long contexts, which can maintain general capabilities (e.g., knowledge, mathematics, and coding) comparable to modern full-attention models like Qwen3-8B and achieve substantial advantages across multiple long-context benchmarks.\n\n- **Efficient Inference on Long Sequences**\n  - Achieves up to 3.5x the inference speed of Qwen3-8B at a sequence length of 256K tokens on A6000D, supports inference at context lengths of up to 1M tokens on both NVIDIA A6000D and 5090 GPUs, whereas Qwen3-8B fails at this length due to out-of-memory (OOM) errors.\n\n### Evaluation Results\n\n#### Efficiency Evaluation\n\nWe benchmarked MiniCPM-SALA (9B) against Qwen3-8B on NVIDIA A6000D and RTX 5090 GPUs to evaluate inference speed and memory efficiency. The results demonstrate a significant performance leap: MiniCPM-SALA not only achieves up to a 2.5x speedup in time-to-first-token (TTFT) but also overcomes the memory bottlenecks of full-attention architectures. While Qwen3-8B suffers from OOM errors at extended lengths, MiniCPM-SALA successfully scales to 1M-token contexts on a single consumer-grade RTX 5090, effectively democratizing ultra-long context inference on edge hardware.\n\n![inference_speed_a6000d](.\u002Fassets\u002Fminicpm_sala\u002Finference_speed_a600d.png)\n\n![inference_speed_5090](.\u002Fassets\u002Fminicpm_sala\u002Finference_speed_5090.png)\n\n#### Long-Context Evaluation\n\nMiniCPM-SALA consistently outperforms other open-source LLMs of similar scale across most involved long-context benchmarks. Specifically, it achieves the highest scores in the RULER and NoLiMa tests at all context lengths (up to 128K) and maintains the highest overall average score of 38.97, suggesting superior performance in handling long-context information processing.\n\n![long_text_evaluation](.\u002Fassets\u002Fminicpm_sala\u002Flong_text_evaluation.png)\n\n#### Ultra-long Context Evaluation\n\nThe evaluation demonstrates that MiniCPM-SALA exhibits effective length extrapolation capabilities, maintaining a score of 81.6 at a 2048K context length despite being trained on only 520K tokens. The model achieves this without auxiliary techniques like YaRN, likely due to its NoPE configuration in sparse attention layers.\n\n![ultra_long_text_evaluation](.\u002Fassets\u002Fminicpm_sala\u002Fultra_long_text_evaluation.png)\n\n#### Standard Evaluation\n\nMiniCPM-SALA achieves an average score of 76.53 across standard benchmarks, outperforming comparable models such as Qwen3-8B and Falcon-H1R-7B. The architecture maintains robust performance in Knowledge, Code, and Math.\n\n![benchmark](.\u002Fassets\u002Fminicpm_sala\u002Fbenchmark.png)\n\n### Inference\n\nTo achieve optimal performance, we recommend using the `Temperature=0.9`.\n\n#### HuggingFace\n\nOur model is readily compatible with 🤗 Hugging Face transformers. You can perform inference with our model as follows:\n\n```python\nimport torch\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\nmodel_path = \"openbmb\u002FMiniCPM-SALA\"\ntokenizer = AutoTokenizer.from_pretrained(model_path)\nmodel = AutoModelForCausalLM.from_pretrained(model_path, trust_remote_code=True, device_map=\"auto\")\nmodel.eval()\n\nprompts = [\"My name is\", \"The capital of China is\"]\nwith torch.no_grad():\n    inputs = tokenizer(prompts, return_tensors=\"pt\").to(model.device)\n    outputs = model.generate(**inputs)\noutput_texts = tokenizer.batch_decode(outputs)\nprint(output_texts)\n```\n\n#### SGLang\n\n##### Requirements\n\n- CUDA 12.x or higher\n- `gcc` \u002F `g++` compiler\n- `uv` package manager (script will check)\n\n##### Installation\n\n```bash\n# Clone repository\ngit clone -b minicpm_sala https:\u002F\u002Fgithub.com\u002FOpenBMB\u002Fsglang.git\ncd sglang\n\n# One-click installation (creates venv and compiles all dependencies)\nbash install_minicpm_sala.sh\n\n# Or specify PyPI mirror\nbash install_minicpm_sala.sh https:\u002F\u002Fmirrors.tuna.tsinghua.edu.cn\u002Fpypi\u002Fweb\u002Fsimple\n```\n\nThe installation script performs the following steps:\n\n1. Creates `sglang_minicpm_sala_env` virtual environment (Python 3.12)\n2. Clones dependencies to `3rdparty\u002F` (infllmv2) and initializes submodules (sparse_kernel)\n3. Installs MiniCPM-SALA (current repo)\n4. Compiles and installs `infllmv2_cuda_impl`\n5. Compiles and installs `sparse_kernel`\n6. Installs `tilelang` & `flash-linear-attention`\n\n##### Usage\n\n```bash\n# Activate environment\nsource sglang_minicpm_sala_env\u002Fbin\u002Factivate\n\n# Launch Inference Server (Replace MODEL_PATH with actual path)\nMODEL_PATH=\u002Fpath\u002Fto\u002Fyour\u002FMiniCPM-SALA\n\npython3 -m sglang.launch_server \\\n    --model ${MODEL_PATH} \\\n    --trust-remote-code \\\n    --disable-radix-cache \\\n    --attention-backend minicpm_flashinfer \\\n    --chunked-prefill-size 8192 \\\n    --max-running-requests 32 \\\n    --skip-server-warmup \\\n    --port 31111 \\\n    --dense-as-sparse\n```\n\n| Parameter | Description |\n|-----------|-------------|\n| `--trust-remote-code` | Allow custom code in model |\n| `--disable-radix-cache` | Disable RadixAttention prefix cache |\n| `--attention-backend minicpm_flashinfer` | Use MiniCPM FlashInfer backend |\n| `--chunked-prefill-size 8192` | Chunked prefill size |\n| `--max-running-requests 32` | Max concurrent requests |\n| `--skip-server-warmup` | Skip server warmup |\n| `--port 31111` | Server port |\n| `--dense-as-sparse` | Use dense-as-sparse mode |\n\n##### Manual Installation\n\nIf the script doesn't work for you, follow these steps:\n\n```bash\n# 0. Ensure uv is installed\npip install uv\n\n# 1. Create venv\nuv venv --python 3.12 sglang_minicpm_sala_env\nsource sglang_minicpm_sala_env\u002Fbin\u002Factivate\n\n# 2. Install SGLang\nuv pip install --upgrade pip setuptools wheel\nuv pip install -e .\u002Fpython[all]\n\n# 3. Compile CUDA Extensions\n# (Ensure dependencies are cloned to 3rdparty\u002F)\ncd 3rdparty\u002Finfllmv2_cuda_impl && python setup.py install && cd ..\u002F..\ncd 3rdparty\u002Fsparse_kernel && python setup.py install && cd ..\u002F..\n\n# 4. Install extra deps\nuv pip install tilelang flash-linear-attention\n```\n\n##### Q&A\n\n**Q: CUDA extension compilation failed?**\n\n- Ensure CUDA 12+ is installed (`nvcc --version`).\n- Ensure `gcc` \u002F `g++` are available.\n- If `CXX` is set to `clang++ -pthread`, manually `export CXX=g++`.\n\n## MiniCPM4 and MiniCPM4.1 Series\n\n\u003Cdiv align=\"center\">\n  \u003Ca href=\"https:\u002F\u002Fwww.youtube.com\u002Fwatch?v=VouXjLHKDUY\">\u003Cimg src=\"https:\u002F\u002Fimg.youtube.com\u002Fvi\u002FVouXjLHKDUY\u002F0.jpg\", width=70%>\u003C\u002Fa>\n\u003C\u002Fdiv>\n\n#### Highlights\nMiniCPM 4.1-8B is the first open-source reasoning LLM with trainable sparse attention:\n\n✅ Strong Reasoning Capability: Surpasses similar-sized models on 15 tasks!\n\n✅ Fast Generation: 3x decoding speedup for reasoning\n\n✅ Efficient Architecture: Trainable sparse attention, frequency-ranked speculative decoding\n\n#### Introduction\nMiniCPM4 and MiniCPM4.1 series are highly efficient large language models (LLMs) designed explicitly for end-side devices, which achieves this efficiency through systematic innovation in four key dimensions: model architecture, training data, training algorithms, and inference systems.\n\n- 🏗️ **Efficient Model Architecture:**\n  - InfLLM-V2 -- Trainable Sparse Attention Mechanism: Adopts a trainable sparse attention mechanism architecture where each token only needs to compute relevance with less than 5% of tokens in 128K long text processing, significantly reducing computational overhead for long texts ([InfLLM-V2 Training Kernels](https:\u002F\u002Fgithub.com\u002FOpenBMB\u002Finfllmv2_cuda_impl))\n\n- 🧠 **Efficient Learning Algorithms:**\n  - Model Wind Tunnel 2.0 -- Efficient Predictable Scaling: Introduces scaling prediction methods for performance of downstream tasks, enabling more precise model training configuration search\n  - BitCPM -- Ultimate Ternary Quantization: Compresses model parameter bit-width to 3 values, achieving 90% extreme model bit-width reduction\n  - Efficient Training Engineering Optimization: Adopts FP8 low-precision computing technology combined with Multi-token Prediction training strategy\n\n- 📚 **High-Quality Training Data:**\n  - UltraClean -- High-quality Pre-training Data Filtering and Generation: Builds iterative data cleaning strategies based on efficient data verification, open-sourcing high-quality Chinese and English pre-training dataset [UltraFinweb](https:\u002F\u002Fhuggingface.co\u002Fdatasets\u002Fopenbmb\u002FUltra-FineWeb)\n  - UltraChat v2 -- High-quality Supervised Fine-tuning Data Generation: Constructs large-scale high-quality supervised fine-tuning datasets covering multiple dimensions including knowledge-intensive data, reasoning-intensive data, instruction-following data, long text understanding data, and tool calling data\n\n- ⚡ **Efficient Inference and Deployment System:**\n  - CPM.cu -- Lightweight and Efficient CUDA Inference Framework: Integrates sparse attention, model quantization, and speculative sampling to achieve efficient prefilling and decoding ([Inference Kernels and Framework](https:\u002F\u002Fgithub.com\u002Fopenbmb\u002Fcpm.cu))\n  - ArkInfer -- Cross-platform Deployment System: Supports efficient deployment across multiple backend environments, providing flexible cross-platform adaptation capabilities\n\n### Evaluation Results\n\n#### Efficiency Evaluation\nOn two typical end-side chips, Jetson AGX Orin and RTX 4090, MiniCPM4 and MiniCPM4.1 demonstrate significantly faster processing speed compared to similar-size models in long text processing tasks. As text length increases, MiniCPM4 and MiniCPM4.1's efficiency advantage becomes more pronounced. On the Jetson AGX Orin platform, compared to Qwen3-8B, MiniCPM4 and MiniCPM4.1 achieves approximately 7x decoding speed improvement.\n\n![benchmark](.\u002Fassets\u002Fminicpm4\u002Fefficiency.png)\n\nMiniCPM4.1 achieves 3x decoding speed improvement in reasoning.\n\n![benchmark](.\u002Fassets\u002Fminicpm4\u002Fminicpm4.1_speed.png)\n\n#### Comprehensive Evaluation\nMiniCPM4 launches end-side versions with 8B and 0.5B parameter scales, both achieving best-in-class performance in their respective categories.\n\n![benchmark](.\u002Fassets\u002Fminicpm4\u002Fbenchmark.png)\n\nMiniCPM4.1 launches end-side versions with 8B parameter scale, achieving best-in-class performance in deep reasoning mode.\n\n![benchmark](.\u002Fassets\u002Fminicpm4\u002Fbenchmark4.1.png)\n\n#### Long Text Evaluation\nMiniCPM4 is pre-trained on 32K long texts and achieves length extension through YaRN technology. In the 128K long text needle-in-a-haystack task, MiniCPM4 demonstrates outstanding performance. MiniCPM4.1 is pre-trained on 64K long texts and achieves length extension through YaRN technology. In the 128K long text needle-in-a-haystack task, MiniCPM4.1 demonstrates outstanding performance.\n\n![long-niah](.\u002Fassets\u002Fminicpm4\u002F128k-niah.png)\n\n### Inference\nMiniCPM 4.1 can be used with following frameworks: Huggingface Transformers, SGLang, vLLM, and CPM.cu. For the ultimate inference speed, we highly recommend CPM.cu.\n\nMiniCPM4\u002FMiniCPM4.1 supports both dense attention inference and sparse attention inference modes, where vLLM and SGLang currently only support dense inference mode. If you want to use sparse inference mode, please use Huggingface Transformers and CPM.cu.\n\n- Dense attention inference: vLLM, SGLang, Huggingface Transformers\n- Sparse attention inference: Huggingface Transformers, CPM.cu\n\n#### Hybird Reasoning Mode\n\nMiniCPM4.1 supports hybrid reasoning mode, which can be used in both deep reasoning mode and non-reasoning mode. To enable hybrid reasoning mode. User can set `enable_thinking=True` in `tokenizer.apply_chat_template` to enable hybrid reasoning mode, and set `enable_thinking=False` to enable non-reasoning mode. Similarly, user can directly add `\u002Fno_think` at the end of the query to enable non-reasoning mode. If not add any special token or add `\u002Fthink` at the end of the query, the model will enable reasoning mode.\n\n```python\n# Enable reasoning mode\nprompt_text = tokenizer.apply_chat_template(\n    messages,\n    tokenize=False,\n    add_generation_prompt=True,\n    enable_thinking=True\n)\n# Enable non-reasoning mode\nprompt_text = tokenizer.apply_chat_template(\n    messages,\n    tokenize=False,\n    add_generation_prompt=True,\n    enable_thinking=False\n)\n```\n\n#### HuggingFace\n\n- **Inference with Dense Attention**\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\nimport torch\ntorch.manual_seed(0)\n\npath = 'openbmb\u002FMiniCPM4.1-8B'\ndevice = \"cuda\"\ntokenizer = AutoTokenizer.from_pretrained(path)\nmodel = AutoModelForCausalLM.from_pretrained(path, torch_dtype=torch.bfloat16, device_map=device, trust_remote_code=True)\n\n# User can directly use the chat interface\n# responds, history = model.chat(tokenizer, \"Write an article about Artificial Intelligence.\", temperature=0.7, top_p=0.7)\n# print(responds)\n\n# User can also use the generate interface\nmessages = [\n    {\"role\": \"user\", \"content\": \"Write an article about Artificial Intelligence.\"},\n]\nprompt_text = tokenizer.apply_chat_template(\n    messages,\n    tokenize=False,\n    add_generation_prompt=True,\n)\nmodel_inputs = tokenizer([prompt_text], return_tensors=\"pt\").to(device)\n\nmodel_outputs = model.generate(\n    **model_inputs,\n    max_new_tokens=32768,\n    top_p=0.95,\n    temperature=0.6\n)\noutput_token_ids = [\n    model_outputs[i][len(model_inputs[i]):] for i in range(len(model_inputs['input_ids']))\n]\n\nresponses = tokenizer.batch_decode(output_token_ids, skip_special_tokens=True)[0]\nprint(responses)\n```\n\n- **Inference with Sparse Attention**\nThis model supports InfLLM v2, a sparse attention mechanism designed for efficient long-sequence inference. It requires the [infllmv2_cuda_impl](https:\u002F\u002Fgithub.com\u002FOpenBMB\u002Finfllmv2_cuda_impl) library.\n\nYou can install it by running the following command:\n\n```bash\ngit clone -b feature_infer https:\u002F\u002Fgithub.com\u002FOpenBMB\u002Finfllmv2_cuda_impl.git\ncd infllmv2_cuda_impl\ngit submodule update --init --recursive\npip install -e . # or python setup.py install \n```\n\nTo enable InfLLM v2, you need to add the `sparse_config` field in `config.json`:\n\n```json\n{\n    ...,\n    \"sparse_config\": {\n        \"kernel_size\": 32,\n        \"kernel_stride\": 16,\n        \"init_blocks\": 1,\n        \"block_size\": 64,\n        \"window_size\": 2048,\n        \"topk\": 64,\n        \"use_nope\": false,\n        \"dense_len\": 8192\n    }\n}\n```\n\nThese parameters control the behavior of InfLLM v2:\n\n* `kernel_size` (default: 32): The size of semantic kernels.\n* `kernel_stride` (default: 16): The stride between adjacent kernels.\n* `init_blocks` (default: 1): The number of initial blocks that every query token attends to. This ensures attention to the beginning of the sequence.\n* `block_size` (default: 64): The block size for key-value blocks.\n* `window_size` (default: 2048): The size of the local sliding window. \n* `topk` (default: 64): The specifies that each token computes attention with only the top-k most relevant key-value blocks.\n* `use_nope` (default: false): Whether to use the NOPE technique in block selection for improved performance.\n* `dense_len` (default: 8192): Since Sparse Attention offers limited benefits for short sequences, the model can use standard (dense) attention for shorter texts. The model will use dense attention for sequences with a token length below `dense_len` and switch to sparse attention for sequences exceeding this length. Set this to `-1` to always use sparse attention regardless of sequence length.\n\n- **Long Context Extension**\nMiniCPM4.1 natively supports context lengths of up to 65,536(64k) tokens. For conversations where the total length (including both input and output) significantly exceeds this limit, we recommend using RoPE scaling techniques for effective handling of long texts. We have validated the model's performance on context lengths of up to 131,072 tokens by modifying the LongRoPE factor.\n\nYou can apply the LongRoPE factor modification by modifying the model files. Specifically, in the `config.json` file, adjust the `rope_scaling` fields.\n\n```json\n{\n    ...,\n    \"rope_scaling\": {\n        \"rope_type\": \"longrope\", \n        \"long_factor\": [0.9982316082870437, 1.033048153422584, 1.0749920956484724, 1.1255096879436193, 1.1863348602111476, 1.259543828902579, 1.3476188888731149, 1.4535223827776373, 1.5807816745852985, 1.7335856049489526, 1.9168922912975785, 2.1365471404135326, 2.3994084200118646, 2.713475511863602, 3.0880118452194134, 3.533650295140154, 4.062463396503134, 4.687974098908333, 5.425075306704039, 6.289818967956352, 7.29902962722721, 8.6357018163639, 10.210822723989212, 12.053807765671676, 14.193944598909404, 16.65780676784363, 19.463620727694074, 22.628311203524586, 26.150106147261315, 30.02526691405111, 34.23183327975347, 38.73811934094828, 43.502489489729555, 48.47627117965394, 53.61139491762471, 58.857366522037935, 64.16798299215064, 69.51359464319125, 74.86555458220285, 80.21497790341579, 85.55322183307433, 90.89611806932027, 96.26245306514224, 101.68269304046481, 107.18619510219668, 112.82253283014026, 118.63764063163615, 119.88866203644656, 120.9462882391725, 121.837565139014, 122.58663780572562, 123.2147719894291, 123.74049454862576, 124.17980424685767, 124.54641761955492, 124.85202548028222, 125.10654406389756, 125.31835105170659, 125.49450117164764, 125.64091910903052, 125.76256945356558, 125.86360463815589, 125.94749252260765, 126.01712561287873],\n        \"short_factor\": [0.9982316082870437, 1.033048153422584, 1.0749920956484724, 1.1255096879436193, 1.1863348602111476, 1.259543828902579, 1.3476188888731149, 1.4535223827776373, 1.5807816745852985, 1.7335856049489526, 1.9168922912975785, 2.1365471404135326, 2.3994084200118646, 2.713475511863602, 3.0880118452194134, 3.533650295140154, 4.062463396503134, 4.687974098908333, 5.425075306704039, 6.289818967956352, 7.29902962722721, 8.6357018163639, 10.210822723989212, 12.053807765671676, 14.193944598909404, 16.65780676784363, 19.463620727694074, 22.628311203524586, 26.150106147261315, 30.02526691405111, 34.23183327975347, 38.73811934094828, 43.502489489729555, 48.47627117965394, 53.61139491762471, 58.857366522037935, 64.16798299215064, 69.51359464319125, 74.86555458220285, 80.21497790341579, 85.55322183307433, 90.89611806932027, 96.26245306514224, 101.68269304046481, 107.18619510219668, 112.82253283014026, 118.63764063163615, 119.88866203644656, 120.9462882391725, 121.837565139014, 122.58663780572562, 123.2147719894291, 123.74049454862576, 124.17980424685767, 124.54641761955492, 124.85202548028222, 125.10654406389756, 125.31835105170659, 125.49450117164764, 125.64091910903052, 125.76256945356558, 125.86360463815589, 125.94749252260765, 126.01712561287873],\n        \"original_max_position_embeddings\": 65536\n    }\n}\n```\n\n#### vLLM\n\n##### Speculative Decoding\n\nFor accelerated inference with speculative decoding using vLLM, follow these steps:\n\n###### 1. Download MiniCPM4.1 Draft Model\n\nFirst, download the MiniCPM4.1 draft model:\n\n```bash\ncd \u002Fyour_path\ngit clone https:\u002F\u002Fhuggingface.co\u002Fopenbmb\u002FMiniCPM4.1-8B-Eagle3\n```\n\n###### 2. Install EAGLE3-Compatible vLLM\n\nThe EAGLE3 vLLM PR has been submitted. For now, use our repository for installation:\n\n```bash\ngit clone https:\u002F\u002Fgithub.com\u002FLDLINGLINGLING\u002Fvllm.git\ncd vllm \npip install -e .\n```\n\n###### 3. Launch vLLM Server with Speculative Decoding\n\nStart the vLLM inference server with speculative decoding enabled. Make sure to update the model path in the speculative-config to point to your downloaded MiniCPM4_1-8B-Eagle3-bf16 folder:\n\n```bash\nVLLM_USE_V1=1 \\\nvllm serve openbmb\u002FMiniCPM4.1-8B \\\n--seed 42 \\\n--trust-remote-code \\\n--speculative-config '{\n  \"model\": \"your\u002Fpath\u002FMiniCPM4_1-8B-Eagle3-bf16\",\n  \"num_speculative_tokens\": 3,\n  \"method\": \"eagle3\",\n  \"draft_tensor_parallel_size\": 1\n}'\n```\n\n###### 4. Client Usage Example\n\nThe client usage remains the same for both standard and speculative decoding:\n\n```python\nimport openai\n\nclient = openai.Client(base_url=\"http:\u002F\u002Flocalhost:8000\u002Fv1\", api_key=\"EMPTY\")\n\nresponse = client.chat.completions.create(\n    model=\"openbmb\u002FMiniCPM4.1-8B\",\n    messages=[\n        {\"role\": \"user\", \"content\": \"Write an article about Artificial Intelligence.\"},\n    ],\n    temperature=0.6,\n    max_tokens=32768,\n    extra_body=dict(add_special_tokens=True),  # Ensures special tokens are added for chat template\n    \n)\n\nprint(response.choices[0].message.content)\n```\n\n###### vLLM Configuration Parameters\n\n- `VLLM_USE_V1=1`: Enables vLLM v1 API\n- `--speculative-config`: JSON configuration for speculative decoding\n  - `model`: Path to the draft model for speculation\n  - `num_speculative_tokens`: Number of speculative tokens (default: 3)\n  - `method`: Speculative decoding method (eagle3)\n  - `draft_tensor_parallel_size`: Tensor parallel size for draft model (default: 1)\n- `--seed`: Random seed for reproducibility\n- `--trust-remote-code`: Allow execution of remote code for custom models\n\n##### Standard Inference (Without Speculative Decoding)\n\nFor now, you need to install the latest version of vLLM.\n\n```bash\npip install -U vllm \\\n    --pre \\\n    --extra-index-url https:\u002F\u002Fwheels.vllm.ai\u002Fnightly\n```\n\nThen you can inference MiniCPM4.1-8B with vLLM:\n```python\nfrom transformers import AutoTokenizer\nfrom vllm import LLM, SamplingParams\n\nmodel_name = \"openbmb\u002FMiniCPM4.1-8B\"\nprompt = [{\"role\": \"user\", \"content\": \"Write an article about Artificial Intelligence.\"}]\n\ntokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)\ninput_text = tokenizer.apply_chat_template(prompt, tokenize=False, add_generation_prompt=True)\n\nllm = LLM(\n    model=model_name,\n    trust_remote_code=True,\n    max_num_batched_tokens=65536,\n    dtype=\"bfloat16\", \n    gpu_memory_utilization=0.8, \n)\nsampling_params = SamplingParams(top_p=0.95, temperature=0.6, max_tokens=32768)\n\noutputs = llm.generate(prompts=input_text, sampling_params=sampling_params)\n\nprint(outputs[0].outputs[0].text)\n```\n\nAlso, you can start the inference server by running the following command:\n> **Note**: In vLLM's chat API, `add_special_tokens` is `False` by default. This means important special tokens—such as the beginning-of-sequence (BOS) token—will not be added automatically. To ensure the input prompt is correctly formatted for the model, you should explicitly set `extra_body={\"add_special_tokens\": True}`.\n\n```bash\nvllm serve openbmb\u002FMiniCPM4.1-8B --trust-remote-code\n```\n\nThen you can use the chat interface by running the following code:\n\n```python\nimport openai\n\nclient = openai.Client(base_url=\"http:\u002F\u002Flocalhost:8000\u002Fv1\", api_key=\"EMPTY\")\n\nresponse = client.chat.completions.create(\n    model=\"openbmb\u002FMiniCPM4.1-8B\",\n    messages=[\n        {\"role\": \"user\", \"content\": \"Write an article about Artificial Intelligence.\"},\n    ],\n    temperature=0.6,\n    max_tokens=32768,\n    extra_body=dict(add_special_tokens=True),  # Ensures special tokens are added for chat template\n)\n\nprint(response.choices[0].message.content)\n```\n\n#### SGLang\n\n##### Speculative Decoding\n\nFor accelerated inference with speculative decoding, follow these steps:\n\n###### 1. Download MiniCPM4.1 Draft Model\n\nFirst, download the MiniCPM4.1 draft model:\n\n```bash\ncd \u002Fyour_path\ngit clone https:\u002F\u002Fhuggingface.co\u002Fopenbmb\u002FMiniCPM4.1-8B-Eagle3\n```\n\n###### 2. Install EAGLE3-Compatible SGLang\n\nThe EAGLE3 adaptation PR has been submitted. For now, use our repository for installation:\n\n```bash\ngit clone https:\u002F\u002Fgithub.com\u002FLDLINGLINGLING\u002Fsglang.git\ncd sglang\npip install -e .\n```\n\n###### 3. Launch SGLang Server with Speculative Decoding\n\nStart the SGLang server with speculative decoding enabled:\n\n```bash\npython -m sglang.launch_server \\\n  --model-path \"openbmb\u002FMiniCPM4.1-8B\" \\\n  --host \"127.0.0.1\" \\\n  --port 30002 \\\n  --mem-fraction-static 0.9 \\\n  --speculative-algorithm EAGLE3 \\\n  --speculative-draft-model-path \"your\u002Fpath\u002FMiniCPM4_1-8B-Eagle3-bf16\" \\\n  --speculative-num-steps 3 \\\n  --speculative-eagle-topk 1 \\\n  --speculative-num-draft-tokens 32 \\\n  --temperature 0.7\n```\n\n###### 4. Client Usage\n\nThe client usage remains the same for both standard and speculative decoding:\n\n```python\nimport openai\n\nclient = openai.Client(base_url=f\"http:\u002F\u002Flocalhost:30002\u002Fv1\", api_key=\"None\")\n\nresponse = client.chat.completions.create(\n    model=\"openbmb\u002FMiniCPM4.1-8B\",\n    messages=[\n        {\"role\": \"user\", \"content\": \"Write an article about Artificial Intelligence.\"},\n    ],\n    temperature=0.6,\n    max_tokens=32768,\n)\n\nprint(response.choices[0].message.content)\n```\n\nNote: Make sure to update the port number in the client code to match the server port (30002 in the speculative decoding example).\n\n###### Configuration Parameters\n\n- `--speculative-algorithm EAGLE3`: Enables EAGLE3 speculative decoding\n- `--speculative-draft-model-path`: Path to the draft model for speculation\n- `--speculative-num-steps`: Number of speculative steps (default: 3)\n- `--speculative-eagle-topk`: Top-k parameter for EAGLE (default: 1)\n- `--speculative-num-draft-tokens`: Number of draft tokens (default: 32)\n- `--mem-fraction-static`: Memory fraction for static allocation (default: 0.9)\n\n##### Standard Inference (Without Speculative Decoding)\n\nFor now, you need to install our forked version of SGLang.\n\n```bash\ngit clone -b openbmb https:\u002F\u002Fgithub.com\u002FOpenBMB\u002Fsglang.git\ncd sglang\n\npip install --upgrade pip\npip install -e \"python[all]\"\n```\n\nYou can start the inference server by running the following command:\n\n```bash\npython -m sglang.launch_server --model openbmb\u002FMiniCPM4.1-8B --trust-remote-code --port 30000 --chat-template chatml\n```\n\nThen you can use the chat interface by running the following command:\n\n```python\nimport openai\n\nclient = openai.Client(base_url=f\"http:\u002F\u002Flocalhost:30000\u002Fv1\", api_key=\"None\")\n\nresponse = client.chat.completions.create(\n    model=\"openbmb\u002FMiniCPM4.1-8B\",\n    messages=[\n        {\"role\": \"user\", \"content\": \"Write an article about Artificial Intelligence.\"},\n    ],\n    temperature=0.6,\n    max_tokens=32768,\n)\n\nprint(response.choices[0].message.content)\n```\n\n\n#### CPM.cu\n\nWe **recommend** using [CPM.cu](https:\u002F\u002Fgithub.com\u002FOpenBMB\u002FCPM.cu) for the inference of MiniCPM4 and MiniCPM4.1. CPM.cu is a CUDA inference framework developed by OpenBMB, which integrates efficient sparse, speculative sampling, and quantization techniques, fully leveraging the efficiency advantages of MiniCPM4 and MiniCPM4.1.\n\nYou can install CPM.cu by running the following command:\n\n```bash\ngit clone https:\u002F\u002Fgithub.com\u002FOpenBMB\u002FCPM.cu.git --recursive\ncd CPM.cu\npython3 setup.py install\n```\n\nYou can run the following command to test the speed of the model.\n\n```bash\npython3 tests\u002Flong_prompt_gen.py # generate prompt.txt\npython3 tests\u002Ftest_generate.py --prompt-file prompt.txt\n```\n\nYou can run the following command to infer with EAGLE3 speculative decoding algorithm.\n\n```bash\npython3 -m cpmcu.cli \\\n    --model-path $BASE_MODEL_PATH \\\n    --draft-model-path $EAGLE3_DRAFT_MODEL_PATH \\\n    --prompt-text \"Tell me about Tsinghua University\" \\\n    --use-eagle3 true\n```\n\nFor more details about CPM.cu, please refer to the repo of [CPM.cu](https:\u002F\u002Fgithub.com\u002FOpenBMB\u002FCPM.cu).\n\n\n#### llama.cpp and Ollama\n\nWe also support inference with [llama.cpp](https:\u002F\u002Fgithub.com\u002Fggml-org\u002Fllama.cpp) and [Ollama](https:\u002F\u002Follama.com\u002F).\n\n##### llama.cpp\n\nYou can download the GGUF format of MiniCPM4.1-8B model from [huggingface](https:\u002F\u002Fhuggingface.co\u002Fopenbmb\u002FMiniCPM4.1-8B-GGUF) and run it with llama.cpp for efficient CPU or GPU inference.\n```\n# case 1: main-cli\n.\u002Fbuild\u002Fbin\u002Fllama-cli -m MiniCPM4.1-8B-Q4_K_M.gguf -p \"Write an article about Artificial Intelligence.\" -n 1500\n\n# case 2: server\n## launch server\n.\u002Fbuild\u002Fbin\u002Fllama-server -m MiniCPM4.1-8B-Q4_K_M.gguf --host 127.0.0.1 --port 8080 -c 4096 -fa on &\n\n## send request\ncurl -X POST http:\u002F\u002F127.0.0.1:8080\u002Fv1\u002Fchat\u002Fcompletions \\\n  -H \"Content-Type: application\u002Fjson\" \\\n  -d '{\n    \"model\": \"gpt-3.5-turbo\",\n    \"messages\": [{\"role\": \"user\", \"content\": \"Write an article about Artificial Intelligence.\"}],\n    \"max_tokens\": 1500\n  }'\n```\n\n##### Ollama\nPlease refer to [model hub](https:\u002F\u002Follama.com\u002Fopenbmb\u002Fminicpm4.1) for model download. After installing ollama package, you can use MiniCPM4.1 with following commands:\n```\nollama run openbmb\u002Fminicpm4.1\n```\n\n### BitCPM4: Quantization\n\nBitCPM4 are ternary quantized models derived from the MiniCPM series models through quantization-aware training (QAT), achieving significant improvements in both training efficiency and model parameter efficiency.\n- Improvements of the training method\n  - Searching hyperparameters with a wind-tunnel on a small model.\n  - Using a two-stage training method: training in high-precision first and then QAT, making the best of the trained high-precision models and significantly reducing the computational resources required for the QAT phase.\n- High parameter efficiency\n  - Achieving comparable performance to full-precision models of similar parameter models with a bit width of only 1.58 bits, demonstrating high parameter efficiency. \n\n#### BitCPM4 Evaluation\n\nBitCPM4's performance is comparable with other full-precision models in same model size.\n![bitcpm-benchmark](.\u002Fassets\u002Fminicpm4\u002Fbitcpm4-benchmark.png)\n\n#### BitCPM4 Inference\n\nBitCPM4's parameters are stored in a fake-quantized format, which supports direct inference within the Huggingface framework.\n\n### MiniCPM4 Application\n\u003Cdetails>\n\u003Csummary>Click to view details about MiniCPM4 Application\u003C\u002Fsummary>\n\n#### MiniCPM4-Survey: Trustworthy Survey Generation\n\n**MiniCPM4-Survey** is an open-source LLM agent model jointly developed by [THUNLP](https:\u002F\u002Fnlp.csai.tsinghua.edu.cn), Renmin University of China and [ModelBest](https:\u002F\u002Fmodelbest.cn\u002Fen). Built on MiniCPM4-8B, it accepts users' quiries as input and autonomously generate trustworthy, long-form survey papers.\n\nKey features include:\n\n- **Plan-Retrieve-Write Survey Generation Framework** — We propose a multi-agent generation framework, which operates through three core stages: planning (defining the overall structure of the survey), retrieval (generating appropriate retrieval keywords), and writing (synthesizing the retrieved information to generate coherent section-level content).\n\n- **High-Quality Dataset Construction** — We gather and process lots of expert-written survey papers to construct a high-quality training dataset. Meanwhile, we collect a large number of research papers to build a retrieval database.\n\n- **Multi-Aspect Reward Design** — We carefully design a reward system with three aspects (structure, content, and citations) to evaluate the quality of the surveys, which is used as the reward function in the RL training stage.\n\n- **Multi-Step RL Training Strategy** — We propose a *Context Manager* to ensure retention of essential information while facilitating efficient reasoning, and we construct *Parallel Environment* to maintain efficient RL training cycles.\n\n##### Demo and Quick Start\n\nSee [here](.\u002Fdemo\u002Fminicpm4\u002FSurveyGeneration\u002FREADME.md)\n\n##### Performance Evaluation\n\n| Method                                      | Relevance | Coverage | Depth | Novelty | Avg.  | Fact Score |\n|---------------------------------------------|-----------|----------|-------|---------|-------|------------|\n| Naive RAG (driven by G2FT)                  | 3.25      | 2.95     | 3.35  | 2.60    | 3.04  | 43.68      |\n| AutoSurvey (driven by G2FT)                 | 3.10      | 3.25     | 3.15  | **3.15**| 3.16  | 46.56      |\n| Webthinker (driven by WTR1-7B)              | 3.30      | 3.00     | 2.75  | 2.50    | 2.89  | --         |\n| Webthinker (driven by QwQ-32B)              | 3.40      | 3.30     | 3.30  | 2.50    | 3.13  | --         |\n| OpenAI Deep Research (driven by GPT-4o)     | 3.50      |**3.95**  | 3.55  | 3.00    | **3.50**  | --         |\n| MiniCPM-4-Survey                            | 3.45      | 3.70     | **3.85** | 3.00    | **3.50**  | **68.73**  |\n|    *w\u002Fo* RL                  | **3.55**  | 3.35     | 3.30  | 2.25    | 3.11  | 50.24      |\n\n*Performance comparison of the survey generation systems. \"G2FT\" stands for Gemini-2.0-Flash-Thinking, and \"WTR1-7B\" denotes Webthinker-R1-7B. FactScore evaluation was omitted for Webthinker, as it does not include citation functionality, and for OpenAI Deep Research, which does not provide citations when exporting the results.*\n\n#### MiniCPM4-MCP: Tool Use with Model Context Protocol\n\n**MiniCPM4-MCP** is an open-source on-device LLM agent model jointly developed by [THUNLP](https:\u002F\u002Fnlp.csai.tsinghua.edu.cn), Renmin University of China and [ModelBest](https:\u002F\u002Fmodelbest.cn\u002Fen), built on [MiniCPM-4](https:\u002F\u002Fhuggingface.co\u002Fopenbmb\u002FMiniCPM4-8B) with 8 billion parameters. It is capable of solving a wide range of real-world tasks by interacting with various tool and data resources through MCP. As of now, MiniCPM4-MCP supports the following:\n\n- Utilization of tools across 16 MCP servers: These servers span various categories, including office, lifestyle, communication, information, and work management.\n\n- Single-tool-calling capability: It can perform single- or multi-step tool calls using a single tool that complies with the MCP.\n\n- Cross-tool-calling capability: It can perform single- or multi-step tool calls using different tools that complies with the MCP.\n\n##### Demo\n\nDemo is available in this [link](.\u002Fdemo\u002Fminicpm4\u002FMCP\u002FREADME_en.md).\n\n##### Performance Evaluation\n\n| MCP Server                  |          | gpt-4o             |              |          | qwen3             |              |      |      minicpm4         |              |\n|-----------------------|----------------|--------------|--------------|---------------|--------------|--------------|----------------|--------------|--------------|\n|                       | func           | param        | value        | func          | param        | value        | func           | param        | value        |\n| Airbnb                | 89.3           | 67.9         | 53.6         | 92.8          | 60.7         | 50.0         | 96.4           | 67.9         | 50.0         |\n| Amap-Maps             | 79.8           | 77.5         | 50.0         | 74.4          | 72.0         | 41.0         | 89.3           | 85.7         | 39.9         |\n| Arxiv-MCP-Server      | 85.7           | 85.7         | 85.7         | 81.8          | 54.5         | 50.0         | 57.1           | 57.1         | 52.4         |\n| Calculator            | 100.0          | 100.0        | 20.0         | 80.0          | 80.0         | 13.3         | 100.0          | 100.0        | 6.67         |\n| Computor-Control-MCP  | 90.0           | 90.0         | 90.0         | 90.0          | 90.0         | 90.0         | 90.0           | 90.0         | 86.7         |\n| Desktop-Commander     | 100.0          | 100.0        | 100.0        | 100.0         | 100.0        | 100.0        | 100.0          | 100.0        | 100.0        |\n| Filesystem            | 63.5           | 63.5         | 31.3         | 69.7          | 69.7         | 26.0         | 83.3           | 83.3         | 42.7         |\n|Github | 92.0 | 80.0 | 58.0 | 80.5 | 50.0 | 27.7 | 62.8 | 25.7 | 17.1 |\n| Gaode                 | 71.1           | 55.6         | 17.8         | 68.8          | 46.6         | 24.4         | 68.9           | 46.7         | 15.6         |\n| MCP-Code-Executor     | 85.0           | 80.0         | 70.0         | 80.0          | 80.0         | 70.0         | 90.0           | 90.0         | 65.0         |\n| MCP-Docx              | 95.8           | 86.7         | 67.1         | 94.9          | 81.6         | 60.1         | 95.1           | 86.6         | 76.1         |\n| PPT                   | 72.6           | 49.8         | 40.9         | 85.9          | 50.7         | 37.5         | 91.2           | 72.1         | 56.7         |\n| PPTx                  | 64.2           | 53.7         | 13.4         | 91.0          | 68.6         | 20.9         | 91.0           | 58.2         | 26.9         |\n| Simple-Time-Server    | 90.0           | 70.0         | 70.0         | 90.0          | 90.0         | 90.0         | 90.0           | 60.0         | 60.0         |\n| Slack                 | 100.0          | 90.0         | 70.0         | 100.0         | 100.0        | 65.0         | 100.0          | 100.0        | 100.0        |\n| Whisper               | 90.0           | 90.0         | 90.0         | 90.0          | 90.0         | 90.0         | 90.0           | 90.0         | 30.0         |\n| **Average**              | **80.2**       | **70.2**     | **49.1**     | **83.5**      | **67.7**     | **43.8**     | **88.3**       | **76.1**     | **51.2**     |\n\n#### MiniCPM Intel AIPC Client: A New Edge Large Model Powerhouse  \n\nDeveloped in collaboration between Mianbi Intelligence and Intel, the MiniCPM Intel AIPC Client is an edge large model client specially designed for devices equipped with Intel Core Ultra series processors. It delivers a low-latency, high-efficiency, and privacy-preserving local large model experience for developers, researchers, and AI enthusiasts. Its core features include:  \n\n##### Key Features  \n- Deep Intel Hardware Adaptation  \nFully compatible with Intel Core Ultra series processors, enabling deep integration with hardware to unleash peak performance. Users can run large models smoothly on local devices without relying on cloud services.  \n\n- Extreme Optimization Based on OpenVINO  \nDeeply optimized with the OpenVINO inference framework, it significantly boosts inference efficiency, reaching up to **80 tokens per second**. This ensures rapid model response for both quick queries and complex task processing.  \n\n- Privacy and Security Assurance  \nAdopting local deployment, all data processing is completed on the device, eliminating privacy risks from cloud uploads. This provides users with peace of mind, especially for scenarios with high data privacy requirements.  \n\n- Catering to Diverse User Groups  \nWhether for developers chasing cutting-edge technologies, researchers focused on academic studies, or enthusiasts eager to explore AI applications, the MiniCPM Intel AIPC Client enables easy access to the power of local large models, opening the door to personalized AI exploration.  \n\n##### System Requirements  \n- Recommended processor: Intel Core Ultra 7 or higher (mobile version)  \n- Recommended RAM: 32GB or above\n\n##### Download\n\n[download](https:\u002F\u002Fgithub.com\u002FOpenBMB\u002FMiniCPM\u002Freleases\u002Ftag\u002F2.4.2)\n\n\u003C\u002Fdetails>\n\n\n## LICENSE\n\n#### Model LICENSE\n\n* This repository and MiniCPM models are released under the [Apache-2.0](https:\u002F\u002Fgithub.com\u002FOpenBMB\u002FMiniCPM\u002Fblob\u002Fmain\u002FLICENSE) License. \n\n#### Statement\n\n* As a language model, MiniCPM generates content by learning from a vast amount of text. \n* However, it does not possess the ability to comprehend or express personal opinions or value judgments. \n* Any content generated by MiniCPM does not represent the viewpoints or positions of the model developers. \n* Therefore, when using content generated by MiniCPM, users should take full responsibility for evaluating and verifying it on their own.\n\n## Institutions\n\nThis project is developed by the following institutions:\n\n- \u003Cimg src=\"assets\u002Fmodelbest.png\" width=\"28px\"> [Modelbest Inc.](https:\u002F\u002Fmodelbest.cn\u002F)\n- \u003Cimg src=\"assets\u002Fthunlp.png\" width=\"28px\"> [THUNLP](https:\u002F\u002Fnlp.csai.tsinghua.edu.cn\u002F)\n- \u003Cimg src=\"assets\u002FRUC.png\" width=\"28px\"> [Gaoling School of Artificial Intelligence of RUC](https:\u002F\u002Flinyankai.github.io\u002F)\n\n## Citation\n\n* Please cite our paper: [MiniCPM4](https:\u002F\u002Farxiv.org\u002Fabs\u002F2506.07900) if you find our work valuable.\n\n```\n@article{minicpm4,\n  title={Minicpm4: Ultra-efficient llms on end devices},\n  author={MiniCPM, Team},\n  journal={arXiv preprint arXiv:2506.07900},\n  year={2025}\n}\n```\n","MiniCPM4 & MiniCPM4.1 是专为终端设备设计的超高效大语言模型，能够实现推理任务生成速度3倍以上的提升。该项目的核心功能包括通过可训练的稀疏注意力机制，在保持高性能的同时大幅提高计算效率，尤其适用于需要在资源受限环境下进行快速推理的应用场景。技术上，它利用了混合推理模型结构和系统级优化策略来减少计算开销，并支持深度推理模式与非推理模式之间的切换。此项目非常适合那些希望在移动设备或边缘计算环境中部署自然语言处理应用的研究者和开发者使用。",2,"2026-06-11 03:48:36","high_star"]