[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"project-72932":3},{"id":4,"name":5,"fullName":6,"owner":7,"repo":5,"description":8,"homepage":9,"htmlUrl":10,"language":11,"languages":10,"totalLinesOfCode":10,"stars":12,"forks":13,"watchers":14,"openIssues":15,"contributorsCount":16,"subscribersCount":16,"size":16,"stars1d":17,"stars7d":18,"stars30d":14,"stars90d":16,"forks30d":16,"starsTrendScore":19,"compositeScore":20,"rankGlobal":10,"rankLanguage":10,"license":21,"archived":22,"fork":22,"defaultBranch":23,"hasWiki":22,"hasPages":22,"topics":24,"createdAt":10,"pushedAt":10,"updatedAt":30,"readmeContent":31,"aiSummary":32,"trendingCount":16,"starSnapshotCount":16,"syncStatus":33,"lastSyncTime":34,"discoverSource":35},72932,"PowerInfer","Tiiny-AI\u002FPowerInfer","Tiiny-AI","High-speed Large Language Model Serving for Local Deployment","",null,"C++",9549,578,102,123,0,15,31,45,39.29,"MIT License",false,"main",[25,26,27,28,29],"large-language-models","llama","llm","llm-inference","local-inference","2026-06-12 02:03:09","# PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPU\n\n## TL;DR\nPowerInfer is a CPU\u002FGPU LLM inference engine leveraging **activation locality** for your device.\n\n\u003Ca href=\"https:\u002F\u002Ftrendshift.io\u002Frepositories\u002F20082\" target=\"_blank\">\u003Cimg src=\"https:\u002F\u002Ftrendshift.io\u002Fapi\u002Fbadge\u002Frepositories\u002F20082\" alt=\"Tiiny-AI%2FPowerInfer | Trendshift\" style=\"width: 250px; height: 55px;\" width=\"250\" height=\"55\"\u002F>\u003C\u002Fa>\n\n[![License: MIT](https:\u002F\u002Fimg.shields.io\u002Fbadge\u002Flicense-MIT-blue.svg)](https:\u002F\u002Fopensource.org\u002Flicenses\u002FMIT)\n\n[Project Kanban](https:\u002F\u002Fgithub.com\u002Forgs\u002FSJTU-IPADS\u002Fprojects\u002F2\u002Fviews\u002F2)\n\n## Latest News 🔥\n- [2026\u002F1\u002F5] We released **[Tiiny AI Pocket Lab](https:\u002F\u002Ftiiny.ai\u002F)**, the world's first pocket-size supercomputer. It runs GPT-OSS-120B (int4) locally at **20 tokens\u002Fs**. Featured at CES 2026.\n- [2025\u002F7\u002F27] We released [SmallThinker-21BA3B-Instruct](https:\u002F\u002Fhuggingface.co\u002FPowerInfer\u002FSmallThinker-21BA3B-Instruct) and [SmallThinker-4BA0.6B-Instruct](https:\u002F\u002Fhuggingface.co\u002FPowerInfer\u002FSmallThinker-4BA0.6B-Instruct). We also released a corresponding framework for efficient [on-device inference](.\u002Fsmallthinker\u002FREADME.md). \n- [2024\u002F6\u002F11] We are thrilled to introduce [PowerInfer-2](https:\u002F\u002Farxiv.org\u002Fabs\u002F2406.06282), our highly optimized inference framework designed specifically for smartphones. With TurboSparse-Mixtral-47B, it achieves an impressive speed of 11.68 tokens per second, which is up to 22 times faster than other state-of-the-art frameworks.\n- [2024\u002F6\u002F11] We are thrilled to present [Turbo Sparse](https:\u002F\u002Farxiv.org\u002Fabs\u002F2406.05955), our TurboSparse models for fast inference. With just $0.1M, we sparsified the original Mistral and Mixtral model to nearly 90% sparsity while maintaining superior performance! For a Mixtral-level model, our TurboSparse-Mixtral activates only **4B** parameters!\n- [2024\u002F5\u002F20] **Competition Recruitment: CCF-TCArch Customized Computing Challenge 2024**. The CCF TCARCH CCC is a national competition organized by the Technical Committee on Computer Architecture (TCARCH) of the China Computer Federation (CCF). This year's competition aims to optimize the PowerInfer inference engine using the open-source ROCm\u002FHIP. More information about the competition can be found [here](https:\u002F\u002Fccf-tcarch-ccc.github.io\u002F2024\u002F).\n- [2024\u002F5\u002F17] We now provide support for AMD devices with ROCm.\n- [2024\u002F3\u002F28] We are trilled to present [Bamboo LLM](https:\u002F\u002Fgithub.com\u002FSJTU-IPADS\u002FBamboo) that achieves both top-level performance and unparalleled speed with PowerInfer! Experience it with Bamboo-7B [Base](https:\u002F\u002Fhuggingface.co\u002FPowerInfer\u002FBamboo-base-v0.1-gguf) \u002F [DPO](https:\u002F\u002Fhuggingface.co\u002FPowerInfer\u002FBamboo-DPO-v0.1-gguf).\n- [2024\u002F3\u002F14] We supported ProSparse Llama 2 ([7B](https:\u002F\u002Fhuggingface.co\u002FSparseLLM\u002Fprosparse-llama-2-7b)\u002F[13B](https:\u002F\u002Fhuggingface.co\u002FSparseLLM\u002Fprosparse-llama-2-13b)), ReLU models with ~90% sparsity, matching original Llama 2's performance (Thanks THUNLP & ModelBest)!\n- [2024\u002F1\u002F11] We supported Windows with GPU inference!\n- [2023\u002F12\u002F24] We released an online [gradio demo](https:\u002F\u002Fpowerinfer-gradio.vercel.app\u002F) for Falcon(ReLU)-40B-FP16!\n- [2023\u002F12\u002F19] We officially released PowerInfer!\n\n## Demo 🔥\n\nhttps:\u002F\u002Fgithub.com\u002FSJTU-IPADS\u002FPowerInfer\u002Fassets\u002F34213478\u002Ffe441a42-5fce-448b-a3e5-ea4abb43ba23\n\nPowerInfer v.s. llama.cpp on a single RTX 4090(24G) running Falcon(ReLU)-40B-FP16 with a 11x speedup!\n\n\u003Csub>Both PowerInfer and llama.cpp were running on the same hardware and fully utilized VRAM on RTX 4090.\u003C\u002Fsub>\n\n> [!NOTE]\n> **Live Demo Online⚡️**\n>\n> Try out our [Gradio server](https:\u002F\u002Fpowerinfer-gradio.vercel.app\u002F) hosting Falcon(ReLU)-40B-FP16 on a RTX 4090!\n>\n> \u003Csub>Experimental and without warranties 🚧\u003C\u002Fsub>\n\n## Abstract\n\nWe introduce PowerInfer, a high-speed Large Language Model (LLM) inference engine on a personal computer (PC)\nequipped with a single consumer-grade GPU. The key underlying the design of PowerInfer is exploiting the high **locality**\ninherent in LLM inference, characterized by a power-law distribution in neuron activation.\n\nThis distribution indicates that a small subset of neurons, termed hot neurons, are consistently activated\nacross inputs, while the majority, cold neurons, vary based on specific inputs.\nPowerInfer exploits such an insight to design a GPU-CPU hybrid inference engine:\nhot-activated neurons are preloaded onto the GPU for fast access, while cold-activated neurons are computed\non the CPU, thus significantly reducing GPU memory demands and CPU-GPU data transfers.\nPowerInfer further integrates adaptive predictors and neuron-aware sparse operators,\noptimizing the efficiency of neuron activation and computational sparsity.\n\nEvaluation shows that PowerInfer attains an average token generation rate of 13.20 tokens\u002Fs, with a peak of 29.08 tokens\u002Fs, across various LLMs (including OPT-175B) on a single NVIDIA RTX 4090 GPU,\nonly 18\\% lower than that achieved by a top-tier server-grade A100 GPU.\nThis significantly outperforms llama.cpp by up to 11.69x while retaining model accuracy.\n\n## Features\nPowerInfer is a high-speed and easy-to-use inference engine for deploying LLMs locally.\n\nPowerInfer is fast with:\n\n- **Locality-centric design**: Utilizes sparse activation and 'hot'\u002F'cold' neuron concept for efficient LLM inference, ensuring high speed with lower resource demands.\n- **Hybrid CPU\u002FGPU Utilization**: Seamlessly integrates memory\u002Fcomputation capabilities of CPU and GPU for a balanced workload and faster processing.\n\nPowerInfer is flexible and easy to use with:\n\n- **Easy Integration**: Compatible with popular [ReLU-sparse models](https:\u002F\u002Fhuggingface.co\u002FSparseLLM).\n- **Local Deployment Ease**: Designed and deeply optimized for local deployment on consumer-grade hardware, enabling low-latency LLM inference and serving on a single GPU.\n- **Backward Compatibility**: While distinct from llama.cpp, you can make use of most of `examples\u002F` the same way as llama.cpp such as server and batched generation. PowerInfer also supports inference with llama.cpp's model weights for compatibility purposes, but there will be no performance gain.\n\nYou can use these models with PowerInfer today:\n\n- Falcon-40B\n- Llama2 family\n- ProSparse Llama2 family\n- Bamboo-7B\n\nWe have tested PowerInfer on the following platforms:\n\n- x86-64 CPUs with AVX2 instructions, with or without NVIDIA GPUs, under **Linux**.\n- x86-64 CPUs with AVX2 instructions, with or without NVIDIA GPUs, under **Windows**.\n- Apple M Chips (CPU only) on **macOS**. (As we do not optimize for Mac, the performance improvement is not significant now.)\n\nAnd new features coming soon:\n\n- Metal backend for sparse inference on macOS\n\nPlease kindly refer to our [Project Kanban](https:\u002F\u002Fgithub.com\u002Forgs\u002FSJTU-IPADS\u002Fprojects\u002F2\u002Fviews\u002F2) for our current focus of development.\n\n## Getting Started\n\n- [Installation](#setup-and-installation)\n- [Model Weights](#model-weights)\n- [Inference](#inference)\n\n## Setup and Installation\n\n### Pre-requisites\n\nPowerInfer requires the following dependencies:\n\n- CMake (3.17+)\n- Python (3.8+) and pip (19.3+), for converting model weights and automatic FFN offloading\n\n### Get the Code\n\n```bash\ngit clone https:\u002F\u002Fgithub.com\u002FTiiny-AI\u002FPowerInfer\ncd PowerInfer\npip install -r requirements.txt # install Python helpers' dependencies\n```\n### Build\n\nIn order to build PowerInfer you have two different options. These commands are supposed to be run from the root directory of the project.\n\nUsing `CMake`(3.17+):\n* If you have an NVIDIA GPU:\n```bash\ncmake -S . -B build -DLLAMA_CUBLAS=ON\ncmake --build build --config Release\n```\n* If you have an AMD GPU:\n```bash\n# Replace '1100' to your card architecture name, you can get it by rocminfo\nCC=\u002Fopt\u002Frocm\u002Fllvm\u002Fbin\u002Fclang CXX=\u002Fopt\u002Frocm\u002Fllvm\u002Fbin\u002Fclang++ cmake -S . -B build -DLLAMA_HIPBLAS=ON -DAMDGPU_TARGETS=gfx1100\ncmake --build build --config Release\n```\n\n* If you have just CPU:\n\n```bash\ncmake -S . -B build\ncmake --build build --config Release\n```\n\n## Model Weights\n\nPowerInfer models are stored in a special format called *PowerInfer GGUF* based on GGUF format, consisting of both LLM weights and predictor weights.\n\n### Download PowerInfer GGUF via Hugging Face\n\nYou can obtain PowerInfer GGUF weights at `*.powerinfer.gguf` as well as profiled model activation statistics for 'hot'-neuron offloading from each Hugging Face repo below.\n\n| Base Model            | PowerInfer GGUF                                                                                               |\n| --------------------- | ------------------------------------------------------------------------------------------------------------- |\n| LLaMA(ReLU)-2-7B      | [PowerInfer\u002FReluLLaMA-7B-PowerInfer-GGUF](https:\u002F\u002Fhuggingface.co\u002FPowerInfer\u002FReluLLaMA-7B-PowerInfer-GGUF)     |\n| LLaMA(ReLU)-2-13B     | [PowerInfer\u002FReluLLaMA-13B-PowerInfer-GGUF](https:\u002F\u002Fhuggingface.co\u002FPowerInfer\u002FReluLLaMA-13B-PowerInfer-GGUF)   |\n| Falcon(ReLU)-40B      | [PowerInfer\u002FReluFalcon-40B-PowerInfer-GGUF](https:\u002F\u002Fhuggingface.co\u002FPowerInfer\u002FReluFalcon-40B-PowerInfer-GGUF) |\n| LLaMA(ReLU)-2-70B     | [PowerInfer\u002FReluLLaMA-70B-PowerInfer-GGUF](https:\u002F\u002Fhuggingface.co\u002FPowerInfer\u002FReluLLaMA-70B-PowerInfer-GGUF)   |\n| ProSparse-LLaMA-2-7B  | [PowerInfer\u002FProSparse-LLaMA-2-7B-GGUF](https:\u002F\u002Fhuggingface.co\u002FPowerInfer\u002Fprosparse-llama-2-7b-gguf)           |\n| ProSparse-LLaMA-2-13B | [PowerInfer\u002FProSparse-LLaMA-2-13B-GGUF](https:\u002F\u002Fhuggingface.co\u002FPowerInfer\u002Fprosparse-llama-2-13b-gguf)         |\n| Bamboo-base-7B 🌟      | [PowerInfer\u002FBamboo-base-v0.1-gguf](https:\u002F\u002Fhuggingface.co\u002FPowerInfer\u002FBamboo-base-v0.1-gguf)                   |\n| Bamboo-DPO-7B 🌟       | [PowerInfer\u002FBamboo-DPO-v0.1-gguf](https:\u002F\u002Fhuggingface.co\u002FPowerInfer\u002FBamboo-DPO-v0.1-gguf)                     |\n\nWe recommend using [`huggingface-cli`](https:\u002F\u002Fhuggingface.co\u002Fdocs\u002Fhuggingface_hub\u002Fguides\u002Fcli) to download the whole model repo. For example, the following command will download [PowerInfer\u002FReluLLaMA-7B-PowerInfer-GGUF](https:\u002F\u002Fhuggingface.co\u002FPowerInfer\u002FReluLLaMA-7B-PowerInfer-GGUF) into the `.\u002FReluLLaMA-7B` directory.\n\n```shell\nhuggingface-cli download --resume-download --local-dir ReluLLaMA-7B --local-dir-use-symlinks False PowerInfer\u002FReluLLaMA-7B-PowerInfer-GGUF\n```\n\nAs such, PowerInfer can automatically make use of the following directory structure for feature-complete model offloading:\n```\n.\n├── *.powerinfer.gguf (Unquantized PowerInfer model)\n├── *.q4.powerinfer.gguf (INT4 quantized PowerInfer model, if available)\n├── activation (Profiled activation statistics for fine-grained FFN offloading)\n│   ├── activation_x.pt (Profiled activation statistics for layer x)\n│   └── ...\n├── *.[q4].powerinfer.gguf.generated.gpuidx (Generated GPU index at runtime for corresponding model)\n```\n\n### Convert from Original Model Weights + Predictor Weights\n\nHugging Face limits single model weight to 50GiB. For unquantized models >= 40B, you can convert PowerInfer GGUF from the original model weights and predictor weights obtained from Hugging Face.\n\n| Base Model            | Original Model                                                                            | Predictor                                                                                                       |\n| --------------------- | ----------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------- |\n| LLaMA(ReLU)-2-7B      | [SparseLLM\u002FReluLLaMA-7B](https:\u002F\u002Fhuggingface.co\u002FSparseLLM\u002FReluLLaMA-7B)                   | [PowerInfer\u002FReluLLaMA-7B-Predictor](https:\u002F\u002Fhuggingface.co\u002FPowerInfer\u002FReluLLaMA-7B-Predictor)                   |\n| LLaMA(ReLU)-2-13B     | [SparseLLM\u002FReluLLaMA-13B](https:\u002F\u002Fhuggingface.co\u002FSparseLLM\u002FReluLLaMA-13B)                 | [PowerInfer\u002FReluLLaMA-13B-Predictor](https:\u002F\u002Fhuggingface.co\u002FPowerInfer\u002FReluLLaMA-13B-Predictor)                 |\n| Falcon(ReLU)-40B      | [SparseLLM\u002FReluFalcon-40B](https:\u002F\u002Fhuggingface.co\u002FSparseLLM\u002FReluFalcon-40B)               | [PowerInfer\u002FReluFalcon-40B-Predictor](https:\u002F\u002Fhuggingface.co\u002FPowerInfer\u002FReluFalcon-40B-Predictor)               |\n| LLaMA(ReLU)-2-70B     | [SparseLLM\u002FReluLLaMA-70B](https:\u002F\u002Fhuggingface.co\u002FSparseLLM\u002FReluLLaMA-70B)                 | [PowerInfer\u002FReluLLaMA-70B-Predictor](https:\u002F\u002Fhuggingface.co\u002FPowerInfer\u002FReluLLaMA-70B-Predictor)                 |\n| ProSparse-LLaMA-2-7B  | [SparseLLM\u002FProSparse-LLaMA-2-7B](https:\u002F\u002Fhuggingface.co\u002FSparseLLM\u002Fprosparse-llama-2-7b)   | [PowerInfer\u002FProSparse-LLaMA-2-7B-Predictor](https:\u002F\u002Fhuggingface.co\u002FPowerInfer\u002Fprosparse-llama-2-7b-predictor)   |\n| ProSparse-LLaMA-2-13B | [SparseLLM\u002FProSparse-LLaMA-2-13B](https:\u002F\u002Fhuggingface.co\u002FSparseLLM\u002Fprosparse-llama-2-13b) | [PowerInfer\u002FProSparse-LLaMA-2-13B-Predictor](https:\u002F\u002Fhuggingface.co\u002FPowerInfer\u002Fprosparse-llama-2-13b-predictor) |\n| Bamboo-base-7B 🌟      | [PowerInfer\u002FBamboo-base-v0.1](https:\u002F\u002Fhuggingface.co\u002FPowerInfer\u002FBamboo-base-v0_1)         | [PowerInfer\u002FBamboo-base-v0.1-predictor](https:\u002F\u002Fhuggingface.co\u002FPowerInfer\u002FBamboo-base-v0.1-predictor)           |\n| Bamboo-DPO-7B 🌟       | [PowerInfer\u002FBamboo-DPO-v0.1](https:\u002F\u002Fhuggingface.co\u002FPowerInfer\u002FBamboo-DPO-v0_1)           | [PowerInfer\u002FBamboo-DPO-v0.1-predictor](https:\u002F\u002Fhuggingface.co\u002FPowerInfer\u002FBamboo-DPO-v0.1-predictor)             |\n\nYou can use the following command to convert the original model weights and predictor weights to PowerInfer GGUF:\n```bash\n# make sure that you have done `pip install -r requirements.txt`\npython convert.py --outfile \u002FPATH\u002FTO\u002FPOWERINFER\u002FGGUF\u002FREPO\u002FMODELNAME.powerinfer.gguf \u002FPATH\u002FTO\u002FORIGINAL\u002FMODEL \u002FPATH\u002FTO\u002FPREDICTOR\n# python convert.py --outfile .\u002FReluLLaMA-70B-PowerInfer-GGUF\u002Fllama-70b-relu.powerinfer.gguf .\u002FSparseLLM\u002FReluLLaMA-70B .\u002FPowerInfer\u002FReluLLaMA-70B-Predictor\n```\nFor the same reason, we suggest keeping the same directory structure as PowerInfer GGUF repos after conversion.\n\n\u003Cdetails>\n\n\u003Csummary>Convert Original models into dense GGUF models(compatible with llama.cpp)\u003C\u002Fsummary>\n\n```bash\npython convert-dense.py --outfile \u002FPATH\u002FTO\u002FDENSE\u002FGGUF\u002FREPO\u002FMODELNAME.gguf \u002FPATH\u002FTO\u002FORIGINAL\u002FMODEL\n# python convert-dense.py --outfile .\u002FBamboo-DPO-v0.1-gguf\u002Fbamboo-7b-dpo-v0.1.gguf --outtype f16 .\u002FBamboo-DPO-v0.1\n```\n\nPlease note that the generated dense GGUF models might not work properly with llama.cpp, as we have altered activation functions (for ReluLLaMA and Prosparse models), or the model architecture (for Bamboo models). The dense GGUF models generated by convert-dense.py can be used for PowerInfer in dense inference mode, but might not work properly with llama.cpp.\n\n\u003C\u002Fdetails>\n\n## Inference\n\nFor CPU-only and CPU-GPU hybrid inference with all available VRAM, you can use the following instructions to run PowerInfer:\n```bash\n.\u002Fbuild\u002Fbin\u002Fmain -m \u002FPATH\u002FTO\u002FMODEL -n $output_token_count -t $thread_num -p $prompt\n# e.g.: .\u002Fbuild\u002Fbin\u002Fmain -m .\u002FReluFalcon-40B-PowerInfer-GGUF\u002Ffalcon-40b-relu.q4.powerinfer.gguf -n 128 -t 8 -p \"Once upon a time\"\n# For Windows: .\\build\\bin\\Release\\main.exe -m .\\ReluFalcon-40B-PowerInfer-GGUF\\falcon-40b-relu.q4.powerinfer.gguf -n 128 -t 8 -p \"Once upon a time\"\n```\n\nIf you want to limit the VRAM usage of GPU:\n```bash\n.\u002Fbuild\u002Fbin\u002Fmain -m \u002FPATH\u002FTO\u002FMODEL -n $output_token_count -t $thread_num -p $prompt --vram-budget $vram_gb\n# e.g.: .\u002Fbuild\u002Fbin\u002Fmain -m .\u002FReluLLaMA-7B-PowerInfer-GGUF\u002Fllama-7b-relu.powerinfer.gguf -n 128 -t 8 -p \"Once upon a time\" --vram-budget 8\n# For Windows: .\\build\\bin\\Release\\main.exe -m .\\ReluLLaMA-7B-PowerInfer-GGUF\\llama-7b-relu.powerinfer.gguf -n 128 -t 8 -p \"Once upon a time\" --vram-budget 8\n```\nUnder CPU-GPU hybrid inference, PowerInfer will automatically offload all dense activation blocks to GPU, then split FFN and offload to GPU if possible.\n\n\u003Cdetails>\n\u003Csummary>Dense inference mode (limited support)\u003C\u002Fsummary>\n\nIf you want to run PowerInfer to infer with the dense variants of the PowerInfer model family, you can use similarly as llama.cpp does:\n\n```bash\n.\u002Fbuild\u002Fbin\u002Fmain -m \u002FPATH\u002FTO\u002FDENSE\u002FMODEL -n $output_token_count -t $thread_num -p $prompt -ngl $num_gpu_layers\n# e.g.: .\u002Fbuild\u002Fbin\u002Fmain -m .\u002FBamboo-base-v0.1-gguf\u002Fbamboo-7b-v0.1.gguf -n 128 -t 8 -p \"Once upon a time\" -ngl 12\n```\n\nSo is the case for other `examples\u002F` like `server` and `batched_generation`. Please note that the dense inference mode is not a \"compatible mode\" for all models. We have altered activation functions (for ReluLLaMA and Prosparse models) in this mode to match with our model family. \n\n\u003C\u002Fdetails>\n\n## Serving, Perplexity Evaluation, and more applications\n\nPowerInfer supports serving and batched generation with the same instructions as llama.cpp. Generally, you can use the same command as llama.cpp, except for `-ngl` argument which has been replaced by `--vram-budget` for PowerInfer. Please refer to the detailed instructions in each `examples\u002F` directory. For example:\n\n- [Serving](.\u002Fexamples\u002Fserver\u002FREADME.md)\n- [Perplexity Evaluation](.\u002Fexamples\u002Fperplexity\u002FREADME.md)\n- [Batched Generation](.\u002Fexamples\u002Fbatched\u002FREADME.md)\n\n## Quantization\n\nPowerInfer has optimized quantization support for INT4(`Q4_0`) models. You can use the following instructions to quantize PowerInfer GGUF model:\n```bash\n.\u002Fbuild\u002Fbin\u002Fquantize \u002FPATH\u002FTO\u002FMODEL \u002FPATH\u002FTO\u002FOUTPUT\u002FQUANTIZED\u002FMODEL Q4_0\n# e.g.: .\u002Fbuild\u002Fbin\u002Fquantize .\u002FReluFalcon-40B-PowerInfer-GGUF\u002Ffalcon-40b-relu.powerinfer.gguf .\u002FReluFalcon-40B-PowerInfer-GGUF\u002Ffalcon-40b-relu.q4.powerinfer.gguf Q4_0\n# For Windows: .\\build\\bin\\Release\\quantize.exe .\\ReluFalcon-40B-PowerInfer-GGUF\\falcon-40b-relu.powerinfer.gguf .\\ReluFalcon-40B-PowerInfer-GGUF\\falcon-40b-relu.q4.powerinfer.gguf Q4_0\n```\nThen you can use the quantized model for inference with PowerInfer with the same instructions as above.\n\n## More Documentation\n- [Performance troubleshooting](.\u002Fdocs\u002Ftoken_generation_performance_tips.md)\n\n## Evaluation\n\nWe evaluated PowerInfer vs. llama.cpp on a single RTX 4090(24G) with a series of FP16 ReLU models under inputs of length 64, and the results are shown below. PowerInfer achieves up to 11x speedup on Falcon 40B and up to 3x speedup on Llama 2 70B.\n\n![github-eval-4090](https:\u002F\u002Fgithub.com\u002FSJTU-IPADS\u002FPowerInfer\u002Fassets\u002F34213478\u002Fd700fa6c-77ba-462f-a2fc-3fd21c898f33)\n\u003Csub>The X axis indicates the output length, and the Y axis represents the speedup compared with llama.cpp. The number above each bar indicates the end-to-end generation speed (total prompting + generation time \u002F total tokens generated, in tokens\u002Fs).\u003C\u002Fsub>\n\nWe also evaluated PowerInfer on a single RTX 2080Ti(11G) with INT4 ReLU models under inputs of length 8, and the results are illustrated in the same way as above. PowerInfer achieves up to 8x speedup on Falcon 40B and up to 3x speedup on Llama 2 70B.\n\n![github-eval-2080ti-q4](https:\u002F\u002Fgithub.com\u002FSJTU-IPADS\u002FPowerInfer\u002Fassets\u002F34213478\u002F0fc1bfc4-aafc-4e82-a865-bec0143aff1a)\n\nPlease refer to our [paper](https:\u002F\u002Fipads.se.sjtu.edu.cn\u002F_media\u002Fpublications\u002Fpowerinfer-20231219.pdf) for more evaluation details.\n\n## FAQs\n1. What if I encountered `CUDA_ERROR_OUT_OF_MEMORY`?\n   - You can try to run with `--reset-gpu-index` argument to rebuild the GPU index for this model to avoid any stale cache.\n   - Due to our current implementation, model offloading might not be as accurate as expected. You can try with `--vram-budget` with a slightly lower value or `--disable-gpu-index` to disable FFN offloading.\n\n2. Does PowerInfer support mistral, original llama, Qwen, ...?\n   - Now we only support models with ReLU\u002FReGLU\u002FSquared ReLU activation function. So we do not support these models now. It's worth mentioning that a [paper](https:\u002F\u002Farxiv.org\u002Fpdf\u002F2310.04564.pdf) has demonstrated that using the ReLU\u002FReGLU activation function has a negligible impact on convergence and performance.\n\n3. Why is there a noticeable downgrade in the performance metrics of our current ReLU model, particularly the 70B model?\n   - In contrast to the typical requirement of around 2T tokens for LLM training, our model's fine-tuning was conducted with only 5B tokens. This insufficient retraining has resulted in the model's inability to regain its original performance. We are actively working on updating to a more capable model, so please stay tuned.\n\n4. What if...\n   - Issues are welcomed! Please feel free to open an issue and attach your running environment and running parameters. We will try our best to help you.\n\n## TODOs\nWe will release the code and data in the following order, please stay tuned!\n\n- [x] Release core code of PowerInfer, supporting Llama-2, Falcon-40B.\n- [x] Support ~~Mistral-7B~~ (Bamboo-7B)\n- [x] Support Windows\n- [ ] Support text-generation-webui\n- [x] Release perplexity evaluation code\n- [ ] Support Metal for Mac\n- [ ] Release code for OPT models\n- [ ] Release predictor training code\n- [x] Support online split for FFN network\n- [ ] Support Multi-GPU\n\n\n## Paper and Citation\nMore technical details can be found in our [paper](https:\u002F\u002Fipads.se.sjtu.edu.cn\u002F_media\u002Fpublications\u002Fpowerinfer-20231219.pdf).\n\nIf you find PowerInfer useful or relevant to your project and research, please kindly cite our paper:\n\n```bibtex\n@misc{song2023powerinfer,\n      title={PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPU},\n      author={Yixin Song and Zeyu Mi and Haotong Xie and Haibo Chen},\n      year={2023},\n      eprint={2312.12456},\n      archivePrefix={arXiv},\n      primaryClass={cs.LG}\n}\n```\n\n## Acknowledgement\nWe are thankful for the easily modifiable operator library [ggml](https:\u002F\u002Fgithub.com\u002Fggerganov\u002Fggml) and execution runtime provided by [llama.cpp](https:\u002F\u002Fgithub.com\u002Fggerganov\u002Fllama.cpp). We also extend our gratitude to [THUNLP](https:\u002F\u002Fnlp.csai.tsinghua.edu.cn\u002F) for their support of ReLU-based sparse models. We also appreciate the research of [Deja Vu](https:\u002F\u002Fproceedings.mlr.press\u002Fv202\u002Fliu23am.html), which inspires PowerInfer.\n","PowerInfer 是一个用于本地部署的高速大语言模型推理引擎。它利用激活局部性优化技术，支持CPU和GPU设备，特别是消费级GPU，以实现高效的模型推理。项目采用了C++开发，具备高度优化的计算性能，并且通过稀疏化技术显著减少了模型参数量，同时保持了优秀的推理速度与准确度。适用于需要在个人电脑、移动设备或边缘计算场景中运行大规模语言模型的应用场合，如自然语言处理任务的快速响应服务。",2,"2026-06-11 03:44:00","high_star"]