[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"project-524":3},{"id":4,"name":5,"fullName":6,"owner":7,"repo":5,"description":8,"homepage":9,"htmlUrl":10,"language":11,"languages":9,"totalLinesOfCode":9,"stars":12,"forks":13,"watchers":14,"openIssues":15,"contributorsCount":9,"subscribersCount":16,"size":16,"stars1d":17,"stars7d":18,"stars30d":19,"stars90d":16,"forks30d":16,"starsTrendScore":20,"compositeScore":21,"rankGlobal":9,"rankLanguage":9,"license":9,"archived":22,"fork":22,"defaultBranch":23,"hasWiki":22,"hasPages":22,"topics":9,"createdAt":9,"pushedAt":9,"updatedAt":24,"readmeContent":25,"aiSummary":26,"trendingCount":16,"starSnapshotCount":16,"syncStatus":27,"lastSyncTime":28,"discoverSource":29},524,"ik_llama.cpp","ikawrakow\u002Fik_llama.cpp","ikawrakow","llama.cpp fork with additional SOTA quants and improved performance",null,"https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp","C++",2722,349,34,44,0,26,98,397,78,29.63,false,"main","2026-06-12 02:00:14","# ik_llama.cpp: llama.cpp fork with better CPU performance\n\n[![License: MIT](https:\u002F\u002Fimg.shields.io\u002Fbadge\u002Flicense-MIT-blue.svg)](https:\u002F\u002Fopensource.org\u002Flicenses\u002FMIT)\n\n## TL;DR\n\nThis repository is a fork of [llama.cpp](https:\u002F\u002Fgithub.com\u002Fggerganov\u002Fllama.cpp) with better CPU and hybrid GPU\u002FCPU performance, new SOTA quantization types, first-class Bitnet support, better DeepSeek performance via MLA, FlashMLA, fused MoE operations and tensor overrides for hybrid GPU\u002FCPU inference, row-interleaved quant packing, etc.\n\n>[!IMPORTANT]\n>If you are running hybrid CPU\u002FGPU inference for MoE models with all or some experts left on the CPU, **do not use -rtr** unless you know what you are doing. The `-rtr` option causes all tensors left in RAM to be repacked to row-interleaved format while loading the model. As not all quantization types have a CUDA implementation, this will result in matrix multiplications with these tensors to be **always done on the CPU**, even when it would have been much better to offload the computation to the GPU, typically resulting in much lower prompt processing speed. Most notably, k-quants (`K2_K, Q3_K, Q4_K, Q5_K, Q6_K`) do not have CUDA row-interleaved implementation.\n\n>[!NOTE]\n>The only fully functional and performant compute backends are CPU (`AVX2` or better, `ARM_NEON` or better) and CUDA (Turing or newer). \n>Please do not enter issues related to ROCm, Vulkan, Metal, old Nvidia GPUs, `AVX` CPUs, etc. They will not get resolved unless you roll up your sleeves and help bring your favorite backend up to speed. With the current regular contributors this project simply does not have the bandwidth to work on all backends available in `llama.cpp`.\n \n>[!IMPORTANT]\n>Do not use quantized models from Unsloth that have `_XL` in their name. These are likely to not work with `ik_llama.cpp`.\n>\n>The above has caused some stir, so to clarify: the Unsloth `_XL` models that are likely to not work are those that contain `f16` tensors (which is never a good idea in the first place). All others are fine.\n\n>[!NOTE]\n>Some users have reported issues with graph parallel (a.k.a. split mode `graph`) and partial GPU offload (using `--cpu-moe` or `--n-cpu-moe` or tensor overrides). If you are using\u002Fwant to use split mode graph and observe gibberish\u002Fincoherent responses, try adding `-cuda graphs=0` to your command line.\n  \n## Quickstart\n\n### Prerequisites\n\n```\ngit clone https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\n\ncd ik_llama.cpp\n```\n\nOn Debian\u002FUbuntu Linux, install the required packages (if using another Linux distro, you need to find the corresponding packages and adapt):\n\n```\napt-get update && apt-get install build-essential git libcurl4-openssl-dev curl libgomp1 cmake\n```\n\n### Build for CPU\n\n```\ncmake -B build -DGGML_NATIVE=ON\n\ncmake --build build --config Release -j$(nproc)\n```\n\nFor AVX-512-capable CPUs (AMD Zen4 \u002F Intel Sapphire Rapids+), see\n[`docs\u002Fbuild.md`](docs\u002Fbuild.md) section \"CPU build flags for AVX-512\" for the\nadditional flags that activate the IQK quantized GEMM kernels (the\n`HAVE_FANCY_SIMD` path). Without those flags, a vanilla `Release` build\nsilently falls back to the AVX2 path on this hardware.\n\n### Build for GPU\n\nInstall Nvidia Drivers and [CUDA Toolkit](https:\u002F\u002Fdeveloper.nvidia.com\u002Fcuda\u002Ftoolkit).\n\n```\ncmake -B build -DGGML_NATIVE=ON -DGGML_CUDA=ON\n\ncmake --build build --config Release -j$(nproc)\n```\n### Step-by-step instructions for a case of a successful Windows build\nhttps:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fblob\u002Fmain\u002Fdocs\u002Fbuild.md\n\n### Run\n\nDownload `.gguf` model files (e.g. [bartowski\u002FQwen_Qwen3-0.6B-IQ4_NL.gguf](https:\u002F\u002Fhuggingface.co\u002Fbartowski\u002FQwen_Qwen3-0.6B-GGUF\u002Fblob\u002Fmain\u002FQwen_Qwen3-0.6B-IQ4_NL.gguf)) to your favorite directory (e.g. `\u002Fmy_local_files\u002Fgguf`).\n\nStart the server with one of the commands (CPU or GPU):\n\n```\n.\u002Fbuild\u002Fbin\u002Fllama-server --model \u002Fmy_local_files\u002Fgguf\u002FQwen_Qwen3-0.6B-IQ4_NL.gguf --ctx-size 4096\n```\n\n```\n.\u002Fbuild\u002Fbin\u002Fllama-server --model \u002Fmy_local_files\u002Fgguf\u002FQwen_Qwen3-0.6B-IQ4_NL.gguf --ctx-size 4096 -ngl 999\n```\n\nThat's all! Open [http:\u002F\u002F127.0.0.1:8080](http:\u002F\u002F127.0.0.1:8080) in Browser start chatting.\n\n\n### [Step by step guide](.\u002Fdocker\u002FREADME.md) for ik_llama.cpp in podman\u002Fdocker container including llama-swap\n\n### [Common parameters and options](.\u002Fdocs\u002Fparameters.md)\n\n## Latest News\n\n\n### Model Support\n\nLlaMA-3-Nemotron [PR 377](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F377), Qwen3 [PR 355](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F355), GLM-4 [PR 344](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F344), Command-A [PR 341](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F341), bitnet-b1.58-2B-4T [PR 337](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F337), LLaMA-4 [PR 321](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F321), Gemma3 [PR 276](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F276),  DeepSeek-V3 [PR 176](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F176), Kimi-2 [PR 609](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F609), dots.llm1 [PR 573](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F573), Hunyuan [PR 565](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F565), GLM-4.5 [PR 668](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F668) (4.5\u002F4.6\u002F4.7\u002FAIR), Ernie 4.5 MOE and 0.3B [PR 759](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F759), grok-2 [PR 782](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F782), Ling\u002FRing (Bailing-MoE2) [PR 833](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F833), Qwen3-VL [PR 883](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F883), SmolLM3 [PR 934](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F934), GigaChat3 [PR 995](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F995), ministral3 [PR 1030](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F1030), Mimo-V2-Flash [PR 1096](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F1096), GLM-4.7-Flash [PR 1168](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F1168), Seed-OSS [PR 1218](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F1218), Step-3.5-Flash [PR 1231](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F1231), GLM-5 [PR 1268](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F1268), Qwen3-Next [PR 1266](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F1266), Qwen3.5-MoE [PR 1288](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F1288) and dense Qwen-3.5 [1326](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F1326), Mistral 4 [PR 1450](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F1450), Bonsai 1-bit [PR 1570](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F1570), Gemma4 [PR 1581](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F1581)\n\n### Quantization\n\n#### Quantization additions\n\n##### Trellis quants (`IQ1_KT`, `IQ2_KT`, `IQ3_KT`, `IQ4_KT`)\n\nInformation and the original CUDA implementation in [PR 113](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F113). Additional implementations: Metal [PR 475](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F475), Neon [PR 471](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F471), CPU [PR 441](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F441). `IQ1_KT` was added more recently in [PR 616](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F616). Note: these are base on a novel, integer-base trellis, which allows to achieve reasonable CPU performance, see [PR 529](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F529) and PRs quoted there for details. \n\n##### IQK quants\n\nInformation can be found in [Discussion 8](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fdiscussions\u002F8).\n\nInitial implementations (Zen4, AVX2, NEON): `IQ5_KS_R4` [PR 426](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F426), `IQ5_KS` [PR 422](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F422), `IQ4_KS_R4` [PR 150](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F150), `IQ5_K_R4` [PR 149](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F149), `IQ2_K_R4` [PR 146](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F146), `IQ3_K_R4` [PR 145](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F145), `IQ4_K_R4` [PR 138](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F138), `IQ4_KSS` [PR 89](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F89), `IQ2_KS` [PR 85](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F85), `IQ4_KS` [PR 83](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F83), `IQ6_K` [PR 14](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F14), `IQ2_K, IQ3_K and IQ5_K` [PR 7](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F7), `IQ4_K` [PR 6](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F6)\n\nCuda implementations:  `IQ4_KS_R4` and `IQ5_KS_R4` [PR 493](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F493), `IQ1_S_R4` [PR 492](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F492), `IQ1_M_R4` [PR 494](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F494). `IQ4_KS_R4` and `IQ5_KS_R4` [PR 462](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F462), `IQ2_K_R4`, `IQ3_K_R4`, `IQ4_K_R4`, `IQ5_K_R4` [PR 461](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F461), `IQ4_K, IQ5_K, IQ6_K` [PR 417](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F417), `IQ2_KS, IQ2_K, IQ3_K` [PR 418](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F417)\n\n`IQ2_KL` is a more recent addition in [PR 602](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F602) \n\n##### Hadamard transforms for K-cache\n\nCPU [PR 1033](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F1033) and CUDA [PR 1034](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F1034)\n\n##### Hadamard transforms for V-cache\n\n[PR 1527](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F1527)\n\n##### MXFP4 as used in gpt-oss models\n\nImplemented for Zen4, AVX2, ARM_NEON, Metal, CUDA [PR 682](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F682) \n\n#### Quantization improvements\n\n* `IQ1_M` [PR 327](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F327), `IQ2_XS` [PR 312](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F312), `Q2_K, Q4_K, Q5_K, Q4_1, Q5_1` [PR 302](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F302), `Q4_0, Q5_0, Q6_0, Q3_K, Q6_K, IQ4_XS, IQ4_NL` [PR 295](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F295)\n* Low perplexity `Q4_0` KV cache [PR 1547](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F1547) [PR 1556](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F1556)\n\n#### Quantization performance improvements \n\n* Much faster CPU prompt processing for all non-interleaved quants. Initial idea in [PR 515](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F515) and [PR 531](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F531), with many follow up PRs to apply to all quantization types for the 3 supported CPU platforms.\n* All quantization types now have quantized matrix multiplication CUDA kernels, see [PR 557](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F515) and several others\n* Faster CPU prompt processing for Trellis quants and MoE models. [PR 488](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F488)\n* Trellis quants: faster CPU prompt processing [PR 482](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F482).\n* Minor (~2%) `iq2_ks` TG performance improvement on CUDA [PR 468](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F468)\n* Faster `IQ3_KT` and `IQ4_KT` [PR 453](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F453)\n* Zen4: Faster PP for `IQ2_KS, IQ4_KS, IQ5_KS` [PR 428](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F428)\n* Fast GEMM\u002FGEMV for `IQ1_S` [PR 212](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F212)\n* AVX-VNNI optimizations [PR 1446](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F1446) [PR 1455](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F1455) [PR 1467](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F1467) [PR 1474](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F1474) [PR 1482](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F1482)\n\n### Features\n\n* New split mode \"graph\" for multi GPU setups [PR 1022](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F1022)\n* Fused delta-net for Qwen3-Next and Qwen3.5-MoE [PR 1315](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F1315) [PR 1333](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F1333) [PR 1362](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F1362) [PR 1373](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F1373)\n* Hadamard transforms for K-cache and V-cache [PR 1033](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F1033) [PR 1034](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F1034) [PR 1527](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F1527)\n* Auto-fit offloaded tensors to available VRAM (MoE and dense models) [PR 1501](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F1501) [PR 1504](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F1504)\n* Checkpoints for recurrent models [PR 1310](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F1310) [PR 1398](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F1398)\n* String ban function for all completions [PR 1185](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F1185) [PR 1243](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F1243)\n* OpenAI `\u002Fv1\u002Fresponses` API endpoint [PR 1184](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F1184)\n* Function call support [PR 628](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F628)\n* jinja template support [PR 677](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F677)\n* Webui: New Features for Conversations, Settings, and Chat Messages [PR 618](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F618)\n* MTP decoding support for GLM-4.x MoE [1270](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F1270)\n* Self speculative decoding, ngram [PR 1261](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F1261)\n* Dynamic control vector management endpoints [PR 1223](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F1223)\n* Legacy quants conversion schemes in `convert_hf_to_gguf.py` [PR 449](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F449), `Q6_0` in [PR 483](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F483)\n* Adaptive-P Sampler [PR 1100](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F1100) implemented as designed by it's author; supported on Webui\n* Multi-modal Vision support in `llama-mtmd-cli` [PR 798](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F798) and in `llama-server` [PR 901](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F901)\n* mikupad as an alternative WebUI [PR 558](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F558)\n* June 8 2025: Webui updated (legacy still available when `--path .\u002Fexamples\u002Fserver\u002Fpublic_legacy` is passed) [PR 481](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F481)\n* June 8 2025: RPC improvements [PR 480](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F480)\n* June 7 2025: Add an endpoint that lists all the saved prompt caches to server [PR 502](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F502)\n* June 6 2025: Make prompt cache saving and restoring MLA aware [PR 497](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F497)\n* June 3 2025: Added samplers, XTC [PR 486](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F486), top-n σ [PR 489](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F489).\n* May 22 2025: Refactor `iqk_mul_mat.cpp` which speeds up compilation time significantly. [PR 435](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F435)\n* May 17 2025: Option to enable or disable the CPU FA kernels [PR 429](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F429).\n* May 12 2025: User can now control if\u002Fwhich operations with tensors held in RAM are offloaded to the GPU. See [PR 405](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F405) \n* May 12 2025: Compatibility issues with mainline `llama.cpp` GGUFs for DeepSeek models with MLA enabled were resolved in [PR 394](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F394). The lower prompt processing performance resulting from using `llama.cpp`-style MLA GGUFs was recovered in [PR 409](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F409).\n* April 21 2025: ik_llama.cpp builds and runs successfully on Android (using termux), see [PR 336](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F336)\n* March 1 2025: Smart Expert Reduction for faster DeepSeek inference [PR 239](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F239) \n* Feb 25 2025: Tensor overrides for better control where model weights are stored (GPU or CPU) [PR 232](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F232)\n* Feb 23 2025: `sweep-bench` - better performance benchmarking [PR 225](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F225)\n* Feb 19 2025: `Q8_KV` - new type for 8-bit KV-cache quantization [PR 208](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F208)\n* March 7 2025: Custom quantization mixes using regular expressions [PR 244](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F244)\n\n### Performance improvements\n\n* Better GPU offload strategy for MoE models when using hybrid HPU\u002FCPU inference, see [PR 520](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F520)\n* Much faster rng sampling [PR 1187](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F1187)\n* May 13 2025: Better CPU FA performance for DeepSeek-Lite. [PR 410](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F410)\n* May 11 2025: Slightly faster flash attention for DeepSeek models on CUDA, along with extending compatibility to Touring or newer GPUs. [PR 408](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F408)\n* May 4 2025: Significant token generation performance improvement on CUDA with Flash Attention for GQA models. For details and benchmarks. [PR 370](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F370) \n* April 17 2025: Better CPU Flash Attention token generation performance. [PR 332](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F332)\n* April 3 2025: Much faster MoE implementation on Metal. [PR 307](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F307) \n* March 25 2025: Better MoE performance on CUDA [PR 283](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F283)\n* March 23 2025: Better batched processing speed for DeepSeek models [PR 282](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F282)\n* March 18 2025: Reduce compute buffer size [PR 237](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F237)\n* March 10 2025: Better TG performance for MoE models on CUDA [PR 248](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F248)\n* Feb 23 2025: Fused FFN ops for faster MoE inference [PR 229](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F229)\n\n### Flash-MLA\n\n* May 7 2025: 🚀 FlashMLA-3 for DeepSeek models on CUDA. [PR 386](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F386). Caveat: Ampere or newer Nvidia GPU required\n* March 21 2025: 🚀 FlashMLA-3: fastest CPU-only inference for DeepSeek models [PR 273](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F273)\n* March 17 2025: 🚀 FlashMLA-2 performance improvements [PR 253](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F253)\n* March 12 2025: Allow `Q8_0` KV cache with FlashMLA-2 on CUDA [PR 265](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F265)\n* March 9 2025: 🚀 FlashMLA on CUDA [PR 247](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F247)\n* March 8 2025: 🚀 Faster FlashMLA CPU implementation [PR 243](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F243)\n* March 3 2025: 🚀 Introducing FlashMLA - MLA with Flash Attention [PR 240](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F240)\n* Feb 27 2025: MLA without transposed cache [PR 235](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F235)\n* Feb 13 2025: Allow `Q8_0` quantized cache with MLA [PR 206](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F206)\n* Feb 11 2025: 🚀 Flash Attention support for DeepSeek models [PR 200](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F200)\n* Feb 9 2025: 🚀 MLA for DeepSeek models [PR 188](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F188)\n\n### Fixes\n\n* Fix bug in MMVQ kernel [PR 446](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F446)\n* Fix AVX2 implementation of `IQ4_K, IQ4_KS, IQ5_K, IQ6_K` [PR 427](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F427) \n* Fix standard attention on the CPU [PR 421](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F421) \n* Fix imatrix calculation for MLA models [PR 411](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F411)\n* Fix new CUDA FA on Touring [PR 413](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F413)\n* Fix SER. CPU: [PR 415](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F415) CUDA: [PR 416](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fpull\u002F416)\n\n## Resources\n\nThere is no single point of reference describing all new `ik_llama.cpp` features. Pull requests often contain detailed information, so browsing the PRs is often the best way to learn about new features and how to use them. In addition\n* [The Wiki page](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fwiki) has performance comparisons to mainline `llama.cpp`\n* [This guide](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fdiscussions\u002F258) is a good place to start if you came here because of DeepSeek models\n* [This discussion](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fdiscussions\u002F266) is about running DeepSeek-V3\u002FR1 on a 16 x 3090 setup\n* [This discussion](https:\u002F\u002Fgithub.com\u002Fikawrakow\u002Fik_llama.cpp\u002Fdiscussions\u002F8) describes the new quantization types available in `ik_llama.cpp`\n\n## Testing\n\n### Function Calls Tests\n\nTo run the function calls test suite:\n\n```bash\ncd build\ncmake --build . --target test-function-calls\n.\u002Fbin\u002Ftest-function-calls\n```\n\nThe test suite covers parser functionality, streaming, error handling, content cleaning, and server integration. All tests should pass to ensure production readiness.\n\n## Contributing\n\nContributions in form of pull requests, issue submissions (bug reports, feature requests), or general discussions, are welcome.\n\n## License\n\n- [subprocess.h](https:\u002F\u002Fgithub.com\u002Fsheredom\u002Fsubprocess.h) - Single-header process launching solution for C and C++ - Public domain\n- [server](example\u002Fserver\u002FREADME.md)\n- [GBNF grammars](grammars\u002FREADME.md)\n\n#### Development documentation\n\n- [How to build](docs\u002Fbuild.md)\n- [Running on Docker](docs\u002Fdocker.md)\n- [Performance troubleshooting](docs\u002Fdevelopment\u002Ftoken_generation_performance_tips.md)\n- [GGML tips & tricks](https:\u002F\u002Fgithub.com\u002Fggml-org\u002Fllama.cpp\u002Fwiki\u002FGGML-Tips-&-Tricks)\n\n#### Seminal papers and background on the models\n\nIf your issue is with model generation quality, then please at least scan the following links and papers to understand the limitations of LLaMA models. This is especially important when choosing an appropriate model size and appreciating both the significant and subtle differences between LLaMA models and ChatGPT:\n- LLaMA:\n    - [Introducing LLaMA: A foundational, 65-billion-parameter large language model](https:\u002F\u002Fai.facebook.com\u002Fblog\u002Flarge-language-model-llama-meta-ai\u002F)\n    - [LLaMA: Open and Efficient Foundation Language Models](https:\u002F\u002Farxiv.org\u002Fabs\u002F2302.13971)\n- GPT-3\n    - [Language Models are Few-Shot Learners](https:\u002F\u002Farxiv.org\u002Fabs\u002F2005.14165)\n- GPT-3.5 \u002F InstructGPT \u002F ChatGPT:\n    - [Aligning language models to follow instructions](https:\u002F\u002Fopenai.com\u002Fresearch\u002Finstruction-following)\n    - [Training language models to follow instructions with human feedback](https:\u002F\u002Farxiv.org\u002Fabs\u002F2203.02155)\n\n## Completions\nCommand-line completion is available for some environments.\n\n#### Bash Completion\n```bash\n$ build\u002Fbin\u002Fllama-cli --completion-bash > ~\u002F.llama-completion.bash\n$ source ~\u002F.llama-completion.bash\n```\nOptionally this can be added to your `.bashrc` or `.bash_profile` to load it\nautomatically. For example:\n```console\n$ echo \"source ~\u002F.llama-completion.bash\" >> ~\u002F.bashrc\n```\n\n## Dependencies\n\n- [yhirose\u002Fcpp-httplib](https:\u002F\u002Fgithub.com\u002Fyhirose\u002Fcpp-httplib) - Single-header HTTP server, used by `llama-server` - MIT license\n- [stb-image](https:\u002F\u002Fgithub.com\u002Fnothings\u002Fstb) - Single-header image format decoder, used by multimodal subsystem - Public domain\n- [nlohmann\u002Fjson](https:\u002F\u002Fgithub.com\u002Fnlohmann\u002Fjson) - Single-header JSON library, used by various tools\u002Fexamples - MIT License\n- [miniaudio.h](https:\u002F\u002Fgithub.com\u002Fmackron\u002Fminiaudio) - Single-header audio format decoder, used by multimodal subsystem - Public domain\n- [subprocess.h](https:\u002F\u002Fgithub.com\u002Fsheredom\u002Fsubprocess.h) - Single-header process launching solution for C and C++ - Public domain\n","ik_llama.cpp 是一个基于 llama.cpp 的改进版本，旨在提高 CPU 和混合 GPU\u002FCPU 性能。该项目引入了最新的量化技术、Bitnet 支持以及通过 MLA、FlashMLA 和融合 MoE 操作优化的 DeepSeek 性能，并支持混合 GPU\u002FCPU 推理的张量覆盖和行交错量化打包。适用于需要高效处理大型语言模型（特别是 MoE 模型）的场景，尤其是在 CPU 和 GPU 协同工作的环境中。项目主要支持现代 CPU（AVX2 或更高版本，ARM NEON 或更高版本）和 CUDA（Turing 或更新版本）作为计算后端。",2,"2026-06-11 02:37:06","trending"]