[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"project-83883":3},{"id":4,"name":5,"fullName":6,"owner":7,"repo":5,"description":8,"homepage":8,"htmlUrl":8,"language":9,"languages":8,"totalLinesOfCode":8,"stars":10,"forks":11,"watchers":12,"openIssues":13,"contributorsCount":14,"subscribersCount":14,"size":14,"stars1d":15,"stars7d":15,"stars30d":15,"stars90d":14,"forks30d":14,"starsTrendScore":16,"compositeScore":17,"rankGlobal":8,"rankLanguage":8,"license":18,"archived":19,"fork":19,"defaultBranch":20,"hasWiki":21,"hasPages":19,"topics":22,"createdAt":8,"pushedAt":8,"updatedAt":23,"readmeContent":24,"aiSummary":8,"trendingCount":14,"starSnapshotCount":14,"syncStatus":25,"lastSyncTime":26,"discoverSource":27},83883,"llama.cpp-MTP-TurboQuant","BoFan-tunning\u002Fllama.cpp-MTP-TurboQuant","BoFan-tunning",null,"C++",100,19,1,6,0,3,9,3.9,"MIT License",false,"merge-mtp-turboquant",true,[],"2026-06-12 02:04:36","# llama.cpp\n\n![llama](https:\u002F\u002Fuser-images.githubusercontent.com\u002F1991296\u002F230134379-7181e485-c521-4d23-a0d6-f7b3b61ba524.png)\n\n[![License: MIT](https:\u002F\u002Fimg.shields.io\u002Fbadge\u002Flicense-MIT-blue.svg)](https:\u002F\u002Fopensource.org\u002Flicenses\u002FMIT)\n[![Release](https:\u002F\u002Fimg.shields.io\u002Fgithub\u002Fv\u002Frelease\u002Fggml-org\u002Fllama.cpp)](https:\u002F\u002Fgithub.com\u002Fggml-org\u002Fllama.cpp\u002Freleases)\n[![Server](https:\u002F\u002Fgithub.com\u002Fggml-org\u002Fllama.cpp\u002Factions\u002Fworkflows\u002Fserver.yml\u002Fbadge.svg)](https:\u002F\u002Fgithub.com\u002Fggml-org\u002Fllama.cpp\u002Factions\u002Fworkflows\u002Fserver.yml)\n\n[Manifesto](https:\u002F\u002Fgithub.com\u002Fggml-org\u002Fllama.cpp\u002Fdiscussions\u002F205) \u002F [ggml](https:\u002F\u002Fgithub.com\u002Fggml-org\u002Fggml) \u002F [ops](https:\u002F\u002Fgithub.com\u002Fggml-org\u002Fllama.cpp\u002Fblob\u002Fmaster\u002Fdocs\u002Fops.md)\n\nLLM inference in C\u002FC++\n\n## llama.cpp MTP+TurboQuant 融合版\n\n整合 **MTP (Multi-Token Prediction)** + **TurboQuant**，推理速度起飞！提升幅度 **2-5 倍**。\n\n> ✅ **Vision 多模态已修复**：MTP 模式现已完美支持图像输入，多模态 + 推测解码同时启用不再崩溃。\n>\n> ✅ **MTP + TurboQuant 完美融合**：同时享受 MTP 推测解码的加速和 TurboQuant KV Cache 压缩的显存节省。\n>\n> ⚠️ **模型要求**：必须配合**内置 MTP 头部**的 GGUF 文件使用（如 Qwen3.6-27B-Q4_K_P_mtp.gguf），普通 GGUF 无法启用 MTP。\n\n### 核心特性\n\n| 特性 | 说明 |\n|------|------|\n| **MTP 推测解码** | 每步预测多个 token，推理吞吐提升 2-5 倍 |\n| **TurboQuant KV Cache** | `-ctk q8_0 -ctv turbo3` 非对称压缩，相比 F16 节省 76% 显存 |\n| **Vision 多模态支持** | MTP + 图像输入同时启用，已修复上游崩溃问题 |\n| **Qwen 3.6 智能思考模板** | 新增增强版 Jinja 模板，实现智能思考支持 |\n| **Tool Calling 完美兼容** | 修复官方模板 9 大缺陷，多层嵌套 JSON 正常渲染 |\n\n---\n\n### 编译（CUDA 版本）\n\n```batch\n@echo off\ncd \u002Fd \"源代码目录如 F:\\llamacpp_MTP_TurboQuant\"\n\n:: ⚡ CUDA 架构号请根据自身显卡修改（例如：75=RTX 2080, 89=RTX 4090, 90=RTX 5090）\ncmake -B build ^\n  -DGGML_CUDA=ON ^\n  -DCMAKE_CUDA_ARCHITECTURES=\"75\" ^\n  -DGGML_CUDA_FA_ALL_QUANTS=ON ^\n  -DGGML_NATIVE=OFF ^\n  -DCMAKE_BUILD_TYPE=Release\n\ncmake --build build --config Release -j --target llama-server llama-cli\n```\n\n> **（根据你的显卡型号修改 `CMAKE_CUDA_ARCHITECTURES`）** 常见对照：\n> - `75` — Turing (RTX 20xx, T4)\n> - `80` — Ampere (RTX 30xx, A100)\n> - `86` — Ampere (RTX 30xx 消费级)\n> - `89` — Ada Lovelace (RTX 40xx)\n> - `120` — Blackwell (RTX 50xx)\n\n---\n\n### 运行（llama-server）\n\n```batch\n@echo off\ncd \u002Fd \"llama-server.exe文件所在目录如 F:\\mtp_llamacpp\\llama.cpp\\build\\bin\\Release\\\"\n\nset CUDA_SCALE_LAUNCH_QUEUES=4x\n\nllama-server.exe -m \"mtp属性gguf文件路径 如D:\\wd3.7\\Qwen3.6-27B-Q4_K_P_mtp.gguf\" --mmproj \"多模态投影文件路径 如D:\\wd3.7\\mmproj-Qwen3.6-27B.gguf\" ^\n  --spec-type mtp --spec-draft-n-max 2 ^\n  -ctk q8_0 -ctv turbo3 ^\n  -c 8000 -b 2048 -ub 512 ^\n  --n-gpu-layers 99 ^\n  --host 0.0.0.0 --port 8080 ^\n  --temp 0.7 --top-k 20 ^\n  -np 1 -fa on ^\n  -t 7 ^\n  --jinja ^\n  --chat-template-file \"聊天模板文件路径如：F:\\llamacpp_MTP_TurboQuant\\3.6_chat_template-v10.jinja\" ^\n  --reasoning auto ^\n  --reasoning-format deepseek\n\npause\n```\n\n> **参数说明（请根据自身硬件调整）**：\n> - `--spec-type mtp` — 启用 MTP 投机解码\n> - `--spec-draft-n-max 2` — MTP 每步预测 2 个候选 token\n> - `-ctk q8_0 -ctv turbo3` — KV Cache 非对称压缩（K 用 8-bit 保精度，V 用 TurboQuant 3-bit 省显存），相比全 F16 节省约 76% 显存\n> - `--mmproj` — 多模态投影文件，启用视觉识别能力\n> - `-c 8000` — 上下文长度（根据显存调整）\n> - `-t 7` — CPU 线程数（根据你的 CPU 核心数调整）\n> - `--n-gpu-layers 99` — 全量 GPU 卸载\n> - `--jinja` + `--chat-template-file` — 使用增强版 Jinja 模板\n\n---\n\n## 3.6_chat_template-v10.jinja — Qwen 3.6 的超级优化模板\n\n### 超级特性 配合上面的 llama-server 运行命令实现 智能判断是否开启*思考*\n\n\n\n该模板专为 llama.cpp 的 minijinja 引擎深度优化，解决了官方模板 9 大缺陷，以下是其在 **llama.cpp 环境下的核心优势**：\n\n| 优势 | 说明 |\n|------|------|\n| **C++ 引擎原生兼容** | 彻底移除 Python 专属语法（`\\|items`、`\\|safe`），使用字典直接取值 + `is iterable` 替代，minijinja 零错误渲染 |\n| **智能自动思考（Auto-Thinking）** | 自动判断用户输入长度：短问题（≤30 字符）跳过思考 → 秒回；长问题（≥300 字符）强制思考 → 深度推理。阈值可通过 `auto_think_short_threshold` \u002F `auto_think_force_threshold` 自定义 |\n| **思考开关标签** | 在 system \u002F user 消息中插入 `\u003C\\|think_off\\|>` 或 `\u003C\\|think_on\\|>` 即可实时切换推理模式，标签在渲染时自动移除，模型完全感知不到 |\n| **`\u003C\u002Fthinking>` 幻觉恢复** | Qwen 3.6 有时会输出 `\u003C\u002Fthinking>` 而非 ` response`，模板自动检测两种闭合标签并动态分割，防止推理流中断 |\n| **思考未闭合自动修复** | 模型在 thinking 块中直接调用 tool 时（未输出 ` response`），模板自动注入闭合标签，防止 XML 标签污染工具调用 |\n| **Tool Call 参数完美兼容** | 支持 string \u002F object 两种参数格式，多层嵌套 JSON 正常渲染，`-ctk q4_1 -ctv q4_1` 缓存下依然稳定 |\n| **多轮工具调用（Agent 友好）** | 正向遍历检测 multi-step tool chain，无用户查询时优雅回退而非崩溃，适配 OpenCode、Docker Agent 等框架 |\n| **对话中 system 消息** | 官方模板在非首条 system 消息时直接崩溃；本模板按时间顺序渲染，兼容所有 agent 框架的中间指令注入 |\n| **`developer` 角色支持** | 完整映射 OpenAI API 的 `developer` 角色 |\n| **Generation Prompt 精细控制** | 模板结尾根据 `enable_thinking` 状态精确输出 ` thinking\\n`（启用思考）或 ` thinking\\n\\n response\\n\\n`（快速回答），引导模型输出格式 |\n\n> **关键参数**：搭配 `--reasoning auto --reasoning-format deepseek` 使用，llama.cpp 可自动解析模板输出的 thinking 块并分离显示，实现类似 DeepSeek 的推理过程可视化。\n\n\n\n## Recent API changes\n\n- [Changelog for `libllama` API](https:\u002F\u002Fgithub.com\u002Fggml-org\u002Fllama.cpp\u002Fissues\u002F9289)\n- [Changelog for `llama-server` REST API](https:\u002F\u002Fgithub.com\u002Fggml-org\u002Fllama.cpp\u002Fissues\u002F9291)\n\n## Hot topics\n\n- **Hugging Face cache migration: models downloaded with `-hf` are now stored in the standard Hugging Face cache directory, enabling sharing with other HF tools.**\n- **[guide : using the new WebUI of llama.cpp](https:\u002F\u002Fgithub.com\u002Fggml-org\u002Fllama.cpp\u002Fdiscussions\u002F16938)**\n- [guide : running gpt-oss with llama.cpp](https:\u002F\u002Fgithub.com\u002Fggml-org\u002Fllama.cpp\u002Fdiscussions\u002F15396)\n- [[FEEDBACK] Better packaging for llama.cpp to support downstream consumers 🤗](https:\u002F\u002Fgithub.com\u002Fggml-org\u002Fllama.cpp\u002Fdiscussions\u002F15313)\n- Support for the `gpt-oss` model with native MXFP4 format has been added | [PR](https:\u002F\u002Fgithub.com\u002Fggml-org\u002Fllama.cpp\u002Fpull\u002F15091) | [Collaboration with NVIDIA](https:\u002F\u002Fblogs.nvidia.com\u002Fblog\u002Frtx-ai-garage-openai-oss) | [Comment](https:\u002F\u002Fgithub.com\u002Fggml-org\u002Fllama.cpp\u002Fdiscussions\u002F15095)\n- Multimodal support arrived in `llama-server`: [#12898](https:\u002F\u002Fgithub.com\u002Fggml-org\u002Fllama.cpp\u002Fpull\u002F12898) | [documentation](.\u002Fdocs\u002Fmultimodal.md)\n- VS Code extension for FIM completions: https:\u002F\u002Fgithub.com\u002Fggml-org\u002Fllama.vscode\n- Vim\u002FNeovim plugin for FIM completions: https:\u002F\u002Fgithub.com\u002Fggml-org\u002Fllama.vim\n- Hugging Face Inference Endpoints now support GGUF out of the box! https:\u002F\u002Fgithub.com\u002Fggml-org\u002Fllama.cpp\u002Fdiscussions\u002F9669\n- Hugging Face GGUF editor: [discussion](https:\u002F\u002Fgithub.com\u002Fggml-org\u002Fllama.cpp\u002Fdiscussions\u002F9268) | [tool](https:\u002F\u002Fhuggingface.co\u002Fspaces\u002FCISCai\u002Fgguf-editor)\n\n----\n\n## Quick start\n\nGetting started with llama.cpp is straightforward. Here are several ways to install it on your machine:\n\n- Install `llama.cpp` using [brew, nix or winget](docs\u002Finstall.md)\n- Run with Docker - see our [Docker documentation](docs\u002Fdocker.md)\n- Download pre-built binaries from the [releases page](https:\u002F\u002Fgithub.com\u002Fggml-org\u002Fllama.cpp\u002Freleases)\n- Build from source by cloning this repository - check out [our build guide](docs\u002Fbuild.md)\n\nOnce installed, you'll need a model to work with. Head to the [Obtaining and quantizing models](#obtaining-and-quantizing-models) section to learn more.\n\nExample command:\n\n```sh\n# Use a local model file\nllama-cli -m my_model.gguf\n\n# Or download and run a model directly from Hugging Face\nllama-cli -hf ggml-org\u002Fgemma-3-1b-it-GGUF\n\n# Launch OpenAI-compatible API server\nllama-server -hf ggml-org\u002Fgemma-3-1b-it-GGUF\n```\n\n## Description\n\nThe main goal of `llama.cpp` is to enable LLM inference with minimal setup and state-of-the-art performance on a wide\nrange of hardware - locally and in the cloud.\n\n- Plain C\u002FC++ implementation without any dependencies\n- Apple silicon is a first-class citizen - optimized via ARM NEON, Accelerate and Metal frameworks\n- AVX, AVX2, AVX512 and AMX support for x86 architectures\n- RVV, ZVFH, ZFH, ZICBOP and ZIHINTPAUSE support for RISC-V architectures\n- 1.5-bit, 2-bit, 3-bit, 4-bit, 5-bit, 6-bit, and 8-bit integer quantization for faster inference and reduced memory use\n- Custom CUDA kernels for running LLMs on NVIDIA GPUs (support for AMD GPUs via HIP and Moore Threads GPUs via MUSA)\n- Vulkan and SYCL backend support\n- CPU+GPU hybrid inference to partially accelerate models larger than the total VRAM capacity\n\nThe `llama.cpp` project is the main playground for developing new features for the [ggml](https:\u002F\u002Fgithub.com\u002Fggml-org\u002Fggml) library.\n\n\u003Cdetails>\n\u003Csummary>Models\u003C\u002Fsummary>\n\nTypically finetunes of the base models below are supported as well.\n\nInstructions for adding support for new models: [HOWTO-add-model.md](docs\u002Fdevelopment\u002FHOWTO-add-model.md)\n\n#### Text-only\n\n- [X] LLaMA 🦙\n- [x] LLaMA 2 🦙🦙\n- [x] LLaMA 3 🦙🦙🦙\n- [X] [Mistral 7B](https:\u002F\u002Fhuggingface.co\u002Fmistralai\u002FMistral-7B-v0.1)\n- [x] [Mixtral MoE](https:\u002F\u002Fhuggingface.co\u002Fmodels?search=mistral-ai\u002FMixtral)\n- [x] [DBRX](https:\u002F\u002Fhuggingface.co\u002Fdatabricks\u002Fdbrx-instruct)\n- [x] [Jamba](https:\u002F\u002Fhuggingface.co\u002Fai21labs)\n- [X] [Falcon](https:\u002F\u002Fhuggingface.co\u002Fmodels?search=tiiuae\u002Ffalcon)\n- [X] [Chinese LLaMA \u002F Alpaca](https:\u002F\u002Fgithub.com\u002Fymcui\u002FChinese-LLaMA-Alpaca) and [Chinese LLaMA-2 \u002F Alpaca-2](https:\u002F\u002Fgithub.com\u002Fymcui\u002FChinese-LLaMA-Alpaca-2)\n- [X] [Vigogne (French)](https:\u002F\u002Fgithub.com\u002Fbofenghuang\u002Fvigogne)\n- [X] [BERT](https:\u002F\u002Fgithub.com\u002Fggml-org\u002Fllama.cpp\u002Fpull\u002F5423)\n- [X] [Koala](https:\u002F\u002Fbair.berkeley.edu\u002Fblog\u002F2023\u002F04\u002F03\u002Fkoala\u002F)\n- [X] [Baichuan 1 & 2](https:\u002F\u002Fhuggingface.co\u002Fmodels?search=baichuan-inc\u002FBaichuan) + [derivations](https:\u002F\u002Fhuggingface.co\u002Fhiyouga\u002Fbaichuan-7b-sft)\n- [X] [Aquila 1 & 2](https:\u002F\u002Fhuggingface.co\u002Fmodels?search=BAAI\u002FAquila)\n- [X] [Starcoder models](https:\u002F\u002Fgithub.com\u002Fggml-org\u002Fllama.cpp\u002Fpull\u002F3187)\n- [X] [Refact](https:\u002F\u002Fhuggingface.co\u002Fsmallcloudai\u002FRefact-1_6B-fim)\n- [X] [MPT](https:\u002F\u002Fgithub.com\u002Fggml-org\u002Fllama.cpp\u002Fpull\u002F3417)\n- [X] [Bloom](https:\u002F\u002Fgithub.com\u002Fggml-org\u002Fllama.cpp\u002Fpull\u002F3553)\n- [x] [Yi models](https:\u002F\u002Fhuggingface.co\u002Fmodels?search=01-ai\u002FYi)\n- [X] [StableLM models](https:\u002F\u002Fhuggingface.co\u002Fstabilityai)\n- [x] [Deepseek models](https:\u002F\u002Fhuggingface.co\u002Fmodels?search=deepseek-ai\u002Fdeepseek)\n- [x] [Qwen models](https:\u002F\u002Fhuggingface.co\u002Fmodels?search=Qwen\u002FQwen)\n- [x] [PLaMo-13B](https:\u002F\u002Fgithub.com\u002Fggml-org\u002Fllama.cpp\u002Fpull\u002F3557)\n- [x] [Phi models](https:\u002F\u002Fhuggingface.co\u002Fmodels?search=microsoft\u002Fphi)\n- [x] [PhiMoE](https:\u002F\u002Fgithub.com\u002Fggml-org\u002Fllama.cpp\u002Fpull\u002F11003)\n- [x] [GPT-2](https:\u002F\u002Fhuggingface.co\u002Fgpt2)\n- [x] [Orion 14B](https:\u002F\u002Fgithub.com\u002Fggml-org\u002Fllama.cpp\u002Fpull\u002F5118)\n- [x] [InternLM2](https:\u002F\u002Fhuggingface.co\u002Fmodels?search=internlm2)\n- [x] [CodeShell](https:\u002F\u002Fgithub.com\u002FWisdomShell\u002Fcodeshell)\n- [x] [Gemma](https:\u002F\u002Fai.google.dev\u002Fgemma)\n- [x] [Mamba](https:\u002F\u002Fgithub.com\u002Fstate-spaces\u002Fmamba)\n- [x] [Grok-1](https:\u002F\u002Fhuggingface.co\u002Fkeyfan\u002Fgrok-1-hf)\n- [x] [Xverse](https:\u002F\u002Fhuggingface.co\u002Fmodels?search=xverse)\n- [x] [Command-R models](https:\u002F\u002Fhuggingface.co\u002Fmodels?search=CohereForAI\u002Fc4ai-command-r)\n- [x] [SEA-LION](https:\u002F\u002Fhuggingface.co\u002Fmodels?search=sea-lion)\n- [x] [GritLM-7B](https:\u002F\u002Fhuggingface.co\u002FGritLM\u002FGritLM-7B) + [GritLM-8x7B](https:\u002F\u002Fhuggingface.co\u002FGritLM\u002FGritLM-8x7B)\n- [x] [OLMo](https:\u002F\u002Fallenai.org\u002Folmo)\n- [x] [OLMo 2](https:\u002F\u002Fallenai.org\u002Folmo)\n- [x] [OLMoE](https:\u002F\u002Fhuggingface.co\u002Fallenai\u002FOLMoE-1B-7B-0924)\n- [x] [Granite models](https:\u002F\u002Fhuggingface.co\u002Fcollections\u002Fibm-granite\u002Fgranite-code-models-6624c5cec322e4c148c8b330)\n- [x] [GPT-NeoX](https:\u002F\u002Fgithub.com\u002FEleutherAI\u002Fgpt-neox) + [Pythia](https:\u002F\u002Fgithub.com\u002FEleutherAI\u002Fpythia)\n- [x] [Snowflake-Arctic MoE](https:\u002F\u002Fhuggingface.co\u002Fcollections\u002FSnowflake\u002Farctic-66290090abe542894a5ac520)\n- [x] [Smaug](https:\u002F\u002Fhuggingface.co\u002Fmodels?search=Smaug)\n- [x] [Poro 34B](https:\u002F\u002Fhuggingface.co\u002FLumiOpen\u002FPoro-34B)\n- [x] [Bitnet b1.58 models](https:\u002F\u002Fhuggingface.co\u002F1bitLLM)\n- [x] [Flan T5](https:\u002F\u002Fhuggingface.co\u002Fmodels?search=flan-t5)\n- [x] [Open Elm models](https:\u002F\u002Fhuggingface.co\u002Fcollections\u002Fapple\u002Fopenelm-instruct-models-6619ad295d7ae9f868b759ca)\n- [x] [ChatGLM3-6b](https:\u002F\u002Fhuggingface.co\u002FTHUDM\u002Fchatglm3-6b) + [ChatGLM4-9b](https:\u002F\u002Fhuggingface.co\u002FTHUDM\u002Fglm-4-9b) + [GLMEdge-1.5b](https:\u002F\u002Fhuggingface.co\u002FTHUDM\u002Fglm-edge-1.5b-chat) + [GLMEdge-4b](https:\u002F\u002Fhuggingface.co\u002FTHUDM\u002Fglm-edge-4b-chat)\n- [x] [GLM-4-0414](https:\u002F\u002Fhuggingface.co\u002Fcollections\u002FTHUDM\u002Fglm-4-0414-67f3cbcb34dd9d252707cb2e)\n- [x] [SmolLM](https:\u002F\u002Fhuggingface.co\u002Fcollections\u002FHuggingFaceTB\u002Fsmollm-6695016cad7167254ce15966)\n- [x] [EXAONE-3.0-7.8B-Instruct](https:\u002F\u002Fhuggingface.co\u002FLGAI-EXAONE\u002FEXAONE-3.0-7.8B-Instruct)\n- [x] [FalconMamba Models](https:\u002F\u002Fhuggingface.co\u002Fcollections\u002Ftiiuae\u002Ffalconmamba-7b-66b9a580324dd1598b0f6d4a)\n- [x] [Jais](https:\u002F\u002Fhuggingface.co\u002Finceptionai\u002Fjais-13b-chat)\n- [x] [Bielik-11B-v2.3](https:\u002F\u002Fhuggingface.co\u002Fcollections\u002Fspeakleash\u002Fbielik-11b-v23-66ee813238d9b526a072408a)\n- [x] [RWKV-7](https:\u002F\u002Fhuggingface.co\u002Fcollections\u002Fshoumenchougou\u002Frwkv7-gxx-gguf)\n- [x] [RWKV-6](https:\u002F\u002Fgithub.com\u002FBlinkDL\u002FRWKV-LM)\n- [x] [QRWKV-6](https:\u002F\u002Fhuggingface.co\u002Frecursal\u002FQRWKV6-32B-Instruct-Preview-v0.1)\n- [x] [GigaChat-20B-A3B](https:\u002F\u002Fhuggingface.co\u002Fai-sage\u002FGigaChat-20B-A3B-instruct)\n- [X] [Trillion-7B-preview](https:\u002F\u002Fhuggingface.co\u002Ftrillionlabs\u002FTrillion-7B-preview)\n- [x] [Ling models](https:\u002F\u002Fhuggingface.co\u002Fcollections\u002FinclusionAI\u002Fling-67c51c85b34a7ea0aba94c32)\n- [x] [LFM2 models](https:\u002F\u002Fhuggingface.co\u002Fcollections\u002FLiquidAI\u002Flfm2-686d721927015b2ad73eaa38)\n- [x] [Hunyuan models](https:\u002F\u002Fhuggingface.co\u002Fcollections\u002Ftencent\u002Fhunyuan-dense-model-6890632cda26b19119c9c5e7)\n- [x] [BailingMoeV2 (Ring\u002FLing 2.0) models](https:\u002F\u002Fhuggingface.co\u002Fcollections\u002FinclusionAI\u002Fling-v2-68bf1dd2fc34c306c1fa6f86)\n\n#### Multimodal\n\n- [x] [LLaVA 1.5 models](https:\u002F\u002Fhuggingface.co\u002Fcollections\u002Fliuhaotian\u002Fllava-15-653aac15d994e992e2677a7e), [LLaVA 1.6 models](https:\u002F\u002Fhuggingface.co\u002Fcollections\u002Fliuhaotian\u002Fllava-16-65b9e40155f60fd046a5ccf2)\n- [x] [BakLLaVA](https:\u002F\u002Fhuggingface.co\u002Fmodels?search=SkunkworksAI\u002FBakllava)\n- [x] [Obsidian](https:\u002F\u002Fhuggingface.co\u002FNousResearch\u002FObsidian-3B-V0.5)\n- [x] [ShareGPT4V](https:\u002F\u002Fhuggingface.co\u002Fmodels?search=Lin-Chen\u002FShareGPT4V)\n- [x] [MobileVLM 1.7B\u002F3B models](https:\u002F\u002Fhuggingface.co\u002Fmodels?search=mobileVLM)\n- [x] [Yi-VL](https:\u002F\u002Fhuggingface.co\u002Fmodels?search=Yi-VL)\n- [x] [Mini CPM](https:\u002F\u002Fhuggingface.co\u002Fmodels?search=MiniCPM)\n- [x] [Moondream](https:\u002F\u002Fhuggingface.co\u002Fvikhyatk\u002Fmoondream2)\n- [x] [Bunny](https:\u002F\u002Fgithub.com\u002FBAAI-DCAI\u002FBunny)\n- [x] [GLM-EDGE](https:\u002F\u002Fhuggingface.co\u002Fmodels?search=glm-edge)\n- [x] [Qwen2-VL](https:\u002F\u002Fhuggingface.co\u002Fcollections\u002FQwen\u002Fqwen2-vl-66cee7455501d7126940800d)\n- [x] [LFM2-VL](https:\u002F\u002Fhuggingface.co\u002Fcollections\u002FLiquidAI\u002Flfm2-vl-68963bbc84a610f7638d5ffa)\n\n\u003C\u002Fdetails>\n\n\u003Cdetails>\n\u003Csummary>Bindings\u003C\u002Fsummary>\n\n- Python: [ddh0\u002Feasy-llama](https:\u002F\u002Fgithub.com\u002Fddh0\u002Feasy-llama)\n- Python: [abetlen\u002Fllama-cpp-python](https:\u002F\u002Fgithub.com\u002Fabetlen\u002Fllama-cpp-python)\n- Go: [go-skynet\u002Fgo-llama.cpp](https:\u002F\u002Fgithub.com\u002Fgo-skynet\u002Fgo-llama.cpp)\n- Node.js: [withcatai\u002Fnode-llama-cpp](https:\u002F\u002Fgithub.com\u002Fwithcatai\u002Fnode-llama-cpp)\n- JS\u002FTS (llama.cpp server client): [lgrammel\u002Fmodelfusion](https:\u002F\u002Fmodelfusion.dev\u002Fintegration\u002Fmodel-provider\u002Fllamacpp)\n- JS\u002FTS (Programmable Prompt Engine CLI): [offline-ai\u002Fcli](https:\u002F\u002Fgithub.com\u002Foffline-ai\u002Fcli)\n- JavaScript\u002FWasm (works in browser): [tangledgroup\u002Fllama-cpp-wasm](https:\u002F\u002Fgithub.com\u002Ftangledgroup\u002Fllama-cpp-wasm)\n- Typescript\u002FWasm (nicer API, available on npm): [ngxson\u002Fwllama](https:\u002F\u002Fgithub.com\u002Fngxson\u002Fwllama)\n- Ruby: [yoshoku\u002Fllama_cpp.rb](https:\u002F\u002Fgithub.com\u002Fyoshoku\u002Fllama_cpp.rb)\n- Rust (more features): [edgenai\u002Fllama_cpp-rs](https:\u002F\u002Fgithub.com\u002Fedgenai\u002Fllama_cpp-rs)\n- Rust (nicer API): [mdrokz\u002Frust-llama.cpp](https:\u002F\u002Fgithub.com\u002Fmdrokz\u002Frust-llama.cpp)\n- Rust (more direct bindings): [utilityai\u002Fllama-cpp-rs](https:\u002F\u002Fgithub.com\u002Futilityai\u002Fllama-cpp-rs)\n- Rust (automated build from crates.io): [ShelbyJenkins\u002Fllm_client](https:\u002F\u002Fgithub.com\u002FShelbyJenkins\u002Fllm_client)\n- C#\u002F.NET: [SciSharp\u002FLLamaSharp](https:\u002F\u002Fgithub.com\u002FSciSharp\u002FLLamaSharp)\n- C#\u002FVB.NET (more features - community license): [LM-Kit.NET](https:\u002F\u002Fdocs.lm-kit.com\u002Flm-kit-net\u002Findex.html)\n- Scala 3: [donderom\u002Fllm4s](https:\u002F\u002Fgithub.com\u002Fdonderom\u002Fllm4s)\n- Clojure: [phronmophobic\u002Fllama.clj](https:\u002F\u002Fgithub.com\u002Fphronmophobic\u002Fllama.clj)\n- React Native: [mybigday\u002Fllama.rn](https:\u002F\u002Fgithub.com\u002Fmybigday\u002Fllama.rn)\n- Java: [kherud\u002Fjava-llama.cpp](https:\u002F\u002Fgithub.com\u002Fkherud\u002Fjava-llama.cpp)\n- Java: [QuasarByte\u002Fllama-cpp-jna](https:\u002F\u002Fgithub.com\u002FQuasarByte\u002Fllama-cpp-jna)\n- Zig: [deins\u002Fllama.cpp.zig](https:\u002F\u002Fgithub.com\u002FDeins\u002Fllama.cpp.zig)\n- Flutter\u002FDart: [netdur\u002Fllama_cpp_dart](https:\u002F\u002Fgithub.com\u002Fnetdur\u002Fllama_cpp_dart)\n- Flutter: [xuegao-tzx\u002FFllama](https:\u002F\u002Fgithub.com\u002Fxuegao-tzx\u002FFllama)\n- PHP (API bindings and features built on top of llama.cpp): [distantmagic\u002Fresonance](https:\u002F\u002Fgithub.com\u002Fdistantmagic\u002Fresonance) [(more info)](https:\u002F\u002Fgithub.com\u002Fggml-org\u002Fllama.cpp\u002Fpull\u002F6326)\n- Guile Scheme: [guile_llama_cpp](https:\u002F\u002Fsavannah.nongnu.org\u002Fprojects\u002Fguile-llama-cpp)\n- Swift [srgtuszy\u002Fllama-cpp-swift](https:\u002F\u002Fgithub.com\u002Fsrgtuszy\u002Fllama-cpp-swift)\n- Swift [ShenghaiWang\u002FSwiftLlama](https:\u002F\u002Fgithub.com\u002FShenghaiWang\u002FSwiftLlama)\n- Delphi [Embarcadero\u002Fllama-cpp-delphi](https:\u002F\u002Fgithub.com\u002FEmbarcadero\u002Fllama-cpp-delphi)\n- Go (no CGo needed): [hybridgroup\u002Fyzma](https:\u002F\u002Fgithub.com\u002Fhybridgroup\u002Fyzma)\n- Android: [llama.android](\u002Fexamples\u002Fllama.android)\n\n\u003C\u002Fdetails>\n\n\u003Cdetails>\n\u003Csummary>UIs\u003C\u002Fsummary>\n\n*(to have a project listed here, it should clearly state that it depends on `llama.cpp`)*\n\n- [AI Sublime Text plugin](https:\u002F\u002Fgithub.com\u002Fyaroslavyaroslav\u002FOpenAI-sublime-text) (MIT)\n- [BonzAI App](https:\u002F\u002Fapps.apple.com\u002Fus\u002Fapp\u002Fbonzai-your-local-ai-agent\u002Fid6752847988) (proprietary)\n- [cztomsik\u002Fava](https:\u002F\u002Fgithub.com\u002Fcztomsik\u002Fava) (MIT)\n- [Dot](https:\u002F\u002Fgithub.com\u002Falexpinel\u002FDot) (GPL)\n- [eva](https:\u002F\u002Fgithub.com\u002Fylsdamxssjxxdd\u002Feva) (MIT)\n- [iohub\u002Fcollama](https:\u002F\u002Fgithub.com\u002Fiohub\u002FcoLLaMA) (Apache-2.0)\n- [janhq\u002Fjan](https:\u002F\u002Fgithub.com\u002Fjanhq\u002Fjan) (AGPL)\n- [johnbean393\u002FSidekick](https:\u002F\u002Fgithub.com\u002Fjohnbean393\u002FSidekick) (MIT)\n- [KanTV](https:\u002F\u002Fgithub.com\u002Fzhouwg\u002Fkantv?tab=readme-ov-file) (Apache-2.0)\n- [KodiBot](https:\u002F\u002Fgithub.com\u002Ffiratkiral\u002Fkodibot) (GPL)\n- [llama.vim](https:\u002F\u002Fgithub.com\u002Fggml-org\u002Fllama.vim) (MIT)\n- [LARS](https:\u002F\u002Fgithub.com\u002Fabgulati\u002FLARS) (AGPL)\n- [Llama Assistant](https:\u002F\u002Fgithub.com\u002Fvietanhdev\u002Fllama-assistant) (GPL)\n- [LlamaLib](https:\u002F\u002Fgithub.com\u002Fundreamai\u002FLlamaLib) (Apache-2.0)\n- [LLMFarm](https:\u002F\u002Fgithub.com\u002Fguinmoon\u002FLLMFarm?tab=readme-ov-file) (MIT)\n- [LLMUnity](https:\u002F\u002Fgithub.com\u002Fundreamai\u002FLLMUnity) (MIT)\n- [LMStudio](https:\u002F\u002Flmstudio.ai\u002F) (proprietary)\n- [LocalAI](https:\u002F\u002Fgithub.com\u002Fmudler\u002FLocalAI) (MIT)\n- [LostRuins\u002Fkoboldcpp](https:\u002F\u002Fgithub.com\u002FLostRuins\u002Fkoboldcpp) (AGPL)\n- [MindMac](https:\u002F\u002Fmindmac.app) (proprietary)\n- [MindWorkAI\u002FAI-Studio](https:\u002F\u002Fgithub.com\u002FMindWorkAI\u002FAI-Studio) (FSL-1.1-MIT)\n- [Mobile-Artificial-Intelligence\u002Fmaid](https:\u002F\u002Fgithub.com\u002FMobile-Artificial-Intelligence\u002Fmaid) (MIT)\n- [Mozilla-Ocho\u002Fllamafile](https:\u002F\u002Fgithub.com\u002FMozilla-Ocho\u002Fllamafile) (Apache-2.0)\n- [nat\u002Fopenplayground](https:\u002F\u002Fgithub.com\u002Fnat\u002Fopenplayground) (MIT)\n- [nomic-ai\u002Fgpt4all](https:\u002F\u002Fgithub.com\u002Fnomic-ai\u002Fgpt4all) (MIT)\n- [ollama\u002Follama](https:\u002F\u002Fgithub.com\u002Follama\u002Follama) (MIT)\n- [oobabooga\u002Ftext-generation-webui](https:\u002F\u002Fgithub.com\u002Foobabooga\u002Ftext-generation-webui) (AGPL)\n- [PocketPal AI](https:\u002F\u002Fgithub.com\u002Fa-ghorbani\u002Fpocketpal-ai) (MIT)\n- [psugihara\u002FFreeChat](https:\u002F\u002Fgithub.com\u002Fpsugihara\u002FFreeChat) (MIT)\n- [ptsochantaris\u002Femeltal](https:\u002F\u002Fgithub.com\u002Fptsochantaris\u002Femeltal) (MIT)\n- [pythops\u002Ftenere](https:\u002F\u002Fgithub.com\u002Fpythops\u002Ftenere) (AGPL)\n- [ramalama](https:\u002F\u002Fgithub.com\u002Fcontainers\u002Framalama) (MIT)\n- [semperai\u002Famica](https:\u002F\u002Fgithub.com\u002Fsemperai\u002Famica) (MIT)\n- [withcatai\u002Fcatai](https:\u002F\u002Fgithub.com\u002Fwithcatai\u002Fcatai) (MIT)\n- [Autopen](https:\u002F\u002Fgithub.com\u002Fblackhole89\u002Fautopen) (GPL)\n\n\u003C\u002Fdetails>\n\n\u003Cdetails>\n\u003Csummary>Tools\u003C\u002Fsummary>\n\n- [akx\u002Fggify](https:\u002F\u002Fgithub.com\u002Fakx\u002Fggify) – download PyTorch models from Hugging Face Hub and convert them to GGML\n- [akx\u002Follama-dl](https:\u002F\u002Fgithub.com\u002Fakx\u002Follama-dl) – download models from the Ollama library to be used directly with llama.cpp\n- [crashr\u002Fgppm](https:\u002F\u002Fgithub.com\u002Fcrashr\u002Fgppm) – launch llama.cpp instances utilizing NVIDIA Tesla P40 or P100 GPUs with reduced idle power consumption\n- [gpustack\u002Fgguf-parser](https:\u002F\u002Fgithub.com\u002Fgpustack\u002Fgguf-parser-go\u002Ftree\u002Fmain\u002Fcmd\u002Fgguf-parser) - review\u002Fcheck the GGUF file and estimate the memory usage\n- [Styled Lines](https:\u002F\u002Fmarketplace.unity.com\u002Fpackages\u002Ftools\u002Fgenerative-ai\u002Fstyled-lines-llama-cpp-model-292902) (proprietary licensed, async wrapper of inference part for game development in Unity3d with pre-built Mobile and Web platform wrappers and a model example)\n- [unslothai\u002Funsloth](https:\u002F\u002Fgithub.com\u002Funslothai\u002Funsloth) – 🦥 exports\u002Fsaves fine-tuned and trained models to GGUF (Apache-2.0)\n\n\u003C\u002Fdetails>\n\n\u003Cdetails>\n\u003Csummary>Infrastructure\u003C\u002Fsummary>\n\n- [Paddler](https:\u002F\u002Fgithub.com\u002Fintentee\u002Fpaddler) - Open-source LLMOps platform for hosting and scaling AI in your own infrastructure\n- [GPUStack](https:\u002F\u002Fgithub.com\u002Fgpustack\u002Fgpustack) - Manage GPU clusters for running LLMs\n- [llama_cpp_canister](https:\u002F\u002Fgithub.com\u002Fonicai\u002Fllama_cpp_canister) - llama.cpp as a smart contract on the Internet Computer, using WebAssembly\n- [llama-swap](https:\u002F\u002Fgithub.com\u002Fmostlygeek\u002Fllama-swap) - transparent proxy that adds automatic model switching with llama-server\n- [Kalavai](https:\u002F\u002Fgithub.com\u002Fkalavai-net\u002Fkalavai-client) - Crowdsource end to end LLM deployment at any scale\n- [llmaz](https:\u002F\u002Fgithub.com\u002FInftyAI\u002Fllmaz) - ☸️ Easy, advanced inference platform for large language models on Kubernetes.\n- [LLMKube](https:\u002F\u002Fgithub.com\u002Fdefilantech\u002Fllmkube) - Kubernetes operator for llama.cpp with multi-GPU and Apple Silicon Metal\n  support\"\n\u003C\u002Fdetails>\n\n\u003Cdetails>\n\u003Csummary>Games\u003C\u002Fsummary>\n\n- [Lucy's Labyrinth](https:\u002F\u002Fgithub.com\u002FMorganRO8\u002FLucys_Labyrinth) - A simple maze game where agents controlled by an AI model will try to trick you.\n\n\u003C\u002Fdetails>\n\n\n## Supported backends\n\n| Backend | Target devices |\n| --- | --- |\n| [Metal](docs\u002Fbuild.md#metal-build) | Apple Silicon |\n| [BLAS](docs\u002Fbuild.md#blas-build) | All |\n| [BLIS](docs\u002Fbackend\u002FBLIS.md) | All |\n| [SYCL](docs\u002Fbackend\u002FSYCL.md) | Intel and Nvidia GPU |\n| [OpenVINO [In Progress]](docs\u002Fbackend\u002FOPENVINO.md) | Intel CPUs, GPUs, and NPUs |\n| [MUSA](docs\u002Fbuild.md#musa) | Moore Threads GPU |\n| [CUDA](docs\u002Fbuild.md#cuda) | Nvidia GPU |\n| [HIP](docs\u002Fbuild.md#hip) | AMD GPU |\n| [ZenDNN](docs\u002Fbuild.md#zendnn) | AMD CPU |\n| [Vulkan](docs\u002Fbuild.md#vulkan) | GPU |\n| [CANN](docs\u002Fbuild.md#cann) | Ascend NPU |\n| [OpenCL](docs\u002Fbackend\u002FOPENCL.md) | Adreno GPU |\n| [IBM zDNN](docs\u002Fbackend\u002FzDNN.md) | IBM Z & LinuxONE |\n| [WebGPU [In Progress]](docs\u002Fbuild.md#webgpu) | All |\n| [RPC](https:\u002F\u002Fgithub.com\u002Fggml-org\u002Fllama.cpp\u002Ftree\u002Fmaster\u002Ftools\u002Frpc) | All |\n| [Hexagon [In Progress]](docs\u002Fbackend\u002Fsnapdragon\u002FREADME.md) | Snapdragon |\n| [VirtGPU](docs\u002Fbackend\u002FVirtGPU.md) | VirtGPU APIR |\n\n## Obtaining and quantizing models\n\nThe [Hugging Face](https:\u002F\u002Fhuggingface.co) platform hosts a [number of LLMs](https:\u002F\u002Fhuggingface.co\u002Fmodels?library=gguf&sort=trending) compatible with `llama.cpp`:\n\n- [Trending](https:\u002F\u002Fhuggingface.co\u002Fmodels?library=gguf&sort=trending)\n- [LLaMA](https:\u002F\u002Fhuggingface.co\u002Fmodels?sort=trending&search=llama+gguf)\n\nYou can either manually download the GGUF file or directly use any `llama.cpp`-compatible models from [Hugging Face](https:\u002F\u002Fhuggingface.co\u002F) or other model hosting sites, by using this CLI argument: `-hf \u003Cuser>\u002F\u003Cmodel>[:quant]`. For example:\n\n```sh\nllama-cli -hf ggml-org\u002Fgemma-3-1b-it-GGUF\n```\n\nBy default, the CLI would download from Hugging Face, you can switch to other options with the environment variable `MODEL_ENDPOINT`. The `MODEL_ENDPOINT` must point to a Hugging Face compatible API endpoint.\n\nAfter downloading a model, use the CLI tools to run it locally - see below.\n\n`llama.cpp` requires the model to be stored in the [GGUF](https:\u002F\u002Fgithub.com\u002Fggml-org\u002Fggml\u002Fblob\u002Fmaster\u002Fdocs\u002Fgguf.md) file format. Models in other data formats can be converted to GGUF using the `convert_*.py` Python scripts in this repo.\n\nThe Hugging Face platform provides a variety of online tools for converting, quantizing and hosting models with `llama.cpp`:\n\n- Use the [GGUF-my-repo space](https:\u002F\u002Fhuggingface.co\u002Fspaces\u002Fggml-org\u002Fgguf-my-repo) to convert to GGUF format and quantize model weights to smaller sizes\n- Use the [GGUF-my-LoRA space](https:\u002F\u002Fhuggingface.co\u002Fspaces\u002Fggml-org\u002Fgguf-my-lora) to convert LoRA adapters to GGUF format (more info: https:\u002F\u002Fgithub.com\u002Fggml-org\u002Fllama.cpp\u002Fdiscussions\u002F10123)\n- Use the [GGUF-editor space](https:\u002F\u002Fhuggingface.co\u002Fspaces\u002FCISCai\u002Fgguf-editor) to edit GGUF meta data in the browser (more info: https:\u002F\u002Fgithub.com\u002Fggml-org\u002Fllama.cpp\u002Fdiscussions\u002F9268)\n- Use the [Inference Endpoints](https:\u002F\u002Fui.endpoints.huggingface.co\u002F) to directly host `llama.cpp` in the cloud (more info: https:\u002F\u002Fgithub.com\u002Fggml-org\u002Fllama.cpp\u002Fdiscussions\u002F9669)\n\nTo learn more about model quantization, [read this documentation](tools\u002Fquantize\u002FREADME.md)\n\n## [`llama-cli`](tools\u002Fcli)\n\n#### A CLI tool for accessing and experimenting with most of `llama.cpp`'s functionality.\n\n- \u003Cdetails open>\n    \u003Csummary>Run in conversation mode\u003C\u002Fsummary>\n\n    Models with a built-in chat template will automatically activate conversation mode. If this doesn't occur, you can manually enable it by adding `-cnv` and specifying a suitable chat template with `--chat-template NAME`\n\n    ```bash\n    llama-cli -m model.gguf\n\n    # > hi, who are you?\n    # Hi there! I'm your helpful assistant! I'm an AI-powered chatbot designed to assist and provide information to users like you. I'm here to help answer your questions, provide guidance, and offer support on a wide range of topics. I'm a friendly and knowledgeable AI, and I'm always happy to help with anything you need. What's on your mind, and how can I assist you today?\n    #\n    # > what is 1+1?\n    # Easy peasy! The answer to 1+1 is... 2!\n    ```\n\n    \u003C\u002Fdetails>\n\n- \u003Cdetails>\n    \u003Csummary>Run in conversation mode with custom chat template\u003C\u002Fsummary>\n\n    ```bash\n    # use the \"chatml\" template (use -h to see the list of supported templates)\n    llama-cli -m model.gguf -cnv --chat-template chatml\n\n    # use a custom template\n    llama-cli -m model.gguf -cnv --in-prefix 'User: ' --reverse-prompt 'User:'\n    ```\n\n    \u003C\u002Fdetails>\n\n- \u003Cdetails>\n    \u003Csummary>Constrain the output with a custom grammar\u003C\u002Fsummary>\n\n    ```bash\n    llama-cli -m model.gguf -n 256 --grammar-file grammars\u002Fjson.gbnf -p 'Request: schedule a call at 8pm; Command:'\n\n    # {\"appointmentTime\": \"8pm\", \"appointmentDetails\": \"schedule a a call\"}\n    ```\n\n    The [grammars\u002F](grammars\u002F) folder contains a handful of sample grammars. To write your own, check out the [GBNF Guide](grammars\u002FREADME.md).\n\n    For authoring more complex JSON grammars, check out https:\u002F\u002Fgrammar.intrinsiclabs.ai\u002F\n\n    \u003C\u002Fdetails>\n\n\n## [`llama-server`](tools\u002Fserver)\n\n#### A lightweight, [OpenAI API](https:\u002F\u002Fgithub.com\u002Fopenai\u002Fopenai-openapi) compatible, HTTP server for serving LLMs.\n\n- \u003Cdetails open>\n    \u003Csummary>Start a local HTTP server with default configuration on port 8080\u003C\u002Fsummary>\n\n    ```bash\n    llama-server -m model.gguf --port 8080\n\n    # Basic web UI can be accessed via browser: http:\u002F\u002Flocalhost:8080\n    # Chat completion endpoint: http:\u002F\u002Flocalhost:8080\u002Fv1\u002Fchat\u002Fcompletions\n    ```\n\n    \u003C\u002Fdetails>\n\n- \u003Cdetails>\n    \u003Csummary>Support multiple-users and parallel decoding\u003C\u002Fsummary>\n\n    ```bash\n    # up to 4 concurrent requests, each with 4096 max context\n    llama-server -m model.gguf -c 16384 -np 4\n    ```\n\n    \u003C\u002Fdetails>\n\n- \u003Cdetails>\n    \u003Csummary>Enable speculative decoding\u003C\u002Fsummary>\n\n    ```bash\n    # the draft.gguf model should be a small variant of the target model.gguf\n    llama-server -m model.gguf -md draft.gguf\n    ```\n\n    \u003C\u002Fdetails>\n\n- \u003Cdetails>\n    \u003Csummary>Serve an embedding model\u003C\u002Fsummary>\n\n    ```bash\n    # use the \u002Fembedding endpoint\n    llama-server -m model.gguf --embedding --pooling cls -ub 8192\n    ```\n\n    \u003C\u002Fdetails>\n\n- \u003Cdetails>\n    \u003Csummary>Serve a reranking model\u003C\u002Fsummary>\n\n    ```bash\n    # use the \u002Freranking endpoint\n    llama-server -m model.gguf --reranking\n    ```\n\n    \u003C\u002Fdetails>\n\n- \u003Cdetails>\n    \u003Csummary>Constrain all outputs with a grammar\u003C\u002Fsummary>\n\n    ```bash\n    # custom grammar\n    llama-server -m model.gguf --grammar-file grammar.gbnf\n\n    # JSON\n    llama-server -m model.gguf --grammar-file grammars\u002Fjson.gbnf\n    ```\n\n    \u003C\u002Fdetails>\n\n\n## [`llama-perplexity`](tools\u002Fperplexity)\n\n#### A tool for measuring the [perplexity](tools\u002Fperplexity\u002FREADME.md) [^1] (and other quality metrics) of a model over a given text.\n\n- \u003Cdetails open>\n    \u003Csummary>Measure the perplexity over a text file\u003C\u002Fsummary>\n\n    ```bash\n    llama-perplexity -m model.gguf -f file.txt\n\n    # [1]15.2701,[2]5.4007,[3]5.3073,[4]6.2965,[5]5.8940,[6]5.6096,[7]5.7942,[8]4.9297, ...\n    # Final estimate: PPL = 5.4007 +\u002F- 0.67339\n    ```\n\n    \u003C\u002Fdetails>\n\n- \u003Cdetails>\n    \u003Csummary>Measure KL divergence\u003C\u002Fsummary>\n\n    ```bash\n    # TODO\n    ```\n\n    \u003C\u002Fdetails>\n\n[^1]: [https:\u002F\u002Fhuggingface.co\u002Fdocs\u002Ftransformers\u002Fperplexity](https:\u002F\u002Fhuggingface.co\u002Fdocs\u002Ftransformers\u002Fperplexity)\n\n## [`llama-bench`](tools\u002Fllama-bench)\n\n#### Benchmark the performance of the inference for various parameters.\n\n- \u003Cdetails open>\n    \u003Csummary>Run default benchmark\u003C\u002Fsummary>\n\n    ```bash\n    llama-bench -m model.gguf\n\n    # Output:\n    # | model               |       size |     params | backend    | threads |          test |                  t\u002Fs |\n    # | ------------------- | ---------: | ---------: | ---------- | ------: | ------------: | -------------------: |\n    # | qwen2 1.5B Q4_0     | 885.97 MiB |     1.54 B | Metal,BLAS |      16 |         pp512 |      5765.41 ± 20.55 |\n    # | qwen2 1.5B Q4_0     | 885.97 MiB |     1.54 B | Metal,BLAS |      16 |         tg128 |        197.71 ± 0.81 |\n    #\n    # build: 3e0ba0e60 (4229)\n    ```\n\n    \u003C\u002Fdetails>\n\n## [`llama-simple`](examples\u002Fsimple)\n\n#### A minimal example for implementing apps with `llama.cpp`. Useful for developers.\n\n- \u003Cdetails>\n    \u003Csummary>Basic text completion\u003C\u002Fsummary>\n\n    ```bash\n    llama-simple -m model.gguf\n\n    # Hello my name is Kaitlyn and I am a 16 year old girl. I am a junior in high school and I am currently taking a class called \"The Art of\n    ```\n\n    \u003C\u002Fdetails>\n\n\n## Contributing\n\n- Contributors can open PRs\n- Collaborators will be invited based on contributions\n- Maintainers can push to branches in the `llama.cpp` repo and merge PRs into the `master` branch\n- Any help with managing issues, PRs and projects is very appreciated!\n- See [good first issues](https:\u002F\u002Fgithub.com\u002Fggml-org\u002Fllama.cpp\u002Fissues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22) for tasks suitable for first contributions\n- Read the [CONTRIBUTING.md](CONTRIBUTING.md) for more information\n- Make sure to read this: [Inference at the edge](https:\u002F\u002Fgithub.com\u002Fggml-org\u002Fllama.cpp\u002Fdiscussions\u002F205)\n- A bit of backstory for those who are interested: [Changelog podcast](https:\u002F\u002Fchangelog.com\u002Fpodcast\u002F532)\n\n## Other documentation\n\n- [cli](tools\u002Fcli\u002FREADME.md)\n- [completion](tools\u002Fcompletion\u002FREADME.md)\n- [server](tools\u002Fserver\u002FREADME.md)\n- [GBNF grammars](grammars\u002FREADME.md)\n\n#### Development documentation\n\n- [How to build](docs\u002Fbuild.md)\n- [Running on Docker](docs\u002Fdocker.md)\n- [Build on Android](docs\u002Fandroid.md)\n- [Performance troubleshooting](docs\u002Fdevelopment\u002Ftoken_generation_performance_tips.md)\n- [GGML tips & tricks](https:\u002F\u002Fgithub.com\u002Fggml-org\u002Fllama.cpp\u002Fwiki\u002FGGML-Tips-&-Tricks)\n\n#### Seminal papers and background on the models\n\nIf your issue is with model generation quality, then please at least scan the following links and papers to understand the limitations of LLaMA models. This is especially important when choosing an appropriate model size and appreciating both the significant and subtle differences between LLaMA models and ChatGPT:\n- LLaMA:\n    - [Introducing LLaMA: A foundational, 65-billion-parameter large language model](https:\u002F\u002Fai.facebook.com\u002Fblog\u002Flarge-language-model-llama-meta-ai\u002F)\n    - [LLaMA: Open and Efficient Foundation Language Models](https:\u002F\u002Farxiv.org\u002Fabs\u002F2302.13971)\n- GPT-3\n    - [Language Models are Few-Shot Learners](https:\u002F\u002Farxiv.org\u002Fabs\u002F2005.14165)\n- GPT-3.5 \u002F InstructGPT \u002F ChatGPT:\n    - [Aligning language models to follow instructions](https:\u002F\u002Fopenai.com\u002Fresearch\u002Finstruction-following)\n    - [Training language models to follow instructions with human feedback](https:\u002F\u002Farxiv.org\u002Fabs\u002F2203.02155)\n\n## XCFramework\nThe XCFramework is a precompiled version of the library for iOS, visionOS, tvOS,\nand macOS. It can be used in Swift projects without the need to compile the\nlibrary from source. For example:\n```swift\n\u002F\u002F swift-tools-version: 5.10\n\u002F\u002F The swift-tools-version declares the minimum version of Swift required to build this package.\n\nimport PackageDescription\n\nlet package = Package(\n    name: \"MyLlamaPackage\",\n    targets: [\n        .executableTarget(\n            name: \"MyLlamaPackage\",\n            dependencies: [\n                \"LlamaFramework\"\n            ]),\n        .binaryTarget(\n            name: \"LlamaFramework\",\n            url: \"https:\u002F\u002Fgithub.com\u002Fggml-org\u002Fllama.cpp\u002Freleases\u002Fdownload\u002Fb5046\u002Fllama-b5046-xcframework.zip\",\n            checksum: \"c19be78b5f00d8d29a25da41042cb7afa094cbf6280a225abe614b03b20029ab\"\n        )\n    ]\n)\n```\nThe above example is using an intermediate build `b5046` of the library. This can be modified\nto use a different version by changing the URL and checksum.\n\n## Completions\nCommand-line completion is available for some environments.\n\n#### Bash Completion\n```bash\n$ build\u002Fbin\u002Fllama-cli --completion-bash > ~\u002F.llama-completion.bash\n$ source ~\u002F.llama-completion.bash\n```\nOptionally this can be added to your `.bashrc` or `.bash_profile` to load it\nautomatically. For example:\n```console\n$ echo \"source ~\u002F.llama-completion.bash\" >> ~\u002F.bashrc\n```\n\n## Dependencies\n\n- [yhirose\u002Fcpp-httplib](https:\u002F\u002Fgithub.com\u002Fyhirose\u002Fcpp-httplib) - Single-header HTTP server, used by `llama-server` - MIT license\n- [stb-image](https:\u002F\u002Fgithub.com\u002Fnothings\u002Fstb) - Single-header image format decoder, used by multimodal subsystem - Public domain\n- [nlohmann\u002Fjson](https:\u002F\u002Fgithub.com\u002Fnlohmann\u002Fjson) - Single-header JSON library, used by various tools\u002Fexamples - MIT License\n- [miniaudio.h](https:\u002F\u002Fgithub.com\u002Fmackron\u002Fminiaudio) - Single-header audio format decoder, used by multimodal subsystem - Public domain\n- [subprocess.h](https:\u002F\u002Fgithub.com\u002Fsheredom\u002Fsubprocess.h) - Single-header process launching solution for C and C++ - Public domain\n",2,"2026-06-11 04:11:46","CREATED_QUERY"]