[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"project-74107":3},{"id":4,"name":5,"fullName":6,"owner":7,"repo":5,"description":8,"homepage":9,"htmlUrl":10,"language":11,"languages":10,"totalLinesOfCode":10,"stars":12,"forks":13,"watchers":14,"openIssues":15,"contributorsCount":16,"subscribersCount":16,"size":16,"stars1d":17,"stars7d":18,"stars30d":19,"stars90d":16,"forks30d":16,"starsTrendScore":20,"compositeScore":21,"rankGlobal":10,"rankLanguage":10,"license":22,"archived":23,"fork":23,"defaultBranch":24,"hasWiki":25,"hasPages":23,"topics":26,"createdAt":10,"pushedAt":10,"updatedAt":27,"readmeContent":28,"aiSummary":29,"trendingCount":16,"starSnapshotCount":16,"syncStatus":30,"lastSyncTime":31,"discoverSource":32},74107,"gemma.cpp","google\u002Fgemma.cpp","google","lightweight, standalone C++ inference engine for Google's Gemma models.","",null,"C++",6939,642,48,22,0,7,27,46,21,96.52,"Apache License 2.0",false,"main",true,[],"2026-06-12 04:01:13","# gemma.cpp\n\ngemma.cpp is a lightweight, standalone C++ inference engine for the Gemma\nfoundation models from Google.\n\nFor additional information about Gemma, see\n[ai.google.dev\u002Fgemma](https:\u002F\u002Fai.google.dev\u002Fgemma). Model weights, including\ngemma.cpp specific artifacts, are\n[available on kaggle](https:\u002F\u002Fwww.kaggle.com\u002Fmodels\u002Fgoogle\u002Fgemma-2).\n\n## Who is this project for?\n\nModern LLM inference engines are sophisticated systems, often with bespoke\ncapabilities extending beyond traditional neural network runtimes. With this\ncomes opportunities for research and innovation through co-design of high level\nalgorithms and low-level computation. However, there is a gap between\ndeployment-oriented C++ inference runtimes, which are not designed for\nexperimentation, and Python-centric ML research frameworks, which abstract away\nlow-level computation through compilation.\n\ngemma.cpp provides a minimalist implementation of Gemma-2, Gemma-3, and\nPaliGemma-2 models, focusing on simplicity and directness rather than full\ngenerality. This is inspired by vertically-integrated model implementations such\nas [ggml](https:\u002F\u002Fgithub.com\u002Fggerganov\u002Fggml),\n[llama.c](https:\u002F\u002Fgithub.com\u002Fkarpathy\u002Fllama2.c), and\n[llama.rs](https:\u002F\u002Fgithub.com\u002Fsrush\u002Fllama2.rs).\n\ngemma.cpp targets experimentation and research use cases. It is intended to be\nstraightforward to embed in other projects with minimal dependencies and also\neasily modifiable with a small ~2K LoC core implementation (along with ~4K LoC\nof supporting utilities). We use the [Google\nHighway](https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fhighway) Library to take advantage of\nportable SIMD for CPU inference.\n\nFor production-oriented edge deployments we recommend standard deployment\npathways using Python frameworks like JAX, Keras, PyTorch, and Transformers\n([all model variations here](https:\u002F\u002Fwww.kaggle.com\u002Fmodels\u002Fgoogle\u002Fgemma)).\n\n## Contributing\n\nCommunity contributions large and small are welcome. See\n[DEVELOPERS.md](https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fgemma.cpp\u002Fblob\u002Fmain\u002FDEVELOPERS.md)\nfor additional notes contributing developers and [join the discord by following\nthis invite link](https:\u002F\u002Fdiscord.gg\u002FH5jCBAWxAe). This project follows\n[Google's Open Source Community\nGuidelines](https:\u002F\u002Fopensource.google.com\u002Fconduct\u002F).\n\n> [!NOTE] Active development is currently done on the `dev` branch. Please open\n> pull requests targeting `dev` branch instead of `main`, which is intended to\n> be more stable.\n\n## What's inside?\n\n-   LLM\n\n    -   CPU-only inference for: Gemma 2-3, PaliGemma 2.\n    -   Sampling with TopK and temperature.\n    -   Backward pass (VJP) and Adam optimizer for Gemma research.\n\n-   Optimizations\n\n    -   Mixed-precision (fp8, bf16, fp32, fp64 bit) GEMM:\n        -   Designed for BF16 instructions, can efficiently emulate them.\n        -   Automatic runtime autotuning 7 parameters per matrix shape.\n    -   Weight compression integrated directly into GEMM:\n        -   Custom fp8 format with 2..3 mantissa bits; tensor scaling.\n        -   Also bf16, f32 and non-uniform 4-bit (NUQ); easy to add new formats.\n\n-   Infrastructure\n\n    -   SIMD: single implementation via Highway. Chooses ISA at runtime.\n    -   Tensor parallelism: CCX-aware, multi-socket thread pool.\n    -   Disk I\u002FO: memory map or parallel read (heuristic with user override).\n    -   Custom format with forward\u002Fbackward-compatible metadata serialization.\n    -   Model conversion from Safetensors, not yet open sourced.\n    -   Portability: Linux, Windows\u002FOS X supported. CMake\u002FBazel. 'Any' CPU.\n\n-   Frontends\n\n    -   C++ APIs with streaming for single query and batched inference.\n    -   Basic interactive command-line app.\n    -   Basic Python bindings (pybind11).\n\n## Quick Start\n\n### System requirements\n\nBefore starting, you should have installed:\n\n- [CMake](https:\u002F\u002Fcmake.org\u002F)\n- [Clang C++ compiler](https:\u002F\u002Fclang.llvm.org\u002Fget_started.html), supporting at\n  least C++17.\n- `tar` for extracting archives from Kaggle.\n\nBuilding natively on Windows requires the Visual Studio 2012 Build Tools with the\noptional Clang\u002FLLVM C++ frontend (`clang-cl`). This can be installed from the\ncommand line with\n[`winget`](https:\u002F\u002Flearn.microsoft.com\u002Fen-us\u002Fwindows\u002Fpackage-manager\u002Fwinget\u002F):\n\n```sh\nwinget install --id Kitware.CMake\nwinget install --id Microsoft.VisualStudio.2022.BuildTools --force --override \"--passive --wait --add Microsoft.VisualStudio.Workload.VCTools;installRecommended --add Microsoft.VisualStudio.Component.VC.Llvm.Clang --add Microsoft.VisualStudio.Component.VC.Llvm.ClangToolset\"\n```\n\n### Step 1: Obtain model weights and tokenizer from Kaggle or Hugging Face Hub\n\nVisit the\n[Kaggle page for Gemma-2](https:\u002F\u002Fwww.kaggle.com\u002Fmodels\u002Fgoogle\u002Fgemma-2\u002FgemmaCpp)\nand select `Model Variations |> Gemma C++`.\n\nOn this tab, the `Variation` dropdown includes the options below. Note bfloat16\nweights are higher fidelity, while 8-bit switched floating point weights enable\nfaster inference. In general, we recommend starting with the `-sfp` checkpoints.\n\n> [!NOTE] **Important**: We strongly recommend starting off with the\n> `gemma2-2b-it-sfp` model to get up and running.\n\nGemma 2 models are named `gemma2-2b-it` for 2B and `9b-it` or `27b-it`. See the\n`ModelPrefix` function in `configs.cc`.\n\n### Step 2: Extract Files\n\nAfter filling out the consent form, the download should proceed to retrieve a\ntar archive file `archive.tar.gz`. Extract files from `archive.tar.gz` (this can\ntake a few minutes):\n\n```\ntar -xf archive.tar.gz\n```\n\nThis should produce a file containing model weights such as `2b-it-sfp.sbs` and\na tokenizer file (`tokenizer.spm`). You may want to move these files to a\nconvenient directory location (e.g. the `build\u002F` directory in this repo).\n\n### Step 3: Build\n\nThe build system uses [CMake](https:\u002F\u002Fcmake.org\u002F). To build the gemma inference\nruntime, create a build directory and generate the build files using `cmake`\nfrom the top-level project directory. Note if you previous ran `cmake` and are\nre-running with a different setting, be sure to delete all files in the `build\u002F`\ndirectory with `rm -rf build\u002F*`.\n\n#### Unix-like Platforms\n```sh\ncmake -B build\n```\n\nAfter running `cmake`, you can enter the `build\u002F` directory and run `make` to\nbuild the `.\u002Fgemma` executable:\n\n```sh\n# Configure `build` directory\ncmake --preset make\n\n# Build project using make\ncmake --build --preset make -j [number of parallel threads to use]\n```\n\nReplace `[number of parallel threads to use]` with a number - the number of\ncores available on your system is a reasonable heuristic. For example, `make -j4\ngemma` will build using 4 threads. If the `nproc` command is available, you can\nuse `make -j$(nproc) gemma` as a reasonable default for the number of threads.\n\nIf you aren't sure of the right value for the `-j` flag, you can simply run\n`make gemma` instead and it should still build the `.\u002Fgemma` executable.\n\n> [!NOTE]\n> On Windows Subsystem for Linux (WSL) users should set the number of\n> parallel threads to 1. Using a larger number may result in errors.\n\nIf the build is successful, you should now have a `gemma` executable in the\n`build\u002F` directory.\n\n#### Windows\n\n```sh\n# Configure `build` directory\ncmake --preset windows\n\n# Build project using Visual Studio Build Tools\ncmake --build --preset windows -j [number of parallel threads to use]\n```\n\nIf the build is successful, you should now have a `gemma.exe` executable in the\n`build\u002F` directory.\n\n#### Bazel\n\n```sh\nbazel build -c opt --cxxopt=-std=c++20 :gemma\n```\n\nIf the build is successful, you should now have a `gemma` executable in the\n`bazel-bin\u002F` directory.\n\n#### Make\n\nIf you prefer Makefiles, @jart has made one available here:\n\nhttps:\u002F\u002Fgithub.com\u002Fjart\u002Fgemma3\u002Fblob\u002Fmain\u002FMakefile\n\n### Step 4: Run\n\nYou can now run `gemma` from inside the `build\u002F` directory.\n\n`gemma` has the following required arguments:\n\nArgument      | Description                  | Example value\n------------- | ---------------------------- | ---------------\n`--weights`   | The compressed weights file. | `2b-it-sfp.sbs`\n`--tokenizer` | The tokenizer file.          | `tokenizer.spm`\n\nExample invocation for the following configuration:\n\n-   weights file `gemma2-2b-it-sfp.sbs` (Gemma2 2B instruction-tuned model,\n    8-bit switched floating point).\n-   Tokenizer file `tokenizer.spm` (can omit for single-format weights files\n    created after 2025-05-06, or output by migrate_weights.cc).\n\n```sh\n.\u002Fgemma \\\n--tokenizer tokenizer.spm --weights gemma2-2b-it-sfp.sbs\n```\n\n### PaliGemma Vision-Language Model\n\nThis repository includes a version of the PaliGemma 2 VLM\n([paper](https:\u002F\u002Farxiv.org\u002Fabs\u002F2412.03555)). We provide a C++ implementation of\nthe PaliGemma 2 model here.\n\nTo use the version of PaliGemma included in this repository, build the gemma\nbinary as noted above in Step 3. Download the compressed weights and tokenizer\nfrom\n[Kaggle](https:\u002F\u002Fwww.kaggle.com\u002Fmodels\u002Fgoogle\u002Fpaligemma-2\u002FgemmaCpp\u002Fpaligemma2-3b-mix-224)\nand run the binary as follows:\n\n```sh\n.\u002Fgemma \\\n--tokenizer paligemma_tokenizer.model \\\n--weights paligemma2-3b-mix-224-sfp.sbs \\\n--image_file paligemma\u002Ftestdata\u002Fimage.ppm\n```\n\nNote that the image reading code is very basic to avoid depending on an image\nprocessing library for now. We currently only support reading binary PPMs (P6).\nSo use a tool like `convert` to first convert your images into that format, e.g.\n\n`convert image.jpeg -resize 224x224^ image.ppm`\n\n(As the image will be resized for processing anyway, we can already resize at\nthis stage for slightly faster loading.)\n\nThe interaction with the image (using the mix-224 checkpoint) may then look\nsomething like this:\n\n```\n> Describe the image briefly\nA large building with two towers in the middle of a city.\n> What type of building is it?\nchurch\n> What color is the church?\ngray\n> caption image\nA large building with two towers stands tall on the water's edge. The building\nhas a brown roof and a window on the side. A tree stands in front of the\nbuilding, and a flag waves proudly from its top. The water is calm and blue,\nreflecting the sky above. A bridge crosses the water, and a red and white boat\nrests on its surface. The building has a window on the side, and a flag on top.\nA tall tree stands in front of the building, and a window on the building is\nvisible from the water. The water is green, and the sky is blue.\n```\n\n### Migrating to single-file format\n\nThere is now a new format for the weights file, which is a single file that\nallows to contain the tokenizer (and the model type) directly. A tool to migrate\nfrom the multi-file format to the single-file format is available.\n\n```sh\nio\u002Fmigrate_weights \\\n  --tokenizer ...\u002Ftokenizer.spm --weights ...\u002Fgemma2-2b-it-sfp.sbs \\\n  --output_weights ...\u002Fgemma2-2b-it-sfp-single.sbs\n```\n\nAfter migration, you can omit the tokenizer argument like this:\n\n```sh\n.\u002Fgemma --weights ...\u002Fgemma2-2b-it-sfp-single.sbs\n```\n\n### Troubleshooting and FAQs\n\n**Problems building in Windows \u002F Visual Studio**\n\nCurrently if you're using Windows, we recommend building in WSL (Windows\nSubsystem for Linux). We are exploring options to enable other build\nconfigurations, see issues for active discussion.\n\n**Model does not respond to instructions and produces strange output**\n\nA common issue is that you are using a pre-trained model, which is not\ninstruction-tuned and thus does not respond to instructions. Make sure you are\nusing an instruction-tuned model (`gemma2-2b-it-sfp`) and not a pre-trained\nmodel (any model with a `-pt` suffix).\n\n**What sequence lengths are supported?**\n\nSee `max_seq_len` in `configs.cc` and `InferenceArgs.seq_len`. For the Gemma 3\nmodels larger than 1B, this is typically 32K but 128K would also work given\nenough RAM. Note that long sequences will be slow due to the quadratic cost of\nattention.\n\n**How do I convert my fine-tune to a `.sbs` compressed model file?**\n\nFor PaliGemma 2 checkpoints, you can use python\u002Fconvert_from_safetensors.py to\nconvert from safetensors format (tested with building via bazel). For an adapter\nmodel, you will likely need to call merge_and_unload() to convert the adapter\nmodel to a single-file format before converting it.\n\nHere is how to use it using a bazel build of the compression library assuming\nlocally installed (venv) torch, numpy, safetensors, absl-py, etc.:\n\n```sh\nbazel build \u002F\u002Fcompression\u002Fpython:compression\nBAZEL_OUTPUT_DIR=\"${PWD}\u002Fbazel-bin\u002Fcompression\"\npython3 -c \"import site; print(site.getsitepackages())\"\n# Use your sites-packages file here:\nln -s $BAZEL_OUTPUT_DIR [...]\u002Fsite-packages\u002Fcompression\npython3 python\u002Fconvert_from_safetensors.py --load_path [...].safetensors.index.json\n```\n\n**What are some easy ways to make the model run faster?**\n\n1.  Make sure you are using the 8-bit switched floating point `-sfp` models.\n    These are half the size of bf16 and thus use less memory bandwidth and cache\n    space.\n2.  Due to auto-tuning, the second and especially third query will be faster.\n3.  If you're on a laptop, make sure power mode is set to maximize performance\n    and saving mode is **off**. For most laptops, the power saving modes get\n    activated automatically if the computer is not plugged in.\n4.  Close other unused cpu-intensive applications.\n5.  On macs, anecdotally we observe a \"warm-up\" ramp-up in speed as performance\n    cores get engaged.\n\nWe're also working on algorithmic and optimization approaches for faster\ninference, stay tuned.\n\n## Usage\n\n`gemma` has different usage modes, controlled by the verbosity flag.\n\nAll usage modes are currently interactive, triggering text generation upon\nnewline input.\n\n| Verbosity       | Usage mode | Details                                       |\n| --------------- | ---------- | --------------------------------------------- |\n| `--verbosity 0` | Minimal | Only prints generation output. Suitable as a CLI tool. |\n| `--verbosity 1` | Default | Standard user-facing terminal UI. |\n| `--verbosity 2` | Detailed | Shows additional developer and debug info. |\n\n### Interactive Terminal App\n\nBy default, verbosity is set to 1, bringing up a terminal-based interactive\ninterface when `gemma` is invoked:\n\n```sh\n$ .\u002Fgemma [...]\n  __ _  ___ _ __ ___  _ __ ___   __ _   ___ _ __  _ __\n \u002F _` |\u002F _ \\ '_ ` _ \\| '_ ` _ \\ \u002F _` | \u002F __| '_ \\| '_ \\\n| (_| |  __\u002F | | | | | | | | | | (_| || (__| |_) | |_) |\n \\__, |\\___|_| |_| |_|_| |_| |_|\\__,_(_)___| .__\u002F| .__\u002F\n  __\u002F |                                    | |   | |\n |___\u002F                                     |_|   |_|\n\n...\n\n*Usage*\n  Enter an instruction and press enter (%C reset conversation, %Q quits).\n\n*Examples*\n  - Write an email to grandma thanking her for the cookies.\n  - What are some historical attractions to visit around Massachusetts?\n  - Compute the nth fibonacci number in javascript.\n  - Write a standup comedy bit about WebGPU programming.\n\n> What are some outdoorsy places to visit around Boston?\n\n[ Reading prompt ] .....................\n\n\n**Boston Harbor and Islands:**\n\n* **Boston Harbor Islands National and State Park:** Explore pristine beaches, wildlife, and maritime history.\n* **Charles River Esplanade:** Enjoy scenic views of the harbor and city skyline.\n* **Boston Harbor Cruise Company:** Take a relaxing harbor cruise and admire the city from a different perspective.\n* **Seaport Village:** Visit a charming waterfront area with shops, restaurants, and a seaport museum.\n\n**Forest and Nature:**\n\n* **Forest Park:** Hike through a scenic forest with diverse wildlife.\n* **Quabbin Reservoir:** Enjoy boating, fishing, and hiking in a scenic setting.\n* **Mount Forest:** Explore a mountain with breathtaking views of the city and surrounding landscape.\n\n...\n```\n\n### Usage as a Command Line Tool\n\nFor using the `gemma` executable as a command line tool, it may be useful to\ncreate an alias for gemma.cpp with arguments fully specified:\n\n```sh\nalias gemma2b=\"~\u002Fgemma.cpp\u002Fbuild\u002Fgemma -- --tokenizer ~\u002Fgemma.cpp\u002Fbuild\u002Ftokenizer.spm --weights ~\u002Fgemma.cpp\u002Fbuild\u002Fgemma2-2b-it-sfp.sbs --verbosity 0\"\n```\n\nReplace the above paths with your own paths to the model and tokenizer paths\nfrom the download.\n\nHere is an example of prompting `gemma` with a truncated input\nfile (using a `gemma2b` alias like defined above):\n\n```sh\ncat configs.h | tail -n 35 | tr '\\n' ' ' | xargs -0 echo \"What does this C++ code do: \" | gemma2b\n```\n\n> [!NOTE]\n> CLI usage of gemma.cpp is experimental and should take context length\n> limitations into account.\n\nThe output of the above command should look like:\n\n```sh\n[ Reading prompt ] [...]\nThis C++ code snippet defines a set of **constants** used in a large language model (LLM) implementation, likely related to the **attention mechanism**.\n\nLet's break down the code:\n[...]\n```\n\n### Incorporating gemma.cpp as a Library in your Project\n\nThe easiest way to incorporate gemma.cpp in your own project is to pull in\ngemma.cpp and dependencies using `FetchContent`. You can add the following to\nyour CMakeLists.txt:\n\n```\ninclude(FetchContent)\n\nFetchContent_Declare(sentencepiece GIT_REPOSITORY https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fsentencepiece GIT_TAG 53de76561cfc149d3c01037f0595669ad32a5e7c)\nFetchContent_MakeAvailable(sentencepiece)\n\nFetchContent_Declare(gemma GIT_REPOSITORY https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fgemma.cpp GIT_TAG origin\u002Fmain)\nFetchContent_MakeAvailable(gemma)\n\nFetchContent_Declare(highway GIT_REPOSITORY https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fhighway.git GIT_TAG 2a16a50ff61071bb25ddef0ce35d92b0e2b9c579)\nFetchContent_MakeAvailable(highway)\n```\n\nNote for the gemma.cpp `GIT_TAG`, you may replace `origin\u002Fmain` for a specific\ncommit hash if you would like to pin the library version.\n\nAfter your executable is defined (substitute your executable name for\n`[Executable Name]` below):\n\n```\ntarget_link_libraries([Executable Name] libgemma hwy hwy_contrib sentencepiece)\nFetchContent_GetProperties(gemma)\nFetchContent_GetProperties(sentencepiece)\ntarget_include_directories([Executable Name] PRIVATE ${gemma_SOURCE_DIR})\ntarget_include_directories([Executable Name] PRIVATE ${sentencepiece_SOURCE_DIR})\n```\n\n### Building gemma.cpp as a Library\n\ngemma.cpp can also be used as a library dependency in your own project. The\nshared library artifact can be built by modifying the make invocation to build\nthe `libgemma` target instead of `gemma`.\n\n> [!NOTE]\n> If you are using gemma.cpp in your own project with the `FetchContent` steps\n> in the previous section, building the library is done automatically by `cmake`\n> and this section can be skipped.\n\nFirst, run `cmake`:\n\n```sh\ncmake -B build\n```\n\nThen, run `make` with the `libgemma` target:\n\n```sh\ncd build\nmake -j [number of parallel threads to use] libgemma\n```\n\nIf this is successful, you should now have a `libgemma` library file in the\n`build\u002F` directory. On Unix platforms, the filename is `libgemma.a`.\n\n## Independent Projects Using gemma.cpp\n\nSome independent projects using gemma.cpp:\n\n- [gemma-cpp-python - Python bindings](https:\u002F\u002Fgithub.com\u002Fnamtranase\u002Fgemma-cpp-python)\n- [lua-cgemma - Lua bindings](https:\u002F\u002Fgithub.com\u002Fufownl\u002Flua-cgemma)\n- [Godot engine demo project](https:\u002F\u002Fgithub.com\u002FRliop913\u002FGemma-godot-demo-project)\n\nIf you would like to have your project included, feel free to get in touch or\nsubmit a PR with a `README.md` edit.\n\n## Acknowledgements and Contacts\n\ngemma.cpp was started in fall 2023 by\n[Austin Huang](mailto:austinvhuang@google.com) and\n[Jan Wassenberg](mailto:janwas@google.com), and subsequently released February\n2024 thanks to contributions from Phil Culliton, Paul Chang, and Dan Zheng.\n\nGriffin support was implemented in April 2024 thanks to contributions by Andrey\nMikhaylov, Eugene Kliuchnikov, Jan Wassenberg, Jyrki Alakuijala, Lode\nVandevenne, Luca Versari, Martin Bruse, Phil Culliton, Sami Boukortt, Thomas\nFischbacher and Zoltan Szabadka. It was removed in 2025-09.\n\nGemma-2 support was implemented in June\u002FJuly 2024 with the help of several\npeople.\n\nPaliGemma support was implemented in September 2024 with contributions from\nDaniel Keysers.\n\n[Jan Wassenberg](mailto:janwas@google.com) has continued to contribute many\nimprovements, including major gains in efficiency, since the initial release.\n\nThis is not an officially supported Google product.\n","gemma.cpp是Google的Gemma模型的轻量级独立C++推理引擎。该项目专注于简化和直接性，提供了Gemma-2、Gemma-3及PaliGemma-2模型的最小化实现，支持CPU上的仅推理操作，并具备TopK采样与温度调节等功能。此外，它还集成了混合精度GEMM（fp8, bf16, fp32, fp64）以优化性能，以及权重压缩技术来减少内存占用。gemma.cpp适合于实验研究场景使用，尤其适用于需要将模型嵌入到其他项目中或进行低级别计算修改的研究人员。其紧凑的设计（约2千行核心代码加上约4千行辅助工具代码）使得它易于集成且依赖少。对于追求高性能边缘部署的应用，建议考虑使用更成熟的Python框架如JAX、Keras等。",2,"2026-06-11 03:48:49","high_star"]