[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"project-92342":3},{"id":4,"name":5,"fullName":6,"owner":7,"repo":5,"description":8,"homepage":9,"htmlUrl":9,"language":10,"languages":9,"totalLinesOfCode":9,"stars":11,"forks":12,"watchers":13,"openIssues":14,"contributorsCount":14,"subscribersCount":14,"size":14,"stars1d":14,"stars7d":14,"stars30d":15,"stars90d":14,"forks30d":14,"starsTrendScore":14,"compositeScore":16,"rankGlobal":9,"rankLanguage":9,"license":17,"archived":18,"fork":18,"defaultBranch":19,"hasWiki":18,"hasPages":18,"topics":20,"createdAt":9,"pushedAt":9,"updatedAt":21,"readmeContent":22,"aiSummary":23,"trendingCount":14,"starSnapshotCount":14,"syncStatus":24,"lastSyncTime":25,"discoverSource":26},92342,"colibri","JustVugg\u002Fcolibri","JustVugg","Run GLM-5.2 (744B MoE) on a 25GB-RAM consumer machine — pure C, zero deps, experts streamed from disk. Tiny engine, immense model. 🐦",null,"C",2313,163,3,0,2238,68.64,"Apache License 2.0",false,"main",[],"2026-07-22 04:02:05","\u003Cp align=\"center\">\n  \u003Cimg src=\"assets\u002Fcolibri.svg\" width=\"500\" alt=\"colibrì — piccolo motore, modello immenso\">\n\u003C\u002Fp>\n\n**Tiny engine, immense model.** Run **GLM-5.2 (744B-parameter MoE)** on a consumer machine with ~25 GB of RAM — in pure C, with zero dependencies, by streaming experts from disk.\n\n```\n$ .\u002Fcoli chat\n  🐦 colibrì v1.0 — GLM-5.2 · 744B MoE · int4 · streaming CPU\n  ✓ pronto in 32s · residente 9.9 GB\n  › ciao!\n  ◆ Ciao! 😊 Come posso aiutarti oggi?\n```\n\n## The idea\n\nA 744B Mixture-of-Experts model activates only ~40B parameters per token — and only ~11 GB of those change from token to token (the routed experts). So:\n\n- the **dense part** (attention, shared experts, embeddings — ~17B params) stays **resident in RAM at int4** (~9.9 GB);\n- the **21,504 routed experts** (75 MoE layers × 256 experts + the MTP head, ~19 MB each at int4) live **on disk** (~370 GB) and are **streamed on demand**, with a per-layer LRU cache, an optional pinned hot-store, and the OS page cache as a free L2.\n\nThe engine is a single C file (`c\u002Fglm.c`, ~1,300 lines) plus small headers. No BLAS, no Python at runtime, no GPU.\n\n## What's implemented\n\n- **Faithful GLM-5.2 (`glm_moe_dsa`) forward** — validated token-exact against a `transformers` oracle (teacher-forcing 32\u002F32, greedy 20\u002F20 on a tiny-random model with the real architecture).\n- **MLA attention** (q\u002Fkv-LoRA, interleaved partial RoPE) with **compressed KV-cache**: 576 floats\u002Ftoken instead of 32,768 (57× smaller — GLM-5.2 has 64 heads and no GQA).\n- **DeepSeek-V3-style sigmoid router** (noaux_tc, routed_scaling_factor), shared expert, first-3-dense layers.\n- **Native MTP speculative decoding** — GLM-5.2's own multi-token-prediction head (layer 78) drafts tokens that the main model verifies in one batched forward. Measured **2.00 tokens\u002Fforward (100% acceptance)** on structured text. Lossless — *and stays lossless under sampling* via rejection sampling (accept draft w.p. p(draft); on reject, resample with the draft banned).\n- **True sampling** — temperature + nucleus, defaults tuned for int4 reality (0.7 \u002F 0.90; the official 1.0 \u002F 0.95 samples quantization noise from the tail).\n- **Integer-dot kernels** (Q8_0-style int8 activations, AVX2 `maddubs`): int8 matmuls 1.4–2.5× faster (119 GFLOP\u002Fs measured), int4 1.8× in batch — routing decided per shape by measurement (int4 single-row stays f32: it measured slower).\n- **MLA weight absorption** (DeepSeek trick) for decode: no per-token k\u002Fv reconstruction — the query absorbs `kv_b`, context is projected after attention. Validated exact: TF 32\u002F32 and generation 20\u002F20 with absorption forced everywhere.\n- **Async expert readahead**: while one block of experts is being multiplied, the kernel is already reading the next (`WILLNEED`).\n- **Quantization kernels**: int8 \u002F packed int4 \u002F packed int2, per-row scales, AVX2, dequant-on-use. Packing validated bit-identical to the int8 container.\n- **DSA sparse attention: in progress** — the lightning-indexer weights (a ~108 GB extraction from the FP8 repo, `--indexer` converter mode) are downloading; the indexer forward lands next. Until then attention is dense and exact for contexts ≤ 2048 tokens.\n- **Batch-union MoE**: in prefill (and MTP verification), each unique expert of the batch is read once and applied to every position that routes to it.\n- **Byte-level BPE tokenizer in C** (GPT-2-style with Unicode-property regex, 320k merges).\n- **RAM safety**: the expert cache is auto-sized from `MemAvailable` at startup — an honest peak projection (working set, KV, MTP row, reconstruction buffers) so the kernel OOM-killer never fires.\n- **Offline FP8→int4 converter** (`c\u002Fconvert_fp8_to_int4.py`): downloads one shard at a time (~5 GB), dequants (128×128 block scales), requantizes to the engine's container, deletes the shard — the 756 GB FP8 checkpoint never needs to exist on disk at once. Resumable.\n\n## Honest numbers (WSL2, 12 cores, 25 GB RAM, NVMe via VHDX)\n\n| metric | value |\n|---|---|\n| model on disk (int4 container) | ~370 GB |\n| resident RAM (dense, int4) | 9.9 GB |\n| load time | ~30 s |\n| peak RSS during chat | ~20 GB (auto-capped) |\n| cold decode cost | ~11 GB disk reads\u002Ftoken (75 layers × 8 experts) |\n| disk ceiling (VHDX random) | ~1 GB\u002Fs → ~0.05–0.1 tok\u002Fs cold |\n| MTP speculation | 2.0 tok\u002Fforward measured |\n\nThis is not fast. It is a 744B frontier-class model **answering correctly on a machine that costs less than one H100 fan**. Warm cache, pinned hot experts and MTP push the useful-response latency down considerably; the physics of the disk does the rest.\n\n### SSD Wear Warning\nCold starts are heavy on random reads (~11 GB\u002Ftoken). Reads themselves are safe, but the OS page cache can generate writes. Heavy use may accelerate wear on cheaper SSDs. Use with caution and monitor your drive health.\n\n## Download the model\n\nA pre-converted **GLM-5.2 int4** model for colibrì is available on Hugging Face:\n\n**https:\u002F\u002Fhuggingface.co\u002Fjlnsrk\u002FGLM-5.2-colibri-int4**\n\nDownload the repository and point `COLI_MODEL` to its directory:\n\n```bash\nCOLI_MODEL=\u002Fpath\u002Fto\u002FGLM-5.2-colibri-int4 .\u002Fcoli chat\n```\n\nThis skips the FP8 → int4 conversion step entirely.\n\nThanks DatPat for your help!\n\n### Quick start\n\n```bash\ncd c\n.\u002Fsetup.sh                      # checks gcc\u002FOpenMP, builds, self-tests\n\n# ONE command does everything model-side: downloads GLM-5.2-FP8 shard by shard\n# (never needs the full 756 GB at once), converts to the int4 container, then\n# converts the MTP head for speculative decoding. Resumable at any point.\n# Conversion (only) needs python with: pip install torch safetensors huggingface_hub numpy\n.\u002Fcoli convert --model \u002Fnvme\u002Fglm52_i4     # ~400 GB free on a real ext4\u002FNVMe path\n\n# chat — RAM budget, expert cache and MTP are all detected automatically:\nCOLI_MODEL=\u002Fnvme\u002Fglm52_i4 .\u002Fcoli chat\n```\n\nThe engine at runtime is pure C — python is only used by the one-time converter.\n\nUseful knobs (env or flags): `--temp T` token sampling temperature (default 0.7 + nucleus 0.90 — tuned for int4; 0 = greedy), `--topp 0.7` adaptive expert top-p (30–40% less disk), `--ngen N` max tokens per answer (`:piu` in chat continues a truncated one), `AUTOPIN=0` disable the learning cache's auto-pin, `THINK=1` enable GLM-5.2's reasoning block, `DRAFT=n` MTP draft depth, `TF=1` teacher-forcing validation.\n\n**The learning cache**: the engine records which experts your usage actually routes to (`.coli_usage` next to the model, updated every turn) and at startup automatically pins the hottest ones in spare RAM. colibrì literally gets faster the more you use it.\n\n## Got a better machine? Try it — here's what to expect\n\ncolibrì was built on deliberately humble hardware (12 cores, 25 GB RAM, NVMe behind a WSL2 VHDX that caps random reads at ~1 GB\u002Fs). **Every one of those constraints is a knob your machine can turn up.** The engine needs: Linux (or WSL2), gcc with OpenMP, AVX2, ≥16 GB RAM, and the ~370 GB int4 model on a local NVMe (ext4 — never a network\u002F9p mount).\n\n**How to test it, in order:**\n\n```bash\ncd c && .\u002Fsetup.sh                 # build + architecture self-test (expects 32\u002F32)\n\n# 1) measure YOUR disk the way the engine uses it (parallel 19 MB random reads):\ngcc -O2 -fopenmp iobench.c -o iobench\n.\u002Fiobench \u002Fpath\u002Fto\u002Fglm52_i4\u002Fout-00069.safetensors 19 64 8 0   # buffered, 8 threads\n.\u002Fiobench \u002Fpath\u002Fto\u002Fglm52_i4\u002Fout-00069.safetensors 19 64 8 1   # O_DIRECT\n\n# 2) chat; watch the per-turn stats line (tok\u002Fs, expert hit-rate, RSS):\nCOLI_MODEL=\u002Fpath\u002Fto\u002Fglm52_i4 .\u002Fcoli chat\n\n# 3) record expert usage, then pin the hottest experts in your spare RAM:\nSTATS=stats.txt .\u002Fcoli chat\nPIN=stats.txt PIN_GB=20 .\u002Fcoli chat        # scale PIN_GB to your free RAM\n\n# 4) quality benchmarks (MMLU\u002FHellaSwag\u002FARC):\n.\u002Fcoli bench\n```\n\n**Back-of-envelope predictions** (decode is disk-bound: a cold token costs ~11.4 GB of expert reads; MTP speculation roughly halves the effective cost; RAM turns cold reads into free cache hits):\n\n| machine | expected |\n|---|---|\n| this dev box (WSL2 VHDX, ~1 GB\u002Fs, 25 GB RAM) | ~0.05–0.1 tok\u002Fs cold — proven baseline |\n| native Linux, PCIe4 NVMe (~3–5 GB\u002Fs random), 32 GB | ~0.5–1 tok\u002Fs |\n| PCIe5 NVMe or 2×NVMe RAID0 (~8–12 GB\u002Fs), 64 GB (PIN ~40 GB of hot experts) | ~2–4 tok\u002Fs |\n| 128–256 GB RAM, 12 cores (hot experts cached) | ~2–4 tok\u002Fs — matmul-bound: ~80 GFLOP\u002Ftoken vs ~250 GFLOP\u002Fs of our AVX2 kernels |\n| same RAM + 24–32 cores, or AVX-512\u002FVNNI kernels | ~5–15 tok\u002Fs — interactive; kernel work is the multiplier |\n\nThese are estimates, not measurements — if you run colibrì on serious hardware, **please open an issue with your numbers**: real datapoints from better machines are exactly what this project needs next.\n\n### Community benchmarks (measured)\n\nReal numbers from real machines, stock build (`setup.sh`, gcc 13), greedy decoding, `--ngen 32`, MTP active:\n\n| machine | disk (iobench, 19 MB × 64, 8 threads) | config | measured |\n|---|---|---|---|\n| Intel Core Ultra 7 270K Plus (24 threads) · WSL2 · 24 GB RAM · NVMe VHDX ([#2](https:\u002F\u002Fgithub.com\u002FJustVugg\u002Fcolibri\u002Fissues\u002F2)) | 1.96 GB\u002Fs buffered · 2.74 GB\u002Fs O_DIRECT | default | 0.07 tok\u002Fs · expert hit 3–4% · RSS 14.1 GB |\n| 〃 | 〃 | `--topp 0.7` | **0.11 tok\u002Fs** · expert hit 11% · RSS 14.7 GB |\n\nTakeaways from this datapoint: with 24 GB of RAM the engine auto-caps the expert cache to 2 slots\u002Flayer, so decode stays cold even on a disk 2–2.7× faster than the dev box — **on small-RAM machines the RAM cap, not the disk, is the binding constraint**, exactly as the table above predicts. And `--topp 0.7` alone bought a clean 1.6× end-to-end speedup.\n\n## Quality benchmark — help wanted\n\nWe have never measured how much the int4 quantization costs in accuracy — the harness is built and wired, but scoring is one forward per answer option, and on the dev box's ~1 GB\u002Fs disk a full run takes the better part of a day. **This is the single most valuable thing a faster machine can contribute.** The code is here and ready; one command runs it end to end (it auto-downloads the datasets on first use):\n\n```bash\ncd c\n.\u002Fcoli bench                                   # hellaswag, arc_challenge, mmlu — 40 questions each\n.\u002Fcoli bench hellaswag --limit 200             # one task, more questions\n.\u002Fcoli bench mmlu arc_challenge --ram 100      # pick tasks, set a RAM budget\n```\n\nIt prints per-task accuracy (log-likelihood scoring, EleutherAI-harness style). Published full-precision GLM-5.2 scores on these tasks sit around 85–95%; if our int4 container lands within a few points, the quantization is validated — if it doesn't, we know to invest in mixed \u002F grouped-scale quantization. **If you have the hardware to run this, please open an issue with the numbers** — it's the measurement the project is missing.\n\n## Supporting the project\n\ncolibrì is a one-person project, written and tested entirely on a 12-core laptop with 25 GB of RAM — the numbers above are the ceiling of what I can measure at home. If this project is useful or interesting to you and you'd like to support its development (better test hardware translates *directly* into a faster engine for everyone: real NVMe scaling data, bigger pinned caches, int2\u002Fint3 quality sweeps on real benchmarks), you can:\n\n- ⭐ star the repo and share it;\n- 🐛 open issues with benchmark numbers from your hardware;\n- 💬 reach out via GitHub issues if you'd like to sponsor development or donate hardware.\n\nEvery contribution, from a datapoint to a disk, moves the ceiling.\n\n## Repo layout\n\n```\nc\u002Fglm.c          the engine (GLM-5.2 forward, streaming MoE, MTP, serve mode)\nc\u002Fst.h           safetensors reader: pread + fadvise, no mmap (RSS stays flat)\nc\u002Ftok.h          byte-level BPE tokenizer in C\nc\u002Fcoli           CLI: chat \u002F run \u002F bench \u002F convert \u002F info\nc\u002Fiobench.c      parallel disk microbenchmark (measures what the engine feels)\nc\u002Fconvert_fp8_to_int4.py   disk-safe FP8 → int4 converter\nc\u002Fmake_glm_oracle.py       tiny-random oracle generator for validation\nc\u002Folmoe.c        stage-A engine (OLMoE), first validation target\n```\n\n## Why \"colibrì\"\n\nThe hummingbird weighs a few grams, hovers in place, and visits a thousand flowers a day. This engine keeps a 744-billion-parameter giant alive on hummingbird rations: 25 GB of RAM, twelve CPU cores, and a lot of disk patience.\n\n## License\n\nApache 2.0. GLM-5.2 weights are released by Z.ai under MIT.\n","Colibri 是一个轻量级 C 语言推理引擎，专为在普通消费级设备（25GB 内存）上高效运行超大规模 MoE 模型 GLM-5.2（744B 参数）而设计。其核心技术包括：纯 C 实现、零外部依赖、磁盘流式加载路由专家（on-demand expert streaming）、整数精度（int4\u002Fint8）计算内核、MLA 注意力压缩 KV 缓存、原生 MTP 多令牌推测解码及异步专家预读。适用于资源受限环境下的本地大模型交互、离线推理与边缘部署，无需 GPU 或 Python 运行时。",2,"2026-07-08 04:30:09","CREATED_QUERY"]