[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"project-96752":3},{"id":4,"name":5,"fullName":6,"owner":7,"repo":5,"description":8,"homepage":9,"htmlUrl":10,"language":11,"languages":10,"totalLinesOfCode":10,"stars":12,"forks":13,"watchers":14,"openIssues":15,"contributorsCount":15,"subscribersCount":15,"size":15,"stars1d":15,"stars7d":15,"stars30d":15,"stars90d":15,"forks30d":15,"starsTrendScore":15,"compositeScore":16,"rankGlobal":10,"rankLanguage":10,"license":17,"archived":18,"fork":18,"defaultBranch":19,"hasWiki":18,"hasPages":20,"topics":21,"createdAt":10,"pushedAt":10,"updatedAt":42,"readmeContent":43,"aiSummary":10,"trendingCount":15,"starSnapshotCount":15,"syncStatus":44,"lastSyncTime":45,"discoverSource":46},96752,"Glyd","surya-koritala\u002FGlyd","surya-koritala","Glyd - Fast lossless compression algorithm","https:\u002F\u002Fgithub.com\u002Fsurya-koritala\u002FGlyd#readme",null,"Rust",167,1,164,0,40.9,"BSD 3-Clause \"New\" or \"Revised\" License",false,"main",true,[22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41],"benchmark","c-abi","cli","compression","data-compression","deduplication","delta-compression","entropy-coding","gzip","jpeg","logs","lossless-compression","lz4","lz77","object-storage","parquet","rust","s3","simd","zstd","2026-09-25 04:01:33","\u003Ch1 align=\"center\">Glyd\u003C\u002Fh1>\n\u003Cp align=\"center\">\u003Cstrong>Glyd - Fast lossless compression algorithm\u003C\u002Fstrong>\u003C\u002Fp>\n\n\u003Cp align=\"center\">\n\u003Ca href=\"https:\u002F\u002Fgithub.com\u002Fsurya-koritala\u002FGlyd\u002Factions\">\u003Cimg alt=\"CI\" src=\"https:\u002F\u002Fgithub.com\u002Fsurya-koritala\u002FGlyd\u002Factions\u002Fworkflows\u002Fci.yml\u002Fbadge.svg\">\u003C\u002Fa>\n\u003Ca href=\"LICENSE\">\u003Cimg alt=\"License: BSD-3-Clause OR GPL-2.0\" src=\"https:\u002F\u002Fimg.shields.io\u002Fbadge\u002Fcodec-BSD--3--Clause%20OR%20GPL--2.0-blue.svg\">\u003C\u002Fa>\n\u003Ca href=\"glyd-store\u002FLICENSE\">\u003Cimg alt=\"Store: BUSL-1.1\" src=\"https:\u002F\u002Fimg.shields.io\u002Fbadge\u002Fstore-BUSL--1.1-blue.svg\">\u003C\u002Fa>\n\u003Cimg alt=\"Rust 1.80+\" src=\"https:\u002F\u002Fimg.shields.io\u002Fbadge\u002Frust-1.80%2B-blue.svg\">\n\u003Cimg alt=\"SIMD: AVX2 | NEON\" src=\"https:\u002F\u002Fimg.shields.io\u002Fbadge\u002FSIMD-AVX2%20%7C%20NEON-orange.svg\">\n\u003Ca href=\"include\u002Fglyd.h\">\u003Cimg alt=\"C ABI\" src=\"https:\u002F\u002Fimg.shields.io\u002Fbadge\u002FC%20ABI-include%2Fglyd.h-brightgreen.svg\">\u003C\u002Fa>\n\u003Ca href=\"https:\u002F\u002Fgithub.com\u002Fsurya-koritala\u002FGlyd\u002Freleases\">\u003Cimg alt=\"Release\" src=\"https:\u002F\u002Fimg.shields.io\u002Fgithub\u002Fv\u002Frelease\u002Fsurya-koritala\u002FGlyd?include_prereleases&label=release\">\u003C\u002Fa>\n\u003C\u002Fp>\n\n\u003Cp align=\"center\">\n\u003Ca href=\"#at-a-glance\">At a glance\u003C\u002Fa> ·\n\u003Ca href=\"#what-glyd-saves-you\">Savings\u003C\u002Fa> ·\n\u003Ca href=\"#quick-start\">Quick start\u003C\u002Fa> ·\n\u003Ca href=\"#levels-and-modes\">Levels and modes\u003C\u002Fa> ·\n\u003Ca href=\"#base-mode-a-version-compressed-against-the-last-one\">Base mode\u003C\u002Fa> ·\n\u003Ca href=\"#record-mode-logs-and-table-dumps-as-columns\">Record mode\u003C\u002Fa> ·\n\u003Ca href=\"#real-data-real-machines\">Real data\u003C\u002Fa> ·\n\u003Ca href=\"#known-gaps\">Known gaps\u003C\u002Fa> ·\n\u003Ca href=\"ROADMAP.md\">Roadmap\u003C\u002Fa> ·\n\u003Ca href=\"#license\">License\u003C\u002Fa>\n\u003C\u002Fp>\n\n---\n\n## At a glance\n\n**Glyd** is a lossless compression library and CLI, written in Rust with a\nC ABI, for the workloads where storage and read CPU decide the bill:\nobject storage, data lakes, logs and telemetry, backups and versioned\nexports, RPC payloads, caches. It is a drop-in alternative to **LZ4**,\n**Snappy** and **zstd**, and it does two things they do not: it turns\nrecord-shaped data (logs, dumps, CSV, JSON lines) into typed columns\nbefore compressing (`-r`), and it compresses a new version of an object\nagainst the old one (`--base`).\n\nEvery number in this README is measured on public data, every decode\ncompared byte for byte with its input, against the reference codec on the\nsame machine and thread count in the same run. The full program and its\nresults: [docs\u002Fbenchmarks\u002Fsuite-2026-09-21.md](docs\u002Fbenchmarks\u002Fsuite-2026-09-21.md).\n\n**The table everyone uses** — the 8.7 GB real-data corpus (logs, JSON\nevents, SQL dumps, Parquet) on AWS Graviton3, ratio · compress MB\u002Fs ·\ndecompress MB\u002Fs. One core, the way zstd's own README reports:\n\n| Codec | Ratio | Compress MB\u002Fs | Decompress MB\u002Fs |\n| :--- | ---: | ---: | ---: |\n| LZ4 | 2.72 | **504** | 1,394 |\n| ⚡ **Glyd default** | 2.82 | 332 | **3,309** |\n| zstd -3 | 3.86 | 313 | 1,425 |\n| ⚡ **Glyd ‑‑max** | **3.98** | 233 | **1,571** |\n\nEight cores, what a server does (Glyd's output decodes in parallel; a\nzstd or LZ4 frame decodes on one thread):\n\n| Codec | Ratio | Compress MB\u002Fs | Decompress MB\u002Fs |\n| :--- | ---: | ---: | ---: |\n| ⚡ **Glyd default** | 2.82 | **2,143** | **22,707** |\n| zstd -3 -T8 | 3.85 | 1,969 | 1,422 |\n| ⚡ **Glyd ‑‑max** | 3.94 | 1,512 | **10,413** |\n| ⚡ **Glyd ‑‑max ‑r** | 4.71 | 643 | **4,040** |\n| LZ4 | 2.72 | 503 | 1,392 |\n| zstd -19 -T8 | 4.66 | 13 | 1,326 |\n| ⚡ **Glyd ‑‑ultra** | 4.66 | 14.5 | **9,620** |\n| ⚡ **Glyd ‑‑ultra ‑r** | **5.22** | 20 | **3,927** |\n\nIn one line: Glyd reads 3–7× faster than zstd on a server and stores 10–70%\nless where the data has structure; it writes at 0.77× zstd -3's speed\n(record mode 0.33×). For data written once and read many times that is the\nright side of the trade; for data written constantly and rarely read,\nzstd -3 or LZ4 still win on write cost.\n\n- 🏪 **The store (`glyd-store`)**: `put` an object and it is kept as a delta against the stored object it most resembles, found by fingerprints, when that pays; chains capped at four. A 39 GB bucket (six Ubuntu image builds, fifteen kernel releases, two months of Wikipedia tables, twelve hours of GitHub events) stores in 1,334 MB against zstd -3's 6,132 MB: **4.6× fewer bytes**, put at 500 MB\u002Fs end to end, every object read back byte-exact.\n- 🗂️ **Record mode (`-r`)**: logs, SQL dumps, CSV and JSON lines as typed columns; logs of varying shape as templates plus typed variables. Telemetry stores 2.5–3.5× less than zstd -3 and 1.5–2× less than zstd -19; application and system logs 1.4–3.3× less than zstd -3 and 1.1–2.1× less than zstd -19; the whole corpus 19% less than zstd -3.\n- 📦 **Packs (`--pack`)**: many small objects as one record-mode stream with an index; 2–4× fewer bytes than zstd + dictionary per object, any one object read back in a millisecond.\n- 🧩 **Shape dictionaries (`--shape`)**: record mode for a single small object. Trained on a sample; a 1–4 KB event or log object stores 1.1–1.9× less than with a zstd dictionary.\n- 🧊 **Cold level (`--cold`)**: context mixing for what is stored for years and read rarely. 1.5–2.6× fewer bytes than zstd -19 on logs, dumps, JSON and text — the zpaq -m5 class at 3–4× its speed — at 1.2–1.5 MB\u002Fs per core each way.\n- 🔁 **Base mode (`--base`)**: a new version against the old one, its content found wherever it moved. Dumps, images and source trees at 1–5% of their plain size; 1.1–2.1× less than `zstd --patch-from` at the fast tier, at 1.8–3× its speed; 15 kernel releases in 228 MB instead of 3 GB.\n- 🔭 **128 MB long-distance matcher** (`--max --long`, `--ultra`, the store): JSON events 22% smaller than zstd -3, 10% smaller than zstd -19.\n- 🚀 **Fastest reads at every ratio**: 8-way interleaved entropy coding and copy-only loops, units that decode one per core.\n- 🛡️ **Verified**: 98 tests, a million-mutation fuzz per run, every earlier format decoded unchanged, the CLI round-tripped with corrupted copies on both AWS machines.\n- 🔌 **Rust, C ABI, CLI**, streaming `std::io` adapters, trained dictionaries for small objects.\n\n---\n\n## What Glyd saves you\n\n> **Try it:** [surya-koritala.github.io\u002FGlyd\u002Fsavings.html](https:\u002F\u002Fsurya-koritala.github.io\u002FGlyd\u002Fsavings.html) — enter what you store and what you compress with today.\n\nBytes stored, against zstd on the same data (measured; the sign is what\nmatters):\n\n| Data | vs zstd -3 (the fast tier) | vs zstd -19 (the slow tier) |\n| :--- | ---: | ---: |\n| **A bucket of versioned objects** — images, releases, dumps, events (`--store`) | **−78%** (2.0–13.9× by family; events, with nothing to share, −23%) | |\n| **Versions** of a dump, image or source tree (`--base`) | **−45 to −53%** vs zstd's fast patch; **−95 to −99%** vs the version alone | −5 to −21% (`--ultra`) |\n| **Telemetry, measurements** as CSV or JSON lines (`-r`) | **−60 to −71%** | **−33 to −52%** |\n| **Application and system logs** (HDFS, Spark, BGL, Android; `-r`) | **−28 to −69%** | **−9 to −53%** |\n| **Cold archives** of logs, dumps, JSON, text (`--cold`, 1 MB\u002Fs per core) | **−52 to −69%** | **−32 to −62%** |\n| **Small objects** (events, log and CSV objects of 1–4 KB) packed (`--pack`) | **−48 to −75%** vs zstd + dictionary per object | |\n| **Access logs** (`-r`) | **−55%** | **−35%** |\n| **SQL dumps** (`-r`) | **−41%** | **−30%** |\n| **JSON events** (API payloads with hashes) | **−22%** | −10% |\n| Whole mixed corpus (`-r`) | **−19%** | −10% |\n| Plain text, binaries, Parquet | ~0% | ~0% (the floor; nothing moves it) |\n\nA terabyte kept a year in S3 Standard, compressed once and read once a\nmonth (Graviton3, CPU billed at the on-demand price): `--max -r` **$61.7**\nagainst zstd -3's $73.3 and zstd -19's $102; at ten reads a month\n`--max` **$81.3**, `--max -r` $82.4, zstd -3 $84.1; at a hundred reads a\nmonth `--max` **$178** against zstd -3's $191 (its reads now cost less\nCPU than zstd's), while `--max -r` is $289: record-mode reads spend 2×\nthe CPU rebuilding the columns\n([report](docs\u002Fbenchmarks\u002Fsuite-2026-09-21.md)). At the scale of\nobject storage (hundreds of exabytes) every 1% fewer bytes is about $250\nmillion a year at list price; the percentages above are what to multiply.\n\n---\n\n## Quick start\n\n```bash\nbrew install surya-koritala\u002Fglyd\u002Fglyd        # macOS \u002F Linux: the glyd and glyd-store CLIs, glyd.h\ncargo install glyd glyd-store                # from crates.io\npip install https:\u002F\u002Fgithub.com\u002Fsurya-koritala\u002FGlyd\u002Freleases\u002Flatest\u002Fdownload\u002Fglyd-0.14.3-py3-none-macosx_11_0_arm64.whl   # or the manylinux x86_64 \u002F aarch64 wheel\n```\n\nEvery [release](https:\u002F\u002Fgithub.com\u002Fsurya-koritala\u002FGlyd\u002Freleases) carries\nthe CLIs, the shared and static libraries and `glyd.h` for Linux\nx86_64, Linux aarch64 and macOS arm64, plus a Python wheel for each.\nBindings: [Python](bindings\u002Fpython\u002FREADME.md), [Go](bindings\u002Fgo\u002Fglyd.go)\n(cgo over `include\u002Fglyd.h`), C (`include\u002Fglyd.h`). Formats:\n[docs\u002Fspec.md](docs\u002Fspec.md). lzbench: `contrib\u002Flzbench\u002Fsetup.sh \u003Ccheckout>`.\n\n```bash\ncargo install --git https:\u002F\u002Fgithub.com\u002Fsurya-koritala\u002FGlyd\n```\n\n```bash\nglyd --max  events.json -o events.glyd            # the zstd -3 slot: fewer bytes, 3-7x faster reads\nglyd --max -r access.log -o access.glyd           # record mode: logs, dumps, CSV, JSON lines as columns\nglyd --ultra -r dump.sql -o dump.glyd             # fewest bytes from a parse; slow to write\nglyd --cold -r dump.sql -o dump.glyd              # fewest bytes of all; 1 MB\u002Fs per core each way\nglyd-store bucket\u002F --put mon.tar tue.tar wed.tar  # the store finds each object's base itself (glyd-store crate)\nglyd-store bucket\u002F --get 2 -o wed.tar            # also --find NAME, --delete ID, --rebase ID, --compact, --verify, --stats\nglyd-store meta\u002F --s3 s3:\u002F\u002Fbucket\u002Fprefix --put wed.tar     # objects in S3 (or any S3-compatible service)\nglyd-store --audit s3:\u002F\u002Fbucket\u002Fprefix            # what the store would save there, from a sample, in dollars\nglyd --base dump-mon.sql dump-tue.sql -o tue.glyd # base mode: Tuesday's dump against Monday's\nglyd -d --base dump-mon.sql tue.glyd -o tue.sql   # decoding a base-mode file needs the base\nglyd    telemetry.bin -o telemetry.glyd           # default: LZ4-class ratio, 22 GB\u002Fs reads on 8 cores\nglyd -d events.glyd -o events.json                # the level and mode are in the stream\nglyd -b bigfile                                   # benchmark every level on your data\n```\n\nRust:\n\n```rust\nlet mut out = Vec::new();\nglyd::compress_parallel_into_max(&input, &mut out);   \u002F\u002F or compress_into_max, _ultra, _cold, compress_into (default)\nlet back = glyd::decompress_parallel(&out)?;          \u002F\u002F any level, any block mix\n\nglyd::compress_records_into_max(&log, &mut out);      \u002F\u002F record mode (-r); decompress() reads it\nglyd::compress_with_base(&old, &new, &mut out, false);\u002F\u002F base mode; decompress_with_base(&old, &out)\nlet mut store = glyd_store::Store::open(\"bucket\u002F\")?;   \u002F\u002F the glyd-store crate: put finds the base, get rebuilds\nlet id = store.put(\"wed.tar\", &data)?;  let back = store.get(id)?;\n```\n\n```python\nimport glyd                                   # bindings\u002Fpython\nc = glyd.compress(data, records=True)         # decompress(c); pack(objects); Store(\"bucket\u002F\").put(name, data)\nglyd::decompress_stream(&out, |batch| file.write_all(batch))?;   \u002F\u002F batches of units, bounded memory\n\n\u002F\u002F Dictionaries for small objects: train once on samples, keep the bytes.\nlet dict = glyd::Dict::train(&samples, 110 * 1024);\nglyd::compress_with_dict(&dict, &doc, &mut out);      \u002F\u002F or compress_with_dict_ultra\nlet back = glyd::decompress_with_dict(&dict, &out)?;\n```\n\nC \u002F C++ \u002F Go \u002F Python (ctypes): link `libglyd` and include [`include\u002Fglyd.h`](include\u002Fglyd.h):\n\n```c\nsize_t cap = glyd_max_compressed_len(n);\nint64_t clen = glyd_compress_max_parallel(src, n, dst, cap);   \u002F\u002F or glyd_compress \u002F _parallel\nint64_t dlen = glyd_decompress_parallel(dst, clen, out, n);\n```\n\n---\n\n## Levels and modes\n\n| Level | Use it for | How it works |\n| :--- | :--- | :--- |\n| **‑‑turbo** (‑t) | Data read far more often than written: caches, assets, KV-cache paging | v6 format, minimum match 10: fewest tokens, one 32-byte copy per token |\n| **default** | The LZ4\u002FSnappy slot with a better ratio and faster reads | v6 format, LZAV-class finder, minimum match 7 |\n| **‑‑fast** (‑1) | LZ4-class compression speed | v6 format, LZ4-class finder, minimum match 5 |\n| **‑‑max** (‑9) | The zstd -3 slot: fewer bytes, 3–7× faster reads on a server | v9 format: 8-way interleaved Huffman literals, tANS sequences with repeat offsets, a double-fast parse over an 8 MB window (zstd -3's, one rule stricter), blocks cut where the bytes' statistics change |\n| **‑‑max ‑‑long** (‑L) | Events, logs, anything that repeats itself across an input | The same, with a 128 MB long-distance matcher run once before the parse (as `zstd --long`): 3–29% fewer bytes on events and logs, at a third more write time |\n| **‑‑max ‑‑dense** (‑D) | Objects written once and read rarely (the store's) | `--long`, in 128 MB units parsed in stripes on all cores: one core's bytes at any core count, 5–9% fewer on files of a few hundred MB; reads scale only with the units |\n| **‑‑ultra** (‑19) | Write once, read many: datasets, release assets | v9 format on an optimal parse: binary-tree finder, every position priced in the coder's own bits ([design](docs\u002Fdesign\u002Fultra-parse.md)) |\n| **‑‑cold** (‑C) | Stored for years, read rarely: archives, compliance holds, the last copy | Context mixing: every bit predicted from eleven contexts (byte orders, the word, the column, the JSON key, the longest earlier match) with bit histories, mixed by two small networks, coded arithmetically; 32 MB units in parallel, 1–1.3 MB\u002Fs per core each way ([design](docs\u002Fdesign\u002Fformat-v7.md#the-cold-level-context-mixing)) |\n\n| Mode | Use it for | How it works |\n| :--- | :--- | :--- |\n| **‑r** record mode | Logs of any shape, SQL dumps, CSV\u002FTSV, JSON lines | Detects the shape (delimited lines, dumps, JSON lines, or templates for logs of varying shape), turns each field, key path or template slot into a typed stream (integer, decimal and date-time deltas, dictionaries with recency ranks, text), compresses those with the level in 32 MB units, rebuilds exactly ([design](docs\u002Fdesign\u002Fformat-v7.md#record-mode-v040-typed-columns-before-the-level)) |\n| **‑‑base** base mode | Versions: nightly dumps, snapshots, images, source trees | Parses each 32 MB of the new version with the region of the old one that holds its content (found through a coarse map of the base) as history; the stream decodes with the same base ([design](docs\u002Fdesign\u002Fformat-v7.md#base-mode-v050-a-version-compressed-against-the-last-one)) |\n\nAll levels write one container; the decoder reads any mix. Blocks are\n256 KB; the parallel paths cut the input into units (one per core, up to\n128 MB) that compress and decode independently. The CLI decodes a batch of\nunits at a time into one reused buffer, so its memory is a batch, not the\nfile.\n\n---\n\n## Real data, real machines\n\nThe verification and benchmark program: 8.7 GB of logs, JSON events,\nSQL dumps and Parquet; zstd -3, zstd -19 and LZ4 on the same AWS machines\n(Graviton3 c7g.2xlarge and Sapphire Rapids c7i.2xlarge) and thread counts;\nsmall objects with dictionaries trained on other days' data; a real S3\nround trip (compress, upload, download, decompress, sha256) costed at list\nprices. Method: [docs\u002Fbenchmarks\u002FREADME.md](docs\u002Fbenchmarks\u002FREADME.md);\nresults with every table: [docs\u002Fbenchmarks\u002Fsuite-2026-09-21.md](docs\u002Fbenchmarks\u002Fsuite-2026-09-21.md);\nraw rows: [benchmarks\u002Fsuite\u002F](benchmarks\u002Fsuite\u002F).\n\nThe short version: `--max` stores 2.2% less than zstd -3 over the corpus\n(24% less on JSON events), decodes 3–7× faster with 8 cores and 1.10×\n(Graviton3) \u002F 0.85× (Sapphire Rapids) on one core, and compresses at\n0.71–0.77× zstd -3's speed. `--ultra` equals zstd -19 (10% smaller on\nJSON). `--max -r` stores 18% less than zstd -3 and 1% less than zstd -19\nat 500–645 MB\u002Fs on 8 cores; `--ultra -r` 11% less than zstd -19.\n\n## Against zstd, xz and brotli on 24 kinds of data\n\nEvery codec's own CLI on one thread, every decode compared byte for\nbyte with its input. Green: Glyd's file is smaller.\n\n![Compression benchmark: Glyd --max bytes against zstd -3 on logs, SQL dumps, JSON, gzip, zip, jar, Office documents, PDF, PNG, JPEG, text, executables and Parquet](docs\u002Fbenchmarks\u002Fcharts\u002Fbytes-fast-tier.svg)\n\n| Data | vs zstd -3 (`--max`) | vs zstd -19, xz, brotli -11 (`--ultra`) |\n| :--- | :--- | :--- |\n| Records: logs, dumps, JSON (`-r`) | **12–61% fewer bytes** | **3–38% fewer** (JSON: 9% more) |\n| Containers: gzip, zip, Office, PDF, PNG, JPEG | **up to 60% fewer** | **14–63% fewer** |\n\nGlyd wins where the data has structure: each field of a record becomes\na column, and the deflate or JPEG inside a container is opened and\nre-created bit for bit. On plain text, executables and Parquet it is\nzstd-class: within 2% at the fast tier, 1–15% larger than xz and\nbrotli -11 at their strongest. Every codec's speed, the strong-tier\nand ratio-against-speed charts, and lz4, bzip2, zpaq and JPEG XL:\n[docs\u002Fbenchmarks\u002Flandscape-2026-09-22.md](docs\u002Fbenchmarks\u002Flandscape-2026-09-22.md).\n\n## The store: compression across objects\n\nInside one object every codec sits on the same floor: on plain bytes\nzstd -19, xz and Glyd `--ultra` land within 5% of each other, and the\nlevers that beat it (columns, base mode, context mixing) each take a\nkind of data. The redundancy of object storage is elsewhere — between\nobjects. A bucket holds builds, snapshots, dumps and releases that are\nnear-copies of earlier ones, and a codec that sees one object at a time\ncannot know it.\n\nThe store is its own crate, `glyd-store` (`glyd-store DIR --put ...`;\nunder the Business Source License, the codec being BSD-3-Clause OR GPL-2.0).\n`Store::put` fingerprints the object (one sparse anchor in 4 KB, the\nsame map base mode uses), looks the fingerprints up in the store's\ntable, takes the stored object sharing the most as the base, and keeps\nthe object as a delta against it (`--base`) when that saves a fifth or\nmore of what it costs alone; else alone at `--max` (record mode where\nit pays; `--ultra` or `--cold` on request). Chains are at most four\nlong; past that the base is the first object of the version's family\n(an object whose base holds under 90% of its fingerprints, a new major\nrelease against the old one, starts a family; its point releases join\nit), so a read is at most five decodes at 6–10 GB\u002Fs and a 6.6 kernel\nis never a delta of 5.15's; `rebase(id)` stores an object read often\nalone again. `get`, `id_of(name)`, `delete` (a deleted object's bytes\nstay while a live chain runs through them), `compact` (frees what no\nlive object needs), `verify` (every object read back and checked).\nThe objects' bytes go through a `Backend`: a directory, or an S3\nbucket over HTTPS (`--s3 s3:\u002F\u002Fbucket\u002Fprefix`, or any S3-compatible\nservice through `AWS_ENDPOINT_URL`). Metadata stays local, but every\nobject's index lines ride beside it in the backend, so `--rebuild`\nremakes a lost metadata directory from the objects alone. When two stored objects score\nwithin 2× of each other as bases, both are tried on the first 32 MB. Measured on a realistic bucket\n(`scripts\u002Fdownload_bucket.sh`, 39 objects, 39.2 GB, each arriving in\norder), every object read back and compared:\n\n| Family | Raw | zstd -3, each object alone | ⚡ **Glyd store** | **Gain** |\n| :--- | ---: | ---: | ---: | ---: |\n| Linux 6.10 releases (15) | 22.5 GB | 3,237 MB | **232 MB** | **13.9× smaller** |\n| Ubuntu 24.04 cloud images (6 builds) | 6.6 GB | 1,880 MB | **361 MB** | **5.2×** |\n| Wikipedia dumps (2 months, 3 tables) | 0.7 GB | 140 MB | **71 MB** | **2.0×** |\n| GitHub events (12 hours) | 9.4 GB | 875 MB | 670 MB | 1.3× (no object is a version of another) |\n| **The bucket** | **39.2 GB** | **6,132 MB (6.4×)** | **1,334 MB (29.4×)** | **4.6× smaller** |\n\nAt a terabyte ([report](docs\u002Fbenchmarks\u002Fstore-gate-2026-09-24.md)):\n1,192 objects, 1.18 TB — 400 kernel point releases, every hour of\nGitHub events in January 2024, five English Wikipedia dumps' tables,\nsix Ubuntu images — put through the store into S3 from one 16-vCPU\ninstance next to the bucket, every object read back and compared\nbyte for byte, then the whole bucket restored by one process and\ncompared again (an earlier run also deleted the metadata directory,\nrebuilt it from the bucket and verified): **46.3 GB stored against\nzstd -3's 153.5 GB, 3.32× fewer bytes (25.6× against raw)**; kernels\n116–286× against raw, Wikipedia tables 19.5×, hourly events 14.1×\n(record mode alone). Put ran at 372 MB\u002Fs (243 in the 2026-09-22 run),\nread-back one object at a time at 461 MB\u002Fs (zstd -3's own read-back\non the same instance: 341 MB\u002Fs) and the restore at 603 MB\u002Fs, S3\nincluded, on that instance.\n\nPut runs at 620 MB\u002Fs end to end over the bucket on ten cores (reading\nthe file, rebuilding the base, writing the delta; a version of the\nlast object put runs at 900 MB\u002Fs, that object being kept in memory as\nthe likeliest next base); verifying the whole bucket reads it back at\n1.5 GB\u002Fs. On a Ryzen 9 7950X3D (16 cores, v0.14.7) a kernel release\narriving as a version of the last one is put at 1,224 MB\u002Fs and read\nback at 1,590; the first of them, alone, at 2,545 MB\u002Fs; an Ubuntu root\nfilesystem with gzip inside as a version at 215 MB\u002Fs (the deflate\nemulation's speed). The same store built on zstd's own\n`--patch-from` would land around 3–4×: our deltas are 1.1–2.1× smaller\nand read 10× faster, and the store design does the rest. In money, a\npetabyte of such data in S3 Standard costs $43K a year with zstd -3\nand $9.4K with the store. Chunk-level dedup, what backup systems do, gains\n1–4× on the same pairs. The store is a directory: `objects\u002F\u003Cid>`, the\nfingerprints, an index, and the fingerprint table — an open-addressing\nhash table mapped from disk (12 bytes per 4 KB stored, kept at most\nhalf full), so the store's memory does not grow with what it holds;\nthe 39 GB bucket's table is 100 MB. Objects under 256 KB have nothing\nto fingerprint and would cost their whole size alone, so `put` gathers\nthem into 2 MB packs (record mode where it pays) and `get` decodes the\npack and slices: 2,000 GitHub events put one by one store at 9.0×\nagainst zstd -3's 3.6× per event.\n\n## Base mode: a version compressed against the last one\n\nMost stored bytes are versions: nightly dumps, snapshots, images,\nsource trees, artifacts rebuilt with small changes. A new version\ncompressed alone costs what the first did; compressed against the old\none it costs the change. `glyd --base old new` parses every 32 MB of\nthe new version with the region of the old one that holds its content\nas history (the long-distance matcher reaches all of it) and writes a stream that\ndecodes with the same base: `glyd -d --base old new.glyd`. Measured on\nconsecutive versions of real objects against zstd 1.5.7's\n`--patch-from`, the same machine and thread count, every rebuild\nbyte-exact (`scripts\u002Fbench_versions.sh`, data from\n`scripts\u002Fdownload_versions.sh`):\n\n| Old → new | zstd -3 --patch-from | zstd -19 --patch-from | ⚡ **Glyd ‑‑max ‑‑base** | **Glyd ‑‑ultra ‑‑base** |\n| :--- | ---: | ---: | ---: | ---: |\n| Wikipedia `page` dumps a month apart (108 MB) | 3.84 MB · 409 MB\u002Fs | 1.30 MB · 2 MB\u002Fs | **1.79 MB · 720 MB\u002Fs** | **1.23 MB** · 6 MB\u002Fs |\n| Ubuntu 24.04 cloud root filesystem, builds 16 days apart (1.1 GB) | 8.82 MB · 654 MB\u002Fs | 5.61 MB · 39 MB\u002Fs | **5.33 MB · 1,590 MB\u002Fs** | **4.60 MB** · 11 MB\u002Fs |\n| Linux 6.10 → 6.10.1 source tar (1.5 GB) | 3.26 MB · 560 MB\u002Fs | 2.58 MB · 30 MB\u002Fs | **3.03 MB · 1,700 MB\u002Fs** | **2.04 MB** · 3 MB\u002Fs |\n\nCompressed alone with `--max` those versions are 33, 287 and 200 MB.\n`--max --base` stores 1.1–2.1× less than zstd's fast patch at 1.8–3×\nits speed, and on the image pair less than zstd's slow patch at 40×\nits speed; `--ultra --base` stores 5–21% less than zstd -19's patch on\nevery pair at the plain `--ultra` speed, which is 3–10× slower than\nzstd -19's patch on the large pairs. Reads run at 6–10 GB\u002Fs. Content\nis found wherever it moved: each unit's region of the base is chosen\nfrom a coarse map of the base (one anchor per KB), so a version with\n48 MB inserted before the kernel tree still costs 18.4 MB (zstd -3\n--patch-from 18.7; a fixed window around the unit's own position,\n33.5). Chunk-level dedup, the backup approach, gains 1–4× on the same\npairs ([experiments\u002Fstructure\u002FREADME.md](experiments\u002Fstructure\u002FREADME.md)).\n\nOver a chain of versions, the Linux 6.10 point releases (15 versions\nof a 1.5 GB tree; `scripts\u002Fdownload_chain.sh`, `scripts\u002Fbench_chain.sh`),\neach version against the one before it, or against 6.10 alone so that\nany version is two reads:\n\n| Stored | Glyd --max --base | zstd -3 --patch-from | Stored one by one |\n| :--- | ---: | ---: | ---: |\n| 6.10 plus 14 point releases, each against the previous | **228 MB** | 260 MB | Glyd --max 3,000 MB · zstd -3 3,236 MB |\n| the same, each against 6.10 | **246 MB** | 265 MB | |\n\nA step costs 1.8 MB (0.12% of the tree; 3.0–3.1 MB when the release\nnumber grows a digit, which touches every path in the archive) and\nthe delta against a base 14 versions old costs 3.6 MB, so rebasing\ninside a release series is not needed. A terabyte of such trees in\nS3 Standard ($276 per stored TB-year) costs $37 a year compressed one\nby one with `--max` ($40 with zstd -3) and $2.8–3.0 with base mode\n($3.2–3.3 with zstd's patch).\n\n## Record mode: logs and table dumps as columns\n\nByte-level matching is a plateau: on real data zstd -19, xz and Glyd\n`--ultra` land within a few percent of each other. The redundancy of a\nlog or a table dump is not in nearby bytes but in the same field of\nevery record. `glyd -r` (record mode, [design](docs\u002Fdesign\u002Fformat-v7.md#record-mode-v040-typed-columns-before-the-level))\ndetects delimited lines, SQL dumps and JSON lines, turns them into one\ntyped stream per field or key path (integer, decimal and date-time\ndeltas, dictionaries with recency ranks, text), compresses those with\nthe chosen level in parallel 32 MB units, and rebuilds the bytes\nexactly. Anything else is left as it is.\n\nThe 8.7 GB benchmark corpus, 10 cores, every decode byte-checked\n(`examples\u002Fbench_suite.rs`, rows in [`benchmarks\u002Fsuite\u002Fm1-max-v0.4.0\u002F`](benchmarks\u002Fsuite\u002Fm1-max-v0.4.0\u002F)):\n\n| Data | Glyd ‑‑max | Glyd ‑‑ultra | ⚡ **Glyd ‑‑max ‑r** | ⚡ **Glyd ‑‑ultra ‑r** | zstd -3 | zstd -19 | **‑‑ultra ‑r vs zstd -19** |\n| :--- | ---: | ---: | ---: | ---: | ---: | ---: | ---: |\n| SQL dumps (3.9 GB) | 4.96 | 6.98 | **8.38** | **10.14** | 4.97 | 7.10 | **1.43× smaller** |\n| Access logs (0.55 GB) | 10.56 | 14.40 | **21.4** | **23.0** | 9.53 | 14.9 | **1.54× smaller** |\n| Pageview logs (0.71 GB) | 3.71 | 4.68 | **4.10** | **4.92** | 3.56 | 4.82 | 1.02× |\n| JSON events (2.6 GB) | **13.26** | **16.40** | 13.26 | 16.40 | 10.46 | 15.07 | **1.09× smaller** |\n| Parquet (1.0 GB) | 1.01 | 1.02 | 1.01 | 1.02 | 1.01 | 1.02 | 1.00× |\n| Whole corpus | 3.94 | 4.66 | **4.71** | **5.22** | 3.85 | 4.66 | **1.12× smaller** |\n\n`--max -r` is 1% smaller than zstd -19 over the corpus while\ncompressing at 640 MB\u002Fs against 13 (8 Graviton3 cores); `--ultra -r` is\n12% smaller than zstd -19, and plain `--ultra` equals it. JSON events\nare not record-shaped (their redundancy is inside each record and\nacross the whole file), so `-r` hands them to the plain level, where\nthe long-distance matcher (repeats up to 128 MB back, on at `--max\n--long` and `--ultra`) does the work: 13.26 and 16.40 against 11.49 and 14.59\nwith the 8 MB window, 27% and 9% smaller than zstd -3 and zstd -19.\nCosts: `--max` compresses the corpus at 2,000 MB\u002Fs (2,400 without the\nmatcher; zstd -3 4,000), record-mode reads run at 4,400 MB\u002Fs instead\nof 6,500-8,700 for plain `--max`.\n\nTelemetry is where the multiples are. Measurements as rows (a cluster\ntrace, daily weather, taxi trips) in CSV or as JSON lines\n(`scripts\u002Fdownload_ext_corpus.sh`, 128 MB slices, 10 cores, rows in\n[`benchmarks\u002Fsuite\u002Fm1-max-v0.4.0\u002Fbench_suite_ext2.txt`](benchmarks\u002Fsuite\u002Fm1-max-v0.4.0\u002Fbench_suite_ext2.txt)):\n\n| Data | ⚡ **Glyd ‑‑max ‑r** | ⚡ **Glyd ‑‑ultra ‑r** | zstd -3 | zstd -19 | **‑‑max ‑r vs zstd -3** | **‑‑ultra ‑r vs zstd -19** |\n| :--- | ---: | ---: | ---: | ---: | ---: | ---: |\n| Alibaba cluster machine usage, CSV | **12.6** | **13.7** | 4.46 | 6.90 | **2.8× smaller** | **2.0× smaller** |\n| the same as JSON lines | **54.6** | **58.9** | 15.7 | 28.7 | **3.5× smaller** | **2.1× smaller** |\n| NOAA daily weather, CSV | **19.6** | **23.1** | 6.99 | 12.0 | **2.8× smaller** | **1.9× smaller** |\n| the same as JSON lines | **47.0** | **58.0** | 18.7 | 31.6 | **2.5× smaller** | **1.8× smaller** |\n| NYC taxi trips, CSV export | **8.93** | **9.37** | 5.64 | 8.38 | **1.6× smaller** | 1.1× smaller |\n| Common Crawl index (a hash per line) | 7.73 | 9.36 | 7.55 | 9.66 | 1.02× | 0.97× |\n\n`--max -r` compresses these at 340-680 MB\u002Fs (10 cores) and reads back\nat 770-2,500 MB\u002Fs. The last row is the honest limit: a line that is\nmostly a hash has nothing a column can model, and API events with\nhashes and free text (GitHub Archive) gain 1.6% from columns and stay\nplain.\n\nLogs whose lines vary in shape (application and system logs) take the\ntemplate shape: each line's template — its text with a hole where every\ntoken holding a digit was — goes into a dictionary, and the tokens\nbecome typed columns keyed by template and slot (loghub 2.0 logs, 128 MB\nof each, 10 cores, every decode byte-checked):\n\n| Log | zstd -3 | zstd -19 | ⚡ **Glyd ‑‑max ‑r** | ⚡ **Glyd ‑‑ultra ‑r** | **‑‑max ‑r vs zstd -3** | **‑‑ultra ‑r vs zstd -19** |\n| :--- | ---: | ---: | ---: | ---: | ---: | ---: |\n| HDFS (Hadoop file system) | 10.5 | 16.0 | **22.0** | **27.5** | **2.1× smaller** | **1.72× smaller** |\n| Spark (application logs) | 14.5 | 25.2 | **47.0** | **53.6** | **3.3× smaller** | **2.12× smaller** |\n| BGL (supercomputer RAS log) | 11.0 | 22.4 | **15.9** | **28.9** | **1.45× smaller** | **1.29× smaller** |\n| Android (system log) | 12.9 | 23.0 | **17.9** | **25.4** | **1.39× smaller** | **1.10× smaller** |\n\n`--max -r` writes these at 260-460 MB\u002Fs and reads them back at\n1,200-1,400 MB\u002Fs.\n\n### Objects opened: gzip, zip, tar, Office documents, jars, PDF, PNG — and JPEG transcoded\n\nMuch of what sits in a bucket is deflate inside a container — gzipped\nlogs (ELB, CloudFront, CloudTrail and flow logs are delivered that\nway), zip and tar archives, .docx\u002F.xlsx\u002F.pptx, .jar, PDF, PNG — and\nto zstd all of it is noise. Containers inside containers open too,\nfour deep: a tar of gzipped logs, a tar.gz of pictures, a deck's\nJPEGs under their deflate entries. Glyd opens the container: every\ndeflate stream inside is decoded to its content along with what it\ntakes to re-encode it bit for bit (Glyd's own reconstruction since\nv0.13.4, `src\u002Freflate\u002F`: zlib's matcher run over the content, only\nwhat differs kept); the plain text is that\ncontent, then every other byte of the object as it was (headers,\ndirectories, stored entries, a tar's files) and the corrections, and\nit takes the level asked for — record mode where it pays, the cold\nlevel, a base, which sees all of it, so versions of an archive share\nwhat they have in common. Containers open from `--max` up; the\ndefault, fast and turbo levels leave them as they are, since a read of\nan opened container re-creates its deflate. `-d` gives back the\nidentical object; `-d --content` gives what `gunzip` would — a gzip's\nmembers' text, a tar.gz's tar — without re-creating the stream, which\nis the expensive part of the read (the 20.7 MB NASA gzip on one\nthread: 0.36 s for the content against 1.58 s for the gzip back; on\nten cores 0.11 s; `gunzip` itself 0.10 s). The store has it as\n`--get ID --content`. The\nobject is also compressed closed, at the same level, and the smaller\nof the two is kept. Measured on this Mac, every decode compared with\nthe input:\n\n| Object | As is | zstd -19 on it | ⚡ **Glyd ‑‑max** | ⚡ **‑‑ultra** | ⚡ **‑‑cold** |\n| :--- | ---: | ---: | ---: | ---: | ---: |\n| NASA access log, gzip -6 (205 MB inside) | 20.7 MB | 20.7 MB | **8.19 MB (−60%)** | 7.67 MB | **6.55 MB (−68%)** |\n| Linux tree, 512 MB, gzip -6 | 72.7 MB | 72.0 MB | **56.1 MB (−23%)** | 42.0 MB | **28.6 MB (−61%)** |\n| zstd source, GitHub zip | 2.73 MB | 2.57 MB | **2.14 MB (−22%)** | 1.66 MB | **1.34 MB (−51%)** |\n| Guava jar, 2,059 entries | 3.05 MB | 2.70 MB | **1.77 MB (−42%)** | 1.42 MB | **1.11 MB (−64%)** |\n| .pptx, 60 slides | 88 KB | 56 KB | **24.7 KB (−72%)** | 21.5 KB | **15.2 KB (−83%)** |\n| .pptx, 12 slides of photos (6.5 MB) | 6.51 MB | 6.50 MB | **5.26 MB (−19%)** | 5.16 MB | **5.07 MB (−22%)** |\n| .docx, 6 PNG screenshots (2.5 MB) | 2.51 MB | 2.50 MB | **2.32 MB (−8%)** | 2.08 MB | **1.65 MB (−34%)** |\n| tar of the 6 photos, and the same tar gzipped | 36.0 MB | 35.5 MB | **27.3 MB (−24%)**, −23% through the gzip | | |\n| tar.gz of a gzipped log, a PDF, a PNG, a .docx (23 MB) | 22.9 MB | 22.9 MB | **10.2 MB (−56%)** | 9.47 MB | **7.94 MB (−65%)** |\n| .xlsx, 30,000 rows | 1.34 MB | 1.23 MB | 1.33 MB (kept closed) | 1.05 MB | **0.47 MB (−65%)** |\n| .docx, 400 sections | 141 KB | 138 KB | 140 KB | 117 KB | **76.7 KB (−46%)** |\n| RFC 8878, PDF | 440 KB | 242 KB | **191 KB (−57%)** | 158 KB | **106 KB (−76%)** |\n| arXiv paper, PDF (pdfTeX) | 2.22 MB | 1.04 MB | **741 KB (−67%)** | 616 KB | **504 KB (−77%)** |\n| arXiv paper with figures, PDF | 6.77 MB | 5.54 MB | **4.24 MB (−37%)** | 3.55 MB | **2.82 MB (−58%)** |\n| PNG photo | 1.83 MB | 1.77 MB | **1.65 MB (−10%)** | 1.52 MB | **1.19 MB (−35%)** |\n| PNG illustration | 669 KB | 663 KB | **634 KB (−5%)** | 540 KB | **448 KB (−33%)** |\n| 6 JPEG photos, 35.9 MB | 35.9 MB | 35.9 MB | **27.3 MB (−24%)** at every level from `--max` up | | |\n\nWhere the container's own deflate was already near what the fast\nlevel does on the content (an Office XML sheet), the fast level keeps\nit closed and the slower levels open it. The cost is the re-encode\nthat makes it exact: about 5 MB\u002Fs of deflate per core in (50 MB\u002Fs of\ncontent), three times that out — and a stream of 16 MB of content or\nmore is cut into chunks that open and close on every core, so on ten\ncores the NASA gzip writes in 2.3 s and reads in 0.26 s (one core: 4.6\nand 1.56), the PDF with figures in 1.9 and 0.59 s (13.7 and 4.3); per\nterabyte of gzipped logs on S3 Standard, about $2 of CPU once against\n$166 a year. GNU gzip's\nstreams open the same as macOS's (checked on Linux: an hour of events\ngzipped, 75.3 → 33.2 MB; a Linux tree, 72.2 → 55.1 MB; byte-exact).\nStreams preflate cannot reproduce, or predicts badly (corrections\nover a quarter of the stream), are kept as they are: 18 of the jar's\n2,059. JPEG takes\na different road: its DCT coefficients are taken out and coded by\nGlyd's own model (v0.14.0, `src\u002Fjpg\u002F`: each coefficient under the\nblocks above and to the left, the first row and column predicted\nfrom pixel continuity across the block edge, the DC from both\nedges), 22–26% fewer bytes on five photos — smaller than Lepton on\nevery one — at 13–22 MB\u002Fs in and 25–46 out on this Mac's cores, the\nidentical JPEG back — inside a zip or an Office document too, stored\nor deflated. Parquet (its columns as records,\n26–40%, the same table rather than the same bytes) is measured in\n[experiments\u002Fresearch](experiments\u002Fresearch\u002FREADME.md#j-re-doing-what-is-already-compressed)\nand not yet built.\n\n### The cold level: context mixing for what is read rarely\n\nEvery LZ codec sits on the same floor: on the data above zstd -19, xz -9\nand Glyd `--ultra` land within 5% of each other. Below that floor is\ncontext mixing — each bit predicted from many contexts at once and coded\nat the mixed probability, with no parse — at 30–100× the CPU. `glyd\n--cold` is that level: eleven predictors (byte orders 1–4, 6 and 8, the\nword and the word before it, the column and the byte above it, the JSON\nkey, the longest earlier match), paq-style bit histories, two mixers,\ntwo SSE stages, 32 MB units coded in parallel. Measured on 64 MB slices\nagainst the strongest tools, one thread each for the references, every\ndecode byte-checked (`experiments\u002Fresearch\u002Fcoldtier.sh`):\n\n| Data | zstd -19 | xz -9 | Glyd ‑‑ultra (‑r) | zpaq -m5 | ⚡ **Glyd ‑‑cold (‑r)** | **vs zstd -19** |\n| :--- | ---: | ---: | ---: | ---: | ---: | ---: |\n| GitHub Archive JSON events | 14.6× | 14.8× | 15.9× | 22.8× · 0.4 MB\u002Fs | **22.5×** · 1.3 MB\u002Fs\u002Fcore | **1.54× smaller** |\n| NASA access log | 15.7× | 15.4× | 26.4× | 31.7× · 0.3 MB\u002Fs | **31.1×** · 1.2 MB\u002Fs\u002Fcore | **1.98× smaller** |\n| enwiki page_props SQL dump | 6.2× | 6.4× | 8.6× | 11.1× · 0.4 MB\u002Fs | **11.6×** · 1.2 MB\u002Fs\u002Fcore | **1.87× smaller** |\n| webster (text, 41 MB) | 4.8× | 4.9× | 4.8× | 7.3× · 0.35 MB\u002Fs | **7.1×** · 1.2 MB\u002Fs\u002Fcore | **1.47× smaller** |\n| HDFS log, 128 MB (`-r`) | 16.0× | | 27.5× | | **33.7×** | **2.1× smaller** |\n| Spark log, 128 MB (`-r`) | 25.2× | | 53.6× | | **65.2×** | **2.6× smaller** |\n\n`--cold` matches zpaq's strongest level within 3% either way at 3–4×\nits speed per core, and reads back at the same speed it writes: a\nterabyte costs about 210 core-hours each way, $8 on Graviton3. Against\nthat, 1.5–2× fewer bytes than zstd -19 saves $3–5 a year per raw\nterabyte in S3 Standard-IA and under $0.50 in Glacier Deep Archive, so\nthe level pays for data kept two years or more in a warm-ish tier and\nread a few times at most, and for bytes that are moved (egress,\nreplication) more than they are read. The encoder holds 400 MB per\nthread.\n\n### Small objects with a shape dictionary\n\nA single event, a small log or CSV has nothing to learn a schema from,\nso small objects got the plain path and its dictionaries. A **shape\ndictionary** (`glyd --shape-train sample -o d.shape`, then `glyd\n--shape d.shape object`) is trained once on a few MB of the data and\ncarries the shape, the frames lines take, the columns' types and the\nvalues dictionary columns usually hold; an object is coded as a\ncompact image (a byte per row, values as ranks and deltas, new values\nas text) through a prepared LZ dictionary trained on such images.\nObjects cut from real files, the dictionaries trained on the first\n4 MB of each, objects from the middle, every decode byte-checked\n(`examples\u002Fshape_gain.rs`):\n\n| Objects | zstd -3 + dict | Glyd --max + Dict | ⚡ **Glyd shape dictionary** | **vs zstd + dict** |\n| :--- | ---: | ---: | ---: | ---: |\n| Alibaba machine usage, JSON lines, 4 KB | 13.0× | 13.1× | **24.3×** | **1.87× smaller** |\n| the same, 1 KB | 10.6× | 10.3× | **14.6×** | **1.37×** |\n| Alibaba machine usage, CSV, 4 KB | 4.4× | 4.2× | **7.9×** | **1.81×** |\n| the same, 1 KB | 3.8× | 3.7× | **5.8×** | **1.53×** |\n| NYC taxi CSV, 1 KB | 3.9× | 3.9× | **5.3×** | **1.35×** |\n| HDFS log, 4 KB | 7.4× | 7.1× | **9.7×** | **1.31×** |\n| the same, 1 KB | 6.0× | 5.6× | **6.7×** | **1.12×** |\n| NASA access log, 4 KB | 6.4× | 6.9× | **7.3×** | **1.14×** |\n| the same, 1 KB | 5.3× | 5.4× | 5.1× | 0.97× |\n\n1.1–1.9× fewer bytes than a zstd dictionary on telemetry and structured\nlogs, nothing on an access log of 1 KB: its bytes are host names and\npaths the object is the first to mention, and a per-object scheme pays\nfor new information whatever it does. Dictionaries are 80–220 KB; an\nobject codes at 60–150 MB\u002Fs and decodes at 55–430 MB\u002Fs on one core.\n\nThe larger lever for small objects is not per-object at all. A\n**pack** (`compress_pack`, `decompress_pack_object`; `glyd --pack\nfiles... -o p.glyd`, `glyd --unpack dir p.glyd`) compresses many small\nobjects as one record-mode stream with an index of their lengths, so\nthey cost what they cost as a file; reading one object is the pack\ndecoded and sliced. The same objects in 1 MB packs at `--max`:\n\n| Objects | zstd -3 + dict, each alone | ⚡ **Glyd pack** | **vs zstd + dict** | One object read |\n| :--- | ---: | ---: | ---: | ---: |\n| JSON events, 1 KB (1,024 per pack) | 10.6× | **42.4×** | **4.0× smaller** | 0.6 ms |\n| JSON events, 4 KB | 13.0× | **39.0×** | **3.0×** | 0.9 ms |\n| NASA access log, 1 KB | 5.3× | **15.2×** | **2.9×** | 0.7 ms |\n| HDFS log, 1 KB | 6.0× | **16.2×** | **2.7×** | 0.5 ms |\n| CSV telemetry, 1 KB | 3.8× | **8.9×** | **2.3×** | 0.9 ms |\n| NYC taxi CSV, 1 KB | 3.9× | **7.6×** | **1.9×** | 1.0 ms |\n\nPacking runs at 40–130 MB\u002Fs on one core. A store that groups small\nobjects — a log shipper, an event stream, an S3 batcher — gets 2–4×\nover per-object dictionaries; one that must compress each object alone\ngets the shape-dictionary table above.\n\n### Small objects\n\nObjects cut from GitHub Archive events, 2,000 per size, each codec with\nits own 110 KB dictionary trained on 2,000 other objects (zstd's\ntrainer for zstd, `Dict::train` for Glyd; zstd's output carries no\nchecksum, Glyd's 4 bytes per object). Apple M1 Max, one core,\n`examples\u002Fsmall_objects.rs` and `examples\u002Fsmall_speed.rs`:\n\n| Object | zstd -3 + dict | ⚡ **Glyd ‑‑max + Dict** | zstd -19 + dict | ⚡ **Glyd ‑‑ultra + Dict** |\n| ---: | ---: | ---: | ---: | ---: |\n| 1 KB | 4.96 | **4.75** | 5.47 | **5.25** |\n| 4 KB | 6.42 | **6.28** | 7.41 | **7.13** |\n| 16 KB | 7.66 | **7.65** | 8.95 | **8.84** |\n\n| Object | Codec | Compress | Decompress |\n| ---: | :--- | ---: | ---: |\n| 1 KB | ⚡ **Glyd ‑‑max + Dict** | 245 MB\u002Fs | **920 MB\u002Fs** |\n| 1 KB | zstd -3 + dict | 454 MB\u002Fs | 1,117 MB\u002Fs |\n| 4 KB | ⚡ **Glyd ‑‑max + Dict** | 321 MB\u002Fs | **1,209 MB\u002Fs** |\n| 4 KB | zstd -3 + dict | 588 MB\u002Fs | 1,440 MB\u002Fs |\n| 16 KB | ⚡ **Glyd ‑‑max + Dict** | 394 MB\u002Fs | **1,674 MB\u002Fs** |\n| 16 KB | zstd -3 + dict | 635 MB\u002Fs | 1,890 MB\u002Fs |\n\nOn small objects zstd is ahead: 1-4% denser at `-3` and 1-4% at `-19`,\n1.6-1.9× faster to compress and 1.1-1.2× faster to decode. Without a\ndictionary Glyd `--max` is 6-12% less dense than zstd -3 on objects\nunder 16 KB.\n\nOn the AWS machines (dictionaries trained on another day's data, 110 KB):\nsizes within 1–3% of zstd's either way, zstd 1.7–2× faster to compress and\n1.4–1.7× faster to decompress per object. Without a dictionary a 1 KB log\nrecord compresses 2.5×; with one, 5.2× (zstd: 2.9× and 5.3×).\n\n### The classic corpus: Silesia, one run, one core (Apple M1 Max; v0.3.0 run)\n\n| Codec | Ratio | Compress MB\u002Fs | **Decompress MB\u002Fs** | |\n| :--- | ---: | ---: | ---: | :--- |\n| ⚡ **Glyd ‑‑max** | **3.254** | 277 | **1,733** | ✅ best ratio; 1.27× zstd -3 decode |\n| zstd -3 | 3.205 | 319 | 1,361 | |\n| zstd -1 | 2.894 | 535 | 1,493 | |\n| LZAV-hi | 2.803 | 91 | 3,185 | |\n| LZAV | 2.450 | 426 | 3,128 | |\n| zstd ‑‑fast=1 | 2.438 | 614 | 2,153 | |\n| zstd ‑‑fast=3 | 2.240 | 684 | 2,307 | |\n| ⚡ **Glyd default** | **2.192** | 312 | **6,507** | ✅ 1.6× liblz4 decode, better ratio |\n| ⚡ **Glyd ‑‑fast** | **2.176** | 501 | **4,670** | ✅ 1.1× liblz4 decode, better ratio |\n| liblz4 | 2.101 | 610 | 4,108 | |\n| lz4_flex | 2.097 | 633 | 3,004 | |\n| snappy | 2.076 | 607 | 1,495 | |\n| zstd ‑‑fast=5 | 2.057 | 746 | 2,484 | |\n| ⚡ **Glyd ‑‑turbo** | 1.884 | 263 | **8,647** | ✅ fastest decode, 2.1× liblz4 |\n\n(`examples\u002Ffield_survey.rs`, taken with other work on the machine, before\nthe 128 MB matcher; the ordering has not changed. This release on the\nsame machine and corpus: `--max` 3.302 at 244 MB\u002Fs and 1,856 MB\u002Fs\ndecode, `--ultra` 3.959 at 3.2 MB\u002Fs and 1,910 MB\u002Fs decode, against\nzstd -3 3.205 \u002F 335 \u002F 1,453 and zstd -19 4.006 \u002F 3.7 \u002F 1,589 in the\nsame runs, `examples\u002Fv7_bench.rs` and `examples\u002Fultra_bench.rs`.)\n\n\u003Cdetails>\n\u003Csummary>\u003Cb>Default level vs liblz4, per file\u003C\u002Fb> (same run)\u003C\u002Fsummary>\n\n| File | Ratio | ⚡ **Glyd MB\u002Fs** | liblz4 MB\u002Fs | Glyd advantage |\n| :--- | ---: | ---: | ---: | ---: |\n| dickens | 1.815 | 5,990 | 5,190 | +15% |\n| mozilla | 1.926 | 5,420 | 4,830 | +12% |\n| mr | 1.902 | 6,650 | 5,570 | +19% |\n| nci | 6.846 | 7,130 | 7,240 | −2% |\n| ooffice | 1.335 | 6,310 | 4,680 | +35% |\n| osdb | 2.294 | 6,270 | 5,110 | +23% |\n| reymont | 2.378 | 5,040 | 4,490 | +12% |\n| samba | 2.858 | 6,380 | 6,140 | +4% |\n| sao | 1.038 | 13,710 | 7,320 | +87% |\n| webster | 2.250 | 4,700 | 4,850 | −3% |\n| xml | 4.949 | 6,420 | 5,560 | +16% |\n| x-ray | 1.000 | 46,600 | 18,100 | +157% |\n\n(AMD Ryzen 9 7950X3D, AVX2 path. On the M1 Max NEON path Glyd wins 12 of 12.)\n\n\u003C\u002Fdetails>\n\n\u003Cdetails>\n\u003Csummary>\u003Cb>\u003Ccode>--max\u003C\u002Fcode> vs zstd -3, per file\u003C\u002Fb> (same run, M1 Max)\u003C\u002Fsummary>\n\n| File | ⚡ **Glyd ratio** | zstd -3 ratio | ⚡ **Glyd MB\u002Fs** | zstd -3 MB\u002Fs |\n| :--- | ---: | ---: | ---: | ---: |\n| dickens | 2.848 | 2.782 | 1,367 | 1,220 |\n| mozilla | 2.801 | 2.810 | 1,734 | 1,316 |\n| mr | 2.827 | 2.811 | 1,519 | 1,320 |\n| nci | 11.735 | 11.840 | 3,696 | 2,746 |\n| ooffice | 1.999 | 1.968 | 1,403 | 1,035 |\n| osdb | 2.903 | 2.880 | 2,243 | 1,731 |\n| reymont | 3.506 | 3.420 | 1,764 | 1,445 |\n| samba | 4.471 | 4.360 | 2,480 | 2,039 |\n| sao | 1.326 | 1.312 | 1,901 | 889 |\n| webster | 3.538 | 3.427 | 1,640 | 1,456 |\n| xml | 8.386 | 8.414 | 3,194 | 2,526 |\n| x-ray | 1.465 | 1.393 | 1,157 | 866 |\n| **Total** | **3.254** | **3.204** | **1,891** | **1,487** |\n\n(Same run; `--max` wins ratio on 8 of 12 files and decode on 12 of 12.)\n\n\u003C\u002Fdetails>\n\n### Reproduce\n\n```bash\nscripts\u002Fdownload_bench_corpus.sh                    # the 8.7 GB real-data corpus (+ training data)\ncargo run --release --example bench_suite -- --large --threads 8 --repeats 3   # every codec, every file, decodes checked\ncargo run --release --example bench_suite -- --small --threads 1               # small objects with dictionaries\nscripts\u002Fdownload_ext_corpus.sh && cargo run --release --example bench_suite -- --large --dir corpus\u002Fext2   # telemetry\nscripts\u002Fdownload_versions.sh && scripts\u002Fbench_versions.sh corpus\u002Fversions\u002Flinux-6.10.tar corpus\u002Fversions\u002Flinux-6.10.1.tar   # base mode vs zstd --patch-from\nscripts\u002Fdownload_chain.sh && scripts\u002Fbench_chain.sh   # 15 Linux point releases as a chain of versions (20 GB)\nexperiments\u002Fresearch\u002Fcoldtier.sh                    # the cold level against xz, brotli, zpaq on 64 MB slices\nscripts\u002Fverify_roundtrip.sh yourfile                # every level and mode through the CLI, corrupted copies\nscripts\u002Fdownload_corpus.sh && cargo run --release --example v7_bench            # Silesia, --max vs zstd -3\nAWS_PROFILE=... scripts\u002Fbench_aws_suite.sh \u003Cbucket> main   # the whole program on Graviton3 + Sapphire Rapids, ~$1.50\n```\n\n---\n\n## How it works\n\nThe compressed block is split into homogeneous streams — tokens, offsets,\nlengths, literals — instead of one interleaved byte stream, so the decoder\npre-decodes 32 tokens per SIMD pass, checks bounds once per chunk and runs\na copy-only loop. `--max` and `--ultra` keep the layout and add 8-way\ninterleaved entropy coding (Huffman literals, tANS sequences with repeat\noffsets), so the entropy decoders run as straight-line SIMD-friendly loops\nand the copies stay a separate pass. A long-distance matcher indexes every\n16th position of a unit with a content-defined anchor and hands the parse\nrepeats up to 128 MB back. Record mode reorders record-shaped text into\none typed stream per field before the level; base mode lays a region of\nthe old version before each unit as history and lets the decoder read it\nin place.\n\nDesign: [docs\u002Fdesign\u002Fformat-v7.md](docs\u002Fdesign\u002Fformat-v7.md) (formats v7–v9,\ndictionaries, record mode, the matcher, base mode) and\n[docs\u002Fdesign\u002Fultra-parse.md](docs\u002Fdesign\u002Fultra-parse.md). Every\nmeasurement, refuted idea and floor: [docs\u002Fengineering-notes.md](docs\u002Fengineering-notes.md),\n[CHANGELOG-BENCH.md](CHANGELOG-BENCH.md) and\n[experiments\u002Fstructure\u002FREADME.md](experiments\u002Fstructure\u002FREADME.md). What\ncomes next: [ROADMAP.md](ROADMAP.md).\n\nSafety: the decoder is fuzzed with a million random mutations per run into\nexact-size buffers with sentinel guards, on every level; corrupted record\nand base envelopes are rejected or decode to a checked length, never a\npanic or an unbounded allocation; every unsafe block carries its bound.\n\n---\n\n## Known gaps\n\n- `--max` on one core of Graviton3 writes at 1.01–1.05× the wall time\n  of `zstd -3` as installed on GitHub events and enwik8, 0.97× on\n  mozilla and 0.88–0.90× on the NASA log and a Wikipedia table dump,\n  0.5–7.8% smaller on each (v0.14.6). zstd's default is two threads (one compressing,\n  one on I\u002FO and the checksum), so the CLI writes on a second thread\n  too; against `zstd -3 --single-thread` it is 1.03–1.26× on all five.\n  What is left on match-dense data is the sequence side: the codes and\n  eight-stream sections written per sequence run more instructions\n  than zstd's single sequence stream, at a higher IPC. On eight cores\n  against `zstd -3 -T8`: 1.08–1.21× on events, the log and mozilla,\n  0.97× on enwik8, 0.83× on the dump. On a Ryzen 9 7950X3D it is\n  1.07–1.26× faster on all five. Against zstd 1.5.7, whose block\n  splitter gains 1.4% on mozilla against ours 0.9%, that file is 0.2%\n  larger. `--long` adds the 128 MB matcher at a third more time. Record mode's transform\n  halves the write speed again (200–400 MB\u002Fs per core).\n- Reads in record mode spend 2–2.7× zstd's CPU rebuilding the columns\n  (5–30 ns per value by column type), which makes zstd -3 the cheaper\n  choice at a hundred CPU-billed reads a month; the plain CLI's\n  one-core reads run at 1.15–1.32× `zstd -d` on Graviton3, 1.00–1.30×\n  on a Ryzen 9 7950X3D and 0.79–0.97× on Sapphire Rapids (v0.14.7).\n- `--ultra --base` runs at the plain `--ultra` speed (3–11 MB\u002Fs on ten\n  M1 cores), 3–10× slower than zstd -19's patch on the large pairs. The\n  encoder holds the old and new versions, a map of 1.6% of the old one,\n  and 128 MB per thread. A unit whose content is spread over two places\n  of the base farther apart than 96 MB has only the denser one in reach.\n- On Sapphire Rapids `--max` decodes at 0.79–0.97× zstd -3 on one core\n  (v0.14.7), against 1.15–1.32× on Graviton3 and 1.00–1.30× on a Ryzen 9\n  7950X3D: what is left there is the decode loop's instructions per\n  cycle on Intel.\n- Small objects with a dictionary: sizes tie, zstd is 1.4–2× faster per\n  object. Record mode works on files, not on single small objects.\n- JSON API events and crawl indexes are 20–65% hashes and random ids once\n  compressed; no column model moves them. Parquet is zstd inside already.\n- `--cold` is symmetric: reads cost what writes cost, 1–1.3 MB\u002Fs per\n  core, so it is for data read a few times in its life, not a tier that\n  serves reads. It is 1% behind zpaq -m5 on text, 6–11% on an\n  executable, a database file and an OS image, and ahead on logs,\n  dumps and JSON.\n- `--ultra` is 0.1–3.5% larger than the better of zstd -19 and -22 on\n  plain text, executables, source and OS trees, images and Parquet\n  (equal or smaller on records and containers), and up to 15% larger\n  than xz -9e and brotli -11 there, which read at 30–125 MB\u002Fs against its\n  500–1,600. zstd 1.5.7's `--max` level is denser still, at 72 minutes\n  per gigabyte.\n- Opened containers read at 1.5–20 MB\u002Fs on one thread (6–7× that on\n  ten cores for streams of 16 MB of content or more): a read re-creates\n  every deflate stream bit for bit. That is why the default, fast and\n  turbo levels leave containers closed.\n- `GlydReader`\u002F`GlydWriter` (std::io streaming) carry v6 levels only.\n\n---\n\n## Releases and versioning\n\nCurrent release: **v0.14.3** ([CHANGELOG.md](CHANGELOG.md), [releases](https:\u002F\u002Fgithub.com\u002Fsurya-koritala\u002FGlyd\u002Freleases)).\nGlyd follows SemVer. The on-disk format is versioned separately in every\nblock header (v6 for default\u002Ffast\u002Fturbo, v9 for `--max` and `--ultra`; v7\nand v8 are read); record and base envelopes carry their own magic. Every\nrelease decodes every earlier format (`tests\u002Fformat_compat.rs` holds the\noutput of each), and a format change always gets a new format number,\nnever a silent reinterpretation. Tags are `vMAJOR.MINOR.PATCH`; each tag\nships with release notes and the benchmark tables measured at that commit.\n\nContributing: open an issue with the measured number for anything that\ntouches speed or ratio (`examples\u002Fbench_suite.rs`, `examples\u002Fv7_bench.rs`\nand `scripts\u002Fbench_versions.sh` print same-run comparisons); pull requests\nrun the full suite including the 1M-mutation fuzz in CI. Every commit\nneeds a `Signed-off-by:` line (`git commit -s`), which accepts the\n[contributor license agreement](CLA.md): you keep your copyright and\ngrant the project the right to use and relicense the contribution. The\nwhole of it is in [CONTRIBUTING.md](CONTRIBUTING.md); the\n[code of conduct](CODE_OF_CONDUCT.md) applies to everyone taking part.\n\n---\n\n## License\n\n- **The codec — the `glyd` crate, the `glyd` CLI, the C ABI, the Python\n  and Go bindings — is under the [BSD 3-Clause License](LICENSE) or, at\n  your option, the [GNU GPL version 2](COPYING)**: the same licenses as\n  zstd, so anything that may ship zstd may ship Glyd, GPLv2 projects such\n  as the Linux kernel included. Use it, embed it, ship it, sell it; keep\n  the notice. That is everything in this repository except the store.\n- **The store — the `glyd-store` crate and CLI — is under the\n  [Business Source License 1.1](glyd-store\u002FLICENSE)**: source available,\n  free for personal, educational, research and other non-commercial use;\n  any commercial production use needs a license\n  (suryakoritala1324@gmail.com); each version converts to Apache-2.0\n  four years after its release.\n\nWhy the split: a codec is adopted by being embedded, and nothing is\nembedded under a source-available license; the store is the product.\n",2,"2026-09-24 02:30:14","CREATED_QUERY"]