[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"project-80257":3},{"id":4,"name":5,"fullName":6,"owner":7,"repo":5,"description":8,"homepage":9,"htmlUrl":10,"language":11,"languages":10,"totalLinesOfCode":10,"stars":12,"forks":13,"watchers":14,"openIssues":14,"contributorsCount":15,"subscribersCount":15,"size":15,"stars1d":16,"stars7d":17,"stars30d":18,"stars90d":15,"forks30d":15,"starsTrendScore":19,"compositeScore":20,"rankGlobal":10,"rankLanguage":10,"license":21,"archived":22,"fork":22,"defaultBranch":23,"hasWiki":24,"hasPages":22,"topics":25,"createdAt":10,"pushedAt":10,"updatedAt":31,"readmeContent":32,"aiSummary":33,"trendingCount":15,"starSnapshotCount":15,"syncStatus":14,"lastSyncTime":34,"discoverSource":35},80257,"opentoken","MrGray17\u002Fopentoken","MrGray17","Token-saving companion for OpenCode — 42 compression layers, zero risk, no caveman speak","https:\u002F\u002Fwww.npmjs.com\u002Fpackage\u002F@mrgray17\u002Fopentoken",null,"TypeScript",142,7,2,0,1,10,39,8,53.61,"MIT License",false,"main",true,[26,27,28,29,30],"compression","llm","opencode","opencode-plugin","token-savings","2026-06-12 04:01:27","# OpenToken\n\n**Token-compression engine for AI coding agents.**\nIntercepts tool output and strips noise before it reaches the LLM.\nWorks with OpenCode, Cursor, Windsurf, Claude Desktop, VS Code Copilot, and any MCP-compatible IDE.\n\n```\n$ git diff HEAD~1                    2,114 tokens of raw diff noise\n$ opentoken wrap \"git diff HEAD~1\"     407 tokens -- 81% reduction\n```\n\n\u003Cp align=\"center\">\n  \u003Ca href=\"https:\u002F\u002Fwww.npmjs.com\u002Fpackage\u002F@mrgray17\u002Fopentoken\">\u003Cimg src=\"https:\u002F\u002Fimg.shields.io\u002Fnpm\u002Fv\u002F@mrgray17\u002Fopentoken?color=0366d6&label=opencode\" \u002F>\u003C\u002Fa>\n  \u003Ca href=\"https:\u002F\u002Fwww.npmjs.com\u002Fpackage\u002F@mrgray17\u002Fopentoken-cli\">\u003Cimg src=\"https:\u002F\u002Fimg.shields.io\u002Fnpm\u002Fv\u002F@mrgray17\u002Fopentoken-cli?color=f97316&label=cli\" \u002F>\u003C\u002Fa>\n  \u003Ca href=\"https:\u002F\u002Fwww.npmjs.com\u002Fpackage\u002F@mrgray17\u002Fopentoken-mcp\">\u003Cimg src=\"https:\u002F\u002Fimg.shields.io\u002Fnpm\u002Fv\u002F@mrgray17\u002Fopentoken-mcp?color=8b5cf6&label=mcp\" \u002F>\u003C\u002Fa>\n  \u003Ca href=\"https:\u002F\u002Fwww.npmjs.com\u002Fpackage\u002F@mrgray17\u002Fopentoken-core\">\u003Cimg src=\"https:\u002F\u002Fimg.shields.io\u002Fnpm\u002Fv\u002F@mrgray17\u002Fopentoken-core?color=22c55e&label=core\" \u002F>\u003C\u002Fa>\n  \u003Ca href=\"https:\u002F\u002Fgithub.com\u002FMrGray17\u002Fopentoken\u002Factions\">\u003Cimg src=\"https:\u002F\u002Fimg.shields.io\u002Fgithub\u002Factions\u002Fworkflow\u002Fstatus\u002FMrGray17\u002Fopentoken\u002Fci.yml?branch=main&label=CI\" \u002F>\u003C\u002Fa>\n  \u003Ca href=\"https:\u002F\u002Fbun.sh\">\u003Cimg src=\"https:\u002F\u002Fimg.shields.io\u002Fbadge\u002Fbun-%E2%89%A51.2.0-fbb744\" \u002F>\u003C\u002Fa>\n  \u003Ca href=\"https:\u002F\u002Fgithub.com\u002FMrGray17\u002Fopentoken\u002Fblob\u002Fmain\u002FLICENSE\">\u003Cimg src=\"https:\u002F\u002Fimg.shields.io\u002Fbadge\u002Flicense-MIT-334155\" \u002F>\u003C\u002Fa>\n\u003C\u002Fp>\n\u003Cp align=\"center\">\n  \u003Cb>431 tests\u003C\u002Fb> · \u003Cb>35 stages\u003C\u002Fb> · \u003Cb>10 command families\u003C\u002Fb> · \u003Cb>zero regressions\u003C\u002Fb>\n\u003C\u002Fp>\n---\n## Why This Exists\nAI coding assistants invoke shell commands -- `git diff`, `npm install`, `grep -r`, `ls -la` -- whose raw output is dominated by noise. ANSI escapes, npm audit spam, CI logs with repeating timestamps, compiler warnings echoed 200 times. Every byte burns input tokens for no semantic benefit.\n\nOpenToken is a transparent filter between tool runtime and LLM context window. Every output passes through 35 compression stages, each ending with a **conservative safety filter**: if a stage makes the output larger, the original is returned untouched.\nThe model sees the same information, reasons the same way, and produces the same answers -- at **50-80% fewer tokens**.\n\n---\n\n## How It Works\n\n```mermaid\nflowchart LR\n    A[\"Tool Output\"] --> B[\"Secrets Redaction\"]\n    B --> C[\"Binary Detection\"]\n    C --> D[\"ANSI Strip\"]\n    D --> E[\"Family Detection\"]\n    E --> F[\"Pipeline Router\"]\n    F --> G[\"git pipeline\"]\n    F --> H[\"npm pipeline\"]\n    F --> I[\"generic pipeline\"]\n    F --> J[\"... 7 more\"]\n    G --> K[\"Conservative Filter\"]\n    H --> K\n    I --> K\n    J --> K\n    K --> L[\"Compressed Output to LLM\"]\n```\n\nPipeline routing selects a specialized chain of 10-20 stages based on command context -- `git status` gets a different pipeline than `cargo build`.\n\n---\n\n## Features\n\n### Noise Reduction\n- **Pre-call rewrites** -- 46 patterns suppress noise *before* execution: adds `--quiet`, `--silent`, `-q`, `-s` flags to npm, yarn, cargo, pip, pytest, curl, docker, make, systemctl, git, and more\n- **ANSI stripping** -- removes terminal color codes and control sequences\n- **Thinking block removal** -- strips XML reasoning, monologue, and scratchpad blocks\n- **JSON cleanup** -- removes null, empty, false, and redundant values\n- **Table whitespace minimization** -- strips padding from CLI table output\n- **Path shortening** -- replaces project-root prefixes with relative paths\n- **Directory grouping** -- collapses repeated directory paths in file listings\n- **Table stripping** -- collapses `docker ps`, `docker images`, `df`, `free`, `ps aux` to essential columns only\n\n### Structural Compression\n- **Diff folding** -- condenses context hunks: `... 14 context lines omitted`\n- **Log folding** -- collapses consecutive identical lines: `8 x error message`\n- **Fold repeats** -- deduplicates non-consecutive identical lines (5+ occurrences)\n- **Line noise normalization** -- replaces timestamps, PIDs, elapsed times with static placeholders\n\n### Advanced Compression\n- **LTSC** -- Lossless Token Sequence Compression (LZ77-sliding window), 18-27%\n- **LZW token substitution** -- high-frequency substrings replaced with single-token markers, 20-40% on repetitive output\n- **Progressive disclosure** -- summary first, full output on demand via offloaded temp files\n- **Reversible compression** -- semantic abbreviation with rewind for full recovery\n- **TOON conversion** -- transforms JSON arrays into tabular format\n\n### Safety\n- **0-risk principle** -- every stage ends with a conservative filter; output only shrinks or stays\n- **Secrets redaction** -- runs before all other processing\n- **Binary detection** -- NUL byte guard prevents corruption of binary streams\n- **Size caps** -- skips compression on inputs exceeding 50KB\n\n### Operations\n- **Auto-tuning** -- per-family effectiveness metrics control whether heavy stages run\n- **Cross-call dedup** -- prevents repeated output across tool calls in the same session\n- **10 command families** -- specialized pipelines: git, npm, cargo, docker, pip, make, test, fs, grep, generic (+ 46 pre-call rewrite patterns)\n- **Stats dashboard** -- session and all-time tracking, per-tool breakdown\n\n\n---\n\n## Comparison\n\n| | OpenToken | RTK | QTK | Caveman | built-in |\n|---|---|---|---|---|---|\n| **Approach** | Full compression engine | CLI proxy | OpenCode plugin | Language mode | Basic truncation |\n| **Token savings** | 55-90% | 60-90% | 60-90% | ~75% (messages) | 20-30% |\n| **Runtime** | Bun (TS, no build) | Rust binary | Bun\u002FNode | Any LLM | built-in |\n| **Stages** | 46+ | ~10 | ~10 | 1 | 1 |\n| **Secrets redaction** | yes | -- | -- | -- | -- |\n| **Progressive** | yes | -- | -- | -- | -- |\n| **Reversible** | yes | -- | -- | -- | -- |\n| **Auto-tuning** | yes | -- | -- | -- | -- |\n| **Stats** | yes | -- | -- | -- | -- |\n| **CLI pipe\u002Fwrap** | yes | yes | -- | -- | -- |\n| **Cross-call dedup** | yes | -- | -- | -- | -- |\n| **0-risk safety filter** | every stage | -- | -- | -- | -- |\n\n\n---\n\n## Quick Start\n\n**Bun v1.2+** recommended. Core also works under **Node.js >=18** via the fs-compat polyfill. [Install Bun](https:\u002F\u002Fbun.sh).\n\n```bash\nnpm i -g @mrgray17\u002Fopentoken     # OpenCode plugin (auto-loads, zero config)\nnpm i -g @mrgray17\u002Fopentoken-mcp # Any IDE via MCP\nbun x @mrgray17\u002Fopentoken-cli    # CLI pipe\u002Fwrap\nnpm i @mrgray17\u002Fopentoken-core   # Library\n```\n\n### Try It\n\n```bash\ngit diff | opentoken -t bash\nopentoken wrap cargo-build\nopentoken stats\n```\n\n---\n## Configuration\n\nWorks with zero configuration. Optional overrides at `~\u002F.config\u002Fopentoken\u002Fconfig.json`:\n\n```json\n{\n  \"enableMetrics\": true,\n  \"safeReadRoot\": \"\u002Fhome\u002Fuser\u002Fprojects\u002Fmyapp\",\n  \"maxOutputBytes\": 1048576,\n  \"enableSymbolIndex\": false\n}\n```\n\nSee [AGENTS.md](https:\u002F\u002Fgithub.com\u002FMrGray17\u002Fopentoken\u002Fblob\u002Fmain\u002FAGENTS.md) for all config fields and defaults.\n\n---\n\n## IDE Integration (MCP)\n\n**Cursor \u002F Windsurf** -- add to `~\u002F.cursor\u002Fmcp.json`:\n\n```json\n{ \"mcpServers\": { \"opentoken\": { \"command\": \"opentoken-mcp\" } } }\n```\n\n**Claude Desktop** -- add to `~\u002F.claude\u002Fclaude_desktop_config.json`:\n\n```json\n{ \"mcpServers\": { \"opentoken\": { \"command\": \"opentoken-mcp\" } } }\n```\n\n**VS Code Copilot** -- add to `.vscode\u002Fmcp.json`:\n\n```json\n{ \"servers\": { \"opentoken\": { \"type\": \"stdio\", \"command\": \"opentoken-mcp\" } } }\n```\n\n---\n\n## Project Structure\n\n```\nopentoken\u002F\n  packages\u002F\n    core\u002Fsrc\u002F              # Universal compression engine (53 files)\n      transform.ts           # Entry: transformToolOutput()\n      precall.ts             # Command rewriting, minified file blocking\n      postcall.ts            # Normalize, fold, minify, strip\n      wrappers.ts            # safeStage, conservativeFilter, routeContent\n      autoescalate.ts        # Progressive compression as context fills\n      rewind.ts              # Reversible compression + abbreviation\n      ltsc.ts                # LZ77-style lossless sequence compression\n      lzw.ts                 # LZW-style token substitution\n      folding.ts             # Diff + log folding\n      progressive.ts         # Summary-first, full on demand\n      skeleton.ts            # AST skeleton extraction\n      families\u002F              # 10 command-family output filters\n      filters\u002F               # 3 tool-specific output filters\n      pipelines\u002F             # 4 tool pipelines + shared utilities\n      utils\u002F                 # 11 utilities (cache, configDir, errors, fs-compat, etc.)\n    cli\u002Fsrc\u002F                 # CLI binary (~260 lines)\n    mcp\u002Fsrc\u002F                 # MCP JSON-RPC server\n    opencode\u002Fsrc\u002F            # OpenCode plugin adapter (~140 lines)\n  tests\u002F\n    core\u002F                    # 21 files, 425 tests\n    opencode\u002F                # 1 file, 6 tests\n```\n\n---\n\n## Design Decisions\n\n**0-risk principle.** Every compression stage is followed by a conservative filter that compares estimated token counts. If a stage produced MORE tokens than its input consumed, the original output is returned untouched. This guarantees compression can never regress quality.\n\n**Bun, with Node.js fallback.** OpenToken targets Bun v1.2+ for native TypeScript execution -- no `tsc`, no `tsup`, no `esbuild`. The core package also works under **Node.js >=18** via a thin `fs-compat.ts` polyfill layer. This means the OpenCode plugin works regardless of whether OpenCode runs on Bun or Node.\n\n**Pipeline architecture.** Each command family (git, npm, cargo, docker, etc.) has a dedicated pipeline of 10-20 stages. A generic pipeline catches everything else. 46 pre-call rewrite patterns suppress noise before execution. The pipeline router detects the command context from the command string and selects the right chain.\n\n**Token estimation, not character counting.** The conservative filter estimates BPE token counts rather than measuring raw character length. This correctly accounts for LTSC\u002FLZW markers which are 2 characters but 2 BPE tokens, preventing false negatives.\n\n**Auto-tuning.** Per-family compression effectiveness is tracked via metrics files. Heavy stages (LTSC, LZW) query this before running -- if a family consistently yields no savings, those stages skip entirely.\n\n---\n\n## Development\n\n```bash\nbun install            # Install dependencies\nbun test               # All 431 tests (Bun test runner)\nbun run typecheck      # tsc --noEmit\nbun run lint           # Biome check\nbun run lint:fix       # Auto-fix with Biome\n```\n\nCI workflow: `typecheck` -> `lint` -> `checks:regex` -> `test`.\n\n### Architecture\n\nTests import from workspace packages: `@mrgray17\u002Fopentoken-core` for core tests, `@mrgray17\u002Fopentoken` for plugin tests. No build step -- Bun resolves workspace packages natively.\n\nSee [AGENTS.md](https:\u002F\u002Fgithub.com\u002FMrGray17\u002Fopentoken\u002Fblob\u002Fmain\u002FAGENTS.md) for developer documentation. Issues and PRs: [GitHub Issues](https:\u002F\u002Fgithub.com\u002FMrGray17\u002Fopentoken\u002Fissues).\n\n---\n\n## Contributors\n\n\u003Ca href=\"https:\u002F\u002Fgithub.com\u002FMrGray17\">\u003Cimg src=\"https:\u002F\u002Fgithub.com\u002FMrGray17.png\" width=\"40\" height=\"40\" alt=\"MrGray17\" style=\"border-radius: 50%;\" \u002F>\u003C\u002Fa>\n\u003Ca href=\"https:\u002F\u002Fgithub.com\u002FOhOkThisIsFine\">\u003Cimg src=\"https:\u002F\u002Fgithub.com\u002FOhOkThisIsFine.png\" width=\"40\" height=\"40\" alt=\"OhOkThisIsFine\" style=\"border-radius: 50%;\" \u002F>\u003C\u002Fa>\n\n---\n\n## License\n\nMIT -- see [LICENSE](.\u002FLICENSE).\n","OpenToken 是一个专为AI编码助手设计的令牌压缩引擎，旨在减少工具输出中的冗余信息。该项目通过42层压缩处理，包括ANSI颜色代码移除、二进制文件检测及秘密数据红化等步骤，在保证原始信息完整性的前提下显著降低了传递给语言模型的数据量，从而实现50-80%的令牌节省。它支持包括OpenCode、Cursor、Windsurf在内的多种IDE环境，并兼容MCP标准。特别适用于需要频繁调用系统命令行工具且对成本敏感的AI辅助开发场景中使用。","2026-06-11 03:59:51","CREATED_QUERY"]