[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"project-92704":3},{"id":4,"name":5,"fullName":6,"owner":7,"repo":5,"description":8,"homepage":9,"htmlUrl":10,"language":11,"languages":10,"totalLinesOfCode":10,"stars":12,"forks":13,"watchers":14,"openIssues":15,"contributorsCount":15,"subscribersCount":15,"size":15,"stars1d":15,"stars7d":15,"stars30d":16,"stars90d":15,"forks30d":15,"starsTrendScore":15,"compositeScore":17,"rankGlobal":10,"rankLanguage":10,"license":18,"archived":19,"fork":19,"defaultBranch":20,"hasWiki":21,"hasPages":19,"topics":22,"createdAt":10,"pushedAt":10,"updatedAt":26,"readmeContent":27,"aiSummary":28,"trendingCount":15,"starSnapshotCount":15,"syncStatus":14,"lastSyncTime":29,"discoverSource":30},92704,"wiki-compiler","Emmimal\u002Fwiki-compiler","Emmimal","A pure-Python compiler that turns raw markdown notes into a linked, linted wiki. Zero dependencies, zero LLM calls, fully deterministic.","https:\u002F\u002Ftowardsdatascience.com\u002Fllm-wikis-are-over-engineered-i-replaced-mine-with-a-pure-python-compiler\u002F",null,"Python",67,8,2,0,9,43.76,"MIT License",false,"main",true,[23,24,25],"andrej-karpathy-llm-wiki","llm-wiki","python","2026-07-22 04:02:06","# wiki-compiler\n\nA pure-Python compiler that turns raw, messy text notes into a linked, linted markdown wiki. No LLM calls, no embeddings, no dependencies.\n\n![Python Version](https:\u002F\u002Fimg.shields.io\u002Fbadge\u002Fpython-3.12%2B-blue) ![License](https:\u002F\u002Fimg.shields.io\u002Fbadge\u002Flicense-MIT-green)\n\nMost LLM wiki tutorials stop at: point an agent at your notes, let it decide what's related, let it rewrite pages. This library handles the deterministic part of that job instead: extracting structure, building the link graph, and validating the result, without a single model call.\n\nRead the full write-up on Towards Data Science → [LLM Wikis Are Over-Engineered. I Replaced Mine With a Pure Python Compiler.](https:\u002F\u002Ftowardsdatascience.com\u002Fllm-wikis-are-over-engineered-i-replaced-mine-with-a-pure-python-compiler\u002F)\n\n## What It Does\n\n```\nRaw Notes (.txt) → Extractor → Graph → Rewriter → Linter → Compiled Wiki (.md)\n```\n\nFour stages, one `compile_wiki()` call:\n\n| Component | Job |\n|---|---|\n| Extractor | Regex scan pulling entity name, aliases, created date, and body text out of inconsistently formatted raw files |\n| Graph | Word-indexed phrase matcher detecting mentions between entities, building a bidirectional reference map |\n| Rewriter | Section-aware markdown compilation; regenerates compiler-owned sections, preserves hand-written Notes |\n| Linter | Structural validation: broken `[[links]]` and orphan pages with zero incoming references |\n\n## Installation\n\n```bash\ngit clone https:\u002F\u002Fgithub.com\u002FEmmimal\u002Fwiki-compiler.git\ncd wiki-compiler\npython init.py\n```\n\nNo dependencies to install. Standard library only.\n\n## Quick Start\n\n```python\nfrom compiler import compile_wiki\n\nresult = compile_wiki(\"raw_notes\", \"compiled_wiki\")\n\nprint(f\"Compiled {len(result['written_paths'])} pages\")\nprint(f\"Broken links: {len(result['lint_report'].broken_links)}\")\nprint(f\"Orphan pages: {len(result['lint_report'].orphan_pages)}\")\n```\n\nOr from the command line:\n\n```bash\npython compiler.py raw_notes\u002F compiled_wiki\u002F\n```\n\n## Running the Tests and Benchmark\n\nSeventeen tests, stdlib `unittest` only, covering every stage plus the full end-to-end pipeline:\n\n```bash\npython -m unittest tests -v\n```\n\nReal per-stage timing at three corpus sizes, using a deterministic synthetic corpus (seed=42):\n\n```bash\npython benchmark.py --files 100 --files 1000 --files 5000\n```\n\n## CLI Reference\n\n```\npython compiler.py raw_dir output_dir [--no-lint]\n\n  raw_dir       Directory of raw .txt source files\n  output_dir    Directory to write compiled .md pages into\n  --no-lint     Skip the lint pass\n```\n\n```\npython benchmark.py [--files N ...] [--seed N]\n\n  --files       Number of files to benchmark at (repeatable)\n  --seed        Random seed for the synthetic corpus generator (default: 42)\n```\n\n## Project Structure\n\n```\nwiki-compiler\u002F\n├── compiler.py       # Orchestrates all four stages behind one function call\n├── extractor.py      # Stage 1: regex metadata extraction\n├── graph.py           # Stage 2: word-indexed mention detection + bidirectional graph\n├── rewriter.py         # Stage 3: section-aware markdown compilation\n├── linter.py            # Stage 4: broken-link and orphan-page validation\n├── generator.py          # Synthetic test corpus generator, for demos and benchmarks\n├── benchmark.py           # Timing harness\n├── init.py                 # Zero-configuration entry point\n└── tests.py                  # 17 unit tests, stdlib only\n```\n\n## Performance (two machines, same deterministic outputs)\n\n| Files | Extract | Graph | Rewrite | Lint | Compile total | Full pipeline | Orphans |\n|---|---|---|---|---|---|---|---|\n| 100 | 22.8 ms | 3.1 ms | 59.4 ms | 86.0 ms | 85.4 ms | 171.4 ms | 13 |\n| 1,000 | 261.5 ms | 47.1 ms | 605.5 ms | 883.9 ms | 914.1 ms | 1,798.0 ms | 133 |\n| 5,000 | 1,398.4 ms | 625.6 ms | 3,446.7 ms | 6,972.5 ms | 5,470.6 ms | 12,443.1 ms | 644 |\n\nOrphan and broken-link counts are identical across every run, on both Linux and Windows. Wall-clock timing varies by hardware and OS; the deterministic outputs don't. `graph` has zero disk I\u002FO and scales the best; `lint` is the most I\u002FO-sensitive stage and the most expensive one at scale.\n\n## When to Use This\n\nWorth it when you have:\n- A folder of local, already-written notes you want structured and cross-referenced\n- A workflow where you want the same output every time you recompile\n- No interest in spending tokens on organizational work an agent would redo on every run\n\nSkip it when you have:\n- Notes that need semantic linking, where related ideas are phrased differently rather than sharing exact terms\n- A need for an agent that also drafts new content, not just links existing text\n- Source data too unstructured for regex-based extraction to make sense of\n\n## Known Limitations\n\n- Mention detection is lexical, not semantic. Two notes describing the same concept in different words won't link automatically.\n- The extractor handles two header styles and optional metadata fields. Wildly inconsistent or multi-language source data would need a more sophisticated extraction layer.\n- Lint performance is I\u002FO-bound and platform-sensitive; expect Windows to run measurably slower than Linux at scale, likely due to filesystem overhead and antivirus scanning.\n\n## License\n\nMIT\n","这是一个纯Python编写的静态维基编译器，将零散的原始文本笔记（.txt）自动转换为结构清晰、双向链接完备且经过校验的Markdown维基（.md）。其核心功能包括：基于正则的元数据提取、词索引驱动的实体提及识别与图构建、保留手工编辑内容的智能重写，以及针对链接断裂和孤立页面的结构化校验。全程无外部依赖、不调用LLM、结果完全可复现。适用于个人知识管理（PKM）、技术文档归档、学术笔记系统等需轻量级、确定性、离线维基生成的场景。","2026-07-10 02:30:12","CREATED_QUERY"]