[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"project-94964":3},{"id":4,"name":5,"fullName":6,"owner":7,"repo":5,"description":8,"homepage":9,"htmlUrl":10,"language":11,"languages":10,"totalLinesOfCode":10,"stars":12,"forks":13,"watchers":14,"openIssues":15,"contributorsCount":15,"subscribersCount":15,"size":15,"stars1d":15,"stars7d":16,"stars30d":16,"stars90d":15,"forks30d":15,"starsTrendScore":17,"compositeScore":18,"rankGlobal":10,"rankLanguage":10,"license":19,"archived":20,"fork":20,"defaultBranch":21,"hasWiki":20,"hasPages":20,"topics":22,"createdAt":10,"pushedAt":10,"updatedAt":29,"readmeContent":30,"aiSummary":31,"trendingCount":15,"starSnapshotCount":15,"syncStatus":32,"lastSyncTime":33,"discoverSource":34},94964,"ai-data-extractor","bawadou\u002Fai-data-extractor","bawadou","Free open-source extractor for AI coding assistant chat histories. Supports Claude Code, Cursor, Windsurf, Aider, Cline\u002FRoo Code, and more.","",null,"Python",537,99,17,0,355,10,80,"MIT License",false,"main",[23,5,24,25,26,27,28],"ai","claude","cursor","cursor-ai","data-extraction","gemini","2026-08-24 04:01:23","# AI Coding Assistant Data Extraction\r\n\r\nExtract your **own local chat history** from AI coding assistants into a single,\r\nnormalized JSONL format - for fine-tuning, personal analytics, or just backing\r\nup years of conversations before an app's local database gets cleared.\r\n\r\n\r\n## Features\r\n\r\nAuto-discovers and extracts complete conversation history, including:\r\n\r\n- User messages & assistant responses\r\n- Code context (file paths, selections, snippets)\r\n- Code diffs \u002F suggested edits, where the tool records them\r\n- Tool calls and their results\r\n- Timestamps, session IDs, project paths, model names - whatever each\r\n  tool's storage actually contains\r\n\r\n## Supported sources\r\n\r\n| # | Tool | Storage | Search location |\r\n|---|------|---------|------------------|\r\n| 1 | **Claude Code** | JSONL, one file per session | `~\u002F.claude\u002Fprojects\u002F**\u002F*.jsonl` |\r\n| 2 | **Codex CLI** | JSONL \"rollout\" files | `~\u002F.codex\u002Fsessions\u002F**\u002Frollout-*.jsonl` |\r\n| 3 | **Cursor** | SQLite (`state.vscdb`) | `~\u002F…\u002FCursor\u002FUser\u002F{global,workspace}Storage` |\r\n| 4 | **Windsurf** | SQLite, undocumented schema (heuristic) | `~\u002F…\u002FWindsurf\u002FUser\u002F{global,workspace}Storage` |\r\n| 5 | **Trae** | SQLite + JSONL, undocumented (heuristic) | `~\u002F…\u002FTrae` |\r\n| 6 | **Continue** | JSON, one file per session | `~\u002F.continue\u002Fsessions\u002F*.json` |\r\n| 7 | **Gemini CLI** | JSON, one file per chat | `~\u002F.gemini\u002Ftmp\u002F\u003Chash>\u002Fchats\u002F*.json` |\r\n| 8 | **OpenCode** | JSON (session\u002Fmessage\u002Fpart trees) | `~\u002F.local\u002Fshare\u002Fopencode\u002Fstorage\u002F` |\r\n| 9 | **Cline \u002F Roo Code** *(new)* | JSON, one folder per task | `\u003Ceditor>\u002FUser\u002FglobalStorage\u002F\u003Cext-id>\u002Ftasks\u002F` |\r\n| 10 | **Aider** *(new)* | Markdown transcript per project | `\u003Cproject>\u002F.aider.chat.history.md` |\r\n\r\nEvery script searches macOS, Linux, and Windows conventions\r\n(`~\u002FLibrary\u002FApplication Support`, `~\u002F.config`, `~\u002F.local\u002Fshare`, `%APPDATA%`,\r\n`%LOCALAPPDATA%`) automatically - you don't need to tell it which OS you're on.\r\n\r\n### Why Cline and Aider\r\n\r\nThey're two of the most-used AI coding tools that the original list left out,\r\nand both have a genuinely different (and instructive) storage shape:\r\n\r\n- **Cline** (and its fork **Roo Code**) is the most popular open-source\r\n  autonomous coding *agent* extension. It stores raw Anthropic-format\r\n  message arrays per task, so its extractor doubles as the simplest example\r\n  of parsing that format if you want to add your own tool later.\r\n- **Aider** is the most popular *terminal-only* pair-programming tool, and\r\n  it's structured completely differently from everything else here: no\r\n  central database, just a markdown transcript sitting in every project\r\n  directory. It's included specifically to prove the toolkit generalizes\r\n  beyond \"SQLite or JSONL in one app-data folder.\"\r\n\r\n## Quick start\r\n\r\n```bash\r\n# No dependencies -- standard library only\r\npython --version   # 3.9+ required, 3.10+ recommended\r\n\r\n# Interactive: pick which sources to extract from a numbered menu\r\npython extract.py\r\n\r\n# Or drive it directly\r\npython extract.py --all\r\npython extract.py --sources cursor,claude_code,aider\r\npython extract.py --list                    # just show what's installed, don't extract\r\npython extract.py --all --merge             # also write all_conversations.jsonl\r\n\r\n# Shorthand for \"extract everything\"\r\n.\u002Fextract_all.sh\r\n```\r\n\r\n### CLI reference\r\n\r\n```\r\npython extract.py [--all] [--sources ids] [--list] [--output-dir DIR]\r\n                    [--search-path PATH ...] [--merge]\r\n\r\n--all                 Extract every supported source, no prompts.\r\n--sources ids         Comma-separated source ids (see table above for names,\r\n                       or run with --list to see them). Skips the menu.\r\n--list                Report what was found for each source without\r\n                       extracting anything -- a fast, safe preview.\r\n--output-dir DIR      Where to write JSONL (default: .\u002Fextracted_data)\r\n--search-path PATH    Extra directory to search, on top of the normal OS\r\n                       locations. Repeatable. Mainly useful for:\r\n                         - Aider, which has no fixed app-data folder and\r\n                           needs to know where your projects live\r\n                         - nonstandard install locations for anything else\r\n--merge               After extracting, also concatenate everything into\r\n                       all_conversations.jsonl\r\n```\r\n\r\nEach extractor can also still be run standalone, same as the original\r\ntoolkit (`python -m extractors.cursor` from the project root, or\r\n`python extractors\u002Fcursor.py`), which is handy while debugging one source.\r\n\r\n## Output format\r\n\r\nEvery run creates timestamped files under `extracted_data\u002F`:\r\n\r\n```\r\nextracted_data\u002F\r\n├── claude_code_conversations_20260816_143022.jsonl\r\n├── cursor_conversations_20260816_143022.jsonl\r\n├── aider_conversations_20260816_143022.jsonl\r\n├── cline_conversations_20260816_143022.jsonl\r\n└── ... one file per source you extracted, plus all_conversations.jsonl if --merge\r\n```\r\n\r\nEach line is one JSON conversation:\r\n\r\n```json\r\n{\r\n  \"messages\": [\r\n    {\r\n      \"role\": \"user\",\r\n      \"content\": \"How do I fix this TypeScript error?\",\r\n      \"code_context\": [\r\n        {\"file\": \"\u002FUsers\u002Fyou\u002Fproject\u002Fsrc\u002Findex.ts\", \"code\": \"const x: string = 123;\"}\r\n      ],\r\n      \"timestamp\": \"2026-01-16T14:30:22Z\"\r\n    },\r\n    {\r\n      \"role\": \"assistant\",\r\n      \"content\": \"The error occurs because you're assigning a number to a string type...\",\r\n      \"tool_use\": [{\"name\": \"edit_file\", \"input\": {\"path\": \"src\u002Findex.ts\"}}],\r\n      \"timestamp\": \"2026-01-16T14:30:25Z\"\r\n    }\r\n  ],\r\n  \"source\": \"cursor-composer\",\r\n  \"session_id\": \"c1a2b3...\",\r\n  \"project_path\": \"\u002FUsers\u002Fyou\u002Fproject\",\r\n  \"name\": \"TypeScript Type Error Fix\",\r\n  \"created_at\": 1705414222000\r\n}\r\n```\r\n\r\nFields vary a bit by source (not every tool records `code_context`, token\r\nusage, or `project_path`) - `messages`, `source`, and `session_id` are the\r\nonly ones you can always rely on.\r\n\r\n## How it works\r\n\r\n1. **Detect the OS** and build a list of plausible data roots\r\n   (`Application Support`, `.config`, `.local\u002Fshare`, `%APPDATA%`, etc).\r\n2. **Search each root** for the tool's known folder name(s).\r\n3. **Read the storage** - JSONL line-by-line, SQLite via a read-only\r\n   connection (so a running app never blocks us), or JSON trees, depending\r\n   on the tool.\r\n4. **Normalize** whatever's found into the `messages[]` schema above.\r\n5. **Write JSONL**, one conversation per line, into `extracted_data\u002F`.\r\n\r\nNothing here ever opens a database for writing, and every reader is wrapped\r\nso that one corrupt or locked file can't take down the whole run - you'll\r\nget a partial result and move on, not a stack trace.\r\n\r\n### A note on Cursor, Windsurf, and Trae\r\n\r\nNone of these publish their storage schema, and it has changed multiple\r\ntimes (Cursor alone has gone through at least three shapes: workspace\r\n`ItemTable` chat, inline composer, and split `bubbleId` composer). The\r\n`cursor.py` extractor implements all three known shapes explicitly.\r\n`windsurf.py` and `trae.py` instead use a **generic heuristic**\r\n(`extractors\u002Fcommon.py::heuristic_extract_chat_from_kv`) that scans\r\nchat-related keys and walks the parsed JSON looking for objects shaped like\r\na role + text pair. It's honest best-effort, not a documented format - if\r\na future version changes shape and stops matching, that's expected; adjust\r\n`KEY_HINTS` in the relevant file or send a PR.\r\n\r\n### A note on Aider\r\n\r\nAider has no central session store - every project directory gets its own\r\n`.aider.chat.history.md`. By default this toolkit scans your home directory\r\nplus a handful of common project-root names (`projects`, `code`, `dev`,\r\n`repos`, `workspace`, `src`, `Documents`) up to 5 directories deep, skipping\r\n`node_modules`, `.git`, and similar. If your projects live somewhere else,\r\npoint at them directly:\r\n\r\n```bash\r\npython extract.py --sources aider --search-path ~\u002Fclient-work --search-path \u002Fmnt\u002Fdata\u002Frepos\r\n```\r\n\r\n## Extending it: adding a new source\r\n\r\nEvery extractor is a small module with the same two-function interface -\r\ncopy the simplest one (`continue_ext.py` is a good template) and fill in:\r\n\r\n```python\r\nDISPLAY_NAME = \"My Tool\"\r\nSOURCE_ID = \"my_tool\"\r\n\r\ndef find_installations(extra_paths: list[Path] | None = None) -> list[Path]:\r\n    \"\"\"Return the directories\u002Ffiles worth scanning.\"\"\"\r\n\r\ndef extract(installations: list[Path]) -> list[dict]:\r\n    \"\"\"Return a list of conversation dicts matching the schema above.\"\"\"\r\n```\r\n\r\nThen register it in `extract.py`'s `REGISTRY` list. `extractors\u002Fcommon.py`\r\nhas the SQLite\u002FJSON\u002FJSONL readers and the two generic heuristics you'll\r\nprobably want.\r\n\r\n## Privacy & security\r\n\r\nThis extracts data from tools running under your own user account. Before\r\nsharing or training on it:\r\n\r\n1. **Scan for secrets**:\r\n   ```bash\r\n   pip install detect-secrets --break-system-packages\r\n   detect-secrets scan extracted_data\u002F*.jsonl\r\n   ```\r\n2. **Review for proprietary code, API keys, and personal file paths** -\r\n   `code_context` and `tool_use` fields are the most likely places to find them.\r\n3. **Don't commit `extracted_data\u002F` to a public repo** (it's already in\r\n   `.gitignore`). Keep it on encrypted storage if it contains client or\r\n   proprietary work.\r\n\r\n## Training use cases\r\n\r\n```python\r\nfrom datasets import load_dataset\r\n\r\ndataset = load_dataset(\"json\", data_files=\"extracted_data\u002F*.jsonl\", split=\"train\")\r\ndataset = dataset.filter(lambda x: any(m[\"role\"] == \"assistant\" for m in x[\"messages\"]))\r\n\r\ndef format_chat(example):\r\n    return {\"text\": tokenizer.apply_chat_template(example[\"messages\"], tokenize=False)}\r\n\r\ndataset = dataset.map(format_chat)\r\n```\r\n\r\n## Troubleshooting\r\n\r\n**\"No installation found\"** - the tool either isn't installed, has no chat\r\nhistory yet, or lives somewhere nonstandard. Pass `--search-path` to point\r\nat it directly, or check `extractors\u002F\u003Ctool>.py`'s `SEARCH_DIRS` \u002F\r\n`APP_DIR_NAMES` constant and add your path there.\r\n\r\n**Cursor\u002FWindsurf database locked** - reads are opened `mode=ro` specifically\r\nso a running editor won't block extraction, but if you still see errors,\r\nclose the app and re-run.\r\n\r\n**Windsurf\u002FTrae found an installation but 0 conversations** - the heuristic\r\nkey-matching in `common.heuristic_extract_chat_from_kv` didn't recognize the\r\ncurrent storage keys. Run `--list` to confirm the app dir was found, then\r\ninspect `state.vscdb`'s `ItemTable`\u002F`cursorDiskKV` keys directly\r\n(`sqlite3 state.vscdb \"SELECT key FROM ItemTable\"`) and add matches to\r\n`KEY_HINTS`.\r\n\r\n## Disclaimer\r\n\r\nThis toolkit extracts **your own** data from AI tools installed on your own\r\nmachine. You're responsible for:\r\n\r\n- having the rights to the extracted data,\r\n- handling any sensitive\u002Fproprietary information appropriately,\r\n- complying with each tool's Terms of Service,\r\n- scanning for secrets before sharing or training on the output.\r\n\r\n## License\r\n\r\nMIT - use freely, including for training ML models.\r\n","这是一个开源的AI编程助手聊天记录提取工具，用于从本地多种AI编码工具中批量导出并标准化用户自己的对话历史数据。核心功能包括自动识别跨平台存储路径（支持macOS\u002FLinux\u002FWindows）、解析JSONL、SQLite及Markdown等异构格式，并统一输出为结构化JSONL，保留消息内容、代码上下文、diff变更、工具调用、时间戳及项目元信息。技术特点为纯Python标准库实现、零外部依赖、支持10+主流工具（如Claude Code、Cursor、Windsurf、Aider、Cline\u002FRoo Code等），且具备可扩展的提取器架构。适用于AI模型微调、个人开发行为分析、知识归档与本地数据备份等场景。",2,"2026-08-19 02:30:03","CREATED_QUERY"]