[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"project-94875":3},{"id":4,"name":5,"fullName":6,"owner":7,"repo":5,"description":8,"homepage":9,"htmlUrl":10,"language":11,"languages":9,"totalLinesOfCode":9,"stars":12,"forks":13,"watchers":14,"openIssues":15,"contributorsCount":9,"subscribersCount":16,"size":16,"stars1d":16,"stars7d":14,"stars30d":14,"stars90d":16,"forks30d":16,"starsTrendScore":16,"compositeScore":17,"rankGlobal":9,"rankLanguage":9,"license":9,"archived":18,"fork":18,"defaultBranch":19,"hasWiki":18,"hasPages":18,"topics":9,"createdAt":9,"pushedAt":9,"updatedAt":20,"readmeContent":21,"aiSummary":22,"trendingCount":16,"starSnapshotCount":16,"syncStatus":23,"lastSyncTime":24,"discoverSource":25},94875,"ai-data-extraction","0xSero\u002Fai-data-extraction","0xSero","extract all your personal data history from cursor, codex, claude-code, windsurf, and trae",null,"https:\u002F\u002Fgithub.com\u002F0xSero\u002Fai-data-extraction","Python",1130,97,5,1,0,53.97,false,"main","2026-08-24 04:01:22","# AI Coding Assistant Training Data Extraction Toolkit\n\nComplete toolkit to extract ALL chat, agent, and code context data from AI coding assistants for machine learning training.\n\n## 🎯 What This Does\n\nAutomatically discovers and extracts **complete conversation history** including:\n- ✅ User messages & AI responses\n- ✅ Code context (file paths, line numbers, snippets)\n- ✅ Code diffs and suggested edits\n- ✅ Multi-file contexts\n- ✅ Tool use and execution results\n- ✅ Timestamps and metadata\n\n## 📦 Included Scripts\n\n### 1. `extract_claude_code.py`\nExtracts from Claude Code \u002F Claude Desktop\n- **Searches**: `~\u002F.claude`, `~\u002F.claude-code`, `~\u002F.claude-local`, `~\u002F.claude-m2`, `~\u002F.claude-zai`\n- **Formats**: JSONL session files\n- **Includes**: Messages, tool use, file contexts, diffs\n\n### 2. `extract_codex.py`\nExtracts from Codex (if installed)\n- **Searches**: `~\u002F.codex`, `~\u002F.codex-local`\n- **Formats**: Rollout JSONL files\n- **Includes**: User\u002Fagent messages, tool results, diffs\n\n### 3. `extract_cursor.py`\nExtracts from Cursor (Chat + Composer + Agent) - ALL VERSIONS\n- **Searches**: `~\u002FLibrary\u002FApplication Support\u002FCursor` (macOS) or equivalent\n- **Formats**: SQLite databases (`state.vscdb`, `cursorDiskKV`)\n- **Handles**:\n  - Old Chat mode (workspace storage)\n  - Composer inline storage (v1.x - messages in composerData array)\n  - Composer separate storage (v1.x-v2.0 transition - messages in bubbleId keys)\n  - Latest Composer\u002FAgent (v2.0+)\n- **Includes**:\n  - Code context, selections, diffs\n  - Suggested edits and code blocks\n  - Tool results and execution outputs\n\n### 4. `extract_trae.py`\nExtracts from Trae\n- **Searches**: `~\u002F.trae`, `~\u002FLibrary\u002FApplication Support\u002FTrae`\n- **Formats**: JSONL and SQLite databases\n- **Includes**: Chat, agent data, tool use, diffs\n\n### 5. `extract_windsurf.py`\nExtracts from Windsurf\n- **Searches**: `~\u002FLibrary\u002FApplication Support\u002FWindsurf` or equivalent\n- **Formats**: SQLite databases (VSCode-like format)\n- **Includes**: Chat, agent\u002Fflow conversations, code context\n\n### 6. `extract_continue.py`\nExtracts from Continue AI Assistant\n- **Searches**: `~\u002F.continue\u002Fsessions\u002F`\n- **Formats**: JSON session files\n- **Includes**:\n  - User\u002Fassistant messages\n  - Tool calls and results\n  - Reasoning blocks\n  - Context items\n  - Workspace information\n\n### 7. `extract_gemini.py`\nExtracts from Google Gemini CLI\n- **Searches**: `~\u002F.gemini\u002Ftmp\u002F[hash]\u002Fchats\u002F`\n- **Formats**: JSON session files\n- **Includes**:\n  - User\u002Fassistant messages\n  - Thoughts (reasoning steps with timestamps)\n  - Token usage breakdown\n  - Model information\n  - Project hash and workspace linking\n\n### 8. `extract_opencode.py`\nExtracts from OpenCode (CLI + Desktop)\n- **Searches**: \n  - CLI: `~\u002F.local\u002Fshare\u002Fopencode\u002Fstorage\u002F` (Linux), `~\u002FLibrary\u002FApplication Support\u002Fopencode` (macOS)\n  - Desktop: `~\u002F.local\u002Fshare\u002Fai.opencode.app` (Linux), `~\u002FLibrary\u002FApplication Support\u002Fai.opencode.app` (macOS)\n- **Formats**: JSON files (sessions\u002Fmessages\u002Fparts) and Tauri .dat files (desktop)\n- **Includes**:\n  - User\u002Fassistant messages with full conversation hierarchy\n  - Tool calls and tool results\n  - Code blocks and text content\n  - Token usage and cost tracking\n  - Model and provider information\n  - Agent mode and session metadata\n  - Project directory and version info\n  - Parent\u002Fchild session relationships\n\n## 🚀 Quick Start\n\n### Installation\n\n```bash\n# No dependencies required - uses Python 3 standard library\npython3 --version  # Ensure Python 3.6+ is installed\n```\n\n### Basic Usage\n\n```bash\n# Extract from Claude Code\npython3 extract_claude_code.py\n\n# Extract from Cursor\npython3 extract_cursor.py\n\n# Extract from Codex\npython3 extract_codex.py\n\n# Extract from Trae\npython3 extract_trae.py\n\n# Extract from Windsurf\npython3 extract_windsurf.py\n\n# Extract from Continue\npython3 extract_continue.py\n\n# Extract from Gemini CLI\npython3 extract_gemini.py\n\n# Extract from OpenCode\npython3 extract_opencode.py\n\n# Extract from ALL tools at once\n.\u002Fextract_all.sh\n```\n\n### Output\n\nAll scripts create an `extracted_data\u002F` directory with timestamped JSONL files:\n\n```\nextracted_data\u002F\n├── claude_code_conversations_20250116_143022.jsonl\n├── cursor_complete_20250116_143045.jsonl\n├── gemini_conversations_20250116_143145.jsonl\n├── codex_conversations_20250116_143102.jsonl\n├── trae_conversations_20250116_143115.jsonl\n├── windsurf_conversations_20250116_143130.jsonl\n├── continue_conversations_20250116_143145.jsonl\n└── opencode_conversations_20250116_143200.jsonl\n```\n\n## 📊 Output Format\n\nEach conversation is a single JSON line in JSONL format:\n\n```json\n{\n  \"messages\": [\n    {\n      \"role\": \"user\",\n      \"content\": \"How do I fix this TypeScript error?\",\n      \"code_context\": [\n        {\n          \"file\": \"\u002FUsers\u002Fuser\u002Fproject\u002Fsrc\u002Findex.ts\",\n          \"code\": \"const x: string = 123;\",\n          \"range\": {\n            \"selectionStartLineNumber\": 10,\n            \"positionLineNumber\": 10\n          }\n        }\n      ],\n      \"timestamp\": \"2025-01-16T14:30:22.123Z\"\n    },\n    {\n      \"role\": \"assistant\",\n      \"content\": \"The error occurs because you're assigning a number to a string type...\",\n      \"suggested_diffs\": [...],\n      \"model\": \"claude-sonnet-4-5\",\n      \"timestamp\": \"2025-01-16T14:30:25.456Z\"\n    }\n  ],\n  \"source\": \"cursor-composer\",\n  \"name\": \"TypeScript Type Error Fix\",\n  \"created_at\": 1705414222000\n}\n```\n\n## 🔍 How It Works\n\n### Auto-Discovery Process\n\nEach script follows this pattern:\n\n1. **Detect Operating System** (macOS, Linux, Windows)\n2. **Search Common Locations**:\n   - macOS: `~\u002FLibrary\u002FApplication Support`, `~\u002F.config`, `~\u002F`\n   - Linux: `~\u002F.config`, `~\u002F.local\u002Fshare`, `~\u002F`\n   - Windows: `%APPDATA%`, `%LOCALAPPDATA%`, `~\u002F`\n3. **Find All Installations** of the target tool\n4. **Scan Storage Locations**:\n   - SQLite databases (`.vscdb`, `.db`)\n   - JSONL session files\n   - Project-specific directories\n5. **Extract Complete Data** including context and diffs\n6. **Save to Organized JSONL** with timestamps\n\n### Storage Formats Handled\n\n#### Claude Code \u002F Codex\n- **Format**: JSONL files (one event per line)\n- **Location**: `~\u002F.claude\u002Fprojects\u002F[project]\u002F[session].jsonl`\n- **Structure**: Event-based with type markers\n\n#### Cursor (v0.43 - v2.0+)\n- **Format**: SQLite databases\n- **Locations**:\n  - Workspace: `~\u002FLibrary\u002FApplication Support\u002FCursor\u002FUser\u002FworkspaceStorage\u002F[hash]\u002Fstate.vscdb`\n  - Global: `~\u002FLibrary\u002FApplication Support\u002FCursor\u002FUser\u002FglobalStorage\u002Fstate.vscdb`\n- **Tables**: `ItemTable` (Chat), `cursorDiskKV` (Composer\u002FAgent)\n- **Storage Evolution**:\n  - **v0.x - v1.x**: Chat mode in workspace `ItemTable`\n  - **v1.x**: Composer inline (messages in `composerData.conversation[]`)\n  - **v1.x - v2.0 transition**: Composer separate (messages in `bubbleId:{composer}:{bubble}` keys)\n  - **v2.0+**: Latest format with enhanced metadata\n- **Keys**:\n  - `workbench.panel.aichat.view.aichat.chatdata` (Chat mode)\n  - `composerData:{uuid}` (Composer metadata + conversation)\n  - `bubbleId:{composer}:{bubble}` (Individual messages - transitional format)\n  - `codeBlockDiff:{id}` (Code block diffs)\n\n#### Trae \u002F Windsurf\n- **Format**: Hybrid (JSONL + SQLite)\n- **Location**: Similar to VSCode\u002FCursor structure\n- **Structure**: VSCode extension data format\n\n## 🎓 Understanding the Data\n\n### Message Roles\n- `user`: Human developer messages\n- `assistant`: AI assistant responses\n\n### Code Context Fields\n- `code_context`: File selections and code snippets\n- `suggested_diffs`: AI-proposed code changes\n- `tool_use`: Code execution, file operations\n- `tool_results`: Execution outputs, diffs applied\n- `diff_histories`: Full edit history\n\n### Metadata Fields\n- `source`: Which tool (e.g., \"cursor-composer\", \"claude-code\")\n- `session_id`\u002F`composer_id`: Unique conversation ID\n- `project_path`: Working directory\n- `timestamp`: Message time\n- `model`: AI model used (if available)\n\n## 🔧 Advanced Usage\n\n### Merge All Extractions\n\n```bash\n# Combine all JSONL files\ncat extracted_data\u002F*.jsonl > all_conversations.jsonl\n\n# Count total conversations\nwc -l all_conversations.jsonl\n\n# Count by source\ngrep -o '\"source\":\"[^\"]*\"' all_conversations.jsonl | sort | uniq -c\n```\n\n### Filter by Date\n\n```python\nimport json\nfrom datetime import datetime\n\nwith open('extracted_data\u002Fcursor_complete_20250116.jsonl') as f:\n    for line in f:\n        conv = json.loads(line)\n        created = conv.get('created_at', 0)\n        if created > 1704067200000:  # After Jan 1, 2024\n            print(json.dumps(conv))\n```\n\n### Extract Only Conversations with Diffs\n\n```python\nimport json\n\nwith open('extracted_data\u002Fcursor_complete.jsonl') as f:\n    for line in f:\n        conv = json.loads(line)\n        if any('suggested_diffs' in m or 'diff_histories' in m\n               for m in conv['messages']):\n            print(json.dumps(conv))\n```\n\n## 📋 Data Quality\n\n### What Gets Extracted\n\n✅ **Complete Conversations**:\n- Both user prompts AND AI responses\n- Multi-turn dialogues\n- Full conversation context\n\n✅ **Code Context**:\n- File paths and names\n- Selected code snippets\n- Line number ranges\n- Multi-file selections\n\n✅ **Diffs and Edits**:\n- Suggested code changes\n- Applied diffs\n- Edit histories\n- File modifications\n\n✅ **Metadata**:\n- Timestamps\n- Project paths\n- Model information\n- Conversation names\n\n### What Might Be Missing\n\n⚠️ **Partial Data**:\n- Conversations without AI responses (user-only)\n- Deleted or archived sessions\n- Corrupted database entries\n\n⚠️ **Privacy Considerations**:\n- May include proprietary code\n- May include API keys\u002Fsecrets\n- May include personal file paths\n\n## 🛡️ Privacy & Security\n\n### Before Using Extracted Data\n\n1. **Scan for Secrets**:\n```bash\npip install detect-secrets\ndetect-secrets scan extracted_data\u002F*.jsonl\n```\n\n2. **Review Sensitive Data**:\n- Check for API keys, passwords, tokens\n- Verify no proprietary code exposed\n- Sanitize file paths if needed\n\n3. **Storage**:\n- Keep on encrypted drives\n- Don't commit to public repositories\n- Secure backups recommended\n\n## 🎯 Training Use Cases\n\n### Direct Fine-Tuning\n\n```python\nfrom datasets import load_dataset\n\ndataset = load_dataset(\n    'json',\n    data_files='extracted_data\u002F*.jsonl',\n    split='train'\n)\n\n# Filter complete conversations\ndataset = dataset.filter(\n    lambda x: any(m['role'] == 'assistant' for m in x['messages'])\n)\n```\n\n### With Unsloth\n\n```python\nfrom unsloth import FastLanguageModel\n\nmodel, tokenizer = FastLanguageModel.from_pretrained(\n    \"unsloth\u002Fqwen2.5-coder-7b-instruct\",\n    max_seq_length=4096,\n    load_in_4bit=True,\n)\n\ndef format_chat(example):\n    return {\n        'text': tokenizer.apply_chat_template(\n            example['messages'],\n            tokenize=False\n        )\n    }\n\ndataset = dataset.map(format_chat)\n```\n\n## 🐛 Troubleshooting\n\n### No installations found\n\n**Problem**: Script reports \"No installations found\"\n\n**Solutions**:\n1. Check if the tool is actually installed\n2. Verify installation location manually\n3. Add custom path to script:\n```python\n# Add to find_XXX_installations() function\nlocations.append(Path(\"\u002Fcustom\u002Fpath\u002Fto\u002Ftool\"))\n```\n\n### Empty extracted_data directory\n\n**Problem**: Extraction completes but no data found\n\n**Solutions**:\n1. Verify you've actually used the tool and have chat history\n2. Check if data is in a non-standard location\n3. Look for database files manually:\n```bash\nfind ~ -name \"*.vscdb\" -o -name \"*.db\" 2>\u002Fdev\u002Fnull\n```\n\n### Database locked errors\n\n**Problem**: SQLite database is locked\n\n**Solutions**:\n1. Close the AI tool before running extraction\n2. Use read-only mode:\n```python\nconn = sqlite3.connect(f'file:{db_path}?mode=ro', uri=True)\n```\n\n### Permission denied\n\n**Problem**: Cannot read certain files\n\n**Solutions**:\n1. Run with appropriate permissions\n2. Check file ownership\n3. Copy databases to accessible location first\n\n## 📚 Platform-Specific Notes\n\n### macOS\n- Uses `~\u002FLibrary\u002FApplication Support` for most tools\n- May need Full Disk Access for system directories\n- SQLite databases typically in `~\u002FLibrary\u002FApplication Support\u002F[Tool]\u002FUser\u002F`\n\n### Linux\n- Uses `~\u002F.config` and `~\u002F.local\u002Fshare`\n- Check `~\u002F.local\u002Fstate` for some tools\n- May use `$XDG_CONFIG_HOME` if set\n\n### Windows\n- Uses `%APPDATA%` and `%LOCALAPPDATA%`\n- Paths: `C:\\Users\\[User]\\AppData\\Roaming\\[Tool]`\n- May need admin privileges for Program Files\n\n## 🔄 Version Compatibility\n\n### Cursor\n- ✅ v2 (0.43+): Composer\u002FAgent in `cursorDiskKV`\n- ✅ v1: Chat in workspace `ItemTable`\n- ⚠️ Pre-v0.43: Different format, limited support\n\n### Claude Code\n- ✅ All versions with JSONL session files\n- ✅ Project-based structure\n\n### Codex\n- ✅ Rollout JSONL format\n- ✅ Time-based session organization\n\n## 📈 Performance Tips\n\n### Large Datasets\n```bash\n# Process in chunks\nsplit -l 1000 all_conversations.jsonl chunk_\n\n# Compress for storage\ngzip extracted_data\u002F*.jsonl\n```\n\n### Speed Optimization\n```python\n# Use multiprocessing for large scans\nfrom multiprocessing import Pool\n\nwith Pool() as pool:\n    results = pool.map(extract_from_db, db_files)\n```\n\n## 🤝 Contributing\n\nFound a new storage format or tool? Contributions welcome!\n\n1. Follow existing script structure\n2. Add auto-discovery logic\n3. Extract complete data (messages + context + diffs)\n4. Output to organized JSONL\n5. Update this README\n\n## 📄 License\n\nMIT License - Use freely for training ML models\n\n## ⚠️ Disclaimer\n\nThis toolkit extracts YOUR OWN data from locally installed AI tools. Users are responsible for:\n- Ensuring they have rights to extracted data\n- Handling sensitive\u002Fproprietary information appropriately\n- Complying with tool Terms of Service\n- Scanning for secrets before sharing\u002Ftraining\n\n---\n\n**Generated**: January 16, 2025\n**Status**: Production Ready\n**Compatibility**: Python 3.6+, macOS\u002FLinux\u002FWindows\n","这是一个面向AI编程助手用户的数据提取工具包，用于批量导出个人在Cursor、Codex、Claude Code、Windsurf、Trae、Continue、Gemini CLI和OpenCode等工具中产生的完整交互历史数据。核心功能包括自动定位本地存储路径（如SQLite数据库、JSONL日志、Tauri .dat文件等），提取对话消息、代码上下文（含文件路径、行号、diff）、工具调用结果、执行输出及元数据（时间戳、模型信息、工作区标识）。技术上支持多格式解析与跨平台路径适配，输出结构化数据便于后续清洗与建模。适用于个人数据归档、私有模型微调训练数据准备、AI行为分析及编码习惯研究等场景。",2,"2026-08-17 02:30:10","trending"]