[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"project-73934":3},{"id":4,"name":5,"fullName":6,"owner":7,"repo":5,"description":8,"homepage":9,"htmlUrl":10,"language":11,"languages":10,"totalLinesOfCode":10,"stars":12,"forks":13,"watchers":14,"openIssues":15,"contributorsCount":16,"subscribersCount":16,"size":16,"stars1d":17,"stars7d":18,"stars30d":19,"stars90d":16,"forks30d":16,"starsTrendScore":20,"compositeScore":21,"rankGlobal":10,"rankLanguage":10,"license":22,"archived":23,"fork":23,"defaultBranch":24,"hasWiki":25,"hasPages":23,"topics":26,"createdAt":10,"pushedAt":10,"updatedAt":38,"readmeContent":39,"aiSummary":40,"trendingCount":16,"starSnapshotCount":16,"syncStatus":17,"lastSyncTime":41,"discoverSource":42},73934,"DevDocs","cyberagiinc\u002FDevDocs","cyberagiinc","Completely free, private, UI based Tech Documentation MCP server. Designed for coders and software developers in mind. Easily integrate into Cursor, Windsurf, Cline, Roo Code, Claude Desktop App ","https:\u002F\u002Fwww.cyberagi.ai",null,"TypeScript",2088,192,11,10,0,2,7,22,6,68.56,"Apache License 2.0",false,"feature-main",true,[27,28,29,30,31,32,33,34,35,36,37],"cline","crawl4ai","cursor","documentation","llm","playwright","python3","scraper","security","typescript","windsurf","2026-06-12 04:01:12","\n# DevDocs by CyberAGI 🚀\n> [!WARNING]\n> 📌 **DevDocs Status**: Not publicly maintained. Enhanced internal version at CyberAGI — public release coming soon. If you have any questions please reach out to info@cyberagi.ai\n\n\u003Cdiv align=\"center\">\n  \u003Cimg src=\"assets\u002Fimage.png\" alt=\"DevDocs Interface\" width=\"800\">\n\n\n  \u003Cp align=\"center\">\n    \u003Cstrong>Turn Weeks of Documentation Research into Hours of Productive Development\u003C\u002Fstrong>\n  \u003C\u002Fp>\n\n  \u003Cp align=\"center\">\n    \u003Ca href=\"#-perfect-for\">Perfect For\u003C\u002Fa> •\n    \u003Ca href=\"#-features\">Features\u003C\u002Fa> •\n    \u003Ca href=\"#-why-devdocs\">Why DevDocs\u003C\u002Fa> •\n    \u003Ca href=\"#-getting-started\">Getting Started\u003C\u002Fa> •\n    \u003Ca href=\"#-scripts-and-their-purpose\">Scripts\u003C\u002Fa> •\n    \u003Ca href=\"#-pricing-comparison\">Compare to FireCrawl\u003C\u002Fa> •\n    \u003Ca href=\"#-join-our-community\">Discord\u003C\u002Fa> •\n    \u003Ca href=\"#-devdocs-roadmap\">DevDocs Roadmap\u003C\u002Fa>\n  \u003C\u002Fp>\n\u003C\u002Fdiv>\n\n## 🚀 Technology Partners\n\n\u003Cdiv align=\"center\" style=\"display: flex; justify-content: center; align-items: center; gap: 30px; flex-wrap: wrap; padding: 20px 0;\">\n\n  \u003Cimg src=\"https:\u002F\u002Fcdn.brandfetch.io\u002FidmJWF3N06\u002Ftheme\u002Flight\u002Fsymbol.svg?c=1dxbfHSJFAPEGdCLU4o5B\" alt=\"Anthropic\" style=\"height: 60px; max-width: 150px; object-fit: contain;\" \u002F>\n  \n  \u003Cimg src=\"https:\u002F\u002Fcdn.brandfetch.io\u002FidR3duQxYl\u002Fw\u002F800\u002Fh\u002F810\u002Ftheme\u002Flight\u002Fsymbol.png?c=1dxbfHSJFAPEGdCLU4o5B\" alt=\"OpenAI\" style=\"height: 60px; max-width: 150px; object-fit: contain;\" \u002F>\n  \n  \u003Cimg src=\"assets\u002Fimage-8.png\" alt=\"CyberAGI\" style=\"height: 60px; max-width: 150px; object-fit: contain;\" \u002F>\n  \n  \u003Cimg src=\"https:\u002F\u002Fraw.githubusercontent.com\u002Funclecode\u002Fcrawl4ai\u002Fmain\u002Fdocs\u002Fassets\u002Fpowered-by-disco.svg\" alt=\"Crawl4AI\" style=\"height: 60px; max-width: 150px; object-fit: contain;\" \u002F>\n\n\u003C\u002Fdiv>\n\n## 🎯 Perfect For\n\n### 🏢 Enterprise Software Developers\nSkip weeks of reading documentation and dealing with technical debt. Implement ANY technology faster by letting DevDocs handle the heavy lifting of documentation understanding.\n\n### 🕸️ Web Scrapers\nPull entire contents of websites with Smart Discovery of Child URLs up to level 5. Perfect for both internal and external website documentation with intelligent crawling.\n\n### 👥 Development Teams\nLeverage internal documentation with built-in MCP servers and Claude integration for intelligent data querying. Transform your team's knowledge base into an actionable resource.\n\n### 🚀 Indie Hackers\nDevDocs + VSCode(cline) + Your Idea = Ship products fast with ANY technology. No more getting stuck in documentation hell when building your next big thing.\n\n## ✨ Features\n\n### 🧠 Intelligent Crawling\n- **Smart Depth Control**: Choose crawl depth from 1-5 levels\n- **Automatic Link Discovery**: Finds and categorizes all related content\n- **Selective Crawling**: Pick exactly what you want to extract\n- **Child URL Detection**: Automatically discovers and maps website structure\n\n### ⚡ Performance & Speed\n- **Parallel Processing**: Crawl multiple pages simultaneously\n- **Smart Caching**: Never waste time on duplicate content\n- **Lazy Loading Support**: Handles modern web apps effortlessly\n- **Rate Limiting**: Respectful crawling that won't overload servers\n\n### 🎯 Content Processing\n- **Clean Extraction**: Get content without the fluff\n- **Multiple Formats**: Export to MD or JSON for LLM fine-tuning\n- **Structured Output**: Logically organized content\n- **MCP Server Integration**: Ready for AI processing\n\n### 🛡️ Enterprise Features\n- **Error Recovery**: Auto-retry on failures\n- **Full Logging**: Track every operation\n- **API Access**: Integrate with your tools\n- **Team Management**: Multiple seats and roles\n\n## 🤔 Why DevDocs?\n\n### The Problem\nDocumentation is everywhere and LLMs are OUTDATED in their knowledge. Reading it, understanding it, and implementing it takes weeks of research and development even for senior engineers. **We cut down that time to hours.**\n\n### Our Solution\nDevDocs brings documentation to you. Point it at any tech documentation URL, and watch as it:\n1. Discovers all related pages to that technology\n2. Extracts meaningful content without the fluff\n3. Organizes information logically inside an MCP server ready for your LLM to query\n4. Presents it in a clean, searchable format in MD or JSON for finetuning LLM purpose\n\n🔥 We want anyone in the world to have the ability to build amazing products quickly using the most cutting edge LLM technology. \n\n## 💰 Pricing Comparison\n\n| Feature | DevDocs | Firecrawl |\n|---------|---------|-----------|\n| Free Tier | Unlimited pages | None |\n| Starting Price | Free Forever | $16\u002Fmonth |\n| Enterprise Plan | Custom | $333\u002Fmonth |\n| Crawl Speed | 1000\u002Fmin | 20\u002Fmin |\n| Depth Levels | Up to 5 | Limited |\n| Team Seats | Unlimited | 1-5 seats |\n| Export Formats | MD, JSON, LLM-ready MCP servers | Limited formats |\n| API Access | Coming Soon | Limited |\n| Model Context Protocol Integration | ✅ | ❌ |\n| Support | Priority Available via Discord | Standard only |\n| Self-hosted (free use) | ✅ | ❌ |\n\n## 🚀 Getting Started\n\nDevDocs is designed to be easy to use with Docker, requiring minimal setup for new users.\n\n### Prerequisites\n\n- [Docker](https:\u002F\u002Fdocs.docker.com\u002Fget-docker\u002F) installed on your system\n- Git for cloning the repository\n\n### Quick Start with Docker (Recommended)\n\nFor Mac\u002FLinux users:\n```bash\n# Clone the repository\ngit clone https:\u002F\u002Fgithub.com\u002Fcyberagiinc\u002FDevDocs.git\n\n# Navigate to the project directory\ncd DevDocs\n\n# Configure environment variables\n# Copy the template file to .env\ncp .env.template .env\n\n# Ensure NEXT_PUBLIC_BACKEND_URL in .env is set correctly (e.g., http:\u002F\u002Flocalhost:24125)\n# This allows the frontend (running in your browser) to communicate with the backend service.\n\n\n# Start all services using Docker\n.\u002Fdocker-start.sh\n```\n\nFor Windows users: Experimental Only (Not Tested Yet)\n```cmd\n# Clone the repository\ngit clone https:\u002F\u002Fgithub.com\u002Fcyberagiinc\u002FDevDocs.git\n\n# Navigate to the project directory\n\ncd DevDocs\n\n# Configure environment variables\n# Copy the template file to .env\n\ncopy .env.template .env\n\n# Ensure NEXT_PUBLIC_BACKEND_URL in .env is set correctly (e.g., http:\u002F\u002Flocalhost:24125)\n\n# This allows the frontend (running in your browser) to communicate with the backend service.\n\n# Prerequisites: Install WSL 2 and Docker Desktop\n# Docker Desktop for Windows requires WSL 2. Please ensure you have WSL 2 installed and running first.\n# 1. Install WSL 2: Follow the official Microsoft guide: https:\u002F\u002Flearn.microsoft.com\u002Fen-us\u002Fwindows\u002Fwsl\u002Finstall\n# 2. Install Docker Desktop for Windows: Download and install from the official Docker website. Docker Desktop includes Docker Compose.\n\n\n\n# Start all services using Docker\ndocker-start.bat\n```\n\u003Cdetails>\n\u003Csummary>Note for Windows Users\u003C\u002Fsummary>\n\n> If you encounter permission issues, you may need to run the script as administrator or manually set permissions on the logs, storage, and crawl_results directories. The script uses the `icacls` command to set permissions, which might require elevated privileges on some Windows systems.\n>\n> **Manually Setting Permissions on Windows**:\n>\n> If you need to manually set permissions, you can do so using either the Windows GUI or command line:\n>\n> **Using Windows Explorer**:\n> 1. Right-click on each directory (logs, storage, crawl_results)\n> 2. Select \"Properties\"\n> 3. Go to the \"Security\" tab\n> 4. Click \"Edit\" to change permissions\n> 5. Click \"Add\" to add users\u002Fgroups\n> 6. Type \"Everyone\" and click \"Check Names\"\n> 7. Click \"OK\"\n> 8. Select \"Everyone\" in the list\n> 9. Check \"Full control\" under \"Allow\"\n> 10. Click \"Apply\" and \"OK\"\n>\n> **Using Command Prompt (as Administrator)**:\n> ```cmd\n> icacls logs \u002Fgrant Everyone:F \u002FT\n> icacls storage \u002Fgrant Everyone:F \u002FT\n> icacls crawl_results \u002Fgrant Everyone:F \u002FT\n> ```\n\u003C\u002Fdetails> \n\n\u003Cdetails>\n\u003Csummary>Note about docker-compose.yml on Windows\u003C\u002Fsummary>\n\n> If you encounter issues with the docker-compose.yml file (such as \"Top-level object must be a mapping\" error), the `docker-start.bat` script automatically fixes this by ensuring the file has the correct format and encoding. This fix is applied every time you run the script, so you don't need to manually modify the file.\n\u003C\u002Fdetails>\n\n\n\nThis single command will:\n1. Create all necessary directories\n2. Set appropriate permissions\n3. Build and start all Docker containers\n4. Monitor the services to ensure they're running properly\n\n### Accessing DevDocs\n\nOnce the services are running:\n- Frontend UI: http:\u002F\u002Flocalhost:3001\n- Backend API: http:\u002F\u002Flocalhost:24125\n- Crawl4AI Service: http:\u002F\u002Flocalhost:11235\n\n### Logs and Monitoring\n\nWhen using Docker, logs can be accessed :\n\n1. **Container Logs** (recommended for debugging):\n   ```bash\n   # View logs from a specific container\n   docker logs devdocs-frontend\n   docker logs devdocs-backend\n   docker logs devdocs-mcp\n   docker logs devdocs-crawl4ai\n   \n   # Follow logs in real-time\n   docker logs -f devdocs-backend\n   ```\n\nTo stop all services, press `Ctrl+C` in the terminal where docker-start is running.\n\n## 📜 Scripts and Their Purpose\n\nDevDocs includes various utility scripts to help with development, testing, and maintenance. Here's a quick reference:\n\n### Startup Scripts\n- `start.sh` \u002F `start.bat` \u002F `start.ps1` - Start all services (frontend, backend, MCP) for local development.\n- `docker-start.sh` \u002F `docker-start.bat` - Start all services using Docker containers.\n\n### MCP Server Scripts\n- `check_mcp_health.sh` - Verify the MCP server's health and configuration status.\n- `restart_and_test_mcp.sh` - Restart Docker containers with updated MCP configuration and test connectivity.\n\n### Crawl4AI Scripts\n- `check_crawl4ai.sh` - Check the status and health of the Crawl4AI service.\n- `debug_crawl4ai.sh` - Run Crawl4AI in debug mode with verbose logging for troubleshooting.\n- `test_crawl4ai.py` - Run tests against the Crawl4AI service to verify functionality.\n- `test_from_container.sh` - Test the Crawl4AI service from within a Docker container.\n\n### Utility Scripts\n- `view_result.sh` - Display crawl results in a formatted view.\n- `find_empty_folders.sh` - Identify empty directories in the project structure.\n- `analyze_empty_folders.sh` - Analyze empty folders and categorize them by risk level.\n- `verify_reorganization.sh` - Verify that code reorganization was successful.\n\nThese scripts are organized in the following directories:\n- Root directory: Main scripts for common operations\n- `scripts\u002Fgeneral\u002F`: General utility scripts\n- `scripts\u002Fdocker\u002F`: Docker-specific scripts\n- `scripts\u002Fmcp\u002F`: MCP server management scripts\n- `scripts\u002Ftest\u002F`: Testing and verification scripts\n\n## 🌍 Built for Developers, by Developers\n\nDevDocs is more than a tool—it's your documentation companion that:\n- **Saves Time**: Turn weeks of research into hours\n- **Improves Understanding**: Get clean, organized documentation\n- **Enables Innovation**: Build faster with any technology\n- **Supports Teams**: Share knowledge efficiently\n- **LLM READY**: Modern times require modern solutions, using devdocs with LLM is extremely easy and intuitive. With minimal configuration you can run Devdocs and Claude App and  recognizes DevDocs's MCP server ready to chat with your data. \n\n## 🛠️ Setting Up the Cline\u002FRoo Cline for Rapid software development.\n\n1. **Open the \"Modes\" Interface**  \n   - In **Roo Code**, click the **+** to create a new Mode-Specific Prompts.\n   \u003Cbr>\n   \n2. **Name**  \n   - Give the mode a **Name** (e.g., `Research_MCP`).  \n   \u003Cbr>\n3. **Role Definition Prompt**\n  \u003Cdetails>\n  \u003Csummary>Prompt\u003C\u002Fsummary>\n\n```\nExpertise and Personality: Expertise: Developer documentation retrieval, technical synthesis, and documentation search. Personality: Systematic, detail-oriented, and precise. Provide well-structured answers with clear references to documentation sections.\n\nBehavioral Mandate: Always use the Table Of Contents and Section Access tools when addressing any query regarding the MCP documentation. Maintain clarity, accuracy, and traceability in your responses.\n```\n  \u003C\u002Fdetails>\n \u003Cbr>\n\n4. **Mode-Specific Custom Instructions Prompt**\n\u003Cdetails>\n\u003Csummary> Prompt \u003C\u002Fsummary>\n\n\n```\n1. Table Of Contents Tool: Returns a full or filtered list of documentation topics. \n2. Section Access Tool: Retrieves the detailed content of specific documentation sections.\n\nGeneral Process: Query Interpretation: Parse the user's query to extract key topics, keywords, and context. Identify the likely relevant sections (e.g., API configurations, error handling) from the query.\n\nDiscovery via Table Of Contents: Use the Table Of Contents tool to search the documentation index for relevant sections. Filter or scan titles and metadata for matching keywords.\n\nDrill-Down Using Section Access: For each identified relevant document or section, use the Section Access tool to retrieve its content. If multiple parts are needed, request all related sections to ensure comprehensive coverage.\n\nSynthesis and Response Formation: Combine the retrieved content into a coherent and complete answer. Reference section identifiers or document paths for traceability. Validate that every aspect of the query has been addressed.\n\nError Handling: If no matching sections are found, adjust the search parameters and retry. Clearly report if the query remains ambiguous or if no relevant documentation is available.\n\nMandatory Tool Usage: \nEnforcement: Every time a query is received that requires information from the MCP server docs, the agent MUST first query the Table Of Contents tool to list potential relevant topics, then use the Section Access tool to retrieve the necessary detailed content.\n\nSearch & Retrieve Workflow: \nInterpret and Isolate: Identify the key terms and data points from the user's query.\n\nIndex Lookup: Immediately query the Table Of Contents tool to obtain a list of relevant documentation sections.\n\nTargeted Retrieval: For each promising section, use the Section Access tool to get complete content.\n\nInformation Synthesis: Merge the retrieved content, ensuring all necessary details are included and clearly referenced.\n\nFallback and Clarification: If initial searches yield insufficient data, adjust the query parameters and retrieve additional sections as needed.\n\nCustom Instruction Loading: Additional custom instructions specific to Research_MCP mode may be loaded from the .clinerules-research-mcp file in your workspace. These may include further refinements or constraints based on evolving documentation structures or query types.\n\nFinal Output Construction: The final answer should be organized, directly address the query, and include clear pointers (e.g., section names or identifiers) back to the MCP documentation. Ensure minimal redundancy while covering all necessary details.\n```\n\n\u003C\u002Fdetails>\n \u003Cbr>\n\n## 🤝 Join Our Community\n\n- 🌟 [Star us on GitHub](https:\u002F\u002Fgithub.com\u002Fcyberagi\u002Fdevdocs)\n- 👋🏽 [Reach out to our founder on Linkedin](https:\u002F\u002Fwww.linkedin.com\u002Fin\u002Fshubhamkhichi\u002F)\n- 💬 [Join our Discord Community](https:\u002F\u002Fdiscord.gg\u002F2594NueRg8)\n\n## 🏆 Success Stories\n\n\"DevDocs turned our 3-week implementation timeline into 2 days. It's not just a crawler, it's a development accelerator.\" - *Senior Engineer at Fortune 100 Company*\n\n\"Launched my SaaS in half the time by using DevDocs to understand and implement new technologies quickly.\" - *Successful Indie Hacker*\n\n## 🛣️ DevDocs Roadmap\n\nThis roadmap outlines the upcoming enhancements and features planned for DevDocs, our advanced web crawling platform powered by Crawl4AI. Each item is designed to leverage Crawl4AI’s capabilities to their fullest, ensuring a robust, efficient, and user-friendly web crawling experience.\n\n⸻\n\n### 1. Enhanced Crawler Logic for Dynamic Content\n\n- Implement `wait_for_images=True` to ensure all images are fully loaded before extraction.\n- Set `scan_full_page=True` to force the crawler to scroll through the entire page, triggering lazy-loaded content.\n- Introduce `scroll_delay` to add delays between scroll steps, allowing content to load properly.\n- Incorporate `wait_for` parameters to wait for specific DOM elements indicative of content loading completion.\n\n---\n\n### 2. Hot Loading with Browser Pooling\n\n- Implement a pool of pre-warmed browser instances to avoid the overhead of launching a new browser for each task.\n- Utilize `use_persistent_context=True` to maintain session data across tasks, reducing the need for repeated logins and setups.\n\n---\n\n### 3. Revamped Docker Containers with Latest DevDocs Integration\n\n- Update Docker images to incorporate the latest DevDocs features and optimizations.\n- Include environment variables for API tokens (`CRAWL4AI_API_TOKEN`) to secure API endpoints.\n- Set appropriate memory limits and resource constraints to optimize performance.\n\n---\n\n### 4. Multi-OS Docker Instance Support\n\n- Create Docker images for different architectures (e.g., `x86_64`, `ARM`) to support a wide range of systems.\n- Implement CI\u002FCD pipelines to build and test images across multiple OS environments, ensuring compatibility and stability.\n\n---\n\n### 5. Memory-Adaptive Crawling\n\n- Integrate DevDocs’ `MemoryAdaptiveDispatcher` to dynamically adjust concurrency based on system memory availability.\n- Implement built-in rate limiting to prevent overwhelming target websites and avoid out-of-memory errors.\n\n---\n\n### 6. PDF Upload and Extraction in UI\n\n- Utilize DevDocs’ capability to export pages as PDFs (`pdf=True`) and extract content from them.\n- Develop frontend components to handle PDF uploads, display extracted content, and allow users to interact with the data.\n\n---\n\n### 7. Hosted Environment with Persistent Storage and Enhanced UX\n\n- Implement BYO-databases solutions to store data privately, crawl results, and configurations across sessions.\n- Design intuitive dashboards and interfaces for users to manage their crawls, view results, and configure settings.\n- Ensure responsive design and accessibility across various browsers.\n\n## Star History\n\n[![Star History Chart](https:\u002F\u002Fapi.star-history.com\u002Fsvg?repos=cyberagiinc\u002FDevDocs&type=Timeline)](https:\u002F\u002Fstar-history.com\u002F#cyberagiinc\u002FDevDocs&Timeline)\n\n\u003Cp align=\"center\">Made with ❤️ by \u003Ca href=\"https:\u002F\u002Fwww.cyberagi.ai\">CyberAGI Inc\u003C\u002Fa> in 🇺🇸\u003C\u002Fp>\n\n\u003Cp align=\"center\">\n  \u003Csub>Make Software Development Better Again \u003Ca href=\"https:\u002F\u002Fgithub.com\u002Fcyberagi\u002Fdevdocs\">Contribute to DevDocs\u003C\u002Fa>\u003C\u002Fsub>\n\u003C\u002Fp>\n\n","DevDocs 是一个专为开发者设计的免费且私有的基于UI的技术文档MCP服务器。它采用TypeScript编写，具备智能爬虫功能，支持自动链接发现、选择性抓取和子URL检测等特性，能够高效地从网站提取技术文档内容。此外，DevDocs还支持与Cursor、Windsurf等多种开发工具集成，并通过Claude桌面应用程序提供智能数据查询服务。该项目非常适合企业软件开发者快速掌握新技术、Web爬虫专家自动化收集网站信息、开发团队构建内部知识库以及独立开发者加速产品开发流程中的文档查阅过程。","2026-06-11 03:47:58","high_star"]