[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"project-474":3},{"id":4,"name":5,"fullName":6,"owner":7,"repo":5,"description":8,"homepage":9,"htmlUrl":10,"language":11,"languages":10,"totalLinesOfCode":10,"stars":12,"forks":13,"watchers":14,"openIssues":15,"contributorsCount":16,"subscribersCount":16,"size":16,"stars1d":17,"stars7d":18,"stars30d":19,"stars90d":16,"forks30d":16,"starsTrendScore":20,"compositeScore":21,"rankGlobal":10,"rankLanguage":10,"license":22,"archived":23,"fork":23,"defaultBranch":24,"hasWiki":25,"hasPages":23,"topics":26,"createdAt":10,"pushedAt":10,"updatedAt":27,"readmeContent":28,"aiSummary":29,"trendingCount":16,"starSnapshotCount":16,"syncStatus":30,"lastSyncTime":31,"discoverSource":32},474,"crawl4ai","unclecode\u002Fcrawl4ai","unclecode","🚀🤖 Crawl4AI: Open-source LLM Friendly Web Crawler & Scraper. Don't be shy, join here: https:\u002F\u002Fdiscord.gg\u002FjP8KfhDhyN","https:\u002F\u002Fcrawl4ai.com",null,"Python",68276,6972,374,21,0,67,501,2884,325,45,"Apache License 2.0",false,"main",true,[],"2026-06-12 02:00:14","# 🚀🤖 Crawl4AI: Open-source LLM Friendly Web Crawler & Scraper.\n\n\u003Cdiv align=\"center\">\n\n\u003Ca href=\"https:\u002F\u002Ftrendshift.io\u002Frepositories\u002F11716\" target=\"_blank\">\u003Cimg src=\"https:\u002F\u002Ftrendshift.io\u002Fapi\u002Fbadge\u002Frepositories\u002F11716\" alt=\"unclecode%2Fcrawl4ai | Trendshift\" style=\"width: 250px; height: 55px;\" width=\"250\" height=\"55\"\u002F>\u003C\u002Fa>\n\n[![GitHub Stars](https:\u002F\u002Fimg.shields.io\u002Fgithub\u002Fstars\u002Funclecode\u002Fcrawl4ai?style=social)](https:\u002F\u002Fgithub.com\u002Funclecode\u002Fcrawl4ai\u002Fstargazers)\n[![GitHub Forks](https:\u002F\u002Fimg.shields.io\u002Fgithub\u002Fforks\u002Funclecode\u002Fcrawl4ai?style=social)](https:\u002F\u002Fgithub.com\u002Funclecode\u002Fcrawl4ai\u002Fnetwork\u002Fmembers)\n\n[![PyPI version](https:\u002F\u002Fbadge.fury.io\u002Fpy\u002Fcrawl4ai.svg)](https:\u002F\u002Fbadge.fury.io\u002Fpy\u002Fcrawl4ai)\n[![Python Version](https:\u002F\u002Fimg.shields.io\u002Fpypi\u002Fpyversions\u002Fcrawl4ai)](https:\u002F\u002Fpypi.org\u002Fproject\u002Fcrawl4ai\u002F)\n[![Downloads](https:\u002F\u002Fstatic.pepy.tech\u002Fbadge\u002Fcrawl4ai\u002Fmonth)](https:\u002F\u002Fpepy.tech\u002Fproject\u002Fcrawl4ai)\n[![GitHub Sponsors](https:\u002F\u002Fimg.shields.io\u002Fgithub\u002Fsponsors\u002Funclecode?style=flat&logo=GitHub-Sponsors&label=Sponsors&color=pink)](https:\u002F\u002Fgithub.com\u002Fsponsors\u002Funclecode)\n\n---\n#### 🚀 Crawl4AI Cloud API — Closed Beta (Launching Soon)\nReliable, large-scale web extraction, now built to be _**drastically more cost-effective**_ than any of the existing solutions.\n\n👉 **Apply [here](https:\u002F\u002Fforms.gle\u002FE9MyPaNXACnAMaqG7) for early access**  \n_We’ll be onboarding in phases and working closely with early users.\nLimited slots._\n\n---\n\n\u003Cp align=\"center\">\n    \u003Ca href=\"https:\u002F\u002Fx.com\u002Fcrawl4ai\">\n      \u003Cimg src=\"https:\u002F\u002Fimg.shields.io\u002Fbadge\u002FFollow%20on%20X-000000?style=for-the-badge&logo=x&logoColor=white\" alt=\"Follow on X\" \u002F>\n    \u003C\u002Fa>\n    \u003Ca href=\"https:\u002F\u002Fwww.linkedin.com\u002Fcompany\u002Fcrawl4ai\">\n      \u003Cimg src=\"https:\u002F\u002Fimg.shields.io\u002Fbadge\u002FFollow%20on%20LinkedIn-0077B5?style=for-the-badge&logo=linkedin&logoColor=white\" alt=\"Follow on LinkedIn\" \u002F>\n    \u003C\u002Fa>\n    \u003Ca href=\"https:\u002F\u002Fdiscord.gg\u002FjP8KfhDhyN\">\n      \u003Cimg src=\"https:\u002F\u002Fimg.shields.io\u002Fbadge\u002FJoin%20our%20Discord-5865F2?style=for-the-badge&logo=discord&logoColor=white\" alt=\"Join our Discord\" \u002F>\n    \u003C\u002Fa>\n  \u003C\u002Fp>\n\u003C\u002Fdiv>\n\nCrawl4AI turns the web into clean, LLM ready Markdown for RAG, agents, and data pipelines. Fast, controllable, battle tested by a 50k+ star community.\n\n[✨ Check out latest update v0.8.6](#-recent-updates)\n\n✨ **New in v0.8.6**: Security hotfix — replaced `litellm` with `unclecode-litellm` due to a PyPI supply chain compromise. If you're on v0.8.5, please upgrade immediately.\n\n✨ Recent v0.8.5: Anti-Bot Detection, Shadow DOM & 60+ Bug Fixes! Automatic 3-tier anti-bot detection with proxy escalation, Shadow DOM flattening, deep crawl cancellation, config defaults API, consent popup removal, and critical security patches. [Release notes →](https:\u002F\u002Fgithub.com\u002Funclecode\u002Fcrawl4ai\u002Fblob\u002Fmain\u002Fdocs\u002Fblog\u002Frelease-v0.8.5.md)\n\n✨ Previous v0.8.0: Crash Recovery & Prefetch Mode! Deep crawl crash recovery with `resume_state` and `on_state_change` callbacks for long-running crawls. New `prefetch=True` mode for 5-10x faster URL discovery. [Release notes →](https:\u002F\u002Fgithub.com\u002Funclecode\u002Fcrawl4ai\u002Fblob\u002Fmain\u002Fdocs\u002Fblog\u002Frelease-v0.8.0.md)\n\n✨ Previous v0.7.8: Stability & Bug Fix Release! 11 bug fixes addressing Docker API issues, LLM extraction improvements, URL handling fixes, and dependency updates. [Release notes →](https:\u002F\u002Fgithub.com\u002Funclecode\u002Fcrawl4ai\u002Fblob\u002Fmain\u002Fdocs\u002Fblog\u002Frelease-v0.7.8.md)\n\n\u003Cdetails>\n  \u003Csummary>🤓 \u003Cstrong>My Personal Story\u003C\u002Fstrong>\u003C\u002Fsummary>\n\nI grew up on an Amstrad, thanks to my dad, and never stopped building. In grad school I specialized in NLP and built crawlers for research. That’s where I learned how much extraction matters.\n\nIn 2023, I needed web-to-Markdown. The “open source” option wanted an account, API token, and $16, and still under-delivered. I went turbo anger mode, built Crawl4AI in days, and it went viral. Now it’s the most-starred crawler on GitHub.\n\nI made it open source for **availability**, anyone can use it without a gate. Now I’m building the platform for **affordability**, anyone can run serious crawls without breaking the bank. If that resonates, join in, send feedback, or just crawl something amazing.\n\u003C\u002Fdetails>\n\n\n\u003Cdetails>\n  \u003Csummary>Why developers pick Crawl4AI\u003C\u002Fsummary>\n\n- **LLM ready output**, smart Markdown with headings, tables, code, citation hints\n- **Fast in practice**, async browser pool, caching, minimal hops\n- **Full control**, sessions, proxies, cookies, user scripts, hooks\n- **Adaptive intelligence**, learns site patterns, explores only what matters\n- **Deploy anywhere**, zero keys, CLI and Docker, cloud friendly\n\u003C\u002Fdetails>\n\n\n## 🚀 Quick Start \n\n1. Install Crawl4AI:\n```bash\n# Install the package\npip install -U crawl4ai\n\n# For pre release versions\npip install crawl4ai --pre\n\n# Run post-installation setup\ncrawl4ai-setup\n\n# Verify your installation\ncrawl4ai-doctor\n```\n\nIf you encounter any browser-related issues, you can install them manually:\n```bash\npython -m playwright install --with-deps chromium\n```\n\n2. Run a simple web crawl with Python:\n```python\nimport asyncio\nfrom crawl4ai import *\n\nasync def main():\n    async with AsyncWebCrawler() as crawler:\n        result = await crawler.arun(\n            url=\"https:\u002F\u002Fwww.nbcnews.com\u002Fbusiness\",\n        )\n        print(result.markdown)\n\nif __name__ == \"__main__\":\n    asyncio.run(main())\n```\n\n3. Or use the new command-line interface:\n```bash\n# Basic crawl with markdown output\ncrwl https:\u002F\u002Fwww.nbcnews.com\u002Fbusiness -o markdown\n\n# Deep crawl with BFS strategy, max 10 pages\ncrwl https:\u002F\u002Fdocs.crawl4ai.com --deep-crawl bfs --max-pages 10\n\n# Use LLM extraction with a specific question\ncrwl https:\u002F\u002Fwww.example.com\u002Fproducts -q \"Extract all product prices\"\n```\n\n## 💖 Support Crawl4AI\n\n> 🎉 **Sponsorship Program Now Open!** After powering 51K+ developers and 1 year of growth, Crawl4AI is launching dedicated support for **startups** and **enterprises**. Be among the first 50 **Founding Sponsors** for permanent recognition in our Hall of Fame.\n\nCrawl4AI is the #1 trending open-source web crawler on GitHub. Your support keeps it independent, innovative, and free for the community — while giving you direct access to premium benefits.\n\n\u003Cdiv align=\"\">\n  \n[![Become a Sponsor](https:\u002F\u002Fimg.shields.io\u002Fbadge\u002FBecome%20a%20Sponsor-pink?style=for-the-badge&logo=github-sponsors&logoColor=white)](https:\u002F\u002Fgithub.com\u002Fsponsors\u002Funclecode)  \n[![Current Sponsors](https:\u002F\u002Fimg.shields.io\u002Fgithub\u002Fsponsors\u002Funclecode?style=for-the-badge&logo=github&label=Current%20Sponsors&color=green)](https:\u002F\u002Fgithub.com\u002Fsponsors\u002Funclecode)\n\n\u003C\u002Fdiv>\n\n### 🤝 Sponsorship Tiers\n\n- **🌱 Believer ($5\u002Fmo)** — Join the movement for data democratization  \n- **🚀 Builder ($50\u002Fmo)** — Priority support & early access to features  \n- **💼 Growing Team ($500\u002Fmo)** — Bi-weekly syncs & optimization help  \n- **🏢 Data Infrastructure Partner ($2000\u002Fmo)** — Full partnership with dedicated support  \n  *Custom arrangements available - see [SPONSORS.md](SPONSORS.md) for details & contact*\n\n**Why sponsor?**  \nNo rate-limited APIs. No lock-in. Build and own your data pipeline with direct guidance from the creator of Crawl4AI.\n\n[See All Tiers & Benefits →](https:\u002F\u002Fgithub.com\u002Fsponsors\u002Funclecode)\n\n\n## ✨ Features \n\n\u003Cdetails>\n\u003Csummary>📝 \u003Cstrong>Markdown Generation\u003C\u002Fstrong>\u003C\u002Fsummary>\n\n- 🧹 **Clean Markdown**: Generates clean, structured Markdown with accurate formatting.\n- 🎯 **Fit Markdown**: Heuristic-based filtering to remove noise and irrelevant parts for AI-friendly processing.\n- 🔗 **Citations and References**: Converts page links into a numbered reference list with clean citations.\n- 🛠️ **Custom Strategies**: Users can create their own Markdown generation strategies tailored to specific needs.\n- 📚 **BM25 Algorithm**: Employs BM25-based filtering for extracting core information and removing irrelevant content. \n\u003C\u002Fdetails>\n\n\u003Cdetails>\n\u003Csummary>📊 \u003Cstrong>Structured Data Extraction\u003C\u002Fstrong>\u003C\u002Fsummary>\n\n- 🤖 **LLM-Driven Extraction**: Supports all LLMs (open-source and proprietary) for structured data extraction.\n- 🧱 **Chunking Strategies**: Implements chunking (topic-based, regex, sentence-level) for targeted content processing.\n- 🌌 **Cosine Similarity**: Find relevant content chunks based on user queries for semantic extraction.\n- 🔎 **CSS-Based Extraction**: Fast schema-based data extraction using XPath and CSS selectors.\n- 🔧 **Schema Definition**: Define custom schemas for extracting structured JSON from repetitive patterns.\n\n\u003C\u002Fdetails>\n\n\u003Cdetails>\n\u003Csummary>🌐 \u003Cstrong>Browser Integration\u003C\u002Fstrong>\u003C\u002Fsummary>\n\n- 🖥️ **Managed Browser**: Use user-owned browsers with full control, avoiding bot detection.\n- 🔄 **Remote Browser Control**: Connect to Chrome Developer Tools Protocol for remote, large-scale data extraction.\n- 👤 **Browser Profiler**: Create and manage persistent profiles with saved authentication states, cookies, and settings.\n- 🔒 **Session Management**: Preserve browser states and reuse them for multi-step crawling.\n- 🧩 **Proxy Support**: Seamlessly connect to proxies with authentication for secure access.\n- ⚙️ **Full Browser Control**: Modify headers, cookies, user agents, and more for tailored crawling setups.\n- 🌍 **Multi-Browser Support**: Compatible with Chromium, Firefox, and WebKit.\n- 📐 **Dynamic Viewport Adjustment**: Automatically adjusts the browser viewport to match page content, ensuring complete rendering and capturing of all elements.\n\n\u003C\u002Fdetails>\n\n\u003Cdetails>\n\u003Csummary>🔎 \u003Cstrong>Crawling & Scraping\u003C\u002Fstrong>\u003C\u002Fsummary>\n\n- 🖼️ **Media Support**: Extract images, audio, videos, and responsive image formats like `srcset` and `picture`.\n- 🚀 **Dynamic Crawling**: Execute JS and wait for async or sync for dynamic content extraction.\n- 📸 **Screenshots**: Capture page screenshots during crawling for debugging or analysis.\n- 📂 **Raw Data Crawling**: Directly process raw HTML (`raw:`) or local files (`file:\u002F\u002F`).\n- 🔗 **Comprehensive Link Extraction**: Extracts internal, external links, and embedded iframe content.\n- 🛠️ **Customizable Hooks**: Define hooks at every step to customize crawling behavior (supports both string and function-based APIs).\n- 💾 **Caching**: Cache data for improved speed and to avoid redundant fetches.\n- 📄 **Metadata Extraction**: Retrieve structured metadata from web pages.\n- 📡 **IFrame Content Extraction**: Seamless extraction from embedded iframe content.\n- 🕵️ **Lazy Load Handling**: Waits for images to fully load, ensuring no content is missed due to lazy loading.\n- 🔄 **Full-Page Scanning**: Simulates scrolling to load and capture all dynamic content, perfect for infinite scroll pages.\n\n\u003C\u002Fdetails>\n\n\u003Cdetails>\n\u003Csummary>🚀 \u003Cstrong>Deployment\u003C\u002Fstrong>\u003C\u002Fsummary>\n\n- 🐳 **Dockerized Setup**: Optimized Docker image with FastAPI server for easy deployment.\n- 🔑 **Secure Authentication**: Built-in JWT token authentication for API security.\n- 🔄 **API Gateway**: One-click deployment with secure token authentication for API-based workflows.\n- 🌐 **Scalable Architecture**: Designed for mass-scale production and optimized server performance.\n- ☁️ **Cloud Deployment**: Ready-to-deploy configurations for major cloud platforms.\n\n\u003C\u002Fdetails>\n\n\u003Cdetails>\n\u003Csummary>🎯 \u003Cstrong>Additional Features\u003C\u002Fstrong>\u003C\u002Fsummary>\n\n- 🕶️ **Stealth Mode**: Avoid bot detection by mimicking real users.\n- 🏷️ **Tag-Based Content Extraction**: Refine crawling based on custom tags, headers, or metadata.\n- 🔗 **Link Analysis**: Extract and analyze all links for detailed data exploration.\n- 🛡️ **Error Handling**: Robust error management for seamless execution.\n- 🔐 **CORS & Static Serving**: Supports filesystem-based caching and cross-origin requests.\n- 📖 **Clear Documentation**: Simplified and updated guides for onboarding and advanced usage.\n- 🙌 **Community Recognition**: Acknowledges contributors and pull requests for transparency.\n\n\u003C\u002Fdetails>\n\n## Try it Now!\n\n✨ Play around with this [![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fdrive\u002F1SgRPrByQLzjRfwoRNq1wSGE9nYY_EE8C?usp=sharing)\n\n✨ Visit our [Documentation Website](https:\u002F\u002Fdocs.crawl4ai.com\u002F)\n\n## Installation 🛠️\n\nCrawl4AI offers flexible installation options to suit various use cases. You can install it as a Python package or use Docker.\n\n\u003Cdetails>\n\u003Csummary>🐍 \u003Cstrong>Using pip\u003C\u002Fstrong>\u003C\u002Fsummary>\n\nChoose the installation option that best fits your needs:\n\n### Basic Installation\n\nFor basic web crawling and scraping tasks:\n\n```bash\npip install crawl4ai\ncrawl4ai-setup # Setup the browser\n```\n\nBy default, this will install the asynchronous version of Crawl4AI, using Playwright for web crawling.\n\n👉 **Note**: When you install Crawl4AI, the `crawl4ai-setup` should automatically install and set up Playwright. However, if you encounter any Playwright-related errors, you can manually install it using one of these methods:\n\n1. Through the command line:\n\n   ```bash\n   playwright install\n   ```\n\n2. If the above doesn't work, try this more specific command:\n\n   ```bash\n   python -m playwright install chromium\n   ```\n\nThis second method has proven to be more reliable in some cases.\n\n---\n\n### Installation with Synchronous Version\n\nThe sync version is deprecated and will be removed in future versions. If you need the synchronous version using Selenium:\n\n```bash\npip install crawl4ai[sync]\n```\n\n---\n\n### Development Installation\n\nFor contributors who plan to modify the source code:\n\n```bash\ngit clone https:\u002F\u002Fgithub.com\u002Funclecode\u002Fcrawl4ai.git\ncd crawl4ai\npip install -e .                    # Basic installation in editable mode\n```\n\nInstall optional features:\n\n```bash\npip install -e \".[torch]\"           # With PyTorch features\npip install -e \".[transformer]\"     # With Transformer features\npip install -e \".[cosine]\"          # With cosine similarity features\npip install -e \".[sync]\"            # With synchronous crawling (Selenium)\npip install -e \".[all]\"             # Install all optional features\n```\n\n\u003C\u002Fdetails>\n\n\u003Cdetails>\n\u003Csummary>🐳 \u003Cstrong>Docker Deployment\u003C\u002Fstrong>\u003C\u002Fsummary>\n\n> 🚀 **Now Available!** Our completely redesigned Docker implementation is here! This new solution makes deployment more efficient and seamless than ever.\n\n### New Docker Features\n\nThe new Docker implementation includes:\n- **Real-time Monitoring Dashboard** with live system metrics and browser pool visibility\n- **Browser pooling** with page pre-warming for faster response times\n- **Interactive playground** to test and generate request code\n- **MCP integration** for direct connection to AI tools like Claude Code\n- **Comprehensive API endpoints** including HTML extraction, screenshots, PDF generation, and JavaScript execution\n- **Multi-architecture support** with automatic detection (AMD64\u002FARM64)\n- **Optimized resources** with improved memory management\n\n### Getting Started\n\n```bash\n# Pull and run the latest release\ndocker pull unclecode\u002Fcrawl4ai:latest\ndocker run -d -p 11235:11235 --name crawl4ai --shm-size=1g unclecode\u002Fcrawl4ai:latest\n\n# Visit the monitoring dashboard at http:\u002F\u002Flocalhost:11235\u002Fdashboard\n# Or the playground at http:\u002F\u002Flocalhost:11235\u002Fplayground\n```\n\n### Quick Test\n\nRun a quick test (works for both Docker options):\n\n```python\nimport requests\n\n# Submit a crawl job\nresponse = requests.post(\n    \"http:\u002F\u002Flocalhost:11235\u002Fcrawl\",\n    json={\"urls\": [\"https:\u002F\u002Fexample.com\"], \"priority\": 10}\n)\nif response.status_code == 200:\n    print(\"Crawl job submitted successfully.\")\n    \nif \"results\" in response.json():\n    results = response.json()[\"results\"]\n    print(\"Crawl job completed. Results:\")\n    for result in results:\n        print(result)\nelse:\n    task_id = response.json()[\"task_id\"]\n    print(f\"Crawl job submitted. Task ID:: {task_id}\")\n    result = requests.get(f\"http:\u002F\u002Flocalhost:11235\u002Ftask\u002F{task_id}\")\n```\n\nFor more examples, see our [Docker Examples](https:\u002F\u002Fgithub.com\u002Funclecode\u002Fcrawl4ai\u002Fblob\u002Fmain\u002Fdocs\u002Fexamples\u002Fdocker_example.py). For advanced configuration, monitoring features, and production deployment, see our [Self-Hosting Guide](https:\u002F\u002Fdocs.crawl4ai.com\u002Fcore\u002Fself-hosting\u002F).\n\n\u003C\u002Fdetails>\n\n---\n\n## 🔬 Advanced Usage Examples 🔬\n\nYou can check the project structure in the directory [docs\u002Fexamples](https:\u002F\u002Fgithub.com\u002Funclecode\u002Fcrawl4ai\u002Ftree\u002Fmain\u002Fdocs\u002Fexamples). Over there, you can find a variety of examples; here, some popular examples are shared.\n\n\u003Cdetails>\n\u003Csummary>📝 \u003Cstrong>Heuristic Markdown Generation with Clean and Fit Markdown\u003C\u002Fstrong>\u003C\u002Fsummary>\n\n```python\nimport asyncio\nfrom crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode\nfrom crawl4ai.content_filter_strategy import PruningContentFilter, BM25ContentFilter\nfrom crawl4ai.markdown_generation_strategy import DefaultMarkdownGenerator\n\nasync def main():\n    browser_config = BrowserConfig(\n        headless=True,  \n        verbose=True,\n    )\n    run_config = CrawlerRunConfig(\n        cache_mode=CacheMode.ENABLED,\n        markdown_generator=DefaultMarkdownGenerator(\n            content_filter=PruningContentFilter(threshold=0.48, threshold_type=\"fixed\", min_word_threshold=0)\n        ),\n        # markdown_generator=DefaultMarkdownGenerator(\n        #     content_filter=BM25ContentFilter(user_query=\"WHEN_WE_FOCUS_BASED_ON_A_USER_QUERY\", bm25_threshold=1.0)\n        # ),\n    )\n    \n    async with AsyncWebCrawler(config=browser_config) as crawler:\n        result = await crawler.arun(\n            url=\"https:\u002F\u002Fdocs.micronaut.io\u002F4.9.9\u002Fguide\u002F\",\n            config=run_config\n        )\n        print(len(result.markdown.raw_markdown))\n        print(len(result.markdown.fit_markdown))\n\nif __name__ == \"__main__\":\n    asyncio.run(main())\n```\n\n\u003C\u002Fdetails>\n\n\u003Cdetails>\n\u003Csummary>🖥️ \u003Cstrong>Executing JavaScript & Extract Structured Data without LLMs\u003C\u002Fstrong>\u003C\u002Fsummary>\n\n```python\nimport asyncio\nfrom crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode\nfrom crawl4ai import JsonCssExtractionStrategy\nimport json\n\nasync def main():\n    schema = {\n    \"name\": \"KidoCode Courses\",\n    \"baseSelector\": \"section.charge-methodology .w-tab-content > div\",\n    \"fields\": [\n        {\n            \"name\": \"section_title\",\n            \"selector\": \"h3.heading-50\",\n            \"type\": \"text\",\n        },\n        {\n            \"name\": \"section_description\",\n            \"selector\": \".charge-content\",\n            \"type\": \"text\",\n        },\n        {\n            \"name\": \"course_name\",\n            \"selector\": \".text-block-93\",\n            \"type\": \"text\",\n        },\n        {\n            \"name\": \"course_description\",\n            \"selector\": \".course-content-text\",\n            \"type\": \"text\",\n        },\n        {\n            \"name\": \"course_icon\",\n            \"selector\": \".image-92\",\n            \"type\": \"attribute\",\n            \"attribute\": \"src\"\n        }\n    ]\n}\n\n    extraction_strategy = JsonCssExtractionStrategy(schema, verbose=True)\n\n    browser_config = BrowserConfig(\n        headless=False,\n        verbose=True\n    )\n    run_config = CrawlerRunConfig(\n        extraction_strategy=extraction_strategy,\n        js_code=[\"\"\"(async () => {const tabs = document.querySelectorAll(\"section.charge-methodology .tabs-menu-3 > div\");for(let tab of tabs) {tab.scrollIntoView();tab.click();await new Promise(r => setTimeout(r, 500));}})();\"\"\"],\n        cache_mode=CacheMode.BYPASS\n    )\n        \n    async with AsyncWebCrawler(config=browser_config) as crawler:\n        \n        result = await crawler.arun(\n            url=\"https:\u002F\u002Fwww.kidocode.com\u002Fdegrees\u002Ftechnology\",\n            config=run_config\n        )\n\n        companies = json.loads(result.extracted_content)\n        print(f\"Successfully extracted {len(companies)} companies\")\n        print(json.dumps(companies[0], indent=2))\n\n\nif __name__ == \"__main__\":\n    asyncio.run(main())\n```\n\n\u003C\u002Fdetails>\n\n\u003Cdetails>\n\u003Csummary>📚 \u003Cstrong>Extracting Structured Data with LLMs\u003C\u002Fstrong>\u003C\u002Fsummary>\n\n```python\nimport os\nimport asyncio\nfrom crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode, LLMConfig\nfrom crawl4ai import LLMExtractionStrategy\nfrom pydantic import BaseModel, Field\n\nclass OpenAIModelFee(BaseModel):\n    model_name: str = Field(..., description=\"Name of the OpenAI model.\")\n    input_fee: str = Field(..., description=\"Fee for input token for the OpenAI model.\")\n    output_fee: str = Field(..., description=\"Fee for output token for the OpenAI model.\")\n\nasync def main():\n    browser_config = BrowserConfig(verbose=True)\n    run_config = CrawlerRunConfig(\n        word_count_threshold=1,\n        extraction_strategy=LLMExtractionStrategy(\n            # Here you can use any provider that Litellm library supports, for instance: ollama\u002Fqwen2\n            # provider=\"ollama\u002Fqwen2\", api_token=\"no-token\", \n            llm_config = LLMConfig(provider=\"openai\u002Fgpt-4o\", api_token=os.getenv('OPENAI_API_KEY')), \n            schema=OpenAIModelFee.schema(),\n            extraction_type=\"schema\",\n            instruction=\"\"\"From the crawled content, extract all mentioned model names along with their fees for input and output tokens. \n            Do not miss any models in the entire content. One extracted model JSON format should look like this: \n            {\"model_name\": \"GPT-4\", \"input_fee\": \"US$10.00 \u002F 1M tokens\", \"output_fee\": \"US$30.00 \u002F 1M tokens\"}.\"\"\"\n        ),            \n        cache_mode=CacheMode.BYPASS,\n    )\n    \n    async with AsyncWebCrawler(config=browser_config) as crawler:\n        result = await crawler.arun(\n            url='https:\u002F\u002Fopenai.com\u002Fapi\u002Fpricing\u002F',\n            config=run_config\n        )\n        print(result.extracted_content)\n\nif __name__ == \"__main__\":\n    asyncio.run(main())\n```\n\n\u003C\u002Fdetails>\n\n\u003Cdetails>\n\u003Csummary>🤖 \u003Cstrong>Using Your own Browser with Custom User Profile\u003C\u002Fstrong>\u003C\u002Fsummary>\n\n```python\nimport os, sys\nfrom pathlib import Path\nimport asyncio, time\nfrom crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode\n\nasync def test_news_crawl():\n    # Create a persistent user data directory\n    user_data_dir = os.path.join(Path.home(), \".crawl4ai\", \"browser_profile\")\n    os.makedirs(user_data_dir, exist_ok=True)\n\n    browser_config = BrowserConfig(\n        verbose=True,\n        headless=True,\n        user_data_dir=user_data_dir,\n        use_persistent_context=True,\n    )\n    run_config = CrawlerRunConfig(\n        cache_mode=CacheMode.BYPASS\n    )\n    \n    async with AsyncWebCrawler(config=browser_config) as crawler:\n        url = \"ADDRESS_OF_A_CHALLENGING_WEBSITE\"\n        \n        result = await crawler.arun(\n            url,\n            config=run_config,\n            magic=True,\n        )\n        \n        print(f\"Successfully crawled {url}\")\n        print(f\"Content length: {len(result.markdown)}\")\n```\n\n\u003C\u002Fdetails>\n\n---\n\n> **💡 Tip:** Some websites may use **CAPTCHA** based verification mechanisms to prevent automated access. If your workflow encounters such challenges, you may optionally integrate a third-party CAPTCHA-handling service such as \u003Cstrong>[CapSolver](https:\u002F\u002Fwww.capsolver.com\u002Fblog\u002FPartners\u002Fcrawl4ai-capsolver\u002F?utm_source=crawl4ai&utm_medium=github_pr&utm_campaign=crawl4ai_integration)\u003C\u002Fstrong>. They support reCAPTCHA v2\u002Fv3, Cloudflare Turnstile, Challenge, AWS WAF, and more. Please ensure that your usage complies with the target website’s terms of service and applicable laws.\n\n## ✨ Recent Updates\n\n\u003Cdetails open>\n\u003Csummary>\u003Cstrong>Version 0.8.6 — Security Hotfix: litellm Supply Chain Fix\u003C\u002Fstrong>\u003C\u002Fsummary>\n\nReplaced `litellm` dependency with `unclecode-litellm` due to a PyPI supply chain compromise affecting the original package. If you're on v0.8.5 or earlier, upgrade immediately.\n\n```bash\npip install -U crawl4ai\n```\n\n\u003C\u002Fdetails>\n\n\u003Cdetails>\n\u003Csummary>\u003Cstrong>Version 0.8.5 Release Highlights - Anti-Bot Detection, Shadow DOM & 60+ Bug Fixes\u003C\u002Fstrong>\u003C\u002Fsummary>\n\nOur biggest release since v0.8.0. Anti-bot detection with proxy escalation, Shadow DOM flattening, deep crawl cancellation, and over 60 bug fixes.\n\n- **🛡️ Anti-Bot Detection & Proxy Escalation**:\n  - 3-tier detection: known vendors, generic block indicators, structural integrity checks\n  - Automatic retry with proxy chain and fallback fetch function\n  ```python\n  from crawl4ai import CrawlerRunConfig\n  from crawl4ai.async_configs import ProxyConfig\n\n  config = CrawlerRunConfig(\n      proxy_config=[ProxyConfig.DIRECT, ProxyConfig(server=\"http:\u002F\u002Fmy-proxy:8080\")],\n      max_retries=2,\n      fallback_fetch_function=my_web_unlocker,\n  )\n  ```\n\n- **🌑 Shadow DOM Flattening**:\n  - Extract content hidden inside shadow DOM components\n  ```python\n  config = CrawlerRunConfig(flatten_shadow_dom=True)\n  ```\n\n- **🛑 Deep Crawl Cancellation**:\n  - Stop long crawls gracefully with `cancel()` or `should_cancel` callback\n  - Works with BFS, DFS, and BestFirst strategies\n\n- **⚙️ Config Defaults API**:\n  - `set_defaults()` \u002F `get_defaults()` \u002F `reset_defaults()` on BrowserConfig and CrawlerRunConfig\n\n- **🔒 Critical Security Fixes**:\n  - RCE via deserialization in Docker `\u002Fcrawl` endpoint — removed `eval()`, added allowlist\n  - Redis CVE-2025-49844 (CVSS 10.0) — upgraded to 7.2.7\n\n- **60+ Bug Fixes** across browser management, proxy, deep crawling, extraction, CLI, and Docker\n\n[Full v0.8.5 Release Notes →](https:\u002F\u002Fgithub.com\u002Funclecode\u002Fcrawl4ai\u002Fblob\u002Fmain\u002Fdocs\u002Fblog\u002Frelease-v0.8.5.md)\n\n\u003C\u002Fdetails>\n\n\u003Cdetails>\n\u003Csummary>\u003Cstrong>Version 0.8.0 Release Highlights - Crash Recovery & Prefetch Mode\u003C\u002Fstrong>\u003C\u002Fsummary>\n\nThis release introduces crash recovery for deep crawls, a new prefetch mode for fast URL discovery, and critical security fixes for Docker deployments.\n\n- **🔄 Deep Crawl Crash Recovery**:\n  - `on_state_change` callback fires after each URL for real-time state persistence\n  - `resume_state` parameter to continue from a saved checkpoint\n  - JSON-serializable state for Redis\u002Fdatabase storage\n  - Works with BFS, DFS, and Best-First strategies\n  ```python\n  from crawl4ai.deep_crawling import BFSDeepCrawlStrategy\n\n  strategy = BFSDeepCrawlStrategy(\n      max_depth=3,\n      resume_state=saved_state,  # Continue from checkpoint\n      on_state_change=save_to_redis,  # Called after each URL\n  )\n  ```\n\n- **⚡ Prefetch Mode for Fast URL Discovery**:\n  - `prefetch=True` skips markdown, extraction, and media processing\n  - 5-10x faster than full processing\n  - Perfect for two-phase crawling: discover first, process selectively\n  ```python\n  config = CrawlerRunConfig(prefetch=True)\n  result = await crawler.arun(\"https:\u002F\u002Fexample.com\", config=config)\n  # Returns HTML and links only - no markdown generation\n  ```\n\n- **🔒 Security Fixes (Docker API)**:\n  - Hooks disabled by default (`CRAWL4AI_HOOKS_ENABLED=false`)\n  - `file:\u002F\u002F` URLs blocked on API endpoints to prevent LFI\n  - `__import__` removed from hook execution sandbox\n\n[Full v0.8.0 Release Notes →](https:\u002F\u002Fgithub.com\u002Funclecode\u002Fcrawl4ai\u002Fblob\u002Fmain\u002Fdocs\u002Fblog\u002Frelease-v0.8.0.md)\n\n\u003C\u002Fdetails>\n\n\u003Cdetails>\n\u003Csummary>\u003Cstrong>Version 0.7.8 Release Highlights - Stability & Bug Fix Release\u003C\u002Fstrong>\u003C\u002Fsummary>\n\nThis release focuses on stability with 11 bug fixes addressing issues reported by the community. No new features, but significant improvements to reliability.\n\n- **🐳 Docker API Fixes**:\n  - Fixed `ContentRelevanceFilter` deserialization in deep crawl requests (#1642)\n  - Fixed `ProxyConfig` JSON serialization in `BrowserConfig.to_dict()` (#1629)\n  - Fixed `.cache` folder permissions in Docker image (#1638)\n\n- **🤖 LLM Extraction Improvements**:\n  - Configurable rate limiter backoff with new `LLMConfig` parameters (#1269):\n    ```python\n    from crawl4ai import LLMConfig\n\n    config = LLMConfig(\n        provider=\"openai\u002Fgpt-4o-mini\",\n        backoff_base_delay=5,           # Wait 5s on first retry\n        backoff_max_attempts=5,          # Try up to 5 times\n        backoff_exponential_factor=3     # Multiply delay by 3 each attempt\n    )\n    ```\n  - HTML input format support for `LLMExtractionStrategy` (#1178):\n    ```python\n    from crawl4ai import LLMExtractionStrategy\n\n    strategy = LLMExtractionStrategy(\n        llm_config=config,\n        instruction=\"Extract table data\",\n        input_format=\"html\"  # Now supports: \"html\", \"markdown\", \"fit_markdown\"\n    )\n    ```\n  - Fixed raw HTML URL variable - extraction strategies now receive `\"Raw HTML\"` instead of HTML blob (#1116)\n\n- **🔗 URL Handling**:\n  - Fixed relative URL resolution after JavaScript redirects (#1268)\n  - Fixed import statement formatting in extracted code (#1181)\n\n- **📦 Dependency Updates**:\n  - Replaced deprecated PyPDF2 with pypdf (#1412)\n  - Pydantic v2 ConfigDict compatibility - no more deprecation warnings (#678)\n\n- **🧠 AdaptiveCrawler**:\n  - Fixed query expansion to actually use LLM instead of hardcoded mock data (#1621)\n\n[Full v0.7.8 Release Notes →](https:\u002F\u002Fgithub.com\u002Funclecode\u002Fcrawl4ai\u002Fblob\u002Fmain\u002Fdocs\u002Fblog\u002Frelease-v0.7.8.md)\n\n\u003C\u002Fdetails>\n\n\u003Cdetails>\n\u003Csummary>\u003Cstrong>Version 0.7.7 Release Highlights - The Self-Hosting & Monitoring Update\u003C\u002Fstrong>\u003C\u002Fsummary>\n\n- **📊 Real-time Monitoring Dashboard**: Interactive web UI with live system metrics and browser pool visibility\n  ```python\n  # Access the monitoring dashboard\n  # Visit: http:\u002F\u002Flocalhost:11235\u002Fdashboard\n\n  # Real-time metrics include:\n  # - System health (CPU, memory, network, uptime)\n  # - Active and completed request tracking\n  # - Browser pool management (permanent\u002Fhot\u002Fcold)\n  # - Janitor cleanup events\n  # - Error monitoring with full context\n  ```\n\n- **🔌 Comprehensive Monitor API**: Complete REST API for programmatic access to all monitoring data\n  ```python\n  import httpx\n\n  async with httpx.AsyncClient() as client:\n      # System health\n      health = await client.get(\"http:\u002F\u002Flocalhost:11235\u002Fmonitor\u002Fhealth\")\n\n      # Request tracking\n      requests = await client.get(\"http:\u002F\u002Flocalhost:11235\u002Fmonitor\u002Frequests\")\n\n      # Browser pool status\n      browsers = await client.get(\"http:\u002F\u002Flocalhost:11235\u002Fmonitor\u002Fbrowsers\")\n\n      # Endpoint statistics\n      stats = await client.get(\"http:\u002F\u002Flocalhost:11235\u002Fmonitor\u002Fendpoints\u002Fstats\")\n  ```\n\n- **⚡ WebSocket Streaming**: Real-time updates every 2 seconds for custom dashboards\n- **🔥 Smart Browser Pool**: 3-tier architecture (permanent\u002Fhot\u002Fcold) with automatic promotion and cleanup\n- **🧹 Janitor System**: Automatic resource management with event logging\n- **🎮 Control Actions**: Manual browser management (kill, restart, cleanup) via API\n- **📈 Production Metrics**: 6 critical metrics for operational excellence with Prometheus integration\n- **🐛 Critical Bug Fixes**:\n  - Fixed async LLM extraction blocking issue (#1055)\n  - Enhanced DFS deep crawl strategy (#1607)\n  - Fixed sitemap parsing in AsyncUrlSeeder (#1598)\n  - Resolved browser viewport configuration (#1495)\n  - Fixed CDP timing with exponential backoff (#1528)\n  - Security update for pyOpenSSL (>=25.3.0)\n\n[Full v0.7.7 Release Notes →](https:\u002F\u002Fgithub.com\u002Funclecode\u002Fcrawl4ai\u002Fblob\u002Fmain\u002Fdocs\u002Fblog\u002Frelease-v0.7.7.md)\n\n\u003C\u002Fdetails>\n\n\u003Cdetails>\n\u003Csummary>\u003Cstrong>Version 0.7.5 Release Highlights - The Docker Hooks & Security Update\u003C\u002Fstrong>\u003C\u002Fsummary>\n\n- **🔧 Docker Hooks System**: Complete pipeline customization with user-provided Python functions at 8 key points\n- **✨ Function-Based Hooks API (NEW)**: Write hooks as regular Python functions with full IDE support:\n  ```python\n  from crawl4ai import hooks_to_string\n  from crawl4ai.docker_client import Crawl4aiDockerClient\n\n  # Define hooks as regular Python functions\n  async def on_page_context_created(page, context, **kwargs):\n      \"\"\"Block images to speed up crawling\"\"\"\n      await context.route(\"**\u002F*.{png,jpg,jpeg,gif,webp}\", lambda route: route.abort())\n      await page.set_viewport_size({\"width\": 1920, \"height\": 1080})\n      return page\n\n  async def before_goto(page, context, url, **kwargs):\n      \"\"\"Add custom headers\"\"\"\n      await page.set_extra_http_headers({'X-Crawl4AI': 'v0.7.5'})\n      return page\n\n  # Option 1: Use hooks_to_string() utility for REST API\n  hooks_code = hooks_to_string({\n      \"on_page_context_created\": on_page_context_created,\n      \"before_goto\": before_goto\n  })\n\n  # Option 2: Docker client with automatic conversion (Recommended)\n  client = Crawl4aiDockerClient(base_url=\"http:\u002F\u002Flocalhost:11235\")\n  results = await client.crawl(\n      urls=[\"https:\u002F\u002Fhttpbin.org\u002Fhtml\"],\n      hooks={\n          \"on_page_context_created\": on_page_context_created,\n          \"before_goto\": before_goto\n      }\n  )\n  # ✓ Full IDE support, type checking, and reusability!\n  ```\n\n- **🤖 Enhanced LLM Integration**: Custom providers with temperature control and base_url configuration\n- **🔒 HTTPS Preservation**: Secure internal link handling with `preserve_https_for_internal_links=True`\n- **🐍 Python 3.10+ Support**: Modern language features and enhanced performance\n- **🛠️ Bug Fixes**: Resolved multiple community-reported issues including URL processing, JWT authentication, and proxy configuration\n\n[Full v0.7.5 Release Notes →](https:\u002F\u002Fgithub.com\u002Funclecode\u002Fcrawl4ai\u002Fblob\u002Fmain\u002Fdocs\u002Fblog\u002Frelease-v0.7.5.md)\n\n\u003C\u002Fdetails>\n\n\u003Cdetails>\n\u003Csummary>\u003Cstrong>Version 0.7.4 Release Highlights - The Intelligent Table Extraction & Performance Update\u003C\u002Fstrong>\u003C\u002Fsummary>\n\n- **🚀 LLMTableExtraction**: Revolutionary table extraction with intelligent chunking for massive tables:\n  ```python\n  from crawl4ai import LLMTableExtraction, LLMConfig\n  \n  # Configure intelligent table extraction\n  table_strategy = LLMTableExtraction(\n      llm_config=LLMConfig(provider=\"openai\u002Fgpt-4.1-mini\"),\n      enable_chunking=True,           # Handle massive tables\n      chunk_token_threshold=5000,     # Smart chunking threshold\n      overlap_threshold=100,          # Maintain context between chunks\n      extraction_type=\"structured\"    # Get structured data output\n  )\n  \n  config = CrawlerRunConfig(table_extraction_strategy=table_strategy)\n  result = await crawler.arun(\"https:\u002F\u002Fcomplex-tables-site.com\", config=config)\n  \n  # Tables are automatically chunked, processed, and merged\n  for table in result.tables:\n      print(f\"Extracted table: {len(table['data'])} rows\")\n  ```\n\n- **⚡ Dispatcher Bug Fix**: Fixed sequential processing bottleneck in arun_many for fast-completing tasks\n- **🧹 Memory Management Refactor**: Consolidated memory utilities into main utils module for cleaner architecture\n- **🔧 Browser Manager Fixes**: Resolved race conditions in concurrent page creation with thread-safe locking\n- **🔗 Advanced URL Processing**: Better handling of raw:\u002F\u002F URLs and base tag link resolution\n- **🛡️ Enhanced Proxy Support**: Flexible proxy configuration supporting both dict and string formats\n\n[Full v0.7.4 Release Notes →](https:\u002F\u002Fgithub.com\u002Funclecode\u002Fcrawl4ai\u002Fblob\u002Fmain\u002Fdocs\u002Fblog\u002Frelease-v0.7.4.md)\n\n\u003C\u002Fdetails>\n\n\u003Cdetails>\n\u003Csummary>\u003Cstrong>Version 0.7.3 Release Highlights - The Multi-Config Intelligence Update\u003C\u002Fstrong>\u003C\u002Fsummary>\n\n- **🕵️ Undetected Browser Support**: Bypass sophisticated bot detection systems:\n  ```python\n  from crawl4ai import AsyncWebCrawler, BrowserConfig\n  \n  browser_config = BrowserConfig(\n      browser_type=\"undetected\",  # Use undetected Chrome\n      headless=True,              # Can run headless with stealth\n      extra_args=[\n          \"--disable-blink-features=AutomationControlled\",\n          \"--disable-web-security\"\n      ]\n  )\n  \n  async with AsyncWebCrawler(config=browser_config) as crawler:\n      result = await crawler.arun(\"https:\u002F\u002Fprotected-site.com\")\n  # Successfully bypass Cloudflare, Akamai, and custom bot detection\n  ```\n\n- **🎨 Multi-URL Configuration**: Different strategies for different URL patterns in one batch:\n```python\nfrom crawl4ai import CrawlerRunConfig, MatchMode, CacheMode\n  \n  configs = [\n      # Documentation sites - aggressive caching\n      CrawlerRunConfig(\n          url_matcher=[\"*docs*\", \"*documentation*\"],\n          cache_mode=CacheMode.WRITE_ONLY,\n          markdown_generator_options={\"include_links\": True}\n      ),\n      \n      # News\u002Fblog sites - fresh content\n      CrawlerRunConfig(\n          url_matcher=lambda url: 'blog' in url or 'news' in url,\n          cache_mode=CacheMode.BYPASS\n      ),\n      \n      # Fallback for everything else\n      CrawlerRunConfig()\n  ]\n  \n  results = await crawler.arun_many(urls, config=configs)\n  # Each URL gets the perfect configuration automatically\n  ```\n\n- **🧠 Memory Monitoring**: Track and optimize memory usage during crawling:\n  ```python\n  from crawl4ai.memory_utils import MemoryMonitor\n  \n  monitor = MemoryMonitor()\n  monitor.start_monitoring()\n  \n  results = await crawler.arun_many(large_url_list)\n  \n  report = monitor.get_report()\n  print(f\"Peak memory: {report['peak_mb']:.1f} MB\")\n  print(f\"Efficiency: {report['efficiency']:.1f}%\")\n  # Get optimization recommendations\n  ```\n\n- **📊 Enhanced Table Extraction**: Direct DataFrame conversion from web tables:\n  ```python\n  result = await crawler.arun(\"https:\u002F\u002Fsite-with-tables.com\")\n  \n  # New way - direct table access\n  if result.tables:\n      import pandas as pd\n      for table in result.tables:\n          df = pd.DataFrame(table['data'])\n          print(f\"Table: {df.shape[0]} rows × {df.shape[1]} columns\")\n  ```\n\n- **💰 GitHub Sponsors**: 4-tier sponsorship system for project sustainability\n- **🐳 Docker LLM Flexibility**: Configure providers via environment variables\n\n[Full v0.7.3 Release Notes →](https:\u002F\u002Fgithub.com\u002Funclecode\u002Fcrawl4ai\u002Fblob\u002Fmain\u002Fdocs\u002Fblog\u002Frelease-v0.7.3.md)\n\n\u003C\u002Fdetails>\n\n\u003Cdetails>\n\u003Csummary>\u003Cstrong>Version 0.7.0 Release Highlights - The Adaptive Intelligence Update\u003C\u002Fstrong>\u003C\u002Fsummary>\n\n- **🧠 Adaptive Crawling**: Your crawler now learns and adapts to website patterns automatically:\n  ```python\n  config = AdaptiveConfig(\n      confidence_threshold=0.7, # Min confidence to stop crawling\n      max_depth=5, # Maximum crawl depth\n      max_pages=20, # Maximum number of pages to crawl\n      strategy=\"statistical\"\n  )\n  \n  async with AsyncWebCrawler() as crawler:\n      adaptive_crawler = AdaptiveCrawler(crawler, config)\n      state = await adaptive_crawler.digest(\n          start_url=\"https:\u002F\u002Fnews.example.com\",\n          query=\"latest news content\"\n      )\n  # Crawler learns patterns and improves extraction over time\n  ```\n\n- **🌊 Virtual Scroll Support**: Complete content extraction from infinite scroll pages:\n  ```python\n  scroll_config = VirtualScrollConfig(\n      container_selector=\"[data-testid='feed']\",\n      scroll_count=20,\n      scroll_by=\"container_height\",\n      wait_after_scroll=1.0\n  )\n  \n  result = await crawler.arun(url, config=CrawlerRunConfig(\n      virtual_scroll_config=scroll_config\n  ))\n  ```\n\n- **🔗 Intelligent Link Analysis**: 3-layer scoring system for smart link prioritization:\n  ```python\n  link_config = LinkPreviewConfig(\n      query=\"machine learning tutorials\",\n      score_threshold=0.3,\n      concurrent_requests=10\n  )\n  \n  result = await crawler.arun(url, config=CrawlerRunConfig(\n      link_preview_config=link_config,\n      score_links=True\n  ))\n  # Links ranked by relevance and quality\n  ```\n\n- **🎣 Async URL Seeder**: Discover thousands of URLs in seconds:\n  ```python\n  seeder = AsyncUrlSeeder(SeedingConfig(\n      source=\"sitemap+cc\",\n      pattern=\"*\u002Fblog\u002F*\",\n      query=\"python tutorials\",\n      score_threshold=0.4\n  ))\n  \n  urls = await seeder.discover(\"https:\u002F\u002Fexample.com\")\n  ```\n\n- **⚡ Performance Boost**: Up to 3x faster with optimized resource handling and memory efficiency\n\nRead the full details in our [0.7.0 Release Notes](https:\u002F\u002Fdocs.crawl4ai.com\u002Fblog\u002Frelease-v0.7.0) or check the [CHANGELOG](https:\u002F\u002Fgithub.com\u002Funclecode\u002Fcrawl4ai\u002Fblob\u002Fmain\u002FCHANGELOG.md).\n\n\u003C\u002Fdetails>\n\n## Version Numbering in Crawl4AI\n\nCrawl4AI follows standard Python version numbering conventions (PEP 440) to help users understand the stability and features of each release.\n\n\u003Cdetails>\n\u003Csummary>📈 \u003Cstrong>Version Numbers Explained\u003C\u002Fstrong>\u003C\u002Fsummary>\n\nOur version numbers follow this pattern: `MAJOR.MINOR.PATCH` (e.g., 0.4.3)\n\n#### Pre-release Versions\nWe use different suffixes to indicate development stages:\n\n- `dev` (0.4.3dev1): Development versions, unstable\n- `a` (0.4.3a1): Alpha releases, experimental features\n- `b` (0.4.3b1): Beta releases, feature complete but needs testing\n- `rc` (0.4.3): Release candidates, potential final version\n\n#### Installation\n- Regular installation (stable version):\n  ```bash\n  pip install -U crawl4ai\n  ```\n\n- Install pre-release versions:\n  ```bash\n  pip install crawl4ai --pre\n  ```\n\n- Install specific version:\n  ```bash\n  pip install crawl4ai==0.4.3b1\n  ```\n\n#### Why Pre-releases?\nWe use pre-releases to:\n- Test new features in real-world scenarios\n- Gather feedback before final releases\n- Ensure stability for production users\n- Allow early adopters to try new features\n\nFor production environments, we recommend using the stable version. For testing new features, you can opt-in to pre-releases using the `--pre` flag.\n\n\u003C\u002Fdetails>\n\n## 📖 Documentation & Roadmap \n\n> 🚨 **Documentation Update Alert**: We're undertaking a major documentation overhaul next week to reflect recent updates and improvements. Stay tuned for a more comprehensive and up-to-date guide!\n\nFor current documentation, including installation instructions, advanced features, and API reference, visit our [Documentation Website](https:\u002F\u002Fdocs.crawl4ai.com\u002F).\n\nTo check our development plans and upcoming features, visit our [Roadmap](https:\u002F\u002Fgithub.com\u002Funclecode\u002Fcrawl4ai\u002Fblob\u002Fmain\u002FROADMAP.md).\n\n\u003Cdetails>\n\u003Csummary>📈 \u003Cstrong>Development TODOs\u003C\u002Fstrong>\u003C\u002Fsummary>\n\n- [x] 0. Graph Crawler: Smart website traversal using graph search algorithms for comprehensive nested page extraction\n- [x] 1. Question-Based Crawler: Natural language driven web discovery and content extraction\n- [x] 2. Knowledge-Optimal Crawler: Smart crawling that maximizes knowledge while minimizing data extraction\n- [x] 3. Agentic Crawler: Autonomous system for complex multi-step crawling operations\n- [x] 4. Automated Schema Generator: Convert natural language to extraction schemas\n- [x] 5. Domain-Specific Scrapers: Pre-configured extractors for common platforms (academic, e-commerce)\n- [x] 6. Web Embedding Index: Semantic search infrastructure for crawled content\n- [x] 7. Interactive Playground: Web UI for testing, comparing strategies with AI assistance\n- [x] 8. Performance Monitor: Real-time insights into crawler operations\n- [ ] 9. Cloud Integration: One-click deployment solutions across cloud providers\n- [x] 10. Sponsorship Program: Structured support system with tiered benefits\n- [ ] 11. Educational Content: \"How to Crawl\" video series and interactive tutorials\n\n\u003C\u002Fdetails>\n\n## 🤝 Contributing \n\nWe welcome contributions from the open-source community. Check out our [contribution guidelines](https:\u002F\u002Fgithub.com\u002Funclecode\u002Fcrawl4ai\u002Fblob\u002Fmain\u002FCONTRIBUTORS.md) for more information.\n\nI'll help modify the license section with badges. For the halftone effect, here's a version with it:\n\nHere's the updated license section:\n\n## 📄 License & Attribution\n\nThis project is licensed under the Apache License 2.0, attribution is recommended via the badges below. See the [Apache 2.0 License](https:\u002F\u002Fgithub.com\u002Funclecode\u002Fcrawl4ai\u002Fblob\u002Fmain\u002FLICENSE) file for details.\n\n### Attribution Requirements\nWhen using Crawl4AI, you must include one of the following attribution methods:\n\n\u003Cdetails>\n\u003Csummary>📈 \u003Cstrong>1. Badge Attribution (Recommended)\u003C\u002Fstrong>\u003C\u002Fsummary>\nAdd one of these badges to your README, documentation, or website:\n\n| Theme | Badge |\n|-------|-------|\n| **Disco Theme (Animated)** | \u003Ca href=\"https:\u002F\u002Fgithub.com\u002Funclecode\u002Fcrawl4ai\">\u003Cimg src=\".\u002Fdocs\u002Fassets\u002Fpowered-by-disco.svg\" alt=\"Powered by Crawl4AI\" width=\"200\"\u002F>\u003C\u002Fa> |\n| **Night Theme (Dark with Neon)** | \u003Ca href=\"https:\u002F\u002Fgithub.com\u002Funclecode\u002Fcrawl4ai\">\u003Cimg src=\".\u002Fdocs\u002Fassets\u002Fpowered-by-night.svg\" alt=\"Powered by Crawl4AI\" width=\"200\"\u002F>\u003C\u002Fa> |\n| **Dark Theme (Classic)** | \u003Ca href=\"https:\u002F\u002Fgithub.com\u002Funclecode\u002Fcrawl4ai\">\u003Cimg src=\".\u002Fdocs\u002Fassets\u002Fpowered-by-dark.svg\" alt=\"Powered by Crawl4AI\" width=\"200\"\u002F>\u003C\u002Fa> |\n| **Light Theme (Classic)** | \u003Ca href=\"https:\u002F\u002Fgithub.com\u002Funclecode\u002Fcrawl4ai\">\u003Cimg src=\".\u002Fdocs\u002Fassets\u002Fpowered-by-light.svg\" alt=\"Powered by Crawl4AI\" width=\"200\"\u002F>\u003C\u002Fa> |\n \n\nHTML code for adding the badges:\n```html\n\u003C!-- Disco Theme (Animated) -->\n\u003Ca href=\"https:\u002F\u002Fgithub.com\u002Funclecode\u002Fcrawl4ai\">\n  \u003Cimg src=\"https:\u002F\u002Fraw.githubusercontent.com\u002Funclecode\u002Fcrawl4ai\u002Fmain\u002Fdocs\u002Fassets\u002Fpowered-by-disco.svg\" alt=\"Powered by Crawl4AI\" width=\"200\"\u002F>\n\u003C\u002Fa>\n\n\u003C!-- Night Theme (Dark with Neon) -->\n\u003Ca href=\"https:\u002F\u002Fgithub.com\u002Funclecode\u002Fcrawl4ai\">\n  \u003Cimg src=\"https:\u002F\u002Fraw.githubusercontent.com\u002Funclecode\u002Fcrawl4ai\u002Fmain\u002Fdocs\u002Fassets\u002Fpowered-by-night.svg\" alt=\"Powered by Crawl4AI\" width=\"200\"\u002F>\n\u003C\u002Fa>\n\n\u003C!-- Dark Theme (Classic) -->\n\u003Ca href=\"https:\u002F\u002Fgithub.com\u002Funclecode\u002Fcrawl4ai\">\n  \u003Cimg src=\"https:\u002F\u002Fraw.githubusercontent.com\u002Funclecode\u002Fcrawl4ai\u002Fmain\u002Fdocs\u002Fassets\u002Fpowered-by-dark.svg\" alt=\"Powered by Crawl4AI\" width=\"200\"\u002F>\n\u003C\u002Fa>\n\n\u003C!-- Light Theme (Classic) -->\n\u003Ca href=\"https:\u002F\u002Fgithub.com\u002Funclecode\u002Fcrawl4ai\">\n  \u003Cimg src=\"https:\u002F\u002Fraw.githubusercontent.com\u002Funclecode\u002Fcrawl4ai\u002Fmain\u002Fdocs\u002Fassets\u002Fpowered-by-light.svg\" alt=\"Powered by Crawl4AI\" width=\"200\"\u002F>\n\u003C\u002Fa>\n\n\u003C!-- Simple Shield Badge -->\n\u003Ca href=\"https:\u002F\u002Fgithub.com\u002Funclecode\u002Fcrawl4ai\">\n  \u003Cimg src=\"https:\u002F\u002Fimg.shields.io\u002Fbadge\u002FPowered%20by-Crawl4AI-blue?style=flat-square\" alt=\"Powered by Crawl4AI\"\u002F>\n\u003C\u002Fa>\n```\n\n\u003C\u002Fdetails>\n\n\u003Cdetails>\n\u003Csummary>📖 \u003Cstrong>2. Text Attribution\u003C\u002Fstrong>\u003C\u002Fsummary>\nAdd this line to your documentation:\n```\nThis project uses Crawl4AI (https:\u002F\u002Fgithub.com\u002Funclecode\u002Fcrawl4ai) for web data extraction.\n```\n\u003C\u002Fdetails>\n\n## 📚 Citation\n\nIf you use Crawl4AI in your research or project, please cite:\n\n```bibtex\n@software{crawl4ai2024,\n  author = {UncleCode},\n  title = {Crawl4AI: Open-source LLM Friendly Web Crawler & Scraper},\n  year = {2024},\n  publisher = {GitHub},\n  journal = {GitHub Repository},\n  howpublished = {\\url{https:\u002F\u002Fgithub.com\u002Funclecode\u002Fcrawl4ai}},\n  commit = {Please use the commit hash you're working with}\n}\n```\n\nText citation format:\n```\nUncleCode. (2024). Crawl4AI: Open-source LLM Friendly Web Crawler & Scraper [Computer software]. \nGitHub. https:\u002F\u002Fgithub.com\u002Funclecode\u002Fcrawl4ai\n```\n\n## 📧 Contact \n\nFor questions, suggestions, or feedback, feel free to reach out:\n\n- GitHub: [unclecode](https:\u002F\u002Fgithub.com\u002Funclecode)\n- Twitter: [@unclecode](https:\u002F\u002Ftwitter.com\u002Funclecode)\n- Website: [crawl4ai.com](https:\u002F\u002Fcrawl4ai.com)\n\nHappy Crawling! 🕸️🚀\n\n## 🗾 Mission\n\nOur mission is to unlock the value of personal and enterprise data by transforming digital footprints into structured, tradeable assets. Crawl4AI empowers individuals and organizations with open-source tools to extract and structure data, fostering a shared data economy.  \n\nWe envision a future where AI is powered by real human knowledge, ensuring data creators directly benefit from their contributions. By democratizing data and enabling ethical sharing, we are laying the foundation for authentic AI advancement.\n\n\u003Cdetails>\n\u003Csummary>🔑 \u003Cstrong>Key Opportunities\u003C\u002Fstrong>\u003C\u002Fsummary>\n \n- **Data Capitalization**: Transform digital footprints into measurable, valuable assets.  \n- **Authentic AI Data**: Provide AI systems with real human insights.  \n- **Shared Economy**: Create a fair data marketplace that benefits data creators.  \n\n\u003C\u002Fdetails>\n\n\u003Cdetails>\n\u003Csummary>🚀 \u003Cstrong>Development Pathway\u003C\u002Fstrong>\u003C\u002Fsummary>\n\n1. **Open-Source Tools**: Community-driven platforms for transparent data extraction.  \n2. **Digital Asset Structuring**: Tools to organize and value digital knowledge.  \n3. **Ethical Data Marketplace**: A secure, fair platform for exchanging structured data.  \n\nFor more details, see our [full mission statement](.\u002FMISSION.md).\n\u003C\u002Fdetails>\n\n## 🌟 Current Sponsors\n\n### 🏢 Enterprise Sponsors & Partners\n\nOur enterprise sponsors and technology partners help scale Crawl4AI to power production-grade data pipelines.\n\n| Company | About | Sponsorship Tier |\n|------|------|----------------------------|\n| \u003Ca href=\"https:\u002F\u002Fwww.thordata.com\u002F?ls=github&lk=crawl4ai\" target=\"_blank\">\u003Cimg src=\"https:\u002F\u002Fgist.github.com\u002Faravindkarnam\u002Fdfc598a67be5036494475acece7e54cf\u002Fraw\u002Fthor_data.svg\" alt=\"Thor Data\" width=\"120\"\u002F>\u003C\u002Fa>  | Leveraging Thordata ensures seamless compatibility with any AI\u002FML workflows and data infrastructure, massively accessing web data with 99.9% uptime, backed by one-on-one customer support. | 🥈 Silver |\n| \u003Ca href=\"https:\u002F\u002Fapp.nstproxy.com\u002Fregister?i=ecOqW9\" target=\"_blank\">\u003Cpicture>\u003Csource width=\"250\" media=\"(prefers-color-scheme: dark)\" srcset=\"https:\u002F\u002Fgist.github.com\u002Faravindkarnam\u002F62f82bd4818d3079d9dd3c31df432cf8\u002Fraw\u002Fnst-light.svg\">\u003Csource width=\"250\" media=\"(prefers-color-scheme: light)\" srcset=\"https:\u002F\u002Fwww.nstproxy.com\u002Flogo.svg\">\u003Cimg alt=\"nstproxy\" src=\"ttps:\u002F\u002Fwww.nstproxy.com\u002Flogo.svg\">\u003C\u002Fpicture>\u003C\u002Fa>  | NstProxy is a trusted proxy provider with over 110M+ real residential IPs, city-level targeting, 99.99% uptime, and low pricing at $0.1\u002FGB, it delivers unmatched stability, scale, and cost-efficiency. | 🥈 Silver |\n| \u003Ca href=\"https:\u002F\u002Fapp.scrapeless.com\u002Fpassport\u002Fregister?utm_source=official&utm_term=crawl4ai\" target=\"_blank\">\u003Cpicture>\u003Csource width=\"250\" media=\"(prefers-color-scheme: dark)\" srcset=\"https:\u002F\u002Fgist.githubusercontent.com\u002Faravindkarnam\u002F0d275b942705604263e5c32d2db27bc1\u002Fraw\u002FScrapeless-light-logo.svg\">\u003Csource width=\"250\" media=\"(prefers-color-scheme: light)\" srcset=\"https:\u002F\u002Fgist.githubusercontent.com\u002Faravindkarnam\u002F22d0525cc0f3021bf19ebf6e11a69ccd\u002Fraw\u002FScrapeless-dark-logo.svg\">\u003Cimg alt=\"Scrapeless\" src=\"https:\u002F\u002Fgist.githubusercontent.com\u002Faravindkarnam\u002F22d0525cc0f3021bf19ebf6e11a69ccd\u002Fraw\u002FScrapeless-dark-logo.svg\">\u003C\u002Fpicture>\u003C\u002Fa>  | Scrapeless provides production-grade infrastructure for Crawling, Automation, and AI Agents, offering Scraping Browser, 4 Proxy Types and Universal Scraping API. | 🥈 Silver |\n| \u003Ca href=\"https:\u002F\u002Fdashboard.capsolver.com\u002Fpassport\u002Fregister?inviteCode=ESVSECTX5Q23\" target=\"_blank\">\u003Cpicture>\u003Csource width=\"120\" media=\"(prefers-color-scheme: dark)\" srcset=\"https:\u002F\u002Fdocs.crawl4ai.com\u002Fuploads\u002Fsponsors\u002F20251013045338_72a71fa4ee4d2f40.png\">\u003Csource width=\"120\" media=\"(prefers-color-scheme: light)\" srcset=\"https:\u002F\u002Fwww.capsolver.com\u002Fassets\u002Fimages\u002Flogo-text.png\">\u003Cimg alt=\"Capsolver\" src=\"https:\u002F\u002Fwww.capsolver.com\u002Fassets\u002Fimages\u002Flogo-text.png\">\u003C\u002Fpicture>\u003C\u002Fa> | AI-powered Captcha solving service. Supports all major Captcha types, including reCAPTCHA, Cloudflare, and more | 🥉 Bronze |\n| \u003Ca href=\"https:\u002F\u002Fkipo.ai\" target=\"_blank\">\u003Cimg src=\"https:\u002F\u002Fdocs.crawl4ai.com\u002Fuploads\u002Fsponsors\u002F20251013045751_2d54f57f117c651e.png\" alt=\"DataSync\" width=\"120\"\u002F>\u003C\u002Fa> | Helps engineers and buyers find, compare, and source electronic & industrial parts in seconds, with specs, pricing, lead times & alternatives.| 🥇 Gold |\n| \u003Ca href=\"https:\u002F\u002Fwww.kidocode.com\u002F\" target=\"_blank\">\u003Cimg src=\"https:\u002F\u002Fdocs.crawl4ai.com\u002Fuploads\u002Fsponsors\u002F20251013045045_bb8dace3f0440d65.svg\" alt=\"Kidocode\" width=\"120\"\u002F>\u003Cp align=\"center\">KidoCode\u003C\u002Fp>\u003C\u002Fa> | Kidocode is a hybrid technology and entrepreneurship school for kids aged 5–18, offering both online and on-campus education. | 🥇 Gold |\n| \u003Ca href=\"https:\u002F\u002Fwww.alephnull.sg\u002F\" target=\"_blank\">\u003Cimg src=\"https:\u002F\u002Fdocs.crawl4ai.com\u002Fuploads\u002Fsponsors\u002F20251013050323_a9e8e8c4c3650421.svg\" alt=\"Aleph null\" width=\"120\"\u002F>\u003C\u002Fa> | Singapore-based  Aleph Null is Asia’s leading edtech hub, dedicated to student-centric, AI-driven education—empowering learners with the tools to thrive in a fast-changing world. | 🥇 Gold |\n\n\n\n### 🧑‍🤝 Individual Sponsors\n\nA heartfelt thanks to our individual supporters! Every contribution helps us keep our opensource mission alive and thriving!\n\n\u003Cp align=\"left\">\n  \u003Ca href=\"https:\u002F\u002Fgithub.com\u002Fhafezparast\">\u003Cimg src=\"https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F14273305?s=60&v=4\" style=\"border-radius:50%;\" width=\"64px;\"\u002F>\u003C\u002Fa>\n  \u003Ca href=\"https:\u002F\u002Fgithub.com\u002Fntohidi\">\u003Cimg src=\"https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F17140097?s=60&v=4\" style=\"border-radius:50%;\"width=\"64px;\"\u002F>\u003C\u002Fa>\n  \u003Ca href=\"https:\u002F\u002Fgithub.com\u002FSjoeborg\">\u003Cimg src=\"https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F17451310?s=60&v=4\" style=\"border-radius:50%;\"width=\"64px;\"\u002F>\u003C\u002Fa>\n  \u003Ca href=\"https:\u002F\u002Fgithub.com\u002Fromek-rozen\">\u003Cimg src=\"https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F30595969?s=60&v=4\" style=\"border-radius:50%;\"width=\"64px;\"\u002F>\u003C\u002Fa>\n  \u003Ca href=\"https:\u002F\u002Fgithub.com\u002FKourosh-Kiyani\">\u003Cimg src=\"https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F34105600?s=60&v=4\" style=\"border-radius:50%;\"width=\"64px;\"\u002F>\u003C\u002Fa>\n  \u003Ca href=\"https:\u002F\u002Fgithub.com\u002FEtherdrake\">\u003Cimg src=\"https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F67021215?s=60&v=4\" style=\"border-radius:50%;\"width=\"64px;\"\u002F>\u003C\u002Fa>\n  \u003Ca href=\"https:\u002F\u002Fgithub.com\u002Fshaman247\">\u003Cimg src=\"https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F211010067?s=60&v=4\" style=\"border-radius:50%;\"width=\"64px;\"\u002F>\u003C\u002Fa>\n  \u003Ca href=\"https:\u002F\u002Fgithub.com\u002Fwork-flow-manager\">\u003Cimg src=\"https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F217665461?s=60&v=4\" style=\"border-radius:50%;\"width=\"64px;\"\u002F>\u003C\u002Fa>\n\u003C\u002Fp>\n\n> Want to join them? [Sponsor Crawl4AI →](https:\u002F\u002Fgithub.com\u002Fsponsors\u002Funclecode)\n\n## Star History\n\n[![Star History Chart](https:\u002F\u002Fapi.star-history.com\u002Fsvg?repos=unclecode\u002Fcrawl4ai&type=Date)](https:\u002F\u002Fstar-history.com\u002F#unclecode\u002Fcrawl4ai&Date)\n","Crawl4AI 是一个开源的、对大语言模型友好的网页爬虫和数据抓取工具。它能够将网页内容转换为干净的Markdown格式，适用于检索增强生成（RAG）、代理及数据流水线等场景。该项目采用Python编写，具备快速可控、支持反爬虫检测与Shadow DOM处理等功能，并且经过了超过5万星标社区的实战检验。Crawl4AI特别适合需要从网络上大规模提取信息并将其结构化以供进一步分析或使用的开发者和研究人员。",2,"2026-06-11 02:36:20","top_all"]