[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"project-4715":3},{"id":4,"name":5,"fullName":6,"owner":7,"repo":5,"description":8,"homepage":9,"htmlUrl":10,"language":11,"languages":10,"totalLinesOfCode":10,"stars":12,"forks":13,"watchers":14,"openIssues":15,"contributorsCount":16,"subscribersCount":16,"size":16,"stars1d":17,"stars7d":18,"stars30d":19,"stars90d":16,"forks30d":16,"starsTrendScore":20,"compositeScore":21,"rankGlobal":10,"rankLanguage":10,"license":22,"archived":23,"fork":23,"defaultBranch":24,"hasWiki":25,"hasPages":23,"topics":26,"createdAt":10,"pushedAt":10,"updatedAt":35,"readmeContent":36,"aiSummary":37,"trendingCount":16,"starSnapshotCount":16,"syncStatus":38,"lastSyncTime":39,"discoverSource":40},4715,"colly","gocolly\u002Fcolly","gocolly","Elegant Scraper and Crawler Framework for Golang","https:\u002F\u002Fgo-colly.org\u002F",null,"Go",25330,1850,314,143,0,4,20,59,21,44.8,"Apache License 2.0",false,"master",true,[27,28,29,30,31,32,33,34],"crawler","crawling","framework","go","golang","scraper","scraping","spider","2026-06-12 02:01:03","# Colly\n\nLightning Fast and Elegant Scraping Framework for Gophers\n\nColly provides a clean interface to write any kind of crawler\u002Fscraper\u002Fspider.\n\nWith Colly you can easily extract structured data from websites, which can be used for a wide range of applications, like data mining, data processing or archiving.\n\n[![GoDoc](https:\u002F\u002Fgodoc.org\u002Fgithub.com\u002Fgocolly\u002Fcolly?status.svg)](https:\u002F\u002Fpkg.go.dev\u002Fgithub.com\u002Fgocolly\u002Fcolly\u002Fv2)\n[![Backers on Open Collective](https:\u002F\u002Fopencollective.com\u002Fcolly\u002Fbackers\u002Fbadge.svg)](#backers) [![Sponsors on Open Collective](https:\u002F\u002Fopencollective.com\u002Fcolly\u002Fsponsors\u002Fbadge.svg)](#sponsors) [![build status](https:\u002F\u002Fgithub.com\u002Fgocolly\u002Fcolly\u002Factions\u002Fworkflows\u002Fci.yml\u002Fbadge.svg)](https:\u002F\u002Fgithub.com\u002Fgocolly\u002Fcolly\u002Factions\u002Fworkflows\u002Fci.yml)\n[![report card](https:\u002F\u002Fimg.shields.io\u002Fbadge\u002Freport%20card-a%2B-ff3333.svg?style=flat-square)](http:\u002F\u002Fgoreportcard.com\u002Freport\u002Fgocolly\u002Fcolly)\n[![view examples](https:\u002F\u002Fimg.shields.io\u002Fbadge\u002Flearn%20by-examples-0077b3.svg?style=flat-square)](https:\u002F\u002Fgithub.com\u002Fgocolly\u002Fcolly\u002Ftree\u002Fmaster\u002F_examples)\n[![Code Coverage](https:\u002F\u002Fimg.shields.io\u002Fcodecov\u002Fc\u002Fgithub\u002Fgocolly\u002Fcolly\u002Fmaster.svg)](https:\u002F\u002Fcodecov.io\u002Fgithub\u002Fgocolly\u002Fcolly?branch=master)\n[![FOSSA Status](https:\u002F\u002Fapp.fossa.io\u002Fapi\u002Fprojects\u002Fgit%2Bgithub.com%2Fgocolly%2Fcolly.svg?type=shield)](https:\u002F\u002Fapp.fossa.io\u002Fprojects\u002Fgit%2Bgithub.com%2Fgocolly%2Fcolly?ref=badge_shield)\n[![Twitter URL](https:\u002F\u002Fimg.shields.io\u002Fbadge\u002Ftwitter-follow-green.svg)](https:\u002F\u002Ftwitter.com\u002Fgocolly)\n\n\n## Sponsor\n\n**[Swiftproxy](https:\u002F\u002Fwww.swiftproxy.net\u002F?ref=colly) – Enterprise Residential Proxy Infrastructure for Scalable Data Access**\n\nSwiftproxy is a high-performance residential proxy network designed for reliable large-scale data collection. Powered by an 80M+ IP pool across 195+ countries, it ensures stable, anonymous, and high-success-rate connections. It integrates seamlessly with tools like Colly to bypass IP blocks and anti-bot systems. Free trial available, plus use code PROXY90 for an exclusive discount.\n\n[![Swiftproxy](assets\u002Fswiftproxy.png)](https:\u002F\u002Fwww.swiftproxy.net\u002F?ref=colly)\n\n\n## Features\n\n-   Clean API\n-   Fast (>1k request\u002Fsec on a single core)\n-   Manages request delays and maximum concurrency per domain\n-   Automatic cookie and session handling\n-   Sync\u002Fasync\u002Fparallel scraping\n-   Caching\n-   Automatic encoding of non-unicode responses\n-   Robots.txt support\n-   Distributed scraping\n-   Configuration via environment variables\n-   Extensions\n\n## Example\n\n```go\n\nimport (\n\t\"fmt\"\n\n\t\"github.com\u002Fgocolly\u002Fcolly\u002Fv2\"\n)\n\nfunc main() {\n\tc := colly.NewCollector()\n\n\t\u002F\u002F Find and visit all links\n\tc.OnHTML(\"a[href]\", func(e *colly.HTMLElement) {\n\t\te.Request.Visit(e.Attr(\"href\"))\n\t})\n\n\tc.OnRequest(func(r *colly.Request) {\n\t\tfmt.Println(\"Visiting\", r.URL)\n\t})\n\n\tc.Visit(\"http:\u002F\u002Fgo-colly.org\u002F\")\n}\n```\n\nSee [examples folder](https:\u002F\u002Fgithub.com\u002Fgocolly\u002Fcolly\u002Ftree\u002Fmaster\u002F_examples) for more detailed examples.\n\n## Installation\n\n`go get github.com\u002Fgocolly\u002Fcolly\u002Fv2`\n\n\n## Bugs\n\nBugs or suggestions? Visit the [issue tracker](https:\u002F\u002Fgithub.com\u002Fgocolly\u002Fcolly\u002Fissues) or join `#colly` on freenode\n\n## Other Projects Using Colly\n\nBelow is a list of public, open source projects that use Colly:\n\n-   [greenpeace\u002Fcheck-my-pages](https:\u002F\u002Fgithub.com\u002Fgreenpeace\u002Fcheck-my-pages) Scraping script to test the Spanish Greenpeace web archive.\n-   [altsab\u002Fgowap](https:\u002F\u002Fgithub.com\u002Faltsab\u002Fgowap) Wappalyzer implementation in Go.\n-   [jesuiscamille\u002Fgoquotes](https:\u002F\u002Fgithub.com\u002Fjesuiscamille\u002Fgoquotes) A quotes scraper, making your day a little better!\n-   [jivesearch\u002Fjivesearch](https:\u002F\u002Fgithub.com\u002Fjivesearch\u002Fjivesearch) A search engine that doesn't track you.\n-   [Leagify\u002Fcolly-draft-prospects](https:\u002F\u002Fgithub.com\u002FLeagify\u002Fcolly-draft-prospects) A scraper for future NFL Draft prospects.\n-   [lucasepe\u002Fgo-ps4](https:\u002F\u002Fgithub.com\u002Flucasepe\u002Fgo-ps4) Search playstation store for your favorite PS4 games using the command line.\n-   [yringler\u002Finside-chassidus-scraper](https:\u002F\u002Fgithub.com\u002Fyringler\u002Finside-chassidus-scraper) Scrapes Rabbi Paltiel's web site for lesson metadata.\n-   [gamedb\u002Fgamedb](https:\u002F\u002Fgithub.com\u002Fgamedb\u002Fgamedb) A database of Steam games.\n-   [lawzava\u002Fscrape](https:\u002F\u002Fgithub.com\u002Flawzava\u002Fscrape) CLI for email scraping from any website.\n-   [eureka101v\u002FWeiboSpiderGo](https:\u002F\u002Fgithub.com\u002Feureka101v\u002FWeiboSpiderGo) A sina weibo(chinese twitter) scraper\n-   [Go-phie\u002Fgophie](https:\u002F\u002Fgithub.com\u002FGo-phie\u002Fgophie) Search, Download and Stream movies from your terminal\n-   [imthaghost\u002Fgoclone](https:\u002F\u002Fgithub.com\u002Fimthaghost\u002Fgoclone) Clone websites to your computer within seconds.\n-   [superiss\u002Fspidy](https:\u002F\u002Fgithub.com\u002Fsuperiss\u002Fspidy) Crawl the web and collect expired domains.\n-   [docker-slim\u002Fdocker-slim](https:\u002F\u002Fgithub.com\u002Fdocker-slim\u002Fdocker-slim) Optimize your Docker containers to make them smaller and better.\n-   [seversky\u002Fgachifinder](https:\u002F\u002Fgithub.com\u002Fseversky\u002Fgachifinder) an agent for asynchronous scraping, parsing and writing to some storages(elasticsearch for now)\n-   [eval-exec\u002Fgoodreads](https:\u002F\u002Fgithub.com\u002Feval-exec\u002Fgoodreads) crawl all tags and all pages of quotes from goodreads.\n\nIf you are using Colly in a project please send a pull request to add it to the list.\n\n## Contributors\n\nThis project exists thanks to all the people who contribute. [[Contribute]](CONTRIBUTING.md).\n\u003Ca href=\"https:\u002F\u002Fgithub.com\u002Fgocolly\u002Fcolly\u002Fgraphs\u002Fcontributors\">\u003Cimg src=\"https:\u002F\u002Fopencollective.com\u002Fcolly\u002Fcontributors.svg?width=890\" \u002F>\u003C\u002Fa>\n\n## Backers\n\nThank you to all our backers! 🙏 [[Become a backer](https:\u002F\u002Fopencollective.com\u002Fcolly#backer)]\n\n\u003Ca href=\"https:\u002F\u002Fopencollective.com\u002Fcolly#backers\" target=\"_blank\">\u003Cimg src=\"https:\u002F\u002Fopencollective.com\u002Fcolly\u002Fbackers.svg?width=890\">\u003C\u002Fa>\n\n## Sponsors\n\nSupport this project by becoming a sponsor. Your logo will show up here with a link to your website. [[Become a sponsor](https:\u002F\u002Fopencollective.com\u002Fcolly#sponsor)]\n\n\u003Ca href=\"https:\u002F\u002Fopencollective.com\u002Fcolly\u002Fsponsor\u002F0\u002Fwebsite\" target=\"_blank\">\u003Cimg src=\"https:\u002F\u002Fopencollective.com\u002Fcolly\u002Fsponsor\u002F0\u002Favatar.svg\">\u003C\u002Fa>\n\u003Ca href=\"https:\u002F\u002Fopencollective.com\u002Fcolly\u002Fsponsor\u002F1\u002Fwebsite\" target=\"_blank\">\u003Cimg src=\"https:\u002F\u002Fopencollective.com\u002Fcolly\u002Fsponsor\u002F1\u002Favatar.svg\">\u003C\u002Fa>\n\u003Ca href=\"https:\u002F\u002Fopencollective.com\u002Fcolly\u002Fsponsor\u002F2\u002Fwebsite\" target=\"_blank\">\u003Cimg src=\"https:\u002F\u002Fopencollective.com\u002Fcolly\u002Fsponsor\u002F2\u002Favatar.svg\">\u003C\u002Fa>\n\u003Ca href=\"https:\u002F\u002Fopencollective.com\u002Fcolly\u002Fsponsor\u002F3\u002Fwebsite\" target=\"_blank\">\u003Cimg src=\"https:\u002F\u002Fopencollective.com\u002Fcolly\u002Fsponsor\u002F3\u002Favatar.svg\">\u003C\u002Fa>\n\u003Ca href=\"https:\u002F\u002Fopencollective.com\u002Fcolly\u002Fsponsor\u002F4\u002Fwebsite\" target=\"_blank\">\u003Cimg src=\"https:\u002F\u002Fopencollective.com\u002Fcolly\u002Fsponsor\u002F4\u002Favatar.svg\">\u003C\u002Fa>\n\u003Ca href=\"https:\u002F\u002Fopencollective.com\u002Fcolly\u002Fsponsor\u002F5\u002Fwebsite\" target=\"_blank\">\u003Cimg src=\"https:\u002F\u002Fopencollective.com\u002Fcolly\u002Fsponsor\u002F5\u002Favatar.svg\">\u003C\u002Fa>\n\u003Ca href=\"https:\u002F\u002Fopencollective.com\u002Fcolly\u002Fsponsor\u002F6\u002Fwebsite\" target=\"_blank\">\u003Cimg src=\"https:\u002F\u002Fopencollective.com\u002Fcolly\u002Fsponsor\u002F6\u002Favatar.svg\">\u003C\u002Fa>\n\u003Ca href=\"https:\u002F\u002Fopencollective.com\u002Fcolly\u002Fsponsor\u002F7\u002Fwebsite\" target=\"_blank\">\u003Cimg src=\"https:\u002F\u002Fopencollective.com\u002Fcolly\u002Fsponsor\u002F7\u002Favatar.svg\">\u003C\u002Fa>\n\u003Ca href=\"https:\u002F\u002Fopencollective.com\u002Fcolly\u002Fsponsor\u002F8\u002Fwebsite\" target=\"_blank\">\u003Cimg src=\"https:\u002F\u002Fopencollective.com\u002Fcolly\u002Fsponsor\u002F8\u002Favatar.svg\">\u003C\u002Fa>\n\u003Ca href=\"https:\u002F\u002Fopencollective.com\u002Fcolly\u002Fsponsor\u002F9\u002Fwebsite\" target=\"_blank\">\u003Cimg src=\"https:\u002F\u002Fopencollective.com\u002Fcolly\u002Fsponsor\u002F9\u002Favatar.svg\">\u003C\u002Fa>\n\n## License\n\n[![FOSSA Status](https:\u002F\u002Fapp.fossa.io\u002Fapi\u002Fprojects\u002Fgit%2Bgithub.com%2Fgocolly%2Fcolly.svg?type=large)](https:\u002F\u002Fapp.fossa.io\u002Fprojects\u002Fgit%2Bgithub.com%2Fgocolly%2Fcolly?ref=badge_large)\n","Colly 是一个为 Go 语言设计的高效且优雅的网页抓取和爬虫框架。它提供了简洁的接口，支持快速（单核上每秒超过1000次请求）的数据抓取与处理，具备自动化的 cookie 和会话管理、请求延迟控制及最大并发数限制等功能，并支持同步、异步和平行抓取模式。此外，Colly 还集成了缓存机制、非Unicode响应编码自动化、Robots.txt遵循以及分布式抓取等特性。适用于需要从网站中提取结构化数据的各种场景，如数据挖掘、信息归档或内容分析等任务。",2,"2026-06-11 03:00:12","top_language"]