[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"project-80496":3},{"id":4,"name":5,"fullName":6,"owner":7,"repo":5,"description":8,"homepage":9,"htmlUrl":9,"language":10,"languages":9,"totalLinesOfCode":9,"stars":11,"forks":12,"watchers":11,"openIssues":12,"contributorsCount":13,"subscribersCount":13,"size":13,"stars1d":13,"stars7d":13,"stars30d":13,"stars90d":13,"forks30d":13,"starsTrendScore":13,"compositeScore":14,"rankGlobal":9,"rankLanguage":9,"license":15,"archived":16,"fork":16,"defaultBranch":17,"hasWiki":18,"hasPages":16,"topics":19,"createdAt":9,"pushedAt":9,"updatedAt":20,"readmeContent":21,"aiSummary":22,"trendingCount":13,"starSnapshotCount":13,"syncStatus":23,"lastSyncTime":24,"discoverSource":25},80496,"supacrawl","davemorin\u002Fsupacrawl","davemorin","Mirror Supabase\u002FPostgres into local SQLite for search and offline analysis",null,"Go",55,5,0,42.33,"MIT License",false,"main",true,[],"2026-06-12 04:01:28","# supacrawl\n\n`supacrawl` mirrors Supabase\u002FPostgres project metadata into local SQLite so you can search, inspect, and query your backend without poking around the Supabase dashboard.\n\nIt follows the local-first shape of tools like `discrawl` and `slacrawl`: crawl once, keep the archive on your machine, use fast local search, and run read-only SQL against the snapshot.\n\n`supacrawl` is part of the OpenClaw crawler family and is intended for teams,\nmaintainers, and agents that need fast local access to Supabase project shape,\npolicies, functions, Storage inventory, and copied table rows.\n\n## Included\n\n- local SQLite archive with FTS5 search\n- Supabase\u002FPostgres schema crawl over a normal Postgres connection URL\n- schemas, tables, columns, indexes, constraints, RLS policies, functions, triggers, extensions\n- optional full table row mirror into SQLite as schema-agnostic JSON rows\n- optional row-copy mode without FTS for smaller archives\n- local archive size reports\n- automatic metadata refresh for read commands, with opt-out flags\n- per-table JSONL\u002FCSV export from the local copy\n- Storage blob downloads from copied `storage.objects` rows\n- encrypted local backup shards with age\n- Supabase Storage buckets and object counts when the `storage` schema is visible\n- `doctor` diagnostics for local archive and database connectivity\n- `metadata --json`, `status --json`, `doctor --json`, and read-only `sql`\n- JSON output for automation with `--json` or `--format json`\n\n## Not Yet Included\n\n- Supabase Management API crawl for edge functions, auth settings, branches, or secrets\n- git-backed archive publish\u002Fsubscribe\n- terminal UI\n- Homebrew\u002Frelease packaging\n- git-backed snapshot sharing through `crawlkit`\n- terminal archive UI through `crawlkit`\n\n## Requirements\n\n- Go 1.26+\n- a Supabase direct or pooler Postgres connection URL\n- enough Postgres privileges to read catalog metadata\n\n## Install\n\nHomebrew:\n\n```bash\nbrew install davemorin\u002Ftap\u002Fsupacrawl\n```\n\nBuild from source:\n\n```bash\ngit clone https:\u002F\u002Fgithub.com\u002Fdavemorin\u002Fsupacrawl.git\ncd supacrawl\ngo build -o bin\u002Fsupacrawl .\u002Fcmd\u002Fsupacrawl\n.\u002Fbin\u002Fsupacrawl version\n```\n\n## Quick Start\n\n```bash\nexport SUPABASE_DB_URL=\"postgres:\u002F\u002Fpostgres.\u003Cref>:\u003Cpassword>@aws-0-us-west-1.pooler.supabase.com:6543\u002Fpostgres?sslmode=require\"\n\ngo run .\u002Fcmd\u002Fsupacrawl init --project-id \u003Cref>\ngo run .\u002Fcmd\u002Fsupacrawl doctor\ngo run .\u002Fcmd\u002Fsupacrawl metadata --json\ngo run .\u002Fcmd\u002Fsupacrawl sync\ngo run .\u002Fcmd\u002Fsupacrawl sync --full\ngo run .\u002Fcmd\u002Fsupacrawl sync --full --no-row-fts\ngo run .\u002Fcmd\u002Fsupacrawl status\ngo run .\u002Fcmd\u002Fsupacrawl status --sync never\ngo run .\u002Fcmd\u002Fsupacrawl size\ngo run .\u002Fcmd\u002Fsupacrawl search \"auth policies\"\ngo run .\u002Fcmd\u002Fsupacrawl diff ~\u002F.supacrawl\u002Fsupacrawl-before.db\ngo run .\u002Fcmd\u002Fsupacrawl export --type jsonl --out companies.jsonl public.companies\ngo run .\u002Fcmd\u002Fsupacrawl storage pull --dir .\u002Fsupabase-storage --limit 10\ngo run .\u002Fcmd\u002Fsupacrawl backup keygen --out ~\u002F.supacrawl\u002Fage.key\ngo run .\u002Fcmd\u002Fsupacrawl backup init --recipient age1...\ngo run .\u002Fcmd\u002Fsupacrawl backup push\ngo run .\u002Fcmd\u002Fsupacrawl backup status\ngo run .\u002Fcmd\u002Fsupacrawl sql \"select schema_name, name, rls_enabled from tables order by schema_name, name limit 20\"\ngo run .\u002Fcmd\u002Fsupacrawl sql \"select row_json from table_rows where schema_name = 'public' and table_name = 'companies' limit 5\"\n```\n\nIf you build the binary, replace `go run .\u002Fcmd\u002Fsupacrawl` with `.\u002Fbin\u002Fsupacrawl`.\n\n## Commands\n\n- `init` creates `~\u002F.supacrawl\u002Fconfig.toml`\n- `doctor` checks the local SQLite archive and Supabase\u002FPostgres connection\n- `metadata` prints command\u002Fcapability metadata for launchers and agents\n- `version` prints the CLI version\n- `sync` crawls metadata into the local archive\n- `sync --data` or `sync --full` also copies base table rows into `table_rows`\n- `status` prints archive counts\n- `report` summarizes schemas and policy coverage\n- `diff` compares the current archive to another archive file\n- `size` reports archive file size and largest copied source tables\n- `search` searches crawled tables, functions, storage buckets, and extensions\n- `export` writes copied source rows for one table as JSONL or CSV\n- `storage pull` downloads Storage blobs into a local directory\n- `backup keygen` creates an age identity and prints the public recipient\n- `backup init` stores local backup settings in config\n- `backup push` writes encrypted JSONL gzip shards plus a manifest\n- `backup status` prints backup manifest metadata\n- `backup pull` decrypts backup shards into a local restore directory\n- `sql` runs read-only SQL against the local SQLite archive\n\n## Configuration\n\nDefault config path:\n\n```text\n~\u002F.supacrawl\u002Fconfig.toml\n```\n\nDefault archive path:\n\n```text\n~\u002F.supacrawl\u002Fsupacrawl.db\n```\n\nThe default secret handling expects the connection string in `SUPABASE_DB_URL`. You can change that during init:\n\n```bash\nsupacrawl init --database-url-env MY_SUPABASE_DB_URL\n```\n\n`--database-url` is available for local experiments, but the env-var flow is the safer default because the config file is durable.\n\nStorage downloads also need:\n\n```bash\nexport SUPABASE_URL=\"https:\u002F\u002F\u003Cref>.supabase.co\"\nexport SUPABASE_SECRET_KEY=\"sb_secret_...\"\n```\n\nIf `SUPABASE_URL` is not set, `supacrawl` will also try `NEXT_PUBLIC_SUPABASE_URL`.\nIf `SUPABASE_SECRET_KEY` is not set, `supacrawl` will also try the legacy\n`SUPABASE_SERVICE_ROLE_KEY` name.\n\nSupabase API keys are separate from the Postgres connection string. Use\n`SUPABASE_DB_URL` for metadata and row sync. Use `SUPABASE_SECRET_KEY` only for\nprivate Storage downloads.\n\n## Read Freshness\n\nRead commands refresh metadata automatically when the local snapshot is stale:\n\n```bash\nsupacrawl status\nsupacrawl search \"profiles\"\nsupacrawl sql \"select count(*) from tables\"\n```\n\nThe default policy is `auto` with a `15m` staleness window. If the database URL\nis not available, `auto` continues with the local archive. Use explicit flags\nwhen you need a deterministic read:\n\n```bash\nsupacrawl status --sync never\nsupacrawl report --sync always\nsupacrawl search --stale-after 1h \"auth.uid\"\n```\n\n## Full Local Copy Mode\n\n`sync --full` mirrors accessible base-table rows into SQLite:\n\n```bash\nsupacrawl sync --full\n```\n\nRows are stored as JSON in `table_rows`:\n\n```sql\nselect\n  schema_name,\n  table_name,\n  json_extract(row_json, '$.id') as id\nfrom table_rows\nwhere schema_name = 'public'\nlimit 20\n```\n\nThis makes the first full-copy mode robust across arbitrary Supabase schemas. It is not yet a byte-for-byte Supabase backup: Edge Function source and project dashboard settings are still future work.\n\nUse `--no-row-fts` when archive size matters more than full-text search over row JSON:\n\n```bash\nsupacrawl sync --full --no-row-fts\n```\n\nUse `--exclude-table` for pathological tables you want to skip during row copy:\n\n```bash\nsupacrawl sync --full --no-row-fts --exclude-table public.enrichments\n```\n\n## Diff\n\nCompare the current archive against another SQLite archive file:\n\n```bash\nsupacrawl diff \u003Cother-archive.db>\n```\n\nThe first diff compares tables, policies, and Storage buckets, including RLS,\npolicy expressions, and public bucket changes.\n\n## Export\n\n```bash\nsupacrawl export --type jsonl --out companies.jsonl public.companies\nsupacrawl export --type csv --out companies.csv public.companies\n```\n\n## Storage Blobs\n\nAfter a full sync has copied `storage.objects`, download blobs:\n\n```bash\nsupacrawl storage pull --dir .\u002Fsupabase-storage\n```\n\nThe downloader preserves bucket\u002Fobject paths under the target directory.\n\n## Encrypted Backups\n\nGenerate a local age identity once:\n\n```bash\nsupacrawl backup keygen --out ~\u002F.supacrawl\u002Fage.key\n```\n\nStore the public recipient in config or in `SUPACRAWL_AGE_RECIPIENT`:\n\n```bash\nsupacrawl backup init --recipient age1...\n```\n\nCreate an encrypted backup from the SQLite archive:\n\n```bash\nsupacrawl backup push\nsupacrawl backup status\n```\n\nRestore decrypted JSONL gzip shards to a directory:\n\n```bash\nsupacrawl backup pull --out .\u002Fsupacrawl-restore\n```\n\n`backup push` is local-only today. It writes encrypted shards under\n`~\u002F.supacrawl\u002Fbackups` by default and does not push to a remote.\n\n## OpenClaw Compatibility\n\n`supacrawl` is designed to fit the OpenClaw local-first crawler family:\n\n- `doctor` is the fastest live sanity check.\n- `metadata --json`, `status --json`, and `doctor --json` are stable surfaces for launchers, agents, and CI.\n- `sync` is metadata-first; `sync --full` is explicit for copying rows.\n- `sync --full` prints per-table progress to stderr so JSON stdout stays parseable.\n- read commands default to a bounded auto-refresh policy and accept `--sync never`.\n- backups are encrypted local shards; private identities are read from env or disk.\n- Secrets are resolved from environment variables and are not written into the archive by default.\n- Analysis happens against local SQLite through read-only `sql`.\n\nSee [docs\u002Fopenclaw.md](docs\u002Fopenclaw.md) for agent rules and convention notes.\n\n## Output Modes\n\n```bash\nsupacrawl status --json\nsupacrawl --format json search \"profiles\"\nsupacrawl --format log sync\n```\n\n## Local Development\n\n```bash\ngo test .\u002F...\ngo vet .\u002F...\ngo build -o bin\u002Fsupacrawl .\u002Fcmd\u002Fsupacrawl\n.\u002Fscripts\u002Fvalidate-local.sh\n```\n","`supacrawl`是一个用于将Supabase\u002FPostgres项目元数据镜像到本地SQLite数据库的工具，便于用户进行搜索、检查和离线分析。其核心功能包括通过标准Postgres连接URL爬取并存储如表结构、索引、约束等信息至本地SQLite档案中，并支持全表行数据作为JSON格式复制。该工具还提供全文搜索引擎FTS5、自动元数据刷新、每表JSONL\u002FCSV导出以及加密本地备份等功能。适用于需要快速本地访问Supabase项目结构及其相关资源（如策略、函数、存储清单）的团队和个人。此外，它还允许直接从本地副本运行只读SQL查询，非常适合那些希望减少对在线服务依赖同时提高工作效率的场景。",2,"2026-06-11 04:00:59","CREATED_QUERY"]