[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"project-95855":3},{"id":4,"name":5,"fullName":6,"owner":7,"repo":5,"description":8,"homepage":9,"htmlUrl":10,"language":11,"languages":10,"totalLinesOfCode":10,"stars":12,"forks":13,"watchers":14,"openIssues":15,"contributorsCount":15,"subscribersCount":15,"size":15,"stars1d":15,"stars7d":15,"stars30d":16,"stars90d":15,"forks30d":15,"starsTrendScore":15,"compositeScore":17,"rankGlobal":10,"rankLanguage":10,"license":18,"archived":19,"fork":19,"defaultBranch":20,"hasWiki":19,"hasPages":19,"topics":21,"createdAt":10,"pushedAt":10,"updatedAt":27,"readmeContent":28,"aiSummary":29,"trendingCount":15,"starSnapshotCount":15,"syncStatus":30,"lastSyncTime":31,"discoverSource":32},95855,"polyledger","nahrek\u002Fpolyledger","nahrek","Resumable Polymarket indexer: CLOB market metadata plus on-chain trades from Polygon, in one DuckDB file you can query with SQL","",null,"Python",621,109,13,0,279,10.12,"MIT License",false,"main",[22,23,24,25,26],"data-science","hypersync","polymarket","polymarketprediction-markets","prediction-market","2026-09-21 02:04:28","# PolyLedger\n\nA resumable indexer for Polymarket market metadata and on-chain trade data, backed by DuckDB.\n\nPolyLedger pulls every market from the Polymarket CLOB API, streams every `OrderFilled` event from Polygon via [Envio HyperSync](https:\u002F\u002Fenvio.dev), and writes both into a single DuckDB file you can query with SQL immediately.\n\n---\n\n## Table of contents\n\n- [Features](#features)\n- [Requirements](#requirements)\n- [Installation](#installation)\n- [Quick start](#quick-start)\n- [CLI reference](#cli-reference)\n- [Data model](#data-model)\n- [Indexed contracts](#indexed-contracts)\n- [Configuration](#configuration)\n- [Design notes](#design-notes)\n- [Development](#development)\n- [Limitations](#limitations)\n- [License](#license)\n\n## Features\n\n- **Resumable by construction.** Rows and the block cursor are committed in a single transaction, so an interrupted run resumes exactly where it stopped.\n- **Idempotent writes.** `order_fills` is keyed on `(transaction_hash, log_index)` and inserted with `ON CONFLICT DO NOTHING`. Re-running a range is a no-op.\n- **DuckDB storage.** Columnar compression, real types, indexes, and SQL joins, all in one file on disk. No server.\n- **Both contract generations.** Decodes the V2 `OrderFilled` layout and the older V1 one, with separate checkpoints per stream.\n- **Schema validation.** Every API response passes through Pydantic models, so an upstream format change fails loudly instead of writing nulls for hours.\n- **Resilient networking.** Exponential backoff with jitter, `Retry-After` support, and a shared token-bucket rate limiter across all REST sources.\n- **Gap recovery.** Token ids missing from the CLOB listing are flagged and backfilled from the Gamma API rather than silently dropped.\n- **Parquet export.** One command dumps every table for use with pandas, polars, or Spark.\n\n## Requirements\n\n- Python 3.11 or newer\n- A HyperSync API token (required since November 2025; the free tier is sufficient). Register at [envio.dev](https:\u002F\u002Fenvio.dev).\n\nThe CLOB and Gamma APIs are public and need no credentials.\n\n## Installation\n\n\n```bat\ngit clone https:\u002F\u002Fgithub.com\u002Fnahrek\u002Fpolyledger\ncd polyledger\npython -m venv .venv\n.venv\\Scripts\\activate.bat\npip install -e .\n```\n\nThen configure the HyperSync token. Copy `.env.example` to `.env`, open it in any\neditor, and set `HYPERSYNC_BEARER_TOKEN`. PolyLedger reads `.env` from the working\ndirectory on startup, so there is no export step. Real environment variables still\ntake precedence if you prefer to set them that way.\n\n## Quick start\n\nRun the full pipeline:\n\n```bash\npolyledger sync\n```\n\nThe first backfill is slow. On the free HyperSync tier it takes anywhere from a few hours to a couple of days depending on how much history you want. It is safe to interrupt at any point. Every subsequent run is incremental and finishes in seconds.\n\nTo verify the setup on a small slice first:\n\n```bash\npolyledger markets                    # metadata only, a few minutes\npolyledger chain --max-blocks 200000  # roughly five days of Polygon history\npolyledger stats\n```\n\n`stats` reports what is actually in the database:\n\n```\n  markets          14231\n  tokens           28455\n  fills            1204773\n  unmatched_fills  392\n  first_block      75014820\n  last_block       75214820\n  first_time       2026-05-14 08:11:23\n  last_time        2026-05-19 03:44:02\n  checkpoints:\n    order_filled:v2: block 75214820, 1204773 rows, updated 2026-09-02 15:11:23\n```\n\nThen query it:\n\n```bash\npolyledger query \"SELECT question, sum(usd_size) AS volume FROM trades GROUP BY 1 ORDER BY 2 DESC LIMIT 10\"\n```\n\n## CLI reference\n\n| Command | Description |\n| --- | --- |\n| `polyledger markets [--backfill]` | Sync market metadata from the CLOB. `--backfill` also resolves unmatched token ids via Gamma. |\n| `polyledger chain [options]` | Index `OrderFilled` logs from Polygon. |\n| `polyledger trades [--drop-unmatched]` | Materialise the joined `trades_mat` table. |\n| `polyledger sync [options]` | Run markets → chain → backfill → trades in order. |\n| `polyledger stats` | Show row counts, block range, and checkpoints. |\n| `polyledger export [--out DIR]` | Write every table to Parquet. |\n| `polyledger query \"SELECT ...\"` | Run one SQL statement and print the result. |\n\n`chain` options:\n\n| Flag | Description |\n| --- | --- |\n| `--contracts v2\\|v1\\|all` | Which exchange generation to index. Default `v2`. |\n| `--from-block N` | Override the start block. Ignored if a checkpoint exists. |\n| `--to-block N` | Stop at this block. |\n| `--max-blocks N` | Index at most this many blocks, then exit. |\n\nGlobal flags: `--db PATH` to use a different database file, `-v` for verbose logging.\n\nThe `polyledger` executable is installed by `pip install -e .`. Without installing, `python -m polyledger` and `python -m polyledger.cli` are equivalent. The entry point is `main()` in `polyledger\u002Fcli.py`.\n\n## Data model\n\n### `trades`\n\nThe main analytical view, joining chain fills to market metadata. Always live and always current. Run `polyledger trades` to materialise it as `trades_mat` if you plan to issue many queries against it.\n\n| Column | Description |\n| --- | --- |\n| `block_time`, `block_number` | Block timestamp (UTC) and height |\n| `question`, `market_slug`, `outcome` | Human-readable context |\n| `token_id`, `condition_id` | Outcome and market identifiers |\n| `maker`, `taker` | Counterparty addresses |\n| `maker_side`, `taker_side` | `BUY` \u002F `SELL`, both sides stated explicitly |\n| `price` | Execution price, 0 to 1 |\n| `shares` | Outcome share quantity |\n| `usd_size` | Notional in dollars |\n| `fee` | Fee from the event |\n| `exchange_name`, `exchange_version` | Which contract emitted the fill |\n| `unmatched_market` | `true` when no market metadata was found |\n\n`price`, `shares`, and `usd_size` are not stored on chain. They are derived from `makerAmountFilled` and `takerAmountFilled` according to the maker's side: when the maker buys, their amount is collateral and the taker's is shares; when the maker sells, it is the other way round. Both are base units with 6 decimals, matching pUSD.\n\nMany datasets collapse the two sides into a single ambiguous `side` column. PolyLedger records both, so you never have to guess whose perspective a row is written from.\n\n### Other tables\n\n| Table | Contents |\n| --- | --- |\n| `markets` | One row per market (`condition_id`), with question, slug, `active`\u002F`closed`\u002F`neg_risk` flags, tick size, and fees |\n| `tokens` | One row per outcome (`token_id`). This is the join key against fills |\n| `order_fills` | Raw decoded logs with no interpretation applied. Build your own join from here if you disagree with the price math |\n| `checkpoints` | One row per indexing stream |\n\nReady-made queries for daily volume, OHLC candles, VWAP, per-address positions, and top traders are in [`examples\u002Fqueries.sql`](examples\u002Fqueries.sql).\n\nThe database is a plain DuckDB file:\n\n```python\nimport duckdb\ncon = duckdb.connect(\"data\u002Fpolyledger.duckdb\", read_only=True)\ndf = con.sql(\"SELECT * FROM trades WHERE token_id = '1001'\").df()\n```\n\nOr export it with `polyledger export` and read the Parquet files from anywhere.\n\n## Indexed contracts\n\nV2 contracts, live since the April 2026 migration, are indexed by default:\n\n| Contract | Address |\n| --- | --- |\n| CTF Exchange V2 | `0xE111180000d2663C0091e4f400237545B87B996B` |\n| NegRisk CTF Exchange V2 | `0xe2222d279d744050d28e00520010520000310F59` |\n| NegRisk CTF Exchange V2 (b) | `0xe2222d002000ba0053cef3375333610f64600036` |\n\nUse `--contracts v1` for the frozen V1 contracts (`0x4bFb41d5…`, `0xC5d563A3…`), or `--contracts all` for both generations. Each stream keeps its own checkpoint, so switching modes will not corrupt a cursor built by earlier runs.\n\nThe two generations emit different event layouts, and the decoder handles both:\n\n```\nV2: OrderFilled(bytes32 orderHash, address maker, address taker, uint8 side,\n                uint256 tokenId, uint256 makerAmountFilled,\n                uint256 takerAmountFilled, uint256 fee,\n                bytes32 builder, bytes32 metadata)\n\nV1: OrderFilled(bytes32 orderHash, address maker, address taker,\n                uint256 makerAssetId, uint256 takerAssetId,\n                uint256 makerAmountFilled, uint256 takerAmountFilled, uint256 fee)\n```\n\nV2 states the side explicitly. V1 does not, so it has to be inferred: asset id `0` is collateral, and whichever leg is zero determines the maker's direction. V1 token-for-token fills between complementary outcomes are skipped, since they have no collateral leg and therefore no meaningful price.\n\nYou do not need to specify a start block. With no checkpoint and no `POLYLEDGER_FROM_BLOCK`, PolyLedger locates the first block containing a matching log. HyperSync skips empty ranges server-side, so this costs a handful of requests rather than a full chain scan.\n\n## Configuration\n\nAll settings are read from environment variables or from a `.env` file in the working directory, which is loaded automatically at startup. Environment variables take precedence.\n\n| Variable | Default | Description |\n| --- | --- | --- |\n| `HYPERSYNC_BEARER_TOKEN` | - | Envio API token. Required |\n| `POLYLEDGER_DB` | `data\u002Fpolyledger.duckdb` | Database file path |\n| `POLYLEDGER_EXPORT_DIR` | `data\u002Fparquet` | Parquet export directory |\n| `POLYLEDGER_CONTRACTS` | `v2` | `v2`, `v1`, or `all` |\n| `POLYLEDGER_FROM_BLOCK` | `0` | `0` means auto-detect |\n| `POLYLEDGER_REORG_BUFFER` | `64` | Blocks to stay behind the chain head |\n| `POLYLEDGER_FLUSH_EVERY` | `50000` | Fills buffered before a checkpointed flush |\n| `POLYLEDGER_HTTP_RPS` | `8` | Request rate limit for CLOB and Gamma |\n| `POLYLEDGER_HTTP_CONCURRENCY` | `8` | Parallel HTTP requests |\n| `POLYLEDGER_HTTP_RETRIES` | `6` | Attempts per request |\n| `POLYLEDGER_HTTP_TIMEOUT` | `30` | Request timeout in seconds |\n\n`REORG_BUFFER` guards against chain reorganisations by stopping 64 blocks short of the head, so a fill from a block that later gets orphaned is never written.\n\n## Design notes\n\n**DuckDB rather than CSV.** Appending to a multi-gigabyte CSV degrades without bound, and any read means parsing every byte. DuckDB gives columnar compression, real types, indexes, and a proper SQL join instead of a `pandas.merge` that has to hold both sides in memory. It remains a single file with no server or daemon.\n\n**Atomic checkpoints.** Rows and the new cursor value are written in one transaction. There is no state where data landed but the cursor did not, or the reverse.\n\n**Idempotent inserts.** Duplicates within a single batch are collapsed before the write, and duplicates against existing rows are dropped by the primary key. A crash mid-batch cannot produce double rows on restart.\n\n**Loud failures on schema drift.** Polymarket changed its response format in April 2026 and broke every consumer built on the old shape. Pydantic validation means the next change surfaces as an error on the first affected row.\n\n**Visible gaps.** Some token ids from older fills are absent from the CLOB listing. Those rows are marked `unmatched_market = true`, counted in `stats`, and recovered from Gamma with `--backfill`. Nothing is discarded; the trade stays in the database, just without human-readable context.\n\n## Development\n\n```bash\npip install -e \".[dev]\"\npytest\n```\n\n41 tests, none requiring network access. They cover decoding for both event versions against fixtures, insert idempotency and transaction rollback, price and side arithmetic, CLOB pagination including the stuck-cursor guard, Gamma batching and fallback, the rate limiter, and the backoff schedule. One test recomputes `topic0` from the textual event signatures and compares it against the hard-coded constants, so a typo in either cannot go unnoticed.\n\n## Limitations\n\n- **Polymarket only.** Kalshi does not settle on chain and needs a different collector. The staged architecture allows for one, but it is not written.\n- **Filled trades only.** Order book state, quotes, and cancellations never reach the chain. Capturing those requires the CLOB WebSocket feed in real time.\n- **Reorgs are avoided, not reconciled.** The indexer stays behind the head rather than rolling back written blocks. This is sufficient for historical analysis and not sufficient for live trading.\n- **Fees come from the event's `fee` field.** Separate `FeeCharged` events are not indexed.\n\n## License\n\nMIT\n","PolyLedger 是一个可断点续传的 Polymarket 链上数据索引器，用于同步 Polymarket CLOB 市场元数据与 Polygon 链上的交易事件（OrderFilled），并统一存储为单个 DuckDB 文件，支持原生 SQL 查询。其核心特点包括：基于事务的断点续传、幂等写入（按 transaction_hash + log_index 去重）、DuckDB 列式存储与本地查询能力、兼容 V1\u002FV2 合约事件解析、Pydantic 严格 schema 校验、指数退避网络容错及缺失 token 的 Gamma API 回填机制。适用于预测市场研究、链上交易行为分析、量化策略回测及数据科学探索等场景。",2,"2026-09-05 02:30:03","CREATED_QUERY"]