[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"project-73349":3},{"id":4,"name":5,"fullName":6,"owner":7,"repo":5,"description":8,"homepage":9,"htmlUrl":10,"language":11,"languages":10,"totalLinesOfCode":10,"stars":12,"forks":13,"watchers":14,"openIssues":15,"contributorsCount":16,"subscribersCount":16,"size":16,"stars1d":17,"stars7d":18,"stars30d":19,"stars90d":16,"forks30d":16,"starsTrendScore":20,"compositeScore":21,"rankGlobal":10,"rankLanguage":10,"license":22,"archived":23,"fork":23,"defaultBranch":24,"hasWiki":25,"hasPages":25,"topics":26,"createdAt":10,"pushedAt":10,"updatedAt":33,"readmeContent":34,"aiSummary":35,"trendingCount":16,"starSnapshotCount":16,"syncStatus":36,"lastSyncTime":37,"discoverSource":38},73349,"etl","supabase\u002Fetl","supabase","Stream your Postgres data anywhere in real-time. Simple Rust building blocks for change data capture (CDC) pipelines.","https:\u002F\u002Fsupabase.github.io\u002Fetl\u002F",null,"Rust",2261,156,29,9,0,1,6,14,3,28.59,"Apache License 2.0",false,"main",true,[27,28,29,30,31,32],"logical-replication","postgres","postgresql","replication","rust","rust-lang","2026-06-12 02:03:12","\u003Cbr \u002F>\n\u003Cp align=\"center\">\n  \u003Ca href=\"https:\u002F\u002Fsupabase.com\">\n    \u003Cpicture>\n      \u003Cimg alt=\"ETL by Supabase\" width=\"100%\" src=\"docs\u002Fassets\u002Fetl-logo-extended.png\">\n    \u003C\u002Fpicture>\n  \u003C\u002Fa>\n\n  \u003Ch1 align=\"center\">ETL\u003C\u002Fh1>\n\n  \u003Cp align=\"center\">\n    \u003Ca href=\"https:\u002F\u002Fgithub.com\u002Fsupabase\u002Fetl\u002Factions\u002Fworkflows\u002Fci.yml\">\n      \u003Cimg alt=\"CI\" src=\"https:\u002F\u002Fgithub.com\u002Fsupabase\u002Fetl\u002Factions\u002Fworkflows\u002Fci.yml\u002Fbadge.svg?branch=main\">\n    \u003C\u002Fa>\n    \u003Ca href=\"https:\u002F\u002Fcoveralls.io\u002Fgithub\u002Fsupabase\u002Fetl?branch=main\">\n      \u003Cimg alt=\"Coverage Status\" src=\"https:\u002F\u002Fcoveralls.io\u002Frepos\u002Fgithub\u002Fsupabase\u002Fetl\u002Fbadge.svg?branch=main\">\n    \u003C\u002Fa>\n    \u003Ca href=\"https:\u002F\u002Fgithub.com\u002Fsupabase\u002Fetl\u002Factions\u002Fworkflows\u002Fdocs.yml\">\n      \u003Cimg alt=\"Docs\" src=\"https:\u002F\u002Fgithub.com\u002Fsupabase\u002Fetl\u002Factions\u002Fworkflows\u002Fdocs.yml\u002Fbadge.svg?branch=main\">\n    \u003C\u002Fa>\n    \u003Ca href=\"https:\u002F\u002Fgithub.com\u002Fsupabase\u002Fetl\u002Factions\u002Fworkflows\u002Fdocker-build.yml\">\n      \u003Cimg alt=\"Docker Build\" src=\"https:\u002F\u002Fgithub.com\u002Fsupabase\u002Fetl\u002Factions\u002Fworkflows\u002Fdocker-build.yml\u002Fbadge.svg?branch=main\">\n    \u003C\u002Fa>\n    \u003Ca href=\"https:\u002F\u002Fgithub.com\u002Fsupabase\u002Fetl\u002Factions\u002Fworkflows\u002Faudit.yml\">\n      \u003Cimg alt=\"Security Audit\" src=\"https:\u002F\u002Fgithub.com\u002Fsupabase\u002Fetl\u002Factions\u002Fworkflows\u002Faudit.yml\u002Fbadge.svg?branch=main\">\n    \u003C\u002Fa>\n    \u003Ca href=\"LICENSE\">\n      \u003Cimg alt=\"License\" src=\"https:\u002F\u002Fimg.shields.io\u002Fbadge\u002FLicense-Apache_2.0-blue.svg\">\n    \u003C\u002Fa>\n    \u003Cbr \u002F>\n    Build real-time Postgres replication applications in Rust\n    \u003Cbr \u002F>\n    \u003Ca href=\"https:\u002F\u002Fsupabase.github.io\u002Fetl\">\u003Cstrong>Documentation\u003C\u002Fstrong>\u003C\u002Fa>\n    ·\n    \u003Ca href=\"https:\u002F\u002Fgithub.com\u002Fsupabase\u002Fetl\u002Ftree\u002Fmain\u002Fcrates\u002Fetl-examples\">\u003Cstrong>Examples\u003C\u002Fstrong>\u003C\u002Fa>\n    ·\n    \u003Ca href=\"https:\u002F\u002Fgithub.com\u002Fsupabase\u002Fetl\u002Fissues\">\u003Cstrong>Issues\u003C\u002Fstrong>\u003C\u002Fa>\n  \u003C\u002Fp>\n\u003C\u002Fp>\n\nETL is a Rust framework by [Supabase](https:\u002F\u002Fsupabase.com) for building\nhigh-performance, real-time data replication apps on Postgres.\n\nIt sits on top of Postgres\n[logical replication](https:\u002F\u002Fwww.postgresql.org\u002Fdocs\u002Fcurrent\u002Fprotocol-logical-replication.html)\nand gives you Rust-native building blocks for copying existing data, streaming\nongoing changes, and writing them to your own destination. Run it as a\nstandalone replicator binary or embed it as a library in your own Rust service.\n\nETL is intentionally cheap to operate: it is one lightweight Rust process on\ntop of Postgres logical replication. You do not need Kafka, Flink, Debezium, or\nanother coordination service to run a pipeline.\n\n## What ETL Does\n\n```mermaid\nflowchart LR\n    Postgres[\"Postgres publication\"] --> ETL[\"ETL\u003Cbr\u002F>copy + stream\"]\n    ETL --> Destination[\"Destination\"]\n```\n\nETL runs as one process that coordinates an initial copy, a continuous\nreplication stream, and a state\u002Fschema store for recovery:\n\n1. **Initial copy** backfills the existing rows covered by a Postgres publication.\n2. **Streaming replication** forwards ongoing inserts, updates, deletes, truncates, and schema events.\n3. **State recovery** lets a durable store resume table state, schema versions, and destination metadata after restarts.\n\n## Why ETL?\n\n| Capability | What it gives you |\n| --- | --- |\n| Real-time replication | Stream Postgres changes as they happen. |\n| Initial copy | Backfill existing table data before CDC begins. |\n| Schema changes | Track simple DDL changes today; destination-specific DDL behavior is documented in [Schema Changes](https:\u002F\u002Fsupabase.github.io\u002Fetl\u002Fexplanation\u002Fschema-changes\u002F). |\n| Cheap operations | Run one lightweight Rust process without Kafka, Flink, Debezium, or extra control-plane infrastructure. |\n| Library or binary | Use ETL as a standalone replicator or embed it in your own Rust application. |\n| Configurable throughput | Tune batching, parallel table sync, retries, and memory backpressure. |\n| Extensible runtime | Implement custom destinations and state\u002Fschema stores. |\n| Typed Rust API | Work with structured events, rows, schemas, and errors. |\n\n## Requirements\n\nETL officially supports and tests against **PostgreSQL 14, 15, 16, 17, and\n18**.\n\n- **PostgreSQL 15+** is recommended for advanced publication features:\n  - Column-level filtering\n  - Row-level filtering with `WHERE` clauses\n  - `FOR ALL TABLES IN SCHEMA` syntax\n- **PostgreSQL 14** is supported with table-level publication filtering.\n\nFor detailed configuration instructions, see the [Configure Postgres documentation](https:\u002F\u002Fsupabase.github.io\u002Fetl\u002Fguides\u002Fconfigure-postgres\u002F).\n\n## Get Started\n\nETL is currently installed from Git while we prepare for a crates.io release.\nChoose the destination features you need.\n\nFor a first production deployment, start with the stable BigQuery module:\n\n```toml\n[dependencies]\netl = { git = \"https:\u002F\u002Fgithub.com\u002Fsupabase\u002Fetl\" }\netl-destinations = { git = \"https:\u002F\u002Fgithub.com\u002Fsupabase\u002Fetl\", features = [\"bigquery\"] }\ntokio = { version = \"1\", features = [\"full\"] }\n```\n\nThen create a pipeline that reads from a Postgres publication and writes to\nBigQuery.\n\n```rust\nuse etl::{\n    config::{\n        BatchConfig, InvalidatedSlotBehavior, MemoryBackpressureConfig, PgConnectionConfig,\n        PipelineConfig, TableSyncCopyConfig, TcpKeepaliveConfig, TlsConfig,\n    },\n    pipeline::Pipeline,\n    store::PostgresStore,\n};\nuse etl_destinations::bigquery::BigQueryDestination;\n\n#[tokio::main]\nasync fn main() -> Result\u003C(), Box\u003Cdyn std::error::Error>> {\n    let pg = PgConnectionConfig {\n        host: \"localhost\".into(),\n        port: 5432,\n        name: \"mydb\".into(),\n        username: \"postgres\".into(),\n        password: Some(\"password\".to_string().into()),\n        tls: TlsConfig { enabled: false, trusted_root_certs: String::new() },\n        keepalive: TcpKeepaliveConfig::default(),\n    };\n\n    let pipeline_id = 1;\n    let store = PostgresStore::new(pipeline_id, pg.clone()).await?;\n    let destination = BigQueryDestination::new_with_key_path(\n        \"my-gcp-project\".into(),\n        \"my_dataset\".into(),\n        \"\u002Fpath\u002Fto\u002Fservice-account-key.json\",\n        None,\n        1,\n        pipeline_id,\n        store.clone(),\n    )\n    .await?;\n\n    let config = PipelineConfig {\n        id: pipeline_id,\n        publication_name: \"my_publication\".into(),\n        pg_connection: pg,\n        batch: BatchConfig {\n            max_fill_ms: 5000,\n            memory_budget_ratio: 0.2,\n            max_bytes: 8 * 1024 * 1024,\n        },\n        table_error_retry_delay_ms: 10_000,\n        table_error_retry_max_attempts: 5,\n        max_table_sync_workers: 4,\n        max_copy_connections_per_table: PipelineConfig::DEFAULT_MAX_COPY_CONNECTIONS_PER_TABLE,\n        memory_refresh_interval_ms: 100,\n        memory_backpressure: Some(MemoryBackpressureConfig::default()),\n        table_sync_copy: TableSyncCopyConfig::default(),\n        invalidated_slot_behavior: InvalidatedSlotBehavior::default(),\n    };\n\n    \u002F\u002F Start the pipeline.\n    let mut pipeline = Pipeline::new(config, store, destination);\n    pipeline.start().await?;\n\n    \u002F\u002F Wait for the pipeline indefinitely.\n    pipeline.wait().await?;\n\n    Ok(())\n}\n```\n\nFor a guided walkthrough, start with\n[Your First Pipeline](https:\u002F\u002Fsupabase.github.io\u002Fetl\u002Fguides\u002Ffirst-pipeline\u002F).\nFor runnable destination examples, see [`etl-examples`](crates\u002Fetl-examples\u002FREADME.md).\n\n## Destinations\n\nETL is designed to be extensible: you can implement your own destination, or use\none of the modules shipped in `etl-destinations`.\n\n| Feature | Destination | Status | Notes |\n| --- | --- | --- | --- |\n| `bigquery` | Google BigQuery | Stable | Full CRUD-capable replication for analytics workloads. |\n| `ducklake` | DuckLake | In progress | Open data lake replication with local or S3-compatible storage. |\n| `iceberg` | Apache Iceberg | Deprecated for now | The module remains available, but new deployments should prefer BigQuery or DuckLake. |\n\nEnable one or more destination modules with crate features:\n\n```toml\n[dependencies]\netl = { git = \"https:\u002F\u002Fgithub.com\u002Fsupabase\u002Fetl\" }\netl-destinations = { git = \"https:\u002F\u002Fgithub.com\u002Fsupabase\u002Fetl\", features = [\"bigquery\", \"ducklake\"] }\n```\n\n## Development\n\nSee [DEVELOPMENT.md](DEVELOPMENT.md) for setup instructions, migration workflows, and development guidelines.\n\n## Contributing\n\nWe welcome pull requests and GitHub issues. We currently cannot accept new custom destinations unless there is significant community demand, as each destination carries a long-term maintenance cost. We are prioritizing core stability, observability, and ergonomics. If you need a destination that is not yet supported, please start a discussion or issue so we can gauge demand before proposing an implementation.\n\n## License\n\nApache‑2.0. See `LICENSE` for details.\n\n---\n\n\u003Cp align=\"center\">\n  Made with ❤️ by the \u003Ca href=\"https:\u002F\u002Fsupabase.com\">Supabase\u003C\u002Fa> team\n\u003C\u002Fp>\n","supabase\u002Fetl 是一个用于实时传输 PostgreSQL 数据的 Rust 框架。它基于 PostgreSQL 的逻辑复制功能，提供了构建高性能、实时数据复制应用所需的 Rust 原生组件，支持数据的初始复制、持续流式更新以及状态恢复等功能。该项目轻量且易于操作，不需要额外的协调服务如 Kafka 或 Flink 即可运行数据管道。适用于需要将 PostgreSQL 数据实时同步到其他系统或存储中的场景，例如数据仓库填充、多数据库环境下的数据一致性维护等。",2,"2026-06-11 03:45:06","high_star"]