[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"project-81460":3},{"id":4,"name":5,"fullName":6,"owner":7,"repo":5,"description":8,"homepage":9,"htmlUrl":10,"language":11,"languages":10,"totalLinesOfCode":10,"stars":12,"forks":13,"watchers":13,"openIssues":14,"contributorsCount":14,"subscribersCount":14,"size":14,"stars1d":15,"stars7d":16,"stars30d":17,"stars90d":14,"forks30d":14,"starsTrendScore":18,"compositeScore":19,"rankGlobal":10,"rankLanguage":10,"license":20,"archived":21,"fork":21,"defaultBranch":22,"hasWiki":21,"hasPages":21,"topics":23,"createdAt":10,"pushedAt":10,"updatedAt":30,"readmeContent":31,"aiSummary":32,"trendingCount":14,"starSnapshotCount":14,"syncStatus":33,"lastSyncTime":34,"discoverSource":35},81460,"quacklake","tobilg\u002Fquacklake","tobilg","A DuckLake data catalog based on quack, deployed to Cloudflare","",null,"TypeScript",56,1,0,3,17,25,20,57.9,"MIT License",false,"main",[24,25,26,27,28,29],"cloudflare","data-catalog","duckdb","ducklake","durable-objects","quack","2026-06-12 04:01:33","![Quacklake](images\u002Fquacklake-compressed.png)\n\n# quacklake\n\n`quacklake` is a Cloudflare Workers \u002F Durable Objects service that speaks DuckDB's experimental Quack HTTP protocol and stores DuckLake catalog metadata in Durable Object SQLite storage.\n\nThe Worker exposes one public Quack endpoint at `\u002Fquack`. Clients authenticate by sending a JWT as the Quack auth string. A valid JWT resolves to one catalog Durable Object and one normalized principal, then quacklake applies the catalog's server-side authorization policy before executing catalog SQL.\n\n## Status\n\nThis is an alpha implementation. It is useful for protocol integration work, local Worker tests, and R2-backed DuckLake metadata smoke tests. It is not a full DuckDB server.\n\nImplemented:\n\n- Quack binary request\u002Fresponse transport through `POST \u002Fquack`.\n- `CONNECTION_REQUEST`, `PREPARE_REQUEST`, `FETCH_REQUEST`, `APPEND_REQUEST`, and `DISCONNECT_MESSAGE`.\n- JWT-only catalog authentication.\n- First-party HS256 quacklake JWT credentials.\n- Third-party OIDC JWT verification through configured providers and JWKS.\n- Catalog auth mappings that select a catalog for verified OIDC principals.\n- Catalog auth policies that authorize SQL and append requests before execution.\n- SQLite-backed query execution with DuckDB-style compatibility rewrites.\n- Planned R2-backed DuckLake `DATA_PATH` assignment and enforcement per catalog.\n- R2-backed DuckLake file discovery for orphan cleanup.\n- Optional trusted-client R2 data leases for catalogs created with `dataAccessMode: \"trusted_client\"`.\n- Result materialization into Quack `DataChunk`s using `@quack-protocol\u002Fsdk`.\n- Basic explicit transaction emulation with snapshot restore on `ROLLBACK`.\n- Worker integration tests through the published `@quack-protocol\u002Fsdk` client.\n- OpenAPI v3 Admin API document at `GET \u002Fapi-docs`.\n\nNot implemented as full DuckDB semantics:\n\n- Complete DuckDB SQL parser or optimizer behavior.\n- Arbitrary DuckDB functions and table functions.\n- Cross-session transactional conflict detection.\n- Complete DuckLake test-suite coverage.\n- OPA\u002FRego policy execution.\n- A server-side data gateway. Trusted-client leases grant raw R2 object access under the catalog data path and do not enforce row or column policy at the storage layer.\n\n## Guides\n\nThe README is intentionally a short project entry point. Detailed operational docs live in `guides\u002F`:\n\n- [Getting Started Guide](.\u002Fguides\u002Fgetting-started.md): simplest production-style Cloudflare deployment with one R2 bucket, first-party JWT auth, default `catalog_only` access, and a DuckDB CLI smoke query.\n- [Authn\u002FAuthz Guide](.\u002Fguides\u002Fauthn-authz.md): JWT-only authentication, first-party credentials, OIDC providers, catalog mappings, catalog policies, policy cookbook, explain output, and troubleshooting.\n- [Cognito End-To-End Guide](.\u002Fguides\u002Fcognito-e2e.md): Cognito user-pool setup, group-based permission profiles, catalog mapping, row and column policy examples, and end-user DuckLake querying.\n- [Microsoft Entra ID End-To-End Guide](.\u002Fguides\u002Fentraid-e2e-md): Entra app registration, group and app-role permission profiles, catalog mapping, row and column policy examples, and end-user DuckLake querying.\n- [Quack, DuckLake, And R2 Guide](.\u002Fguides\u002Fquack-ducklake.md): DuckDB Quack secrets, SDK usage, DuckLake attachment, planned R2 `DATA_PATH` enforcement, trusted-client R2 leases, R2 bucket listing, diagnostics, and file inventory endpoints.\n- [Local Development And Configuration Guide](.\u002Fguides\u002Flocal-development.md): dependencies, Wrangler configuration, local secrets, development commands, local Worker health checks, and OpenAPI discovery.\n\nThe machine-readable Admin API reference is served by a running Worker:\n\n```sh\ncurl http:\u002F\u002Flocalhost:8787\u002Fapi-docs\n```\n\n## Architecture\n\nThere are two Durable Object classes:\n\n- `CatalogRegistry`: global registry for catalog ids, first-party credential metadata, OIDC providers, catalog auth mappings, and catalog auth policies.\n- `QuackCatalogObject`: one SQLite-backed Durable Object database per catalog.\n\nRequest flow:\n\n1. A client sends a Quack `CONNECTION_REQUEST` with a JWT auth string.\n2. The Worker asks `CatalogRegistry` to verify and resolve the JWT.\n3. The JWT resolves to one catalog id, one `QuackCatalogObject`, one normalized principal, and the current catalog policy version.\n4. The catalog object opens a session and stores the auth context.\n5. The Worker signs `{ catalogId, sessionId }` into the public Quack `connection_id`.\n6. Later Quack messages include that signed connection id and route directly to the catalog Durable Object.\n7. `PREPARE_REQUEST` and `APPEND_REQUEST` are authorized against the stored session principal and policy before execution.\n\n## Project Layout\n\n- `src\u002Findex.ts`: Worker fetch handler, `\u002Fquack`, CORS, `\u002Fapi-docs`, and `\u002Fadmin\u002F*` routes.\n- `src\u002Fopenapi.ts`: OpenAPI v3 Admin API document.\n- `src\u002Fregistry.ts`: catalog, credential, OIDC provider, mapping, and policy registry Durable Object.\n- `src\u002Fauth.ts`: shared authentication, mapping, policy, and session auth types.\n- `src\u002Fauthz.ts`: SQL classification and internal policy evaluator.\n- `src\u002Fcatalog.ts`: Quack protocol Durable Object and per-session auth enforcement.\n- `src\u002Fsql-compat.ts`: SQL execution orchestration, session state, schema tracking, transactions, and result chunking.\n- `src\u002Fducklake-metadata.ts`: DuckLake-specific catalog query and migration shims that SQLite cannot execute directly.\n- `src\u002Fsql-rewrite.ts`: DuckDB-to-SQLite SQL text rewrites and column-definition parsing.\n- `src\u002Fsql-names.ts`: schema-qualified identifier normalization helpers.\n- `src\u002Fsql-types.ts`: shared SQL execution, result, schema, and transaction snapshot types.\n- `src\u002Fquack-values.ts`: value and logical type conversion between SQLite and Quack.\n- `src\u002Fcrypto.ts`: signed connection ids and constant-time comparisons.\n- `test\u002Fquack-worker.test.ts`: Worker integration tests through `QuackClient`.\n- `test\u002Fauth.test.ts`: JWT, OIDC, mapping, policy, protocol, OpenAPI, and explain tests.\n- `test\u002Fauthz.test.ts`: SQL authorization classifier and policy evaluator tests.\n- `test\u002Ffile-listing.test.ts`: R2\u002Ffile-listing helper tests.\n- `test\u002Fquack-values.test.ts`: Quack value and logical type conversion tests.\n- `guides\u002F`: focused operator and developer guides.\n- `scripts\u002Fcreate-jwt.sh`: creates a first-party personal JWT and installs a broad personal catalog policy.\n- `scripts\u002Fsetup-cognito.sh`: creates Cognito user-pool resources for OIDC smoke tests and registration.\n- `scripts\u002Fregister-cognito-idp.sh`: registers Cognito as a quacklake OIDC provider and installs group-based mapping\u002Fpolicy rules.\n- `wrangler.example.jsonc`: tracked Worker, Durable Object, R2, migration, and runtime configuration template.\n\n## Quick Start\n\nInstall dependencies:\n\n```sh\npnpm install\n```\n\nCreate `.dev.vars` for local development:\n\n```dotenv\nADMIN_TOKEN=admin-test-token\nQUACKLAKE_JWT_SECRET=replace-with-long-random-local-jwt-secret\nCONNECTION_SIGNING_SECRET=replace-with-long-random-local-signing-secret\n```\n\nRun checks:\n\n```sh\npnpm run typecheck\npnpm run test\npnpm run test:coverage\n```\n\nRun the Worker locally:\n\n```sh\npnpm run dev\n```\n\nHealth check:\n\n```sh\ncurl http:\u002F\u002Flocalhost:8787\u002F\n```\n\nExpected shape:\n\n```json\n{\n  \"name\": \"quacklake\",\n  \"protocol\": \"quack\",\n  \"endpoint\": \"\u002Fquack\",\n  \"apiDocs\": \"\u002Fapi-docs\"\n}\n```\n\nCreate a local catalog and first-party JWT credential:\n\n```sh\ncurl -s -X POST http:\u002F\u002Flocalhost:8787\u002Fadmin\u002Fcatalogs \\\n  -H 'Authorization: Bearer admin-test-token' \\\n  -H 'Content-Type: application\u002Fjson' \\\n  -d '{\"catalogId\":\"default\",\"scopes\":[\"catalog.admin\"]}'\n```\n\nInstall a permissive bootstrap policy for local setup:\n\n```sh\ncurl -s -X PUT http:\u002F\u002Flocalhost:8787\u002Fadmin\u002Fcatalogs\u002Fdefault\u002Fauth-policy \\\n  -H 'Authorization: Bearer admin-test-token' \\\n  -H 'Content-Type: application\u002Fjson' \\\n  -d '{\"version\":1,\"defaultEffect\":\"allow\",\"rules\":[]}'\n```\n\nFor production policies, OIDC, and troubleshooting, use [Authn\u002FAuthz Guide](.\u002Fguides\u002Fauthn-authz.md).\n\n## Configuration Summary\n\n`wrangler.example.jsonc` shows the Worker configuration shape. Copy it to `wrangler.jsonc` and edit the local copy before running Wrangler commands.\n\nImportant runtime vars:\n\n- `QUACK_FETCH_ROWS_PER_CHUNK`: default `1024`.\n- `QUACK_FETCH_CHUNKS_PER_BATCH`: default `12`.\n- `QUACKLAKE_JWT_ISSUER`: first-party JWT issuer, default `quacklake`.\n- `QUACKLAKE_JWT_AUDIENCE`: first-party JWT audience, default `quacklake:quack`.\n- `QUACKLAKE_JWT_DEFAULT_TTL_SECONDS`: first-party credential lifetime, default one year.\n- `DUCKLAKE_R2_BINDINGS`: JSON map from DuckLake bucket name to Worker R2 binding name, for example `{\"\u003Cbucket-name>\":\"DUCKLAKE_R2\"}`. Every usable DuckLake data bucket must also appear in `wrangler.jsonc` `r2_buckets`.\n\nTrusted-client lease vars, only needed when using `dataAccessMode: \"trusted_client\"`:\n\n- `DUCKLAKE_R2_DATA_LEASE_TTL_SECONDS`: trusted-client R2 data lease TTL, clamped to 30-120 seconds. Default `60`.\n- `R2_ACCOUNT_ID`: Cloudflare account id used when locally signing R2 temporary credentials.\n- `R2_ENDPOINT`: S3-compatible R2 endpoint, for example `https:\u002F\u002F\u003Caccount-id>.r2.cloudflarestorage.com`.\n\nRuntime secrets:\n\n- `ADMIN_TOKEN`: bearer secret required for every `\u002Fadmin\u002F*` route.\n- `QUACKLAKE_JWT_SECRET`: HS256 signing key for first-party quacklake JWT credentials.\n- `CONNECTION_SIGNING_SECRET`: HMAC secret used to sign Quack connection ids.\n- `R2_ACCESS_KEY_ID`: parent R2 S3 access key id used only when issuing trusted-client data leases.\n- `R2_SECRET_ACCESS_KEY`: parent R2 S3 secret access key used only when issuing trusted-client data leases.\n\nFor a deployed Worker using the default name:\n\n```sh\npnpm exec wrangler secret put ADMIN_TOKEN --name quacklake\npnpm exec wrangler secret put QUACKLAKE_JWT_SECRET --name quacklake\npnpm exec wrangler secret put CONNECTION_SIGNING_SECRET --name quacklake\n```\n\nIf you enable `dataAccessMode: \"trusted_client\"` for any catalog, also set the parent R2 S3 credentials used for local temporary-credential signing:\n\n```sh\npnpm exec wrangler secret put R2_ACCESS_KEY_ID --name quacklake\npnpm exec wrangler secret put R2_SECRET_ACCESS_KEY --name quacklake\n```\n\nThe value passed to admin calls, including `scripts\u002Fcreate-jwt.sh --admin-token`, must exactly match the deployed `ADMIN_TOKEN` secret.\n\nSee [Local Development And Configuration Guide](.\u002Fguides\u002Flocal-development.md) for local and deployed secret setup.\n\n## Client Usage Summary\n\nUse a JWT as the Quack secret token value:\n\nUse the `core_nightly` DuckDB extension builds for `quack` and `ducklake`; those\nbuilds contain bugfixes required by the current quacklake workflows.\n\n```sql\nFORCE INSTALL quack FROM core_nightly;\nFORCE INSTALL ducklake FROM core_nightly;\nLOAD quack;\nLOAD ducklake;\n\nCREATE OR REPLACE SECRET quacklake_catalog (\n  TYPE quack,\n  TOKEN '\u003Cjwt>',\n  SCOPE 'quack:\u003Cworker-host>:443'\n);\n```\n\n`POST \u002Fadmin\u002Fcatalogs` assigns the catalog a planned `DATA_PATH` of `r2:\u002F\u002F\u003Cbucket>\u002Fcatalogs\u002F\u003CcatalogId>\u002F` and returns `ducklake.secretSql` and `ducklake.attachSql` for copy\u002Fpaste bootstrap. `ducklake.secretSql` contains the one-time-visible JWT and must be treated as secret material.\n\nFor DuckLake, create a separate storage secret scoped to the planned bucket and prefix:\n\n```sql\nCREATE OR REPLACE SECRET lake_r2 (\n  TYPE s3,\n  PROVIDER config,\n  KEY_ID '\u003Cr2-access-key-id>',\n  SECRET '\u003Cr2-secret-access-key>',\n  ENDPOINT '\u003Caccount-id>.r2.cloudflarestorage.com',\n  URL_STYLE 'path',\n  REGION 'auto',\n  SCOPE 'r2:\u002F\u002F\u003Cbucket>\u002Fcatalogs\u002F\u003CcatalogId>\u002F'\n);\n\nATTACH 'ducklake:quack:\u003Cworker-host>:443' AS lake (\n  DATA_PATH 'r2:\u002F\u002F\u003Cbucket>\u002Fcatalogs\u002F\u003CcatalogId>\u002F'\n);\n```\n\nManual storage secrets are still the default `catalog_only` setup. For trusted clients, create the catalog with `dataAccessMode: \"trusted_client\"` and call `POST \u002Fcatalog\u002Fdata-lease` with the same catalog JWT to receive short-lived R2 credentials for the planned catalog `DATA_PATH`.\n\nFor server-side DuckLake maintenance paths such as `read_blob()` orphan discovery, and for validating trusted-client lease paths, the Worker also needs an R2 bucket binding mapped through `DUCKLAKE_R2_BINDINGS`. See [Quack, DuckLake, And R2 Guide](.\u002Fguides\u002Fquack-ducklake.md) for Worker R2 binding setup, client storage secrets, trusted-client leases, R2 bucket listing, diagnostics, and file inventory examples.\n\n## Notes\n\n- Keep one catalog id per independent DuckLake `DATA_PATH`.\n- Additional credentials for a catalog are credential rotations or app-specific credentials; they do not create a new metadata store.\n- Signed connection ids depend on `CONNECTION_SIGNING_SECRET`; rotating it invalidates all active client sessions.\n- First-party credentials depend on `QUACKLAKE_JWT_SECRET`; rotating it requires credential reissue.\n- OPA\u002FRego is intentionally not implemented in v1, but the internal explain input\u002Foutput shape is OPA-compatible enough to support a future OPA Wasm backend.\n","quacklake 是一个基于 Cloudflare Workers 和 Durable Objects 的服务，它实现了 DuckDB 的实验性 Quack HTTP 协议，并将 DuckLake 数据目录元数据存储在 Durable Object SQLite 存储中。其核心功能包括通过 `\u002Fquack` 端点处理二进制请求\u002F响应传输、支持多种 Quack 请求类型、JWT 认证以及基于 OIDC 的第三方认证和授权策略应用。此外，该项目还计划实现 R2 存储支持下的数据路径分配与文件发现等特性。尽管当前版本为 alpha 版本，尚未完全实现所有 DuckDB 功能，但 quacklake 已经能够满足协议集成测试、本地 Worker 测试及 R2 支持的 DuckLake 元数据冒烟测试的需求。适用于需要轻量级云原生数据库解决方案且对安全性有一定要求的应用场景。",2,"2026-06-11 04:05:09","CREATED_QUERY"]