[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"project-1136":3},{"id":4,"name":5,"fullName":6,"owner":7,"repo":5,"description":8,"homepage":9,"htmlUrl":10,"language":11,"languages":10,"totalLinesOfCode":10,"stars":12,"forks":13,"watchers":14,"openIssues":15,"contributorsCount":16,"subscribersCount":16,"size":16,"stars1d":17,"stars7d":18,"stars30d":19,"stars90d":16,"forks30d":16,"starsTrendScore":20,"compositeScore":21,"rankGlobal":10,"rankLanguage":10,"license":22,"archived":23,"fork":23,"defaultBranch":24,"hasWiki":23,"hasPages":25,"topics":26,"createdAt":10,"pushedAt":10,"updatedAt":31,"readmeContent":32,"aiSummary":33,"trendingCount":16,"starSnapshotCount":16,"syncStatus":34,"lastSyncTime":35,"discoverSource":36},1136,"firnflow","gordonmurray\u002Ffirnflow","gordonmurray","The cost efficiency of S3 with the speed of local RAM. A multi-tenant vector and full-text search engine featuring a tiered RAM → NVMe → S3 architecture for microsecond latency on   top of object storage","https:\u002F\u002Ffirnflow.io\u002F",null,"Rust",393,22,1,14,0,4,8,54,12,4.09,"Apache License 2.0",false,"main",true,[27,28,29,30],"foyer","lance","lancedb","s3","2026-06-12 02:00:23","# Firn\n\n**Firn** is a high-performance, multi-tenant vector and full-text search engine backed by object storage (AWS S3, MinIO, Cloudflare R2, Tigris, DigitalOcean Spaces). It is designed as a credible open-source alternative to turbopuffer, proving that a professional-grade tiered storage architecture (**RAM → NVMe → S3**) is achievable entirely from open-source components. See [Storage backends](#storage-backends) for the full compatibility matrix.\n\nThe cost efficiency of S3 with the speed of local RAM. A multi-tenant vector and full-text search engine backed by S3. Built with LanceDB and Foyer for microsecond-scale search latency on top of object storage.\n\n## 25 Seconds to 72 Microseconds\n\nOn real-world cloud infrastructure (AWS S3), a raw linear scan of 100,000 vectors can take **25 seconds** per query. By pairing **LanceDB** with a tiered **foyer** (RAM + NVMe) cache, **Firn** collapses that bottleneck:\n\n*   **Cold Query (S3 Linear Scan):** ~25.1s\n*   **Cold Query (ANN Indexed):** ~979ms (**25x faster**)\n*   **Warm Query (Internal Engine):** **~72µs** (**350,000x faster**)\n*   **End-to-End HTTP (Warm):** **\u003C 5ms** (including network RTT and JSON overhead)\n\nEvery cache hit results in **zero** S3 requests, directly reducing your cloud bill while providing \"instant\" search response times.\n\n### Demo\n\nCold query, warm query, full-text search, and cache proof, all in 60 seconds. This demo runs against local MinIO; on real AWS S3 the cold query takes 25 seconds instead of 109ms, making the cache speedup even more dramatic.\n\n![Firn demo](bench\u002Fdemo.gif)\n\n## Architecture\n\n**Firn** is built on a \"Tiered Storage\" philosophy:\n\n1.  **L1: RAM Cache** (via foyer): Sub-microsecond access for the most frequent queries.\n2.  **L2: NVMe Cache** (via foyer): Fast, durable cache for high-volume search results.\n3.  **L3: Object Storage** (via LanceDB + S3\u002FMinIO): The \"Source of Truth\" where every namespace is isolated under its own S3 prefix.\n\n### Key Technologies\n*   **axum:** High-performance async REST API.\n*   **LanceDB:** Vector and BM25 search engine that runs natively on object storage.\n*   **foyer:** Advanced hybrid cache (RAM + NVMe) with LFU\u002FLRU eviction.\n*   **Prometheus:** Full operational visibility into cache hits, misses, and S3 request savings.\n\n## Storage backends\n\nFirn's correctness depends on S3's `If-None-Match: *` precondition behaving as a strictly linearisable compare-and-swap across concurrent writers. LanceDB's commit protocol uses it to serialise manifest updates, so a backend that ignores or incorrectly handles this header will silently lose writes. Every provider below has been tested with the same two checks: a sequential conditional-PUT pre-flight, and an 8-writer x 100-row concurrent stress (the passing backends additionally survive 100 consecutive runs). The test harness is in `crates\u002Ffirnflow-core\u002Ftests\u002F`.\n\n| Provider | Supported | Reason |\n| --- | :---: | --- |\n| **AWS S3** (`eu-west-1` validated) | ✅ | Strict CAS, clean pass on 100-run stress. |\n| **MinIO** (self-hosted \u002F local) | ✅ | Reference implementation for the S3 protocol; clean pass on 100-run stress. |\n| **Cloudflare R2** | ✅ | `If-None-Match: *` honoured correctly; 100-run stress clean. Per-iteration latency is roughly 7x AWS due to R2's multi-region commit path, but correctness is what the gate checks. Zero egress makes this the most interesting non-AWS target. Use path-style addressing. |\n| **Backblaze B2** (S3 compat layer) | ❌ | Returns `HTTP 501 NotImplemented` on the first PutObject with `If-None-Match: *`. B2's native API supports conditional writes via `X-Bz-*` headers, but the S3-compat gateway does not translate them. Loud failure: easy to detect, not usable for Firn. |\n| **Tigris** (dual-region + single-region) | ✅ | `If-None-Match: *` honoured on concurrent commits; 100-run stress clean on both dual-region and single-region buckets as of 2026-04-19 after an upstream CAS fix. Use path-style addressing on `t3.storage.dev`. |\n| **DigitalOcean Spaces** (`lon1` validated) | ✅ | Strict CAS, 100-run stress clean. Per-iteration latency ~3.10s, in the same band as AWS `eu-west-1` and the fastest non-AWS backend tested. Use the regional endpoint (`https:\u002F\u002F\u003Cregion>.digitaloceanspaces.com`), not the virtual-hosted form, with path-style addressing. |\n| **Google Cloud Storage** (via S3 interop) | ❌ | Silently ignores `If-None-Match: *`: the second conditional PUT returns `200 OK` instead of `412`. Concurrent stress loses 6\u002F8 writers every run, deterministically. GCS's native `x-goog-if-generation-match: 0` works correctly; only the S3-interop translation is broken. A future release could support GCS via LanceDB's native GCS backend rather than the S3 layer. |\n\nThe two dedicated tests live at `crates\u002Ffirnflow-core\u002Ftests\u002Fs3_conditional_writes.rs` and `crates\u002Ffirnflow-core\u002Ftests\u002Flance_concurrent_writes.rs`. Both are `#[ignore]`'d and require credentials to run. If you want to evaluate a backend not in the table, copy a block from either file and point it at your own bucket.\n\n## Features\n\n*   **Multi-tenant by Design:** Each namespace maps to an isolated S3 prefix (`s3:\u002F\u002Fbucket\u002Fnamespace\u002F`) with near-zero idle cost.\n*   **Instant Invalidation:** A \"Generation Counter\" strategy ensures that after a write, all stale search results for that namespace are invalidated in $O(1)$ time.\n*   **CAS Consistency:** Verified concurrency safety using S3's `If-None-Match: *` to prevent data loss when multiple writers fight for the same bucket.\n*   **Zero-Copy Ready:** Optimized serialization via `bincode` (with architectural triggers to move to `rkyv` if needed).\n*   **Operational Excellence:** Native Prometheus metrics tracking cache hit rates and S3 request count (the primary signal for cost savings).\n\n## Quickstart\n\n### 1. Launch the Stack\nEverything you need (MinIO storage + Firn API) is orchestrated via Docker Compose:\n\n```bash\ngit clone https:\u002F\u002Fgithub.com\u002Fgordonmurray\u002Ffirnflow\ncd firnflow\ndocker compose up --build\n```\n\n### 2. Upsert a Vector\n\nThe API is live at `http:\u002F\u002Flocalhost:3000`. Save a vector to the `demo` namespace:\n\n```bash\ncurl -X POST http:\u002F\u002Flocalhost:3000\u002Fns\u002Fdemo\u002Fupsert \\\n     -H 'Content-Type: application\u002Fjson' \\\n     -d '{\n       \"rows\": [\n         {\"id\": 1, \"vector\": [1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]}\n       ]\n     }'\n```\n\n### 3. Perform a Search\nQuery the same namespace for the nearest neighbor:\n\n```bash\ncurl -X POST http:\u002F\u002Flocalhost:3000\u002Fns\u002Fdemo\u002Fquery \\\n     -H 'Content-Type: application\u002Fjson' \\\n     -d '{\"vector\": [1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], \"k\": 1}'\n```\n\n### 4. Check the Savings\nSee how much S3 traffic you've avoided:\n\n```bash\ncurl http:\u002F\u002Flocalhost:3000\u002Fmetrics | grep s3_requests\n```\n\n## Authentication\n\nFirn ships with optional bearer-token authentication on the REST API. Both keys are opt-in:\n\n| Env var | Tier | Routes |\n| :--- | :--- | :--- |\n| `FIRNFLOW_API_KEY` | read\u002Fwrite | `upsert`, `query`, `list`, `warmup` |\n| `FIRNFLOW_ADMIN_API_KEY` | admin (destructive) | `delete`, `index`, `fts-index`, `scalar-index`, `compact` |\n| `FIRNFLOW_METRICS_TOKEN` | metrics | `\u002Fmetrics` (otherwise public) |\n\nHeader format on every protected request: `Authorization: Bearer \u003Ctoken>`.\n\nIf `FIRNFLOW_ADMIN_API_KEY` is unset, the read\u002Fwrite key authorises admin routes too (single-key fallback). Set both keys to a different value to lock destructive operations behind a separate credential. If neither key is set the API stays open and logs a single startup `WARN` — this preserves the default-open posture of the local-dev compose stack.\n\n```bash\nexport FIRNFLOW_API_KEY=$(openssl rand -hex 32)\ncurl -X POST http:\u002F\u002Flocalhost:3000\u002Fns\u002Fdemo\u002Fupsert \\\n     -H \"Authorization: Bearer $FIRNFLOW_API_KEY\" \\\n     -H 'Content-Type: application\u002Fjson' \\\n     -d '{\"rows\": [...]}'\n```\n\n**This is service-level authentication.** Any holder of `FIRNFLOW_API_KEY` can read or write any namespace; any holder of `FIRNFLOW_ADMIN_API_KEY` can additionally delete or rebuild indexes on any namespace. If you need per-tenant namespace isolation, place Firn behind an authenticating gateway that enforces tenant-to-namespace authorisation. See [`docs\u002Fconfiguration.html`](https:\u002F\u002Ffirnflow.io\u002Fconfiguration.html) for the rate-limiting knobs (`FIRNFLOW_RATE_LIMIT_RPS`, `FIRNFLOW_RATE_LIMIT_BURST`, `FIRNFLOW_PREAUTH_IP_LIMIT_RPS`) and the `FIRNFLOW_TRUST_PROXY_HEADERS` switch for deployments behind a load balancer.\n\n## API Surface\n\n| Endpoint | Method | Auth | Description |\n| :--- | :--- | :--- | :--- |\n| `\u002Fhealth` | `GET` | open | Liveness check |\n| `\u002Fmetrics` | `GET` | metrics or open | Prometheus exposition format |\n| `\u002Fns\u002F{ns}` | `DELETE` | admin | Removes all data (S3 + Cache) for a namespace |\n| `\u002Fns\u002F{ns}\u002Fupsert` | `POST` | read\u002Fwrite | Insert\u002Fupdate vectors and data |\n| `\u002Fns\u002F{ns}\u002Fquery` | `POST` | read\u002Fwrite | Vector, FTS, or hybrid search |\n| `\u002Fns\u002F{ns}\u002Flist` | `GET` | read\u002Fwrite | Cursor-paginated list ordered by `_ingested_at` |\n| `\u002Fns\u002F{ns}\u002Fwarmup` | `POST` | read\u002Fwrite | Non-blocking cache pre-warm hint |\n| `\u002Fns\u002F{ns}\u002Findex` | `POST` | admin | Build IVF_PQ vector index (async, returns 202) |\n| `\u002Fns\u002F{ns}\u002Ffts-index` | `POST` | admin | Build BM25 full-text search index (async, returns 202) |\n| `\u002Fns\u002F{ns}\u002Fscalar-index` | `POST` | admin | Build BTree index on `_ingested_at` to accelerate `\u002Flist` (async, returns 202) |\n| `\u002Fns\u002F{ns}\u002Fcompact` | `POST` | admin | Compact and prune data files (async, returns 202) |\n\nAuth column: `open` = no header required; `read\u002Fwrite` = `FIRNFLOW_API_KEY` (or `FIRNFLOW_ADMIN_API_KEY`); `admin` = `FIRNFLOW_ADMIN_API_KEY` if configured, otherwise `FIRNFLOW_API_KEY` via the single-key fallback. `\u002Fmetrics` is `metrics` when `FIRNFLOW_METRICS_TOKEN` is set, otherwise `open`.\n\n## Development and Benchmarking\n\n**Firn** uses a containerized toolchain. No local Rust installation is required.\n\n```bash\n# Run the full test suite (requires MinIO)\n.\u002Fscripts\u002Fcargo test --workspace -- --ignored\n\n# Run the cold-vs-warm latency benchmark\n.\u002Fscripts\u002Fcargo run --release -p firnflow-bench\n```\n\nBenchmark results are committed at `bench\u002Fresults\u002Fcold_vs_warm.md`.\n","Firn是一个高性能的多租户向量和全文搜索引擎，它通过分层存储架构（RAM → NVMe → S3）实现了在对象存储上的微秒级延迟。项目使用Rust语言开发，结合了LanceDB和Foyer技术，能够在本地RAM的速度下提供S3的成本效益。Firn特别适合需要快速搜索响应时间同时又希望控制云成本的场景，如大规模数据检索、实时数据分析等。其架构设计确保了即使在冷查询情况下也能显著提升性能，而热查询则几乎可以实现即时响应。",2,"2026-06-11 02:41:52","CREATED_QUERY"]