[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"project-74035":3},{"id":4,"name":5,"fullName":6,"owner":7,"repo":5,"description":8,"homepage":9,"htmlUrl":10,"language":11,"languages":10,"totalLinesOfCode":10,"stars":12,"forks":13,"watchers":14,"openIssues":15,"contributorsCount":16,"subscribersCount":16,"size":16,"stars1d":17,"stars7d":18,"stars30d":19,"stars90d":16,"forks30d":16,"starsTrendScore":20,"compositeScore":21,"rankGlobal":10,"rankLanguage":10,"license":22,"archived":23,"fork":23,"defaultBranch":24,"hasWiki":23,"hasPages":23,"topics":25,"createdAt":10,"pushedAt":10,"updatedAt":27,"readmeContent":28,"aiSummary":29,"trendingCount":16,"starSnapshotCount":16,"syncStatus":30,"lastSyncTime":31,"discoverSource":32},74035,"3FS","deepseek-ai\u002F3FS","deepseek-ai"," A high-performance distributed file system designed to address the challenges of AI training and inference workloads. ","",null,"C++",9956,1056,92,120,0,4,18,78,12,40.07,"MIT License",false,"main",[26],"distributed-file-system","2026-06-12 02:03:21","#  Fire-Flyer File System\n\n[![Build](https:\u002F\u002Fgithub.com\u002Fdeepseek-ai\u002F3fs\u002Factions\u002Fworkflows\u002Fbuild.yml\u002Fbadge.svg)](https:\u002F\u002Fgithub.com\u002Fdeepseek-ai\u002F3fs\u002Factions\u002Fworkflows\u002Fbuild.yml)\n[![License](https:\u002F\u002Fimg.shields.io\u002Fbadge\u002FLICENSE-MIT-blue.svg)](LICENSE)\n\nThe Fire-Flyer File System (3FS) is a high-performance distributed file system designed to address the challenges of AI training and inference workloads. It leverages modern SSDs and RDMA networks to provide a shared storage layer that simplifies development of distributed applications. Key features and benefits of 3FS include:\n\n- Performance and Usability\n  - **Disaggregated Architecture** Combines the throughput of thousands of SSDs and the network bandwidth of hundreds of storage nodes, enabling applications to access storage resource in a locality-oblivious manner.\n  - **Strong Consistency** Implements Chain Replication with Apportioned Queries (CRAQ) for strong consistency, making application code simple and easy to reason about.\n  - **File Interfaces** Develops stateless metadata services backed by a transactional key-value store (e.g., FoundationDB). The file interface is well known and used everywhere. There is no need to learn a new storage API.\n\n- Diverse Workloads\n  - **Data Preparation** Organizes outputs of data analytics pipelines into hierarchical directory structures and manages a large volume of intermediate outputs efficiently.\n  - **Dataloaders** Eliminates the need for prefetching or shuffling datasets by enabling random access to training samples across compute nodes.\n  - **Checkpointing** Supports high-throughput parallel checkpointing for large-scale training.\n  - **KVCache for Inference** Provides a cost-effective alternative to DRAM-based caching, offering high throughput and significantly larger capacity.\n\n## Documentation\n\n* [Design Notes](docs\u002Fdesign_notes.md)\n* [Setup Guide](deploy\u002FREADME.md)\n* [USRBIO API Reference](src\u002Flib\u002Fapi\u002FUsrbIo.md)\n* [P Specifications](.\u002Fspecs\u002FREADME.md)\n\n## Performance\n\n### 1. Peak throughput\n\nThe following figure demonstrates the throughput of read stress test on a large 3FS cluster. This cluster consists of 180 storage nodes, each equipped with 2×200Gbps InfiniBand NICs and sixteen 14TiB NVMe SSDs. Approximately 500+ client nodes were used for the read stress test, with each client node configured with 1x200Gbps InfiniBand NIC. The final aggregate read throughput reached approximately 6.6 TiB\u002Fs with background traffic from training jobs.\n\n![Large block read throughput under stress test on a 180-node cluster](docs\u002Fimages\u002Fpeak_throughput.jpg)\n\nTo benchmark 3FS, please use our [fio engine for USRBIO](benchmarks\u002Ffio_usrbio\u002FREADME.md).\n\n### 2. GraySort\n\nWe evaluated [smallpond](https:\u002F\u002Fgithub.com\u002Fdeepseek-ai\u002Fsmallpond) using the GraySort benchmark, which measures sort performance on large-scale datasets. Our implementation adopts a two-phase approach: (1) partitioning data via shuffle using the prefix bits of keys, and (2) in-partition sorting. Both phases read\u002Fwrite data from\u002Fto 3FS.\n\nThe test cluster comprised 25 storage nodes (2 NUMA domains\u002Fnode, 1 storage service\u002FNUMA, 2×400Gbps NICs\u002Fnode) and 50 compute nodes (2 NUMA domains, 192 physical cores, 2.2 TiB RAM, and 1×200 Gbps NIC\u002Fnode). Sorting 110.5 TiB of data across 8,192 partitions completed in 30 minutes and 14 seconds, achieving an average throughput of *3.66 TiB\u002Fmin*.\n\n![](docs\u002Fimages\u002Fgray_sort_server.png)\n![](docs\u002Fimages\u002Fgray_sort_client.png)\n\n### 3. KVCache\n\nKVCache is a technique used to optimize the LLM inference process. It avoids redundant computations by caching the key and value vectors of previous tokens in the decoder layers.\nThe top figure demonstrates the read throughput of all KVCache clients (1×400Gbps NIC\u002Fnode), highlighting both peak and average values, with peak throughput reaching up to 40 GiB\u002Fs. The bottom figure presents the IOPS of removing ops from garbage collection (GC) during the same time period.\n\n![KVCache Read Throughput](.\u002Fdocs\u002Fimages\u002Fkvcache_read_throughput.png)\n![KVCache GC IOPS](.\u002Fdocs\u002Fimages\u002Fkvcache_gc_iops.png)\n\n## Check out source code\n\nClone 3FS repository from GitHub:\n\n\tgit clone https:\u002F\u002Fgithub.com\u002Fdeepseek-ai\u002F3fs\n\nWhen `deepseek-ai\u002F3fs` has been cloned to a local file system, run the\nfollowing commands to check out the submodules:\n\n```bash\ncd 3fs\ngit submodule update --init --recursive\n.\u002Fpatches\u002Fapply.sh\n```\n\n## Install dependencies\n\nInstall dependencies:\n\n```bash\n# for Ubuntu 20.04.\napt install cmake libuv1-dev liblz4-dev liblzma-dev libdouble-conversion-dev libdwarf-dev libunwind-dev \\\n  libaio-dev libgflags-dev libgoogle-glog-dev libgtest-dev libgmock-dev clang-format-14 clang-14 clang-tidy-14 lld-14 \\\n  libgoogle-perftools-dev google-perftools libssl-dev libclang-rt-14-dev gcc-10 g++-10 libboost1.71-all-dev build-essential\n\n# for Ubuntu 22.04.\napt install cmake libuv1-dev liblz4-dev liblzma-dev libdouble-conversion-dev libdwarf-dev libunwind-dev \\\n  libaio-dev libgflags-dev libgoogle-glog-dev libgtest-dev libgmock-dev clang-format-14 clang-14 clang-tidy-14 lld-14 \\\n  libgoogle-perftools-dev google-perftools libssl-dev gcc-12 g++-12 libboost-all-dev build-essential\n\n# for openEuler 2403sp1\nyum install cmake libuv-devel lz4-devel xz-devel double-conversion-devel libdwarf-devel libunwind-devel \\\n    libaio-devel gflags-devel glog-devel gtest-devel gmock-devel clang-tools-extra clang lld \\\n    gperftools-devel gperftools openssl-devel gcc gcc-c++ boost-devel\n\n# for OpenCloudOS 9 and TencentOS 4\ndnf install epol-release wget git meson cmake perl lld gcc gcc-c++ autoconf lz4 lz4-devel xz xz-devel \\\n    double-conversion-devel libdwarf-devel libunwind-devel libaio-devel gflags-devel glog-devel \\\n    libuv-devel gmock-devel gperftools gperftools-devel openssl-devel boost-static boost-devel mono-devel \\\n    libevent-devel libibverbs-devel numactl-devel python3-devel\n```\n\nInstall other build prerequisites:\n\n- [`libfuse`](https:\u002F\u002Fgithub.com\u002Flibfuse\u002Flibfuse\u002Freleases\u002Ftag\u002Ffuse-3.16.1) 3.16.1 or newer version\n- [FoundationDB](https:\u002F\u002Fapple.github.io\u002Ffoundationdb\u002Fgetting-started-linux.html) 7.1 or newer version\n- [Rust](https:\u002F\u002Fwww.rust-lang.org\u002Ftools\u002Finstall) toolchain: minimal 1.75.0, recommended 1.85.0 or newer version (latest stable version) \n\n## Build 3FS\n\nBuild 3FS in `build` folder:\n\n```bash\n# Replace \u003Cmethod> with 'g++10' or 'g++11' based on your environment\ncmake -S . -B build \\\n      -DCMAKE_CXX_COMPILER=clang++-14 -DCMAKE_C_COMPILER=clang-14 \\\n      -DCMAKE_BUILD_TYPE=RelWithDebInfo -DCMAKE_EXPORT_COMPILE_COMMANDS=ON \\\n      -DSHUFFLE_METHOD=\u003Cmethod>\ncmake --build build -j 32\n```\n\nDue to the historical use of `std::shuffle`, binaries compiled with different compiler versions (e.g., `g++10` vs. `g++11 +`) may be incompatible ([issue](https:\u002F\u002Fgithub.com\u002Fdeepseek-ai\u002F3FS\u002Fissues\u002F368)). To resolve this, you must explicitly specify `-DSHUFFLE_METHOD` during compilation to lock in a consistent shuffle algorithm:\n\n- Existing Clusters: Use the method corresponding to the compiler version previously used to deploy the cluster (`g++10` or `g++11`).\n- New Clusters: You can choose either `g++10` or `g++11`. However, once the cluster is deployed, you must stay with the same configuration for all future builds to maintain compatibility.\n\n### Build 3FS use Docker\n- For TencentOS-4:  `docker pull docker.io\u002Ftencentos\u002Ftencentos4-deepseek3fs-build:latest`\n- For OpenCloudOS-9:  `docker pull docker.io\u002Fopencloudos\u002Fopencloudos9-deepseek3fs-build:latest`\n  \n## Run a test cluster\n\nFollow instructions in [setup guide](deploy\u002FREADME.md) to run a test cluster.\n\n## Report Issues\n\nPlease visit https:\u002F\u002Fgithub.com\u002Fdeepseek-ai\u002F3fs\u002Fissues to report issues.\n","Fire-Flyer File System (3FS) 是一个高性能分布式文件系统，专为应对AI训练和推理工作负载的挑战而设计。它利用现代SSD和RDMA网络提供了一个共享存储层，简化了分布式应用的开发。3FS采用解耦架构，结合数千个SSD的吞吐量和数百个存储节点的网络带宽，实现了数据访问的本地性透明；通过实现CRAQ协议确保强一致性，使应用程序代码简单易懂；并提供了基于事务性键值存储的状态无元数据服务，使用广泛熟知的文件接口。该系统适用于多种场景，包括数据准备、数据加载器、检查点以及推断时的KV缓存等，尤其适合需要高效处理大规模数据集的AI训练与推理任务。",2,"2026-06-11 03:48:30","high_star"]