[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"project-70987":3},{"id":4,"name":5,"fullName":6,"owner":7,"repo":5,"description":8,"homepage":9,"htmlUrl":10,"language":11,"languages":10,"totalLinesOfCode":10,"stars":12,"forks":13,"watchers":14,"openIssues":15,"contributorsCount":16,"subscribersCount":16,"size":16,"stars1d":17,"stars7d":18,"stars30d":19,"stars90d":16,"forks30d":16,"starsTrendScore":20,"compositeScore":21,"rankGlobal":10,"rankLanguage":10,"license":22,"archived":23,"fork":23,"defaultBranch":24,"hasWiki":23,"hasPages":25,"topics":26,"createdAt":10,"pushedAt":10,"updatedAt":27,"readmeContent":28,"aiSummary":29,"trendingCount":16,"starSnapshotCount":16,"syncStatus":30,"lastSyncTime":31,"discoverSource":32},70987,"xformers","facebookresearch\u002Fxformers","facebookresearch","Hackable and optimized Transformers building blocks, supporting a composable construction.","https:\u002F\u002Ffacebookresearch.github.io\u002Fxformers\u002F",null,"Python",10489,775,75,364,0,4,10,35,12,43.67,"Other",false,"main",true,[],"2026-06-12 02:02:46","\u003Cimg src=\".\u002Fdocs\u002Fassets\u002Flogo.png\" width=800>\n\n[![Open in Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002Ffacebookresearch\u002Fxformers\u002Fblob\u002Fmain\u002Fdocs\u002Fsource\u002Fxformers_mingpt.ipynb)\n\u003Cbr\u002F>\u003C!--\n![PyPI](https:\u002F\u002Fimg.shields.io\u002Fpypi\u002Fv\u002Fxformers)\n![PyPI - License](https:\u002F\u002Fimg.shields.io\u002Fpypi\u002Fl\u002Fxformers)\n[![Documentation Status](https:\u002F\u002Fgithub.com\u002Ffacebookresearch\u002Fxformers\u002Factions\u002Fworkflows\u002Fgh-pages.yml\u002Fbadge.svg)](https:\u002F\u002Fgithub.com\u002Ffacebookresearch\u002Fxformers\u002Factions\u002Fworkflows\u002Fgh-pages.yml\u002Fbadge.svg)\n-->\n[![CircleCI](https:\u002F\u002Fcircleci.com\u002Fgh\u002Ffacebookresearch\u002Fxformers.svg?style=shield)](https:\u002F\u002Fapp.circleci.com\u002Fpipelines\u002Fgithub\u002Ffacebookresearch\u002Fxformers\u002F)\n[![Codecov](https:\u002F\u002Fcodecov.io\u002Fgh\u002Ffacebookresearch\u002Fxformers\u002Fbranch\u002Fmain\u002Fgraph\u002Fbadge.svg?token=PKGKDR4JQM)](https:\u002F\u002Fcodecov.io\u002Fgh\u002Ffacebookresearch\u002Fxformers)\n[![black](https:\u002F\u002Fimg.shields.io\u002Fbadge\u002Fcode%20style-black-000000.svg)](https:\u002F\u002Fgithub.com\u002Fpsf\u002Fblack)\n\u003Cbr\u002F>\n[![PRs welcome](https:\u002F\u002Fimg.shields.io\u002Fbadge\u002FPRs-welcome-brightgreen.svg)](CONTRIBUTING.md)\n\u003C!--\n[![Downloads](https:\u002F\u002Fpepy.tech\u002Fbadge\u002Fxformers)](https:\u002F\u002Fpepy.tech\u002Fproject\u002Fxformers)\n-->\n--------------------------------------------------------------------------------\n\n## xFormers - Toolbox to Accelerate Research on Transformers\n\nxFormers is:\n- **Customizable building blocks**: Independent\u002Fcustomizable building blocks that can be used without boilerplate code. The components are domain-agnostic and xFormers is used by researchers in vision, NLP and more.\n- **Research first**: xFormers contains bleeding-edge components, that are not yet available in mainstream libraries like PyTorch.\n- **Built with efficiency in mind**: Because speed of iteration matters, components are as fast and memory-efficient as possible. xFormers contains its own CUDA kernels, but dispatches to other libraries when relevant.\n\n## Installing xFormers\n\n* **(RECOMMENDED, linux & win) Install latest stable with pip**: Requires [PyTorch 2.10.0](https:\u002F\u002Fpytorch.org\u002Fget-started\u002Flocally\u002F)\n\n```bash\n# [linux & win] cuda 12.6 version\npip3 install -U xformers --index-url https:\u002F\u002Fdownload.pytorch.org\u002Fwhl\u002Fcu126\n# [linux & win] cuda 12.8 version\npip3 install -U xformers --index-url https:\u002F\u002Fdownload.pytorch.org\u002Fwhl\u002Fcu128\n# [linux & win] cuda 13.0 version\npip3 install -U xformers --index-url https:\u002F\u002Fdownload.pytorch.org\u002Fwhl\u002Fcu130\n# [linux only] (EXPERIMENTAL) rocm 7.1 version\npip3 install -U xformers --index-url https:\u002F\u002Fdownload.pytorch.org\u002Fwhl\u002Frocm7.1\n```\n\n* **Development binaries**:\n\n```bash\n# Same requirements as for the stable version above\npip install --pre -U xformers\n```\n\n* **Install from source**: If you want to use with another version of PyTorch for instance (including nightly-releases)\n\n```bash\n# (Optional) Makes the build much faster\npip install ninja\n# Set TORCH_CUDA_ARCH_LIST if running and building on different GPU types\n# NOTE: pytorch must already be installed!\npip install -v --no-build-isolation -U git+https:\u002F\u002Fgithub.com\u002Ffacebookresearch\u002Fxformers.git@main#egg=xformers\n# (this can take dozens of minutes)\n```\n\n\n## Benchmarks\n\n**Memory-efficient MHA**\n![Benchmarks for ViTS](.\u002Fdocs\u002Fplots\u002Fmha\u002Fmha_vit.png)\n*Setup: A100 on f16, measured total time for a forward+backward pass*\n\nNote that this is exact attention, not an approximation, just by calling [`xformers.ops.memory_efficient_attention`](https:\u002F\u002Ffacebookresearch.github.io\u002Fxformers\u002Fcomponents\u002Fops.html#xformers.ops.memory_efficient_attention)\n\n**More benchmarks**\n\nxFormers provides many components, and more benchmarks are available in [BENCHMARKS.md](BENCHMARKS.md).\n\n### (Optional) Testing the installation\n\nThis command will provide information on an xFormers installation, and what kernels are built\u002Favailable:\n\n```python\npython -m xformers.info\n```\n\n## Using xFormers\n\n### Key Features\n\n1. Optimized building blocks, beyond PyTorch primitives\n   1. Memory-efficient exact attention - up to 10x faster\n   2. sparse attention\n   3. block-sparse attention\n   4. fused softmax\n   5. fused linear layer\n   6. fused layer norm\n   7. fused dropout(activation(x+bias))\n   8. fused SwiGLU\n\n### Install troubleshooting\n\n\n* NVCC and the current CUDA runtime match. Depending on your setup, you may be able to change the CUDA runtime with `module unload cuda; module load cuda\u002Fxx.x`, possibly also `nvcc`\n* the version of GCC that you're using matches the current NVCC capabilities\n* the `TORCH_CUDA_ARCH_LIST` env variable is set to the architectures that you want to support. A suggested setup (slow to build but comprehensive) is `export TORCH_CUDA_ARCH_LIST=\"6.0;6.1;6.2;7.0;7.2;7.5;8.0;8.6\"`\n* If the build from source OOMs, it's possible to reduce the parallelism of ninja with `MAX_JOBS` (eg `MAX_JOBS=2`)\n* If getting error message `Filename longer than 260 characters` on Windows, make sure long paths are enabled at OS level, and also execute the command `git config --global core.longpaths true`\n\n\n### License\n\nxFormers has a BSD-style license, as found in the [LICENSE](LICENSE) file.\nIt includes code from the [triton-lang\u002Fkernels](https:\u002F\u002Fgithub.com\u002Ftriton-lang\u002Fkernels) repo.\n\n## Citing xFormers\n\nIf you use xFormers in your publication, please cite it by using the following BibTeX entry.\n\n``` bibtex\n@Misc{xFormers2022,\n  author =       {Benjamin Lefaudeux and Francisco Massa and Diana Liskovich and Wenhan Xiong and Vittorio Caggiano and Sean Naren and Min Xu and Jieru Hu and Marta Tintore and Susan Zhang and Patrick Labatut and Daniel Haziza and Luca Wehrstedt and Jeremy Reizenstein and Grigory Sizov},\n  title =        {xFormers: A modular and hackable Transformer modelling library},\n  howpublished = {\\url{https:\u002F\u002Fgithub.com\u002Ffacebookresearch\u002Fxformers}},\n  year =         {2022}\n}\n```\n\n## Credits\n\nThe following repositories are used in xFormers, either in close to original form or as an inspiration:\n\n* [Sputnik](https:\u002F\u002Fgithub.com\u002Fgoogle-research\u002Fsputnik)\n* [GE-SpMM](https:\u002F\u002Fgithub.com\u002Fhgyhungry\u002Fge-spmm)\n* [Triton](https:\u002F\u002Fgithub.com\u002Fopenai\u002Ftriton)\n* [LucidRain Reformer](https:\u002F\u002Fgithub.com\u002Flucidrains\u002Freformer-pytorch)\n* [RevTorch](https:\u002F\u002Fgithub.com\u002FRobinBruegger\u002FRevTorch)\n* [Nystromformer](https:\u002F\u002Fgithub.com\u002Fmlpen\u002FNystromformer)\n* [FairScale](https:\u002F\u002Fgithub.com\u002Ffacebookresearch\u002Ffairscale\u002F)\n* [Pytorch Image Models](https:\u002F\u002Fgithub.com\u002Frwightman\u002Fpytorch-image-models)\n* [CUTLASS](https:\u002F\u002Fgithub.com\u002Fnvidia\u002Fcutlass)\n* [Flash-Attention](https:\u002F\u002Fgithub.com\u002FHazyResearch\u002Fflash-attention)\n","xFormers 是一个用于构建和优化Transformer模型的可组合构建块工具箱。它提供了高度自定义的组件，无需冗余代码即可使用，适用于视觉、自然语言处理等多个领域。项目包含了前沿的研究成果，这些成果尚未在PyTorch等主流库中出现。xFormers特别注重执行效率与内存利用，通过定制的CUDA内核实现快速迭代，同时也能智能地调用其他库以提升性能。该工具非常适合需要高效开发和研究新型Transformer架构的场景，如学术研究、工业应用中的模型探索等。",2,"2026-06-11 03:35:18","high_star"]