[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"project-72214":3},{"id":4,"name":5,"fullName":6,"owner":7,"repo":5,"description":8,"homepage":9,"htmlUrl":10,"language":11,"languages":10,"totalLinesOfCode":10,"stars":12,"forks":13,"watchers":14,"openIssues":15,"contributorsCount":16,"subscribersCount":16,"size":16,"stars1d":17,"stars7d":18,"stars30d":19,"stars90d":16,"forks30d":16,"starsTrendScore":18,"compositeScore":20,"rankGlobal":10,"rankLanguage":10,"license":21,"archived":22,"fork":22,"defaultBranch":23,"hasWiki":22,"hasPages":22,"topics":24,"createdAt":10,"pushedAt":10,"updatedAt":25,"readmeContent":26,"aiSummary":27,"trendingCount":16,"starSnapshotCount":16,"syncStatus":17,"lastSyncTime":28,"discoverSource":29},72214,"Vim","hustvl\u002FVim","hustvl","[ICML 2024] Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model","",null,"Python",3879,287,30,108,0,2,6,19,68.28,"Apache License 2.0",false,"main",[],"2026-06-12 04:01:04","\u003Cdiv align=\"center\">\n\u003Ch1>Vision Mamba \u003C\u002Fh1>\n\u003Ch3>Efficient Visual Representation Learning with Bidirectional State Space Model\u003C\u002Fh3>\n\n[Lianghui Zhu](https:\u002F\u002Fgithub.com\u002FUnrealluver)\u003Csup>1\u003C\u002Fsup> \\*,[Bencheng Liao](https:\u002F\u002Fgithub.com\u002FLegendBC)\u003Csup>1\u003C\u002Fsup> \\*,[Qian Zhang](https:\u002F\u002Fscholar.google.com\u002Fcitations?user=pCY-bikAAAAJ&hl=zh-CN)\u003Csup>2\u003C\u002Fsup>, [Xinlong Wang](https:\u002F\u002Fwww.xloong.wang\u002F)\u003Csup>3\u003C\u002Fsup>, [Wenyu Liu](http:\u002F\u002Feic.hust.edu.cn\u002Fprofessor\u002Fliuwenyu\u002F)\u003Csup>1\u003C\u002Fsup>, [Xinggang Wang](https:\u002F\u002Fxwcv.github.io\u002F)\u003Csup>1 :email:\u003C\u002Fsup>\n\n\u003Csup>1\u003C\u002Fsup>  Huazhong University of Science and Technology, \u003Csup>2\u003C\u002Fsup>  Horizon Robotics,  \u003Csup>3\u003C\u002Fsup> Beijing Academy of Artificial Intelligence\n\n(\\*) equal contribution, (\u003Csup>:email:\u003C\u002Fsup>) corresponding author.\n\nICML 2024 ([conference paper](https:\u002F\u002Ficml.cc\u002Fvirtual\u002F2024\u002Fposter\u002F33768)), ArXiv Preprint ([arXiv 2401.09417](https:\u002F\u002Farxiv.org\u002Fabs\u002F2401.09417)), HuggingFace Page ([🤗 2401.09417](https:\u002F\u002Fhuggingface.co\u002Fpapers\u002F2401.09417))\n\n\n\u003C\u002Fdiv>\n\n\n#\n\n\n\n### News\n* **` May. 2nd, 2024`:** Vision Mamba (Vim) is accepted by ICML2024. 🎉 Conference page can be found [here](https:\u002F\u002Ficml.cc\u002Fvirtual\u002F2024\u002Fpaper_metadata_from_author\u002F33768).\n\n* **` Feb. 10th, 2024`:** We update Vim-tiny\u002Fsmall weights and training scripts. By placing the class token at middle, Vim achieves improved results. Further details can be found in code and our updated [arXiv](https:\u002F\u002Farxiv.org\u002Fabs\u002F2401.09417).\n\n* **` Jan. 18th, 2024`:** We released our paper on Arxiv. Code\u002FModels are coming soon. Please stay tuned! ☕️\n\n\n## Abstract\nRecently the state space models (SSMs) with efficient hardware-aware designs, i.e., the Mamba deep learning model, have shown great potential for long sequence modeling. Meanwhile building efficient and generic vision backbones purely upon SSMs is an appealing direction. However, representing visual data is challenging for SSMs due to the position-sensitivity of visual data and the requirement of global context for visual understanding. In this paper, we show that the reliance on self-attention for visual representation learning is not necessary and propose a new generic vision backbone with bidirectional Mamba blocks (Vim), which marks the image sequences with position embeddings and compresses the visual representation with bidirectional state space models. On ImageNet classification, COCO object detection, and ADE20k semantic segmentation tasks, Vim achieves higher performance compared to well-established vision transformers like DeiT, while also demonstrating significantly improved computation & memory efficiency. For example, Vim is 2.8x faster than DeiT and saves 86.8% GPU memory when performing batch inference to extract features on images with a resolution of 1248x1248. The results demonstrate that Vim is capable of overcoming the computation & memory constraints on performing Transformer-style understanding for high-resolution images and it has great potential to be the next-generation backbone for vision foundation models.\n\n\n\u003Cdiv align=\"center\">\n\u003Cimg src=\"assets\u002Fvim_teaser_v1.7.png\" \u002F>\n\u003C\u002Fdiv>\n\n## Overview\n\u003Cdiv align=\"center\">\n\u003Cimg src=\"assets\u002Fvim_pipeline_v1.9.png\" \u002F>\n\u003C\u002Fdiv>\n\n## Envs. for Pretraining\n\n- NVIDIA GPUs:\n  - Python 3.10.13\n\n    - `conda create -n your_env_name python=3.10.13`\n\n  - torch 2.1.1 + cu118\n    - `pip install torch==2.1.1 torchvision==0.16.1 torchaudio==2.1.1 --index-url https:\u002F\u002Fdownload.pytorch.org\u002Fwhl\u002Fcu118`\n\n- AMD GPUs:\n  - A [Docker image](https:\u002F\u002Fhub.docker.com\u002Fr\u002Frocm\u002Fpytorch\u002Ftags) is recommended, such as `rocm\u002Fpytorch:rocm6.2_ubuntu20.04_py3.9_pytorch_release_2.1.2`. For step-by-step instructions, please refer to [Vision Mamba on AMD GPU with ROCm](https:\u002F\u002Frocm.blogs.amd.com\u002Fartificial-intelligence\u002Fvision-mamba\u002FREADME.html#vision-mamba).\n\n\n- Requirements: vim_requirements.txt\n  - `pip install -r vim\u002Fvim_requirements.txt`\n\n- Install ``causal_conv1d`` and ``mamba``\n  - `pip install -e causal_conv1d>=1.1.0`\n  - `pip install -e mamba-1p1p1`\n  \n  \n\n\n## Train Your Vim\n\n`bash vim\u002Fscripts\u002Fpt-vim-t.sh`\n\n## Train Your Vim at Finer Granularity\n`bash vim\u002Fscripts\u002Fft-vim-t.sh`\n\n## Model Weights\n\n| Model | #param. | Top-1 Acc. | Top-5 Acc. | Hugginface Repo |\n|:------------------------------------------------------------------:|:-------------:|:----------:|:----------:|:----------:|\n| [Vim-tiny](https:\u002F\u002Fhuggingface.co\u002Fhustvl\u002FVim-tiny-midclstok)    |       7M       |   76.1   | 93.0 | https:\u002F\u002Fhuggingface.co\u002Fhustvl\u002FVim-tiny-midclstok |\n| [Vim-tiny\u003Csup>+\u003C\u002Fsup>](https:\u002F\u002Fhuggingface.co\u002Fhustvl\u002FVim-tiny-midclstok)    |       7M       |   78.3   | 94.2 | https:\u002F\u002Fhuggingface.co\u002Fhustvl\u002FVim-tiny-midclstok |\n| [Vim-small](https:\u002F\u002Fhuggingface.co\u002Fhustvl\u002FVim-small-midclstok)    |       26M       |   80.5   | 95.1 | https:\u002F\u002Fhuggingface.co\u002Fhustvl\u002FVim-small-midclstok |\n| [Vim-small\u003Csup>+\u003C\u002Fsup>](https:\u002F\u002Fhuggingface.co\u002Fhustvl\u002FVim-small-midclstok)    |       26M       |   81.6   | 95.4 | https:\u002F\u002Fhuggingface.co\u002Fhustvl\u002FVim-small-midclstok |\n| [Vim-base](https:\u002F\u002Fhuggingface.co\u002Fhustvl\u002FVim-base-midclstok)    |       98M       |   81.9   | 95.8 | https:\u002F\u002Fhuggingface.co\u002Fhustvl\u002FVim-base-midclstok |\n\n**Notes:**\n- \u003Csup>+\u003C\u002Fsup> means that we finetune at finer granularity with short schedule.\n## Evaluation on Provided Weights\nTo evaluate `Vim-Ti` on ImageNet-1K, run:\n```bash\npython main.py --eval --resume \u002Fpath\u002Fto\u002Fckpt --model vim_tiny_patch16_224_bimambav2_final_pool_mean_abs_pos_embed_with_midclstok_div2 --data-path \u002Fpath\u002Fto\u002Fimagenet\n```\n## Acknowledgement :heart:\nThis project is based on Mamba ([paper](https:\u002F\u002Farxiv.org\u002Fabs\u002F2312.00752), [code](https:\u002F\u002Fgithub.com\u002Fstate-spaces\u002Fmamba)), Causal-Conv1d ([code](https:\u002F\u002Fgithub.com\u002FDao-AILab\u002Fcausal-conv1d)), DeiT ([paper](https:\u002F\u002Farxiv.org\u002Fabs\u002F2012.12877), [code](https:\u002F\u002Fgithub.com\u002Ffacebookresearch\u002Fdeit)). Thanks for their wonderful works.\n\n## Citation\nIf you find Vim is useful in your research or applications, please consider giving us a star 🌟 and citing it by the following BibTeX entry.\n\n```bibtex\n @inproceedings{vim,\n  title={Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model},\n  author={Zhu, Lianghui and Liao, Bencheng and Zhang, Qian and Wang, Xinlong and Liu, Wenyu and Wang, Xinggang},\n  booktitle={Forty-first International Conference on Machine Learning}\n}\n```\n","Vision Mamba (Vim) 是一个基于双向状态空间模型的高效视觉表示学习框架。该项目通过引入位置嵌入和双向状态空间模型来压缩视觉表示，从而在不依赖自注意力机制的情况下实现高效的视觉数据处理。其核心功能包括在图像分类、目标检测和语义分割等任务上表现出色，并且在计算速度和内存使用方面优于现有的视觉变换器如DeiT。例如，在处理高分辨率图像时，Vim比DeiT快2.8倍，并节省了86.8%的GPU内存。适用于需要高性能和资源效率的计算机视觉应用场景，特别是在处理大规模或高分辨率图像数据集时。","2026-06-11 03:40:54","high_star"]