[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"project-71064":3},{"id":4,"name":5,"fullName":6,"owner":7,"repo":5,"description":8,"homepage":9,"htmlUrl":9,"language":10,"languages":9,"totalLinesOfCode":9,"stars":11,"forks":12,"watchers":13,"openIssues":14,"contributorsCount":15,"subscribersCount":15,"size":15,"stars1d":16,"stars7d":17,"stars30d":18,"stars90d":15,"forks30d":15,"starsTrendScore":19,"compositeScore":20,"rankGlobal":9,"rankLanguage":9,"license":21,"archived":22,"fork":23,"defaultBranch":24,"hasWiki":23,"hasPages":23,"topics":25,"createdAt":9,"pushedAt":9,"updatedAt":26,"readmeContent":27,"aiSummary":28,"trendingCount":15,"starSnapshotCount":15,"syncStatus":29,"lastSyncTime":30,"discoverSource":31},71064,"dino","facebookresearch\u002Fdino","facebookresearch","PyTorch code for Vision Transformers training with the Self-Supervised learning method DINO",null,"Python",7590,1045,1,101,0,5,12,38,15,84.86,"Apache License 2.0",true,false,"main",[],"2026-06-12 04:00:58",":new: *Please check out our more recent [DINOv2](https:\u002F\u002Fgithub.com\u002Ffacebookresearch\u002Fdinov2) effort in the same line of work.*\n\n# Self-Supervised Vision Transformers with DINO\n\nPyTorch implementation and pretrained models for DINO. For details, see **Emerging Properties in Self-Supervised Vision Transformers**.  \n[[`blogpost`](https:\u002F\u002Fai.facebook.com\u002Fblog\u002Fdino-paws-computer-vision-with-self-supervised-transformers-and-10x-more-efficient-training)] [[`arXiv`](https:\u002F\u002Farxiv.org\u002Fabs\u002F2104.14294)] [[`Yannic Kilcher's video`](https:\u002F\u002Fwww.youtube.com\u002Fwatch?v=h3ij3F3cPIk)]\n\n\u003Cdiv align=\"center\">\n  \u003Cimg width=\"100%\" alt=\"DINO illustration\" src=\".github\u002Fdino.gif\">\n\u003C\u002Fdiv>\n\n## Pretrained models\nYou can choose to download only the weights of the pretrained backbone used for downstream tasks, or the full checkpoint which contains backbone and projection head weights for both student and teacher networks. We also provide the backbone in `onnx` format, as well as detailed arguments and training\u002Fevaluation logs. Note that `DeiT-S` and `ViT-S` names refer exactly to the same architecture.\n\n\u003Ctable>\n  \u003Ctr>\n    \u003Cth>arch\u003C\u002Fth>\n    \u003Cth>params\u003C\u002Fth>\n    \u003Cth>k-nn\u003C\u002Fth>\n    \u003Cth>linear\u003C\u002Fth>\n    \u003Cth colspan=\"6\">download\u003C\u002Fth>\n  \u003C\u002Ftr>\n  \u003Ctr>\n    \u003Ctd>ViT-S\u002F16\u003C\u002Ftd>\n    \u003Ctd>21M\u003C\u002Ftd>\n    \u003Ctd>74.5%\u003C\u002Ftd>\n    \u003Ctd>77.0%\u003C\u002Ftd>\n    \u003Ctd>\u003Ca href=\"https:\u002F\u002Fdl.fbaipublicfiles.com\u002Fdino\u002Fdino_deitsmall16_pretrain\u002Fdino_deitsmall16_pretrain.pth\">backbone only\u003C\u002Fa>\u003C\u002Ftd>\n    \u003Ctd>\u003Ca href=\"https:\u002F\u002Fdl.fbaipublicfiles.com\u002Fdino\u002Fdino_deitsmall16_pretrain\u002Fdino_deitsmall16_pretrain_full_checkpoint.pth\">full ckpt\u003C\u002Fa>\u003C\u002Ftd>\n    \u003Ctd>\u003Ca href=\"https:\u002F\u002Fdl.fbaipublicfiles.com\u002Fdino\u002Fdino_deitsmall16_pretrain\u002Fdino_deits16.onnx\">onnx\u003C\u002Fa>\u003C\u002Ftd>\n    \u003Ctd>\u003Ca href=\"https:\u002F\u002Fdl.fbaipublicfiles.com\u002Fdino\u002Fdino_deitsmall16_pretrain\u002Fargs.txt\">args\u003C\u002Fa>\u003C\u002Ftd>\n    \u003Ctd>\u003Ca href=\"https:\u002F\u002Fdl.fbaipublicfiles.com\u002Fdino\u002Fdino_deitsmall16_pretrain\u002Fdino_deitsmall16_pretrain_log.txt\">logs\u003C\u002Fa>\u003C\u002Ftd>\n    \u003Ctd>\u003Ca href=\"https:\u002F\u002Fdl.fbaipublicfiles.com\u002Fdino\u002Fdino_deitsmall16_pretrain\u002Fdino_deitsmall16_pretrain_eval_linear_log.txt\">eval logs\u003C\u002Fa>\u003C\u002Ftd>\n  \u003C\u002Ftr>\n  \u003Ctr>\n    \u003Ctd>ViT-S\u002F8\u003C\u002Ftd>\n    \u003Ctd>21M\u003C\u002Ftd>\n    \u003Ctd>78.3%\u003C\u002Ftd>\n    \u003Ctd>79.7%\u003C\u002Ftd>\n    \u003Ctd>\u003Ca href=\"https:\u002F\u002Fdl.fbaipublicfiles.com\u002Fdino\u002Fdino_deitsmall8_pretrain\u002Fdino_deitsmall8_pretrain.pth\">backbone only\u003C\u002Fa>\u003C\u002Ftd>\n    \u003Ctd>\u003Ca href=\"https:\u002F\u002Fdl.fbaipublicfiles.com\u002Fdino\u002Fdino_deitsmall8_pretrain\u002Fdino_deitsmall8_pretrain_full_checkpoint.pth\">full ckpt\u003C\u002Fa>\u003C\u002Ftd>\n    \u003Ctd>\u003Ca href=\"https:\u002F\u002Fdl.fbaipublicfiles.com\u002Fdino\u002Fdino_deitsmall8_pretrain\u002Fdino_deits8.onnx\">onnx\u003C\u002Fa>\u003C\u002Ftd>\n    \u003Ctd>\u003Ca href=\"https:\u002F\u002Fdl.fbaipublicfiles.com\u002Fdino\u002Fdino_deitsmall8_pretrain\u002Fargs.txt\">args\u003C\u002Fa>\u003C\u002Ftd>\n    \u003Ctd>\u003Ca href=\"https:\u002F\u002Fdl.fbaipublicfiles.com\u002Fdino\u002Fdino_deitsmall8_pretrain\u002Fdino_deitsmall8_pretrain_log.txt\">logs\u003C\u002Fa>\u003C\u002Ftd>\n    \u003Ctd>\u003Ca href=\"https:\u002F\u002Fdl.fbaipublicfiles.com\u002Fdino\u002Fdino_deitsmall8_pretrain\u002Fdino_deitsmall8_pretrain_eval_linear_log.txt\">eval logs\u003C\u002Fa>\u003C\u002Ftd>\n  \u003C\u002Ftr>\n  \u003Ctr>\n    \u003Ctd>ViT-B\u002F16\u003C\u002Ftd>\n    \u003Ctd>85M\u003C\u002Ftd>\n    \u003Ctd>76.1%\u003C\u002Ftd>\n    \u003Ctd>78.2%\u003C\u002Ftd>\n    \u003Ctd>\u003Ca href=\"https:\u002F\u002Fdl.fbaipublicfiles.com\u002Fdino\u002Fdino_vitbase16_pretrain\u002Fdino_vitbase16_pretrain.pth\">backbone only\u003C\u002Fa>\u003C\u002Ftd>\n    \u003Ctd>\u003Ca href=\"https:\u002F\u002Fdl.fbaipublicfiles.com\u002Fdino\u002Fdino_vitbase16_pretrain\u002Fdino_vitbase16_pretrain_full_checkpoint.pth\">full ckpt\u003C\u002Fa>\u003C\u002Ftd>\n    \u003Ctd>\u003Ca href=\"https:\u002F\u002Fdl.fbaipublicfiles.com\u002Fdino\u002Fdino_vitbase16_pretrain\u002Fdino_vitb16.onnx\">onnx\u003C\u002Fa>\u003C\u002Ftd>\n    \u003Ctd>\u003Ca href=\"https:\u002F\u002Fdl.fbaipublicfiles.com\u002Fdino\u002Fdino_vitbase16_pretrain\u002Fargs.txt\">args\u003C\u002Fa>\u003C\u002Ftd>\n    \u003Ctd>\u003Ca href=\"https:\u002F\u002Fdl.fbaipublicfiles.com\u002Fdino\u002Fdino_vitbase16_pretrain\u002Fdino_vitbase16_pretrain_log.txt\">logs\u003C\u002Fa>\u003C\u002Ftd>\n    \u003Ctd>\u003Ca href=\"https:\u002F\u002Fdl.fbaipublicfiles.com\u002Fdino\u002Fdino_vitbase16_pretrain\u002Fdino_vitbase16_pretrain_eval_linear_log.txt\">eval logs\u003C\u002Fa>\u003C\u002Ftd>\n  \u003C\u002Ftr>\n  \u003Ctr>\n    \u003Ctd>ViT-B\u002F8\u003C\u002Ftd>\n    \u003Ctd>85M\u003C\u002Ftd>\n    \u003Ctd>77.4%\u003C\u002Ftd>\n    \u003Ctd>80.1%\u003C\u002Ftd>\n    \u003Ctd>\u003Ca href=\"https:\u002F\u002Fdl.fbaipublicfiles.com\u002Fdino\u002Fdino_vitbase8_pretrain\u002Fdino_vitbase8_pretrain.pth\">backbone only\u003C\u002Fa>\u003C\u002Ftd>\n    \u003Ctd>\u003Ca href=\"https:\u002F\u002Fdl.fbaipublicfiles.com\u002Fdino\u002Fdino_vitbase8_pretrain\u002Fdino_vitbase8_pretrain_full_checkpoint.pth\">full ckpt\u003C\u002Fa>\u003C\u002Ftd>\n    \u003Ctd>\u003Ca href=\"https:\u002F\u002Fdl.fbaipublicfiles.com\u002Fdino\u002Fdino_vitbase8_pretrain\u002Fdino_vitb8.onnx\">onnx\u003C\u002Fa>\u003C\u002Ftd>\n    \u003Ctd>\u003Ca href=\"https:\u002F\u002Fdl.fbaipublicfiles.com\u002Fdino\u002Fdino_vitbase8_pretrain\u002Fargs.txt\">args\u003C\u002Fa>\u003C\u002Ftd>\n    \u003Ctd>\u003Ca href=\"https:\u002F\u002Fdl.fbaipublicfiles.com\u002Fdino\u002Fdino_vitbase8_pretrain\u002Fdino_vitbase8_pretrain_log.txt\">logs\u003C\u002Fa>\u003C\u002Ftd>\n    \u003Ctd>\u003Ca href=\"https:\u002F\u002Fdl.fbaipublicfiles.com\u002Fdino\u002Fdino_vitbase8_pretrain\u002Fdino_vitbase8_pretrain_eval_linear_log.txt\">eval logs\u003C\u002Fa>\u003C\u002Ftd>\n  \u003C\u002Ftr>\n  \u003Ctr>\n    \u003Ctd>ResNet-50\u003C\u002Ftd>\n    \u003Ctd>23M\u003C\u002Ftd>\n    \u003Ctd>67.5%\u003C\u002Ftd>\n    \u003Ctd>75.3%\u003C\u002Ftd>\n    \u003Ctd>\u003Ca href=\"https:\u002F\u002Fdl.fbaipublicfiles.com\u002Fdino\u002Fdino_resnet50_pretrain\u002Fdino_resnet50_pretrain.pth\">backbone only\u003C\u002Fa>\u003C\u002Ftd>\n    \u003Ctd>\u003Ca href=\"https:\u002F\u002Fdl.fbaipublicfiles.com\u002Fdino\u002Fdino_resnet50_pretrain\u002Fdino_resnet50_pretrain_full_checkpoint.pth\">full ckpt\u003C\u002Fa>\u003C\u002Ftd>\n    \u003Ctd>\u003Ca href=\"https:\u002F\u002Fdl.fbaipublicfiles.com\u002Fdino\u002Fdino_resnet50_pretrain\u002Fdino_resnet50.onnx\">onnx\u003C\u002Fa>\u003C\u002Ftd>\n    \u003Ctd>\u003Ca href=\"https:\u002F\u002Fdl.fbaipublicfiles.com\u002Fdino\u002Fdino_resnet50_pretrain\u002Fargs.txt\">args\u003C\u002Fa>\u003C\u002Ftd>\n    \u003Ctd>\u003Ca href=\"https:\u002F\u002Fdl.fbaipublicfiles.com\u002Fdino\u002Fdino_resnet50_pretrain\u002Fdino_resnet50_pretrain_log.txt\">logs\u003C\u002Fa>\u003C\u002Ftd>\n    \u003Ctd>\u003Ca href=\"https:\u002F\u002Fdl.fbaipublicfiles.com\u002Fdino\u002Fdino_resnet50_pretrain\u002Fdino_resnet50_pretrain_eval_linear_log.txt\">eval logs\u003C\u002Fa>\u003C\u002Ftd>\n  \u003C\u002Ftr>\n\u003C\u002Ftable>\n\nWe also release XCiT models ([[`arXiv`](https:\u002F\u002Farxiv.org\u002Fabs\u002F2106.09681)] [[`code`](https:\u002F\u002Fgithub.com\u002Ffacebookresearch\u002Fxcit)]) trained with DINO:\n\u003Ctable>\n  \u003Ctr>\n    \u003Cth>arch\u003C\u002Fth>\n    \u003Cth>params\u003C\u002Fth>\n    \u003Cth>k-nn\u003C\u002Fth>\n    \u003Cth>linear\u003C\u002Fth>\n    \u003Cth colspan=\"5\">download\u003C\u002Fth>\n  \u003C\u002Ftr>\n  \u003Ctr>\n    \u003Ctd>xcit_small_12_p16\u003C\u002Ftd>\n    \u003Ctd>26M\u003C\u002Ftd>\n    \u003Ctd>76.0%\u003C\u002Ftd>\n    \u003Ctd>77.8%\u003C\u002Ftd>\n    \u003Ctd>\u003Ca href=\"https:\u002F\u002Fdl.fbaipublicfiles.com\u002Fdino\u002Fdino_xcit_small_12_p16_pretrain\u002Fdino_xcit_small_12_p16_pretrain.pth\">backbone only\u003C\u002Fa>\u003C\u002Ftd>\n    \u003Ctd>\u003Ca href=\"https:\u002F\u002Fdl.fbaipublicfiles.com\u002Fdino\u002Fdino_xcit_small_12_p16_pretrain\u002Fdino_xcit_small_12_p16_pretrain_full_checkpoint.pth\">full ckpt\u003C\u002Fa>\u003C\u002Ftd>\n    \u003Ctd>\u003Ca href=\"https:\u002F\u002Fdl.fbaipublicfiles.com\u002Fdino\u002Fdino_xcit_small_12_p16_pretrain\u002Fargs.txt\">args\u003C\u002Fa>\u003C\u002Ftd>\n    \u003Ctd>\u003Ca href=\"https:\u002F\u002Fdl.fbaipublicfiles.com\u002Fdino\u002Fdino_xcit_small_12_p16_pretrain\u002Fdino_xcit_small_12_p16_pretrain_log.txt\">logs\u003C\u002Fa>\u003C\u002Ftd>\n    \u003Ctd>\u003Ca href=\"https:\u002F\u002Fdl.fbaipublicfiles.com\u002Fdino\u002Fdino_xcit_small_12_p16_pretrain\u002Fdino_xcit_small_12_p16_pretrain_eval_linear_log.txt\">eval\u003C\u002Fa>\u003C\u002Ftd>\n  \u003C\u002Ftr>\n  \u003Ctr>\n    \u003Ctd>xcit_small_12_p8\u003C\u002Ftd>\n    \u003Ctd>26M\u003C\u002Ftd>\n    \u003Ctd>77.1%\u003C\u002Ftd>\n    \u003Ctd>79.2%\u003C\u002Ftd>\n    \u003Ctd>\u003Ca href=\"https:\u002F\u002Fdl.fbaipublicfiles.com\u002Fdino\u002Fdino_xcit_small_12_p8_pretrain\u002Fdino_xcit_small_12_p8_pretrain.pth\">backbone only\u003C\u002Fa>\u003C\u002Ftd>\n    \u003Ctd>\u003Ca href=\"https:\u002F\u002Fdl.fbaipublicfiles.com\u002Fdino\u002Fdino_xcit_small_12_p8_pretrain\u002Fdino_xcit_small_12_p8_pretrain_full_checkpoint.pth\">full ckpt\u003C\u002Fa>\u003C\u002Ftd>\n    \u003Ctd>\u003Ca href=\"https:\u002F\u002Fdl.fbaipublicfiles.com\u002Fdino\u002Fdino_xcit_small_12_p8_pretrain\u002Fargs.txt\">args\u003C\u002Fa>\u003C\u002Ftd>\n    \u003Ctd>\u003Ca href=\"https:\u002F\u002Fdl.fbaipublicfiles.com\u002Fdino\u002Fdino_xcit_small_12_p8_pretrain\u002Fdino_xcit_small_12_p8_pretrain_log.txt\">logs\u003C\u002Fa>\u003C\u002Ftd>\n    \u003Ctd>\u003Ca href=\"https:\u002F\u002Fdl.fbaipublicfiles.com\u002Fdino\u002Fdino_xcit_small_12_p8_pretrain\u002Fdino_xcit_small_12_p8_pretrain_eval_linear_log.txt\">eval\u003C\u002Fa>\u003C\u002Ftd>\n  \u003C\u002Ftr>\n  \u003Ctr>\n    \u003Ctd>xcit_medium_24_p16\u003C\u002Ftd>\n    \u003Ctd>84M\u003C\u002Ftd>\n    \u003Ctd>76.4%\u003C\u002Ftd>\n    \u003Ctd>78.8%\u003C\u002Ftd>\n    \u003Ctd>\u003Ca href=\"https:\u002F\u002Fdl.fbaipublicfiles.com\u002Fdino\u002Fdino_xcit_medium_24_p16_pretrain\u002Fdino_xcit_medium_24_p16_pretrain.pth\">backbone only\u003C\u002Fa>\u003C\u002Ftd>\n    \u003Ctd>\u003Ca href=\"https:\u002F\u002Fdl.fbaipublicfiles.com\u002Fdino\u002Fdino_xcit_medium_24_p16_pretrain\u002Fdino_xcit_medium_24_p16_pretrain_full_checkpoint.pth\">full ckpt\u003C\u002Fa>\u003C\u002Ftd>\n    \u003Ctd>\u003Ca href=\"https:\u002F\u002Fdl.fbaipublicfiles.com\u002Fdino\u002Fdino_xcit_medium_24_p16_pretrain\u002Fargs.txt\">args\u003C\u002Fa>\u003C\u002Ftd>\n    \u003Ctd>\u003Ca href=\"https:\u002F\u002Fdl.fbaipublicfiles.com\u002Fdino\u002Fdino_xcit_medium_24_p16_pretrain\u002Fdino_xcit_medium_24_p16_pretrain_log.txt\">logs\u003C\u002Fa>\u003C\u002Ftd>\n    \u003Ctd>\u003Ca href=\"https:\u002F\u002Fdl.fbaipublicfiles.com\u002Fdino\u002Fdino_xcit_medium_24_p16_pretrain\u002Fdino_xcit_medium_24_p16_pretrain_eval_linear_log.txt\">eval\u003C\u002Fa>\u003C\u002Ftd>\n  \u003C\u002Ftr>\n  \u003Ctr>\n    \u003Ctd>xcit_medium_24_p8\u003C\u002Ftd>\n    \u003Ctd>84M\u003C\u002Ftd>\n    \u003Ctd>77.9%\u003C\u002Ftd>\n    \u003Ctd>80.3%\u003C\u002Ftd>\n    \u003Ctd>\u003Ca href=\"https:\u002F\u002Fdl.fbaipublicfiles.com\u002Fdino\u002Fdino_xcit_medium_24_p8_pretrain\u002Fdino_xcit_medium_24_p8_pretrain.pth\">backbone only\u003C\u002Fa>\u003C\u002Ftd>\n    \u003Ctd>\u003Ca href=\"https:\u002F\u002Fdl.fbaipublicfiles.com\u002Fdino\u002Fdino_xcit_medium_24_p8_pretrain\u002Fdino_xcit_medium_24_p8_pretrain_full_checkpoint.pth\">full ckpt\u003C\u002Fa>\u003C\u002Ftd>\n    \u003Ctd>\u003Ca href=\"https:\u002F\u002Fdl.fbaipublicfiles.com\u002Fdino\u002Fdino_xcit_medium_24_p8_pretrain\u002Fargs.txt\">args\u003C\u002Fa>\u003C\u002Ftd>\n    \u003Ctd>\u003Ca href=\"https:\u002F\u002Fdl.fbaipublicfiles.com\u002Fdino\u002Fdino_xcit_medium_24_p8_pretrain\u002Fdino_xcit_medium_24_p8_pretrain_log.txt\">logs\u003C\u002Fa>\u003C\u002Ftd>\n    \u003Ctd>\u003Ca href=\"https:\u002F\u002Fdl.fbaipublicfiles.com\u002Fdino\u002Fdino_xcit_medium_24_p8_pretrain\u002Fdino_xcit_medium_24_p8_pretrain_eval_linear_log.txt\">eval\u003C\u002Fa>\u003C\u002Ftd>\n  \u003C\u002Ftr>\n\u003C\u002Ftable>\n\n### Pretrained models on PyTorch Hub\n```python\nimport torch\nvits16 = torch.hub.load('facebookresearch\u002Fdino:main', 'dino_vits16')\nvits8 = torch.hub.load('facebookresearch\u002Fdino:main', 'dino_vits8')\nvitb16 = torch.hub.load('facebookresearch\u002Fdino:main', 'dino_vitb16')\nvitb8 = torch.hub.load('facebookresearch\u002Fdino:main', 'dino_vitb8')\nxcit_small_12_p16 = torch.hub.load('facebookresearch\u002Fdino:main', 'dino_xcit_small_12_p16')\nxcit_small_12_p8 = torch.hub.load('facebookresearch\u002Fdino:main', 'dino_xcit_small_12_p8')\nxcit_medium_24_p16 = torch.hub.load('facebookresearch\u002Fdino:main', 'dino_xcit_medium_24_p16')\nxcit_medium_24_p8 = torch.hub.load('facebookresearch\u002Fdino:main', 'dino_xcit_medium_24_p8')\nresnet50 = torch.hub.load('facebookresearch\u002Fdino:main', 'dino_resnet50')\n```\n\n## Training\n\n### Documentation\nPlease install [PyTorch](https:\u002F\u002Fpytorch.org\u002F) and download the [ImageNet](https:\u002F\u002Fimagenet.stanford.edu\u002F) dataset. This codebase has been developed with python version 3.6, PyTorch version 1.7.1, CUDA 11.0 and torchvision 0.8.2. The exact arguments to reproduce the models presented in our paper can be found in the `args` column of the [pretrained models section](https:\u002F\u002Fgithub.com\u002Ffacebookresearch\u002Fdino#pretrained-models). For a glimpse at the full documentation of DINO training please run:\n```\npython main_dino.py --help\n```\n\n### Vanilla DINO training :sauropod:\nRun DINO with ViT-small network on a single node with 8 GPUs for 100 epochs with the following command. Training time is 1.75 day and the resulting checkpoint should reach 69.3% on k-NN eval and 74.0% on linear eval. We provide [training](https:\u002F\u002Fdl.fbaipublicfiles.com\u002Fdino\u002Fexample_runs_logs\u002Fdino_vanilla_deitsmall16_log.txt) and [linear evaluation](https:\u002F\u002Fdl.fbaipublicfiles.com\u002Fdino\u002Fexample_runs_logs\u002Fdino_vanilla_deitsmall16_eval.txt) logs (with batch size 256 at evaluation time) for this run to help reproducibility.\n```\npython -m torch.distributed.launch --nproc_per_node=8 main_dino.py --arch vit_small --data_path \u002Fpath\u002Fto\u002Fimagenet\u002Ftrain --output_dir \u002Fpath\u002Fto\u002Fsaving_dir\n```\n\n### Multi-node training\nWe use Slurm and [submitit](https:\u002F\u002Fgithub.com\u002Ffacebookincubator\u002Fsubmitit) (`pip install submitit`). To train on 2 nodes with 8 GPUs each (total 16 GPUs):\n```\npython run_with_submitit.py --nodes 2 --ngpus 8 --arch vit_small --data_path \u002Fpath\u002Fto\u002Fimagenet\u002Ftrain --output_dir \u002Fpath\u002Fto\u002Fsaving_dir\n```\n\n\u003Cdetails>\n\u003Csummary>\nDINO with ViT-base network.\n\u003C\u002Fsummary>\n\n```\npython run_with_submitit.py --nodes 2 --ngpus 8 --use_volta32 --arch vit_base  --data_path \u002Fpath\u002Fto\u002Fimagenet\u002Ftrain --output_dir \u002Fpath\u002Fto\u002Fsaving_dir\n```\n\n\u003C\u002Fdetails>\n\n### Boosting DINO performance :t-rex:\nYou can improve the performance of the vanilla run by:\n- training for more epochs: `--epochs 300`,\n- increasing the teacher temperature: `--teacher_temp 0.07 --warmup_teacher_temp_epochs 30`.\n- removing last layer normalization (only safe with `--arch vit_small`): `--norm_last_layer false`,\n\n\u003Cdetails>\n\u003Csummary>\nFull command.\n\u003C\u002Fsummary>\n\n```\npython run_with_submitit.py --arch vit_small --epochs 300 --teacher_temp 0.07 --warmup_teacher_temp_epochs 30 --norm_last_layer false --data_path \u002Fpath\u002Fto\u002Fimagenet\u002Ftrain --output_dir \u002Fpath\u002Fto\u002Fsaving_dir\n```\n\n\u003C\u002Fdetails>\n\nThe resulting pretrained model should reach 73.3% on k-NN eval and 76.0% on linear eval. Training time is 2.6 days with 16 GPUs. We provide [training](https:\u002F\u002Fdl.fbaipublicfiles.com\u002Fdino\u002Fexample_runs_logs\u002Fdino_boost_deitsmall16_log.txt) and [linear evaluation](https:\u002F\u002Fdl.fbaipublicfiles.com\u002Fdino\u002Fexample_runs_logs\u002Fdino_boost_deitsmall16_eval.txt) logs (with batch size 256 at evaluation time) for this run to help reproducibility.\n\n### ResNet-50 and other convnets trainings\nThis code also works for training DINO on convolutional networks, like ResNet-50 for example. We highly recommend to adapt some optimization arguments in this case. For example following is a command to train DINO on ResNet-50 on a single node with 8 GPUs for 100 epochs. We provide [training logs](https:\u002F\u002Fdl.fbaipublicfiles.com\u002Fdino\u002Fexample_runs_logs\u002Fdino_rn50_log.txt) and [final checkpoint](https:\u002F\u002Fdl.fbaipublicfiles.com\u002Fdino\u002Fexample_runs_logs\u002Fdino_rn50_checkpoint.pth) for this run.\n```\npython -m torch.distributed.launch --nproc_per_node=8 main_dino.py --arch resnet50 --optimizer sgd --lr 0.03 --weight_decay 1e-4 --weight_decay_end 1e-4 --global_crops_scale 0.14 1 --local_crops_scale 0.05 0.14 --data_path \u002Fpath\u002Fto\u002Fimagenet\u002Ftrain --output_dir \u002Fpath\u002Fto\u002Fsaving_dir\n```\n\n## Self-attention visualization\nYou can look at the self-attention of the [CLS] token on the different heads of the last layer by running:\n```\npython visualize_attention.py\n```\n\n\u003Cdiv align=\"center\">\n  \u003Cimg width=\"100%\" alt=\"Self-attention from a Vision Transformer with 8x8 patches trained with DINO\" src=\".github\u002Fattention_maps.png\">\n\u003C\u002Fdiv>\n\n## Self-attention video generation\nYou can generate videos like the one on the blog post with `video_generation.py`.\n\nhttps:\u002F\u002Fuser-images.githubusercontent.com\u002F46140458\u002F116817761-47885e80-ab68-11eb-9975-d61d5a919e13.mp4\n\nExtract frames from input video and generate attention video:\n```\npython video_generation.py  --pretrained_weights dino_deitsmall8_pretrain.pth \\\n    --input_path input\u002Fvideo.mp4 \\\n    --output_path output\u002F \\\n    --fps 25\n```\n\nUse folder of frames already extracted and generate attention video:\n```\npython video_generation.py  --pretrained_weights dino_deitsmall8_pretrain.pth \\\n    --input_path output\u002Fframes\u002F \\\n    --output_path output\u002F \\\n    --resize 256 \\\n```\n\nOnly generate video from folder of attention maps images:\n```\npython video_generation.py --input_path output\u002Fattention \\\n    --output_path output\u002F \\\n    --video_only \\\n    --video_format avi\n```\n\n\n## Evaluation: k-NN classification on ImageNet\nTo evaluate a simple k-NN classifier with a single GPU on a pre-trained model, run:\n```\npython -m torch.distributed.launch --nproc_per_node=1 eval_knn.py --data_path \u002Fpath\u002Fto\u002Fimagenet\n```\nIf you choose not to specify `--pretrained_weights`, then DINO reference weights are used by default. If you want instead to evaluate checkpoints from a run of your own, you can run for example:\n```\npython -m torch.distributed.launch --nproc_per_node=1 eval_knn.py --pretrained_weights \u002Fpath\u002Fto\u002Fcheckpoint.pth --checkpoint_key teacher --data_path \u002Fpath\u002Fto\u002Fimagenet \n```\n\n## Evaluation: Linear classification on ImageNet\nTo train a supervised linear classifier on frozen weights on a single node with 8 gpus, run:\n```\npython -m torch.distributed.launch --nproc_per_node=8 eval_linear.py --data_path \u002Fpath\u002Fto\u002Fimagenet\n```\n\nWe release the logs and weights from evaluating the different models:\n\n\u003Ctable>\n  \u003Ctr>\n    \u003Cth>arch\u003C\u002Fth>\n    \u003Cth>top-1 ImageNet\u003C\u002Fth>\n    \u003Cth colspan=\"2\">linear evaluation\u003C\u002Fth>\n  \u003C\u002Ftr>\n  \u003Ctr>\n    \u003Ctd>ViT-S\u002F16\u003C\u002Ftd>\n    \u003Ctd>77.0%\u003C\u002Ftd>\n    \u003Ctd>\u003Ca href=\"https:\u002F\u002Fdl.fbaipublicfiles.com\u002Fdino\u002Fdino_deitsmall16_pretrain\u002Fdino_deitsmall16_linearweights.pth\">linear weights\u003C\u002Fa>\u003C\u002Ftd>\n    \u003Ctd>\u003Ca href=\"https:\u002F\u002Fdl.fbaipublicfiles.com\u002Fdino\u002Fdino_deitsmall16_pretrain\u002Fdino_deitsmall16_pretrain_eval_linear_log.txt\">logs\u003C\u002Fa>\u003C\u002Ftd>\n  \u003C\u002Ftr>\n  \u003Ctr>\n    \u003Ctd>ViT-S\u002F8\u003C\u002Ftd>\n    \u003Ctd>79.7%\u003C\u002Ftd>\n    \u003Ctd>\u003Ca href=\"https:\u002F\u002Fdl.fbaipublicfiles.com\u002Fdino\u002Fdino_deitsmall8_pretrain\u002Fdino_deitsmall8_linearweights.pth\">linear weights\u003C\u002Fa>\u003C\u002Ftd>\n    \u003Ctd>\u003Ca href=\"https:\u002F\u002Fdl.fbaipublicfiles.com\u002Fdino\u002Fdino_deitsmall8_pretrain\u002Fdino_deitsmall8_pretrain_eval_linear_log.txt\">logs\u003C\u002Fa>\u003C\u002Ftd>\n  \u003C\u002Ftr>\n  \u003Ctr>\n    \u003Ctd>ViT-B\u002F16\u003C\u002Ftd>\n    \u003Ctd>78.2%\u003C\u002Ftd>\n    \u003Ctd>\u003Ca href=\"https:\u002F\u002Fdl.fbaipublicfiles.com\u002Fdino\u002Fdino_vitbase16_pretrain\u002Fdino_vitbase16_linearweights.pth\">linear weights\u003C\u002Fa>\u003C\u002Ftd>\n    \u003Ctd>\u003Ca href=\"https:\u002F\u002Fdl.fbaipublicfiles.com\u002Fdino\u002Fdino_vitbase16_pretrain\u002Fdino_vitbase16_pretrain_eval_linear_log.txt\">logs\u003C\u002Fa>\u003C\u002Ftd>\n  \u003C\u002Ftr>\n  \u003Ctr>\n    \u003Ctd>ViT-B\u002F8\u003C\u002Ftd>\n    \u003Ctd>80.1%\u003C\u002Ftd>\n    \u003Ctd>\u003Ca href=\"https:\u002F\u002Fdl.fbaipublicfiles.com\u002Fdino\u002Fdino_vitbase8_pretrain\u002Fdino_vitbase8_linearweights.pth\">linear weights\u003C\u002Fa>\u003C\u002Ftd>\n    \u003Ctd>\u003Ca href=\"https:\u002F\u002Fdl.fbaipublicfiles.com\u002Fdino\u002Fdino_vitbase8_pretrain\u002Fdino_vitbase8_pretrain_eval_linear_log.txt\">logs\u003C\u002Fa>\u003C\u002Ftd>\n  \u003C\u002Ftr>\n  \u003Ctr>\n    \u003Ctd>xcit_small_12_p16\u003C\u002Ftd>\n    \u003Ctd>77.8%\u003C\u002Ftd>\n    \u003Ctd>\u003Ca href=\"https:\u002F\u002Fdl.fbaipublicfiles.com\u002Fdino\u002Fdino_xcit_small_12_p16_pretrain\u002Fdino_xcit_small_12_p16_linearweights.pth\">linear weights\u003C\u002Fa>\u003C\u002Ftd>\n    \u003Ctd>\u003Ca href=\"https:\u002F\u002Fdl.fbaipublicfiles.com\u002Fdino\u002Fdino_xcit_small_12_p16_pretrain\u002Fdino_xcit_small_12_p16_pretrain_eval_linear_log.txt\">logs\u003C\u002Fa>\u003C\u002Ftd>\n  \u003C\u002Ftr>\n  \u003Ctr>\n    \u003Ctd>xcit_small_12_p8\u003C\u002Ftd>\n    \u003Ctd>79.2%\u003C\u002Ftd>\n    \u003Ctd>\u003Ca href=\"https:\u002F\u002Fdl.fbaipublicfiles.com\u002Fdino\u002Fdino_xcit_small_12_p8_pretrain\u002Fdino_xcit_small_12_p8_linearweights.pth\">linear weights\u003C\u002Fa>\u003C\u002Ftd>\n    \u003Ctd>\u003Ca href=\"https:\u002F\u002Fdl.fbaipublicfiles.com\u002Fdino\u002Fdino_xcit_small_12_p8_pretrain\u002Fdino_xcit_small_12_p8_pretrain_eval_linear_log.txt\">logs\u003C\u002Fa>\u003C\u002Ftd>\n  \u003C\u002Ftr>\n  \u003Ctr>\n    \u003Ctd>xcit_medium_24_p16\u003C\u002Ftd>\n    \u003Ctd>78.8%\u003C\u002Ftd>\n    \u003Ctd>\u003Ca href=\"https:\u002F\u002Fdl.fbaipublicfiles.com\u002Fdino\u002Fdino_xcit_medium_24_p16_pretrain\u002Fdino_xcit_medium_24_p16_linearweights.pth\">linear weights\u003C\u002Fa>\u003C\u002Ftd>\n    \u003Ctd>\u003Ca href=\"https:\u002F\u002Fdl.fbaipublicfiles.com\u002Fdino\u002Fdino_xcit_medium_24_p16_pretrain\u002Fdino_xcit_medium_24_p16_pretrain_eval_linear_log.txt\">logs\u003C\u002Fa>\u003C\u002Ftd>\n  \u003C\u002Ftr>\n  \u003Ctr>\n    \u003Ctd>xcit_medium_24_p8\u003C\u002Ftd>\n    \u003Ctd>80.3%\u003C\u002Ftd>\n    \u003Ctd>\u003Ca href=\"https:\u002F\u002Fdl.fbaipublicfiles.com\u002Fdino\u002Fdino_xcit_medium_24_p8_pretrain\u002Fdino_xcit_medium_24_p8_linearweights.pth\">linear weights\u003C\u002Fa>\u003C\u002Ftd>\n    \u003Ctd>\u003Ca href=\"https:\u002F\u002Fdl.fbaipublicfiles.com\u002Fdino\u002Fdino_xcit_medium_24_p8_pretrain\u002Fdino_xcit_medium_24_p8_pretrain_eval_linear_log.txt\">logs\u003C\u002Fa>\u003C\u002Ftd>\n  \u003C\u002Ftr>\n  \u003Ctr>\n    \u003Ctd>ResNet-50\u003C\u002Ftd>\n    \u003Ctd>75.3%\u003C\u002Ftd>\n    \u003Ctd>\u003Ca href=\"https:\u002F\u002Fdl.fbaipublicfiles.com\u002Fdino\u002Fdino_resnet50_pretrain\u002Fdino_resnet50_linearweights.pth\">linear weights\u003C\u002Fa>\u003C\u002Ftd>\n    \u003Ctd>\u003Ca href=\"https:\u002F\u002Fdl.fbaipublicfiles.com\u002Fdino\u002Fdino_resnet50_pretrain\u002Fdino_resnet50_pretrain_eval_linear_log.txt\">logs\u003C\u002Fa>\u003C\u002Ftd>\n  \u003C\u002Ftr>\n\u003C\u002Ftable>\n\nYou can check the performance of the pretrained weights on ImageNet validation set by running the following command lines:\n```\npython eval_linear.py --evaluate --arch vit_small --patch_size 16 --data_path \u002Fpath\u002Fto\u002Fimagenet\u002Ftrain\n```\n\n```\npython eval_linear.py --evaluate --arch vit_small --patch_size 8 --data_path \u002Fpath\u002Fto\u002Fimagenet\u002Ftrain\n```\n\n```\npython eval_linear.py --evaluate --arch vit_base --patch_size 16 --n_last_blocks 1 --avgpool_patchtokens true --data_path \u002Fpath\u002Fto\u002Fimagenet\u002Ftrain\n```\n\n```\npython eval_linear.py --evaluate --arch vit_base --patch_size 8 --n_last_blocks 1 --avgpool_patchtokens true --data_path \u002Fpath\u002Fto\u002Fimagenet\u002Ftrain\n```\n\n```\npython eval_linear.py --evaluate --arch resnet50 --data_path \u002Fpath\u002Fto\u002Fimagenet\u002Ftrain\n```\n\n## Evaluation: DAVIS 2017 Video object segmentation\nPlease verify that you're using pytorch version 1.7.1 since we are not able to reproduce the results with most recent pytorch 1.8.1 at the moment.\n\n**Step 1: Prepare DAVIS 2017 data**  \n```\ncd $HOME\ngit clone https:\u002F\u002Fgithub.com\u002Fdavisvideochallenge\u002Fdavis-2017 && cd davis-2017\n.\u002Fdata\u002Fget_davis.sh\n```\n\n**Step 2: Video object segmentation**  \n```\npython eval_video_segmentation.py --data_path $HOME\u002Fdavis-2017\u002FDAVIS\u002F --output_dir \u002Fpath\u002Fto\u002Fsaving_dir\n```\n\n**Step 3: Evaluate the obtained segmentation**  \n```\ngit clone https:\u002F\u002Fgithub.com\u002Fdavisvideochallenge\u002Fdavis2017-evaluation $HOME\u002Fdavis2017-evaluation\npython $HOME\u002Fdavis2017-evaluation\u002Fevaluation_method.py --task semi-supervised --results_path \u002Fpath\u002Fto\u002Fsaving_dir --davis_path $HOME\u002Fdavis-2017\u002FDAVIS\u002F\n```\n\n## Evaluation: Image Retrieval on revisited Oxford and Paris\nStep 1: Prepare revisited Oxford and Paris by following [this repo](https:\u002F\u002Fgithub.com\u002Ffilipradenovic\u002Frevisitop).\n\nStep 2: Image retrieval (if you do not specify weights with `--pretrained_weights` then by default [DINO weights pretrained on Google Landmark v2 dataset](https:\u002F\u002Fdl.fbaipublicfiles.com\u002Fdino\u002Fdino_vitsmall16_googlelandmark_pretrain\u002Fdino_vitsmall16_googlelandmark_pretrain.pth) will be used).\n\nParis:\n```\npython -m torch.distributed.launch --use_env --nproc_per_node=1 eval_image_retrieval.py --imsize 512 --multiscale 1 --data_path \u002Fpath\u002Fto\u002Frevisited_paris_oxford\u002F --dataset rparis6k\n```\n\nOxford:\n```\npython -m torch.distributed.launch --use_env --nproc_per_node=1 eval_image_retrieval.py --imsize 224 --multiscale 0 --data_path \u002Fpath\u002Fto\u002Frevisited_paris_oxford\u002F --dataset roxford5k\n```\n\n## Evaluation: Copy detection on Copydays\nStep 1: Prepare [Copydays dataset](https:\u002F\u002Flear.inrialpes.fr\u002F~jegou\u002Fdata.php#copydays).\n\nStep 2 (opt): Prepare a set of image distractors and a set of images on which to learn the whitening operator.\nIn our paper, we use 10k random images from YFCC100M as distractors and 20k random images from YFCC100M (different from the distractors) for computing the whitening operation.\n\nStep 3: Run copy detection:\n```\npython -m torch.distributed.launch --use_env --nproc_per_node=1 eval_copy_detection.py --data_path \u002Fpath\u002Fto\u002Fcopydays\u002F --whitening_path \u002Fpath\u002Fto\u002Fwhitening_data\u002F --distractors_path \u002Fpath\u002Fto\u002Fdistractors\u002F\n```\nWe report result on the strong subset. For example in the stdout from the command above we get: `eval on strong mAP=0.858`.\n\n## License\nThis repository is released under the Apache 2.0 license as found in the [LICENSE](LICENSE) file.\n\n## Citation\nIf you find this repository useful, please consider giving a star :star: and citation :t-rex::\n```\n@inproceedings{caron2021emerging,\n  title={Emerging Properties in Self-Supervised Vision Transformers},\n  author={Caron, Mathilde and Touvron, Hugo and Misra, Ishan and J\\'egou, Herv\\'e  and Mairal, Julien and Bojanowski, Piotr and Joulin, Armand},\n  booktitle={Proceedings of the International Conference on Computer Vision (ICCV)},\n  year={2021}\n}\n```\n","DINO 是一个基于 PyTorch 的自监督学习方法，用于训练视觉变换器（Vision Transformers）。项目提供了预训练模型和详细的实现代码，支持多种架构如 ViT-S\u002F16、ViT-S\u002F8 和 ViT-B\u002F16，并且可以下载仅包含骨干网络权重的版本或完整的检查点。其核心技术特点是通过自监督学习使得模型在不需要大量标注数据的情况下也能达到较好的性能表现。适用于需要高效利用未标记图像数据进行预训练，进而提升下游任务效果的各种计算机视觉场景中，比如图像分类、目标检测等。",2,"2026-06-11 03:35:43","high_star"]