[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"project-9679":3},{"id":4,"name":5,"fullName":6,"owner":7,"repo":5,"description":8,"homepage":9,"htmlUrl":10,"language":11,"languages":10,"totalLinesOfCode":10,"stars":12,"forks":13,"watchers":14,"openIssues":15,"contributorsCount":16,"subscribersCount":16,"size":16,"stars1d":16,"stars7d":17,"stars30d":18,"stars90d":16,"forks30d":16,"starsTrendScore":16,"compositeScore":19,"rankGlobal":10,"rankLanguage":10,"license":20,"archived":21,"fork":21,"defaultBranch":22,"hasWiki":23,"hasPages":21,"topics":24,"createdAt":10,"pushedAt":10,"updatedAt":29,"readmeContent":30,"aiSummary":31,"trendingCount":16,"starSnapshotCount":16,"syncStatus":32,"lastSyncTime":33,"discoverSource":34},9679,"first-order-model","AliaksandrSiarohin\u002Ffirst-order-model","AliaksandrSiarohin","This repository contains the source code for the paper First Order Motion Model for Image Animation ","https:\u002F\u002Faliaksandrsiarohin.github.io\u002Ffirst-order-model-website\u002F",null,"Jupyter Notebook",15003,3274,346,314,0,3,10,45,"MIT License",false,"master",true,[25,26,27,28],"deep-learning","generative-model","image-animation","motion-retargeting","2026-06-12 02:02:11","\u003Cb>!!! Check out our new [paper](https:\u002F\u002Farxiv.org\u002Fpdf\u002F2104.11280.pdf) and [framework](https:\u002F\u002Fgithub.com\u002Fsnap-research\u002Farticulated-animation) improved for articulated objects\u003C\u002Fb>\n\n# First Order Motion Model for Image Animation\n\nThis repository contains the source code for the paper [First Order Motion Model for Image Animation](https:\u002F\u002Fpapers.nips.cc\u002Fpaper\u002F8935-first-order-motion-model-for-image-animation) by Aliaksandr Siarohin, [Stéphane Lathuilière](http:\u002F\u002Fstelat.eu), [Sergey Tulyakov](http:\u002F\u002Fstulyakov.com), [Elisa Ricci](http:\u002F\u002Felisaricci.eu\u002F) and [Nicu Sebe](http:\u002F\u002Fdisi.unitn.it\u002F~sebe\u002F). \n\n[Hugging Face Spaces](https:\u002F\u002Fhuggingface.co\u002Fspaces\u002Fabhishek\u002Ffirst-order-motion-model)\n\n## Example animations\n\nThe videos on the left show the driving videos. The first row on the right for each dataset shows the source videos. The bottom row contains the animated sequences with motion transferred from the driving video and object taken from the source image. We trained a separate network for each task.\n\n### VoxCeleb Dataset\n![Screenshot](sup-mat\u002Fvox-teaser.gif)\n### Fashion Dataset\n![Screenshot](sup-mat\u002Ffashion-teaser.gif)\n### MGIF Dataset\n![Screenshot](sup-mat\u002Fmgif-teaser.gif)\n\n\n### Installation\n\nWe support ```python3```. To install the dependencies run:\n```\npip install -r requirements.txt\n```\n\n### YAML configs\n\nThere are several configuration (```config\u002Fdataset_name.yaml```) files one for each `dataset`. See ```config\u002Ftaichi-256.yaml``` to get description of each parameter.\n\n\n### Pre-trained checkpoint\nCheckpoints can be found under following link: [google-drive](https:\u002F\u002Fdrive.google.com\u002Fopen?id=1PyQJmkdCsAkOYwUyaj_l-l0as-iLDgeH) or [yandex-disk](https:\u002F\u002Fyadi.sk\u002Fd\u002FlEw8uRm140L_eQ).\n\n### Animation Demo\nTo run a demo, download checkpoint and run the following command:\n```\npython demo.py  --config config\u002Fdataset_name.yaml --driving_video path\u002Fto\u002Fdriving --source_image path\u002Fto\u002Fsource --checkpoint path\u002Fto\u002Fcheckpoint --relative --adapt_scale\n```\nThe result will be stored in ```result.mp4```.\n\nThe driving videos and source images should be cropped before it can be used in our method. To obtain some semi-automatic crop suggestions you can use ```python crop-video.py --inp some_youtube_video.mp4```. It will generate commands for crops using ffmpeg. In order to use the script, face-alligment library is needed:\n```\ngit clone https:\u002F\u002Fgithub.com\u002F1adrianb\u002Fface-alignment\ncd face-alignment\npip install -r requirements.txt\npython setup.py install\n```\n\n### Animation demo with Docker\n\nIf you are having trouble getting the demo to work because of library compatibility issues,\nand you're running Linux, you might try running it inside a Docker container, which would\ngive you better control over the execution environment.\n\nRequirements: Docker 19.03+ and [nvidia-docker](https:\u002F\u002Fgithub.com\u002FNVIDIA\u002Fnvidia-docker)\ninstalled and able to successfully run the `nvidia-docker` usage tests.\n\nWe'll first build the container.\n\n```\ndocker build -t first-order-model .\n```\n\nAnd now that we have the container available locally, we can use it to run the demo.\n\n```\ndocker run -it --rm --gpus all \\\n       -v $HOME\u002Ffirst-order-model:\u002Fapp first-order-model \\\n       python3 demo.py --config config\u002Fvox-256.yaml \\\n           --driving_video driving.mp4 \\\n           --source_image source.png  \\ \n           --checkpoint vox-cpk.pth.tar \\ \n           --result_video result.mp4 \\\n           --relative --adapt_scale\n```\n\n### Colab Demo \n\n[![Open In Colab](https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg)](https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002FAliaksandrSiarohin\u002Ffirst-order-model\u002Fblob\u002Fmaster\u002Fdemo.ipynb) [![Open in Kaggle](https:\u002F\u002Fkaggle.com\u002Fstatic\u002Fimages\u002Fopen-in-kaggle.svg)](https:\u002F\u002Fkaggle.com\u002Fkernels\u002Fwelcome?src=https:\u002F\u002Fgithub.com\u002FAliaksandrSiarohin\u002Ffirst-order-model\u002Fblob\u002Fmaster\u002Fdemo.ipynb)\n\n@graphemecluster prepared a GUI demo for the Google Colab. It also works in Kaggle. For the source code, see [```demo.ipynb```](https:\u002F\u002Fgithub.com\u002FAliaksandrSiarohin\u002Ffirst-order-model\u002Fblob\u002Fmaster\u002Fdemo.ipynb).\n\nFor the old demo, see [```old_demo.ipynb```](https:\u002F\u002Fgithub.com\u002FAliaksandrSiarohin\u002Ffirst-order-model\u002Fblob\u002Fmaster\u002Fold_demo.ipynb).\n\n### Face-swap\nIt is possible to modify the method to perform face-swap using supervised segmentation masks.\n![Screenshot](sup-mat\u002Fface-swap.gif)\nFor both unsupervised and supervised video editing, such as face-swap, please refer to [Motion Co-Segmentation](https:\u002F\u002Fgithub.com\u002FAliaksandrSiarohin\u002Fmotion-cosegmentation).\n\n\n### Training\n\nTo train a model on specific dataset run:\n```\nCUDA_VISIBLE_DEVICES=0,1,2,3 python run.py --config config\u002Fdataset_name.yaml --device_ids 0,1,2,3\n```\nThe code will create a folder in the log directory (each run will create a time-stamped new directory).\nCheckpoints will be saved to this folder.\nTo check the loss values during training see ```log.txt```.\nYou can also check training data reconstructions in the ```train-vis``` subfolder.\nBy default the batch size is tuned to run on 2 or 4 Titan-X gpu (apart from speed it does not make much difference). You can change the batch size in the train_params in corresponding ```.yaml``` file.\n\n### Evaluation on video reconstruction\n\nTo evaluate the reconstruction performance run:\n```\nCUDA_VISIBLE_DEVICES=0 python run.py --config config\u002Fdataset_name.yaml --mode reconstruction --checkpoint path\u002Fto\u002Fcheckpoint\n```\nYou will need to specify the path to the checkpoint,\nthe ```reconstruction``` subfolder will be created in the checkpoint folder.\nThe generated video will be stored to this folder, also generated videos will be stored in ```png``` subfolder in loss-less '.png' format for evaluation.\nInstructions for computing metrics from the paper can be found: https:\u002F\u002Fgithub.com\u002FAliaksandrSiarohin\u002Fpose-evaluation.\n\n### Image animation\n\nIn order to animate videos run:\n```\nCUDA_VISIBLE_DEVICES=0 python run.py --config config\u002Fdataset_name.yaml --mode animate --checkpoint path\u002Fto\u002Fcheckpoint\n```\nYou will need to specify the path to the checkpoint,\nthe ```animation``` subfolder will be created in the same folder as the checkpoint.\nYou can find the generated video there and its loss-less version in the ```png``` subfolder.\nBy default video from test set will be randomly paired, but you can specify the \"source,driving\" pairs in the corresponding ```.csv``` files. The path to this file should be specified in corresponding ```.yaml``` file in pairs_list setting.\n\nThere are 2 different ways of performing animation:\nby using **absolute** keypoint locations or by using **relative** keypoint locations.\n\n1) \u003Ci>Animation using absolute coordinates:\u003C\u002Fi> the animation is performed using the absolute positions of the driving video and appearance of the source image.\nIn this way there are no specific requirements for the driving video and source appearance that is used.\nHowever this usually leads to poor performance since irrelevant details such as shape is transferred.\nCheck animate parameters in ```taichi-256.yaml``` to enable this mode.\n\n\u003Cimg src=\"sup-mat\u002Fabsolute-demo.gif\" width=\"512\"> \n\n2) \u003Ci>Animation using relative coordinates:\u003C\u002Fi> from the driving video we first estimate the relative movement of each keypoint,\nthen we add this movement to the absolute position of keypoints in the source image.\nThis keypoint along with source image is used for animation. This usually leads to better performance, however this requires\nthat the object in the first frame of the video and in the source image have the same pose\n\n\u003Cimg src=\"sup-mat\u002Frelative-demo.gif\" width=\"512\"> \n\n\n### Datasets\n\n1) **Bair**. This dataset can be directly [downloaded](https:\u002F\u002Fyadi.sk\u002Fd\u002FRr-fjn-PdmmqeA).\n\n2) **Mgif**. This dataset can be directly [downloaded](https:\u002F\u002Fyadi.sk\u002Fd\u002F5VdqLARizmnj3Q).\n\n3) **Fashion**. Follow the instruction on dataset downloading [from](https:\u002F\u002Fvision.cs.ubc.ca\u002Fdatasets\u002Ffashion\u002F).\n\n4) **Taichi**. Follow the instructions in [data\u002Ftaichi-loading](data\u002Ftaichi-loading\u002FREADME.md) or instructions from https:\u002F\u002Fgithub.com\u002FAliaksandrSiarohin\u002Fvideo-preprocessing. \n\n5) **Nemo**. Please follow the [instructions](https:\u002F\u002Fwww.uva-nemo.org\u002F) on how to download the dataset. Then the dataset should be preprocessed using scripts from https:\u002F\u002Fgithub.com\u002FAliaksandrSiarohin\u002Fvideo-preprocessing.\n \n6) **VoxCeleb**. Please follow the instruction from https:\u002F\u002Fgithub.com\u002FAliaksandrSiarohin\u002Fvideo-preprocessing.\n\n\n### Training on your own dataset\n1) Resize all the videos to the same size e.g 256x256, the videos can be in '.gif', '.mp4' or folder with images.\nWe recommend the later, for each video make a separate folder with all the frames in '.png' format. This format is loss-less, and it has better i\u002Fo performance.\n\n2) Create a folder ```data\u002Fdataset_name``` with 2 subfolders ```train``` and ```test```, put training videos in the ```train``` and testing in the ```test```.\n\n3) Create a config ```config\u002Fdataset_name.yaml```, in dataset_params specify the root dir the ```root_dir:  data\u002Fdataset_name```. Also adjust the number of epoch in train_params.\n\n#### Additional notes\n\nCitation:\n\n```\n@InProceedings{Siarohin_2019_NeurIPS,\n  author={Siarohin, Aliaksandr and Lathuilière, Stéphane and Tulyakov, Sergey and Ricci, Elisa and Sebe, Nicu},\n  title={First Order Motion Model for Image Animation},\n  booktitle = {Conference on Neural Information Processing Systems (NeurIPS)},\n  month = {December},\n  year = {2019}\n}\n```\n","该项目实现了一个基于深度学习的图像动画技术，能够将一段视频中的动作迁移到另一张静态图片上，使图片内容根据视频动作进行动态展示。其核心功能包括利用生成模型实现图像动画化与运动重定向，并支持多种数据集如VoxCeleb、Fashion等。项目采用Jupyter Notebook编写，提供了详细的安装指南、预训练模型及演示脚本，便于用户快速上手实验。此外，还支持通过Docker容器部署以解决潜在的环境兼容性问题。适用于需要创建个性化动画效果或研究图像处理技术的研究者和开发者。",2,"2026-06-11 03:24:11","top_topic"]