[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"project-72420":3},{"id":4,"name":5,"fullName":6,"owner":7,"repo":5,"description":8,"homepage":9,"htmlUrl":10,"language":11,"languages":10,"totalLinesOfCode":10,"stars":12,"forks":13,"watchers":14,"openIssues":15,"contributorsCount":16,"subscribersCount":16,"size":16,"stars1d":17,"stars7d":18,"stars30d":19,"stars90d":16,"forks30d":16,"starsTrendScore":20,"compositeScore":21,"rankGlobal":10,"rankLanguage":10,"license":22,"archived":23,"fork":23,"defaultBranch":24,"hasWiki":23,"hasPages":23,"topics":25,"createdAt":10,"pushedAt":10,"updatedAt":26,"readmeContent":27,"aiSummary":28,"trendingCount":16,"starSnapshotCount":16,"syncStatus":29,"lastSyncTime":30,"discoverSource":31},72420,"MusePose","TMElyralab\u002FMusePose","TMElyralab","MusePose: a Pose-Driven Image-to-Video Framework for Virtual Human Generation","",null,"Python",2689,202,39,50,0,4,18,23,12,78.22,"Other",false,"main",[],"2026-06-12 04:01:05","# MusePose\n\nMusePose: a Pose-Driven Image-to-Video Framework for Virtual Human Generation. \n\nZhengyan Tong,\nChao Li,\nZhaokang Chen,\nBin Wu\u003Csup>†\u003C\u002Fsup>,\nWenjiang Zhou\n(\u003Csup>†\u003C\u002Fsup>Corresponding Author, benbinwu@tencent.com)\n\nLyra Lab, Tencent Music Entertainment\n\n\n**[github](https:\u002F\u002Fgithub.com\u002FTMElyralab\u002FMusePose)**    **[huggingface](https:\u002F\u002Fhuggingface.co\u002FTMElyralab\u002FMusePose)**    **space (comming soon)**    **Project (comming soon)**    **Technical report (comming soon)**\n\n[MusePose](https:\u002F\u002Fgithub.com\u002FTMElyralab\u002FMusePose) is an image-to-video generation framework for virtual human under control signal such as pose. The current released model was an implementation of [AnimateAnyone](https:\u002F\u002Fgithub.com\u002FHumanAIGC\u002FAnimateAnyone) by optimizing [Moore-AnimateAnyone](https:\u002F\u002Fgithub.com\u002FMooreThreads\u002FMoore-AnimateAnyone).\n\n`MusePose` is the last building block of **the Muse opensource serie**. Together with [MuseV](https:\u002F\u002Fgithub.com\u002FTMElyralab\u002FMuseV) and [MuseTalk](https:\u002F\u002Fgithub.com\u002FTMElyralab\u002FMuseTalk), we hope the community can join us and march towards the vision where a virtual human can be generated end2end with native ability of full body movement and interaction. Please stay tuned for our next milestone!\n\nWe really appreciate [AnimateAnyone](https:\u002F\u002Fgithub.com\u002FHumanAIGC\u002FAnimateAnyone) for their academic paper and [Moore-AnimateAnyone](https:\u002F\u002Fgithub.com\u002FMooreThreads\u002FMoore-AnimateAnyone) for their code base, which have significantly expedited the development of the AIGC community and [MusePose](https:\u002F\u002Fgithub.com\u002FTMElyralab\u002FMusePose).\n\nUpdate:\n1. We release train codes of MusePose now!\n\n## Overview\n[MusePose](https:\u002F\u002Fgithub.com\u002FTMElyralab\u002FMusePose) is a diffusion-based and pose-guided virtual human video generation framework.  \nOur main contributions could be summarized as follows:\n1. The released model can generate dance videos of the human character in a reference image under the given pose sequence. The result quality exceeds almost all current open source models within the same topic.\n2. We release the `pose align` algorithm so that users could align arbitrary dance videos to arbitrary reference images, which **SIGNIFICANTLY** improved inference performance and enhanced model usability.\n3. We have fixed several important bugs and made some improvement based on the code of [Moore-AnimateAnyone](https:\u002F\u002Fgithub.com\u002FMooreThreads\u002FMoore-AnimateAnyone).\n\n## Demos\n\u003Ctable class=\"center\">\n    \n\u003Ctr>\n    \u003Ctd width=50% style=\"border: none\">\n        \u003Cvideo controls autoplay loop src=\"https:\u002F\u002Fgithub.com\u002FTMElyralab\u002FMusePose\u002Fassets\u002F47803475\u002Fbb52ca3e-8a5c-405a-8575-7ab42abca248\" muted=\"false\">\u003C\u002Fvideo>\n    \u003C\u002Ftd>\n    \u003Ctd width=50% style=\"border: none\">\n        \u003Cvideo controls autoplay loop src=\"https:\u002F\u002Fgithub.com\u002FTMElyralab\u002FMusePose\u002Fassets\u002F47803475\u002F6667c9ae-8417-49a1-bbbb-fe1695404c23\" muted=\"false\">\u003C\u002Fvideo>\n    \u003C\u002Ftd>\n\u003C\u002Ftr>\n\n\u003Ctr>\n    \u003Ctd width=50% style=\"border: none\">\n        \u003Cvideo controls autoplay loop src=\"https:\u002F\u002Fgithub.com\u002FTMElyralab\u002FMusePose\u002Fassets\u002F47803475\u002F7f7a3aaf-2720-4b50-8bca-3257acce4733\" muted=\"false\">\u003C\u002Fvideo>\n    \u003C\u002Ftd>\n    \u003Ctd width=50% style=\"border: none\">\n        \u003Cvideo controls autoplay loop src=\"https:\u002F\u002Fgithub.com\u002FTMElyralab\u002FMusePose\u002Fassets\u002F47803475\u002Fc56f7e9c-d94d-494e-88e6-62a4a3c1e016\" muted=\"false\">\u003C\u002Fvideo>\n    \u003C\u002Ftd>\n\u003C\u002Ftr>\n\n\n\u003Ctr>\n    \u003Ctd width=50% style=\"border: none\">\n        \u003Cvideo controls autoplay loop src=\"https:\u002F\u002Fgithub.com\u002FTMElyralab\u002FMusePose\u002Fassets\u002F47803475\u002F00a9faec-2453-4834-ad1f-44eb0ec8247d\" muted=\"false\">\u003C\u002Fvideo>\n    \u003C\u002Ftd>\n    \u003Ctd width=50% style=\"border: none\">\n        \u003Cvideo controls autoplay loop src=\"https:\u002F\u002Fgithub.com\u002FTMElyralab\u002FMusePose\u002Fassets\u002F47803475\u002F41ad26b3-d477-4975-bf29-73a3c9ed0380\" muted=\"false\">\u003C\u002Fvideo>\n    \u003C\u002Ftd>\n\u003C\u002Ftr>\n\n\u003Ctr>\n    \u003Ctd width=50% style=\"border: none\">\n        \u003Cvideo controls autoplay loop src=\"https:\u002F\u002Fgithub.com\u002FTMElyralab\u002FMusePose\u002Fassets\u002F47803475\u002F2bbebf98-6805-4f1b-b769-537f69cc0e4b\" muted=\"false\">\u003C\u002Fvideo>\n    \u003C\u002Ftd>\n    \u003Ctd width=50% style=\"border: none\">\n        \u003Cvideo controls autoplay loop src=\"https:\u002F\u002Fgithub.com\u002FTMElyralab\u002FMusePose\u002Fassets\u002F47803475\u002F1b2b97d0-0ae9-49a6-83ba-b3024ae64f08\" muted=\"false\">\u003C\u002Fvideo>\n    \u003C\u002Ftd>\n\u003C\u002Ftr>\n\n\u003C\u002Ftable>\n\n\n## News\n- [05\u002F27\u002F2024] Release `MusePose` and pretrained models.\n- [05\u002F31\u002F2024] Support [Comfyui-MusePose](https:\u002F\u002Fgithub.com\u002FTMElyralab\u002FComfyui-MusePose)\n- [06\u002F14\u002F2024] Bug Fixed in `inference_v2.yaml`.\n- [03\u002F04\u002F2025] Release train codes.\n\n## Todo:\n- [x] release our trained models and inference codes of MusePose.\n- [x] release pose align algorithm.\n- [x] Comfyui-MusePose\n- [x] training guidelines.\n- [ ] Huggingface Gradio demo.\n- [ ] a improved architecture and model (may take longer).\n\n\n# Getting Started\nWe provide a detailed tutorial about the installation and the basic usage of MusePose for new users:\n\n## Installation\nTo prepare the Python environment and install additional packages such as opencv, diffusers, mmcv, etc., please follow the steps below:\n\n### Build environment\n\nWe recommend a python version >=3.10 and cuda version =11.7. Then build environment as follows:\n\n```shell\npip install -r requirements.txt\n```\n\n### mmlab packages\n```bash\npip install --no-cache-dir -U openmim \nmim install mmengine \nmim install \"mmcv>=2.0.1\" \nmim install \"mmdet>=3.1.0\" \nmim install \"mmpose>=1.1.0\" \n```\n\n\n### Download weights\nYou can download weights manually as follows:\n\n1. Download our trained [weights](https:\u002F\u002Fhuggingface.co\u002FTMElyralab\u002FMusePose).\n\n2. Download the weights of other components:\n   - [sd-image-variations-diffusers](https:\u002F\u002Fhuggingface.co\u002Flambdalabs\u002Fsd-image-variations-diffusers\u002Ftree\u002Fmain\u002Funet)\n   - [sd-vae-ft-mse](https:\u002F\u002Fhuggingface.co\u002Fstabilityai\u002Fsd-vae-ft-mse)\n   - [dwpose](https:\u002F\u002Fhuggingface.co\u002Fyzd-v\u002FDWPose\u002Ftree\u002Fmain)\n   - [yolox](https:\u002F\u002Fdownload.openmmlab.com\u002Fmmdetection\u002Fv2.0\u002Fyolox\u002Fyolox_l_8x8_300e_coco\u002Fyolox_l_8x8_300e_coco_20211126_140236-d3bd2b23.pth) - Make sure to rename to `yolox_l_8x8_300e_coco.pth`\n   - [image_encoder](https:\u002F\u002Fhuggingface.co\u002Flambdalabs\u002Fsd-image-variations-diffusers\u002Ftree\u002Fmain\u002Fimage_encoder)\n   - [control_v11p_sd15_openpose](https:\u002F\u002Fhuggingface.co\u002Flllyasviel\u002Fcontrol_v11p_sd15_openpose\u002Fblob\u002Fmain\u002Fdiffusion_pytorch_model.bin) (for training only)\n   - [animatediff](https:\u002F\u002Fhuggingface.co\u002Fguoyww\u002Fanimatediff\u002Fblob\u002Fmain\u002Fmm_sd_v15_v2.ckpt) (for training only)\n\nFinally, these weights should be organized in `pretrained_weights` as follows:\n```\n.\u002Fpretrained_weights\u002F\n|-- MusePose\n|   |-- denoising_unet.pth\n|   |-- motion_module.pth\n|   |-- pose_guider.pth\n|   └── reference_unet.pth\n|-- dwpose\n|   |-- dw-ll_ucoco_384.pth\n|   └── yolox_l_8x8_300e_coco.pth\n|-- sd-image-variations-diffusers\n|   └── unet\n|       |-- config.json\n|       └── diffusion_pytorch_model.bin\n|-- image_encoder\n|   |-- config.json\n|   └── pytorch_model.bin\n|-- sd-vae-ft-mse\n|   |-- config.json\n|   └── diffusion_pytorch_model.bin\n|-- control_v11p_sd15_openpose\n|   └── diffusion_pytorch_model.bin\n└── animatediff\n    └── mm_sd_v15_v2.ckpt\n```\n\n## Quickstart\n### Inference\n#### Preparation\nPrepare your referemce images and dance videos in the folder ```.\u002Fassets``` and organnized as the example: \n```\n.\u002Fassets\u002F\n|-- images\n|   └── ref.png\n└── videos\n    └── dance.mp4\n```\n\n#### Pose Alignment\nGet the aligned dwpose of the reference image:\n```\npython pose_align.py --imgfn_refer .\u002Fassets\u002Fimages\u002Fref.png --vidfn .\u002Fassets\u002Fvideos\u002Fdance.mp4\n```\nAfter this, you can see the pose align results in ```.\u002Fassets\u002Fposes```, where ```.\u002Fassets\u002Fposes\u002Falign\u002Fimg_ref_video_dance.mp4``` is the aligned dwpose and the ```.\u002Fassets\u002Fposes\u002Falign_demo\u002Fimg_ref_video_dance.mp4``` is for debug.\n\n#### Inferring MusePose\nAdd the path of the reference image and the aligned dwpose to the test config file ```.\u002Fconfigs\u002Ftest_stage_2.yaml``` as the example:\n```\ntest_cases:\n  \".\u002Fassets\u002Fimages\u002Fref.png\":\n    - \".\u002Fassets\u002Fposes\u002Falign\u002Fimg_ref_video_dance.mp4\"\n```\n\nThen, simply run\n```\npython test_stage_2.py --config .\u002Fconfigs\u002Ftest_stage_2.yaml\n```\n```.\u002Fconfigs\u002Ftest_stage_2.yaml``` is the path to the inference configuration file.\n\nFinally, you can see the output results in ```.\u002Foutput\u002F```\n\n##### Reducing VRAM cost\nIf you want to reduce the VRAM cost, you could set the width and height for inference. For example,\n```\npython test_stage_2.py --config .\u002Fconfigs\u002Ftest_stage_2.yaml -W 512 -H 512\n```\nIt will generate the video at 512 x 512 first, and then resize it back to the original size of the pose video.\n\nCurrently, it takes 16GB VRAM to run on 512 x 512 x 48 and takes 28GB VRAM to run on 768 x 768 x 48. However, it should be noticed that the inference resolution would affect the final results (especially face region).\n\n#### Face Enhancement\n\nIf you want to enhance the face region to have a better consistency of the face, you could use [FaceFusion](https:\u002F\u002Fgithub.com\u002Ffacefusion\u002Ffacefusion). You could use the `face-swap` function to swap the face in the reference image to the generated video.\n\n### Training\n1. Prepare  \n    First, put all your dance videos in a folder such as `.\u002Fxxx`  \n    Next, `python extract_dwpose_keypoints.py --video_dir .\u002Fxxx`. The extracted dwpose_keypoints will be saved in `.\u002Fxxx_dwpose_keypoints`.  \n    Then, `python draw_dwpose.py --video_dir .\u002Fxxx`. The rendered dwpose videos will be saved in `.\u002Fxxx_dwpose_without_face` if `draw_face=False`. The rendered dwpose videos will be saved in `.\u002Fxxx_dwpose` if `draw_face=True`.  \n    Finally, `python extract_meta_info_multiple_dataset.py --video_dirs .\u002Fxxx --dataset_name xxx`  \n        You will get a json file to record the path of all data. `.\u002Fmeta\u002Fxxx.json` \n\n2. Config your accelerate and deepspeed  \n    `pip install accelerate`  \n    use cmd `accelerate config` to config your deepspeed according to your machine. \n    We use zero 2 without any offload and our machine has 8x80GB GPU.\n\n3. Config the yaml file for training  \n    stage 1  \n    `.\u002Fconfigs\u002Ftrain_stage_1.yaml` \n    stage 2    \n    `.\u002Fconfigs\u002Ftrain_stage_2.yaml`  \n\n4. Launch Training  \n    stage 1    \n        `accelerate launch train_stage_1_multiGPU.py --config configs\u002Ftrain_stage_1.yaml`  \n    stage 2   \n        `accelerate launch train_stage_2_multiGPU.py --config configs\u002Ftrain_stage_2.yaml`\n\n\n\n\n# Acknowledgement\n1. We thank [AnimateAnyone](https:\u002F\u002Fgithub.com\u002FHumanAIGC\u002FAnimateAnyone) for their technical report, and have refer much to [Moore-AnimateAnyone](https:\u002F\u002Fgithub.com\u002FMooreThreads\u002FMoore-AnimateAnyone) and [diffusers](https:\u002F\u002Fgithub.com\u002Fhuggingface\u002Fdiffusers).\n1. We thank open-source components like [AnimateDiff](https:\u002F\u002Fanimatediff.github.io\u002F), [dwpose](https:\u002F\u002Fgithub.com\u002FIDEA-Research\u002FDWPose), [Stable Diffusion](https:\u002F\u002Fgithub.com\u002FCompVis\u002Fstable-diffusion), etc.. \n\nThanks for open-sourcing!\n\n# Limitations\n- Detail consitency: some details of the original character are not well preserved (e.g. face region and complex clothing).\n- Noise and flickering: we observe noise and flicking in complex background. \n\n# Citation\n```bib\n@article{musepose,\n  title={MusePose: a Pose-Driven Image-to-Video Framework for Virtual Human Generation},\n  author={Tong, Zhengyan and Li, Chao and Chen, Zhaokang and Wu, Bin and Zhou, Wenjiang},\n  journal={arxiv},\n  year={2024}\n}\n```\n# Disclaimer\u002FLicense\n1. `code`: The code of MusePose is released under the MIT License. There is no limitation for both academic and commercial usage.\n1. `model`: The trained model are available for non-commercial research purposes only.\n1. `other opensource model`: Other open-source models used must comply with their license, such as `ft-mse-vae`, `dwpose`, etc..\n1. The testdata are collected from internet, which are available for non-commercial research purposes only.\n1. `AIGC`: This project strives to impact the domain of AI-driven video generation positively. Users are granted the freedom to create videos using this tool, but they are expected to comply with local laws and utilize it responsibly. The developers do not assume any responsibility for potential misuse by users.\n","MusePose 是一个基于姿态驱动的图像到视频生成框架，用于虚拟人物的创建。其核心功能包括根据给定的姿态序列生成参考图像中人物的舞蹈视频，并且通过发布的姿态对齐算法显著提升了推理性能和模型实用性。该框架采用扩散模型技术，生成质量超越了当前大多数同类开源模型。MusePose 适用于需要创建具有全身动作能力的虚拟人物场景，如虚拟主播、游戏角色等。作为 Muse 开源系列的一部分，它与 MuseV 和 MuseTalk 协同工作，旨在实现从头到尾生成具备自然交互能力的虚拟人物。",2,"2026-06-11 03:41:58","high_star"]