[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"project-72394":3},{"id":4,"name":5,"fullName":6,"owner":7,"repo":5,"description":8,"homepage":9,"htmlUrl":10,"language":11,"languages":10,"totalLinesOfCode":10,"stars":12,"forks":13,"watchers":14,"openIssues":15,"contributorsCount":16,"subscribersCount":16,"size":16,"stars1d":17,"stars7d":17,"stars30d":18,"stars90d":16,"forks30d":16,"starsTrendScore":18,"compositeScore":19,"rankGlobal":10,"rankLanguage":10,"license":20,"archived":21,"fork":21,"defaultBranch":22,"hasWiki":23,"hasPages":21,"topics":24,"createdAt":10,"pushedAt":10,"updatedAt":31,"readmeContent":32,"aiSummary":33,"trendingCount":16,"starSnapshotCount":16,"syncStatus":17,"lastSyncTime":34,"discoverSource":35},72394,"MuseV","TMElyralab\u002FMuseV","TMElyralab","MuseV: Infinite-length and High Fidelity Virtual Human Video Generation with Visual Conditioned Parallel Denoising","",null,"Python",2842,305,34,66,0,2,6,65.06,"Other",false,"main",true,[25,26,27,28,29,30],"diffusion","human-video-generation","image2video","infinite-length","musev","video-generation","2026-06-12 04:01:04","# MuseV [English](README.md) [中文](README-zh.md)\n\n\u003Cfont size=5>MuseV: Infinite-length and High Fidelity Virtual Human Video Generation with Visual Conditioned  Parallel Denoising\n\u003C\u002Fbr>\nZhiqiang Xia \u003Csup>\\*\u003C\u002Fsup>,\nZhaokang Chen\u003Csup>\\*\u003C\u002Fsup>,\nBin Wu\u003Csup>†\u003C\u002Fsup>,\nChao Li,\nKwok-Wai Hung,\nChao Zhan,\nYingjie He, \nWenjiang Zhou\n(\u003Csup>*\u003C\u002Fsup>co-first author, \u003Csup>†\u003C\u002Fsup>Corresponding Author, benbinwu@tencent.com)\n\u003C\u002Ffont>\n\nLyra Lab, Tencent Music Entertainment\n\n**[github](https:\u002F\u002Fgithub.com\u002FTMElyralab\u002FMuseV)**    **[huggingface](https:\u002F\u002Fhuggingface.co\u002FTMElyralab\u002FMuseV)**  **[HuggingfaceSpace](https:\u002F\u002Fhuggingface.co\u002Fspaces\u002FAnchorFake\u002FMuseVDemo)**  **[project](https:\u002F\u002Ftmelyralab.github.io\u002FMuseV_Page\u002F)**    **Technical report (comming soon)**\n\n\nWe have setup **the world simulator vision since March 2023, believing diffusion models can simulate the world**. `MuseV` was a milestone achieved around **July 2023**. Amazed by the progress of Sora, we decided to opensource `MuseV`, hopefully it will benefit the community. Next we will move on to the promising diffusion+transformer scheme.\n\n\nUpdate: \n\n1. We have released \u003Ca href=\"https:\u002F\u002Fgithub.com\u002FTMElyralab\u002FMuseTalk\" style=\"font-size:24px; color:red;\">MuseTalk\u003C\u002Fa>, a real-time high quality lip sync model, which can be applied with MuseV as a complete virtual human generation solution.\n2. :new: We are thrilled to announce that [MusePose](https:\u002F\u002Fgithub.com\u002FTMElyralab\u002FMusePose\u002F) has been released. MusePose is an image-to-video generation framework for virtual human under control signal like pose. Together with MuseV and MuseTalk, we hope the community can join us and march towards the vision where a virtual human can be generated end2end with native ability of full body movement and interaction.\n\n# Overview\n`MuseV` is a diffusion-based virtual human video generation framework, which \n1. supports **infinite length** generation using a novel **Visual Conditioned Parallel Denoising scheme**.\n2. checkpoint available for virtual human video generation trained on human dataset.\n3. supports Image2Video, Text2Image2Video, Video2Video.\n4. compatible with the **Stable Diffusion ecosystem**, including `base_model`, `lora`, `controlnet`, etc. \n5. supports multi reference image technology, including `IPAdapter`, `ReferenceOnly`, `ReferenceNet`, `IPAdapterFaceID`.\n6. training codes (comming very soon).\n\n# Important bug fixes\n1. `musev_referencenet_pose`: model_name of `unet`, `ip_adapter` of Command is not correct, please use `musev_referencenet_pose` instead of `musev_referencenet`.\n\n# News\n- [03\u002F27\u002F2024] release `MuseV` project and trained model `musev`, `muse_referencenet`.\n- [03\u002F30\u002F2024] add huggingface space gradio to generate video in gui\n\n## Model\n### Overview of model structure\n![model_structure](.\u002Fdata\u002Fmodels\u002Fmusev_structure.png)\n### Parallel denoising\n![parallel_denoise](.\u002Fdata\u002F\u002Fmodels\u002Fparallel_denoise.png)\n\n## Cases\nAll frames were generated directly from text2video model, without any post process.\nMoreCase is in **[project](https:\u002F\u002Ftmelyralab.github.io\u002FMuseV_Page\u002F)**, including **1-2 minute video**.\n\n\u003C!-- # TODO: \u002F\u002F use youtu video link? -->\nExamples bellow can be accessed at `configs\u002Ftasks\u002Fexample.yaml`\n\n\n### Text\u002FImage2Video\n\n#### Human\n\n\u003Ctable class=\"center\">\n  \u003Ctr style=\"font-weight: bolder;text-align:center;\">\n        \u003Ctd width=\"50%\">image\u003C\u002Ftd>\n        \u003Ctd width=\"45%\">video \u003C\u002Ftd>\n        \u003Ctd width=\"5%\">prompt\u003C\u002Ftd>\n  \u003C\u002Ftr>\n\n  \u003Ctr>\n    \u003Ctd>\n      \u003Cimg src=.\u002Fdata\u002Fimages\u002Fyongen.jpeg width=\"400\">\n    \u003C\u002Ftd>\n    \u003Ctd >\n     \u003Cvideo src=\"https:\u002F\u002Fgithub.com\u002FTMElyralab\u002FMuseV\u002Fassets\u002F163980830\u002F732cf1fd-25e7-494e-b462-969c9425d277\" width=\"100\" controls preload>\u003C\u002Fvideo>\n    \u003C\u002Ftd>\n    \u003Ctd>(masterpiece, best quality, highres:1),(1boy, solo:1),(eye blinks:1.8),(head wave:1.3)\n    \u003C\u002Ftd>\n  \u003C\u002Ftr>\n\n  \u003Ctr>\n    \u003Ctd>\n      \u003Cimg src=.\u002Fdata\u002Fimages\u002Fseaside4.jpeg width=\"400\">\n    \u003C\u002Ftd>\n    \u003Ctd>\n      \u003Cvideo src=\"https:\u002F\u002Fgithub.com\u002FTMElyralab\u002FMuseV\u002Fassets\u002F163980830\u002F9b75a46c-f4e6-45ef-ad02-05729f091c8f\" width=\"100\" controls preload>\u003C\u002Fvideo>\n    \u003C\u002Ftd>   \n    \u003Ctd>\n    (masterpiece, best quality, highres:1), peaceful beautiful sea scene\n    \u003C\u002Ftd>\n  \u003C\u002Ftr>\n  \u003Ctr>\n    \u003Ctd>\n      \u003Cimg src=.\u002Fdata\u002Fimages\u002Fseaside_girl.jpeg width=\"400\">\n    \u003C\u002Ftd>\n    \u003Ctd>\n      \u003Cvideo src=\"https:\u002F\u002Fgithub.com\u002FTMElyralab\u002FMuseV\u002Fassets\u002F163980830\u002Fd0f3b401-09bf-4018-81c3-569ec24a4de9\" width=\"100\" controls preload>\u003C\u002Fvideo>\n    \u003C\u002Ftd>\n    \u003Ctd>\n      (masterpiece, best quality, highres:1), peaceful beautiful sea scene\n    \u003C\u002Ftd>\n  \u003C\u002Ftr>\n  \u003C!-- guitar  -->\n  \u003Ctr>\n    \u003Ctd>\n      \u003Cimg src=.\u002Fdata\u002Fimages\u002Fboy_play_guitar.jpeg width=\"400\">\n    \u003C\u002Ftd>\n    \u003Ctd>\n      \u003Cvideo src=\"https:\u002F\u002Fgithub.com\u002FTMElyralab\u002FMuseV\u002Fassets\u002F163980830\u002F61bf955e-7161-44c8-a498-8811c4f4eb4f\" width=\"100\" controls preload>\u003C\u002Fvideo>\n    \u003C\u002Ftd>\n    \u003Ctd>\n       (masterpiece, best quality, highres:1), playing guitar\n    \u003C\u002Ftd>\n  \u003C\u002Ftr>\n  \u003Ctr>\n    \u003Ctd>\n      \u003Cimg src=.\u002Fdata\u002Fimages\u002Fgirl_play_guitar2.jpeg width=\"400\">\n    \u003C\u002Ftd>\n    \u003Ctd>\n      \u003Cvideo src=\"https:\u002F\u002Fgithub.com\u002FTMElyralab\u002FMuseV\u002Fassets\u002F163980830\u002F40982aa7-9f6a-4e44-8ef6-3f185d284e6a\" width=\"100\" controls preload>\u003C\u002Fvideo>\n    \u003C\u002Ftd>\n    \u003Ctd>\n      (masterpiece, best quality, highres:1), playing guitar\n    \u003C\u002Ftd>\n  \u003C\u002Ftr>\n  \u003C!-- famous people -->\n  \u003Ctr>\n    \u003Ctd>\n      \u003Cimg src=.\u002Fdata\u002Fimages\u002Fdufu.jpeg width=\"400\">\n    \u003C\u002Ftd>\n    \u003Ctd>\n      \u003Cvideo src=\"https:\u002F\u002Fgithub.com\u002FTMElyralab\u002FMuseV\u002Fassets\u002F163980830\u002F28294baa-b996-420f-b1fb-046542adf87d\" width=\"100\" controls preload>\u003C\u002Fvideo>\n    \u003C\u002Ftd>\n    \u003Ctd>\n    (masterpiece, best quality, highres:1),(1man, solo:1),(eye blinks:1.8),(head wave:1.3),Chinese ink painting style\n    \u003C\u002Ftd>\n  \u003C\u002Ftr>\n\n  \u003Ctr>\n    \u003Ctd>\n      \u003Cimg src=.\u002Fdata\u002Fimages\u002FMona_Lisa.jpg width=\"400\">\n    \u003C\u002Ftd>\n    \u003Ctd>\n      \u003Cvideo src=\"https:\u002F\u002Fgithub.com\u002FTMElyralab\u002FMuseV\u002Fassets\u002F163980830\u002F1ce11da6-14c6-4dcd-b7f9-7a5f060d71fb\" width=\"100\" controls preload>\u003C\u002Fvideo>\n    \u003C\u002Ftd>   \n    \u003Ctd>\n      (masterpiece, best quality, highres:1),(1girl, solo:1),(beautiful face,\n    soft skin, costume:1),(eye blinks:{eye_blinks_factor}),(head wave:1.3)\n    \u003C\u002Ftd>\n  \u003C\u002Ftr>\n\u003C\u002Ftable >\n\n#### Scene\n\n\u003Ctable class=\"center\">\n    \u003Ctr style=\"font-weight: bolder;text-align:center;\">\n        \u003Ctd width=\"35%\">image\u003C\u002Ftd>\n        \u003Ctd width=\"50%\">video\u003C\u002Ftd>\n        \u003Ctd width=\"15%\">prompt\u003C\u002Ftd>\n    \u003C\u002Ftr>\n\n  \u003Ctr>\n    \u003Ctd>\n      \u003Cimg src=.\u002Fdata\u002Fimages\u002Fwaterfall4.jpeg width=\"400\">\n    \u003C\u002Ftd>\n    \u003Ctd>\n      \u003Cvideo src=\"https:\u002F\u002Fgithub.com\u002FTMElyralab\u002FMuseV\u002Fassets\u002F163980830\u002F852daeb6-6b58-4931-81f9-0dddfa1b4ea5\" width=\"100\" controls preload>\u003C\u002Fvideo>\n    \u003C\u002Ftd>\n    \u003Ctd>\n      (masterpiece, best quality, highres:1), peaceful beautiful waterfall, an\n    endless waterfall\n    \u003C\u002Ftd>\n  \u003C\u002Ftr>\n\n  \u003Ctr>\n    \u003Ctd>\n      \u003Cimg src=.\u002Fdata\u002Fimages\u002Fseaside2.jpeg width=\"400\">\n    \u003C\u002Ftd>\n    \u003Ctd>\n      \u003Cvideo src=\"https:\u002F\u002Fgithub.com\u002FTMElyralab\u002FMuseV\u002Fassets\u002F163980830\u002F4a4d527a-6203-411f-afe9-31c992d26816\" width=\"100\" controls preload>\u003C\u002Fvideo>\n    \u003C\u002Ftd>\n    \u003Ctd>(masterpiece, best quality, highres:1), peaceful beautiful sea scene\n    \u003C\u002Ftd>\n  \u003C\u002Ftr>\n\u003C\u002Ftable >\n\n### VideoMiddle2Video\n\n**pose2video**\nIn `duffy` mode, pose of the vision condition frame is not aligned with the first frame of control video. `posealign` will solve the problem.\n\n\u003Ctable class=\"center\">\n    \u003Ctr style=\"font-weight: bolder;text-align:center;\">\n        \u003Ctd width=\"25%\">image\u003C\u002Ftd>\n        \u003Ctd width=\"65%\">video\u003C\u002Ftd>\n        \u003Ctd width=\"10%\">prompt\u003C\u002Ftd>\n    \u003C\u002Ftr>\n  \u003Ctr>\n    \u003Ctd>\n      \u003Cimg src=.\u002Fdata\u002Fimages\u002Fspark_girl.png width=\"200\">\n      \u003Cimg src=.\u002Fdata\u002Fimages\u002Fcyber_girl.png width=\"200\">\n    \u003C\u002Ftd>\n    \u003Ctd>\n        \u003Cvideo src=\"https:\u002F\u002Fgithub.com\u002FTMElyralab\u002FMuseV\u002Fassets\u002F163980830\u002F484cc69d-c316-4464-a55b-3df929780a8e\" width=\"400\" controls preload>\u003C\u002Fvideo>\n    \u003C\u002Ftd>\n    \u003Ctd>\n      (masterpiece, best quality, highres:1) , a girl is dancing, animation\n    \u003C\u002Ftd>\n  \u003C\u002Ftr>\n  \u003Ctr>   \n    \u003Ctd>\n      \u003Cimg src=.\u002Fdata\u002Fimages\u002Fduffy.png width=\"400\">\n    \u003C\u002Ftd>\n    \u003Ctd>\n      \u003Cvideo src=\"https:\u002F\u002Fgithub.com\u002FTMElyralab\u002FMuseV\u002Fassets\u002F163980830\u002Fc44682e6-aafc-4730-8fc1-72825c1bacf2\" width=\"400\" controls preload>\u003C\u002Fvideo>\n    \u003C\u002Ftd>\n    \u003Ctd>\n      (masterpiece, best quality, highres:1), is dancing, animation\n    \u003C\u002Ftd>\n  \u003C\u002Ftr>\n\u003C\u002Ftable >\n\n### MuseTalk\nThe character of talk, `Sun Xinying` is a supermodel KOL. You can follow her on [douyin](https:\u002F\u002Fwww.douyin.com\u002Fuser\u002FMS4wLjABAAAAWDThbMPN_6Xmm_JgXexbOii1K-httbu2APdG8DvDyM8).\n\n\u003Ctable class=\"center\">\n    \u003Ctr style=\"font-weight: bolder;\">\n        \u003Ctd width=\"35%\">name\u003C\u002Ftd>\n        \u003Ctd width=\"50%\">video\u003C\u002Ftd>\n    \u003C\u002Ftr>\n\n  \u003Ctr>\n    \u003Ctd>\n       talk\n    \u003C\u002Ftd>\n    \u003Ctd>\n      \u003Cvideo src=\"https:\u002F\u002Fgithub.com\u002FTMElyralab\u002FMuseV\u002Fassets\u002F163980830\u002F951188d1-4731-4e7f-bf40-03cacba17f2f\" width=\"100\" controls preload>\u003C\u002Fvideo>\n    \u003C\u002Ftd>\n  \u003Ctr>\n    \u003Ctd>\n       sing\n    \u003C\u002Ftd>\n    \u003Ctd>\n      \u003Cvideo src=\"https:\u002F\u002Fgithub.com\u002FTMElyralab\u002FMuseV\u002Fassets\u002F163980830\u002F50b8ffab-9307-4836-99e5-947e6ce7d112\" width=\"100\" controls preload>\u003C\u002Fvideo>\n    \u003C\u002Ftd>\n  \u003C\u002Ftr>\n\u003C\u002Ftable >\n\n\n# TODO:\n- [ ] technical report (comming soon).\n- [ ] training codes.\n- [ ] release pretrained unet model, which is trained with controlnet、referencenet、IPAdapter, which is better on pose2video.\n- [ ] support diffusion transformer generation framework.\n- [ ] release `posealign` module\n\n# Quickstart\nPrepare python environment and install extra package like `diffusers`, `controlnet_aux`, `mmcm`.\n\n## Third party integration\nThanks for the third-party integration, which makes installation and use more convenient for everyone.\nWe also hope you note that we have not verified, maintained, or updated third-party. Please refer to this project for specific results.\n\n### [ComfyUI](https:\u002F\u002Fgithub.com\u002Fchaojie\u002FComfyUI-MuseV)\n### [One click integration package in windows](https:\u002F\u002Fwww.bilibili.com\u002Fvideo\u002FBV1ux4y1v7pF\u002F?vd_source=fe03b064abab17b79e22a692551405c3)\nnetdisk:https:\u002F\u002Fwww.123pan.com\u002Fs\u002FPf5Yjv-Bb9W3.html\n\ncode: glut\n\n## Prepare environment\nYou are recommended to use `docker` primarily to prepare python environment.\n### prepare python env\n**Attention**: we only test with docker, there are maybe trouble with conda, or requirement. We will try to fix it. Use `docker` Please.\n\n#### Method 1: docker\n1. pull docker image\n```bash\ndocker pull anchorxia\u002Fmusev:latest\n```\n2. run docker\n```bash\ndocker run --gpus all -it --entrypoint \u002Fbin\u002Fbash anchorxia\u002Fmusev:latest\n```\nThe default conda env is `musev`.\n\n#### Method 2: conda \ncreate conda environment from environment.yaml\n```\nconda env create --name musev --file .\u002Fenvironment.yml\n```\n#### Method 3: pip requirements\n```\npip install -r requirements.txt\n```\n#### Prepare mmlab package\nif not use docker, should install mmlab package additionally.\n```bash\npip install --no-cache-dir -U openmim \nmim install mmengine \nmim install \"mmcv>=2.0.1\" \nmim install \"mmdet>=3.1.0\" \nmim install \"mmpose>=1.1.0\" \n```\n\n### Prepare custom package \u002F modified package\n#### clone\n```bash\ngit clone --recursive https:\u002F\u002Fgithub.com\u002FTMElyralab\u002FMuseV.git\n```\n#### prepare PYTHONPATH\n```bash\ncurrent_dir=$(pwd)\nexport PYTHONPATH=${PYTHONPATH}:${current_dir}\u002FMuseV\nexport PYTHONPATH=${PYTHONPATH}:${current_dir}\u002FMuseV\u002FMMCM\nexport PYTHONPATH=${PYTHONPATH}:${current_dir}\u002FMuseV\u002Fdiffusers\u002Fsrc\nexport PYTHONPATH=${PYTHONPATH}:${current_dir}\u002FMuseV\u002Fcontrolnet_aux\u002Fsrc\ncd MuseV\n```\n\n1. `MMCM`: multi media, cross modal process package。\n1. `diffusers`: modified diffusers package based on [diffusers](https:\u002F\u002Fgithub.com\u002Fhuggingface\u002Fdiffusers)\n1. `controlnet_aux`: modified based on [controlnet_aux](https:\u002F\u002Fgithub.com\u002FTMElyralab\u002Fcontrolnet_aux)\n\n\n## Download models\n```bash\ngit clone https:\u002F\u002Fhuggingface.co\u002FTMElyralab\u002FMuseV .\u002Fcheckpoints\n```\n- `motion`: text2video model, trained on tiny `ucf101` and tiny `webvid` dataset, approximately 60K videos text pairs. GPU memory consumption  testing on `resolution`$=512*512$, `time_size=12`.\n    - `musev\u002Funet`: only has and train `unet` motion module. `GPU memory consumption` $\\approx 8G$. \n    - `musev_referencenet`: train `unet` module, `referencenet`, `IPAdapter`. `GPU memory consumption` $\\approx 12G$. \n        - `unet`: `motion` module, which has `to_k`, `to_v` in `Attention` layer refer to `IPAdapter`\n        - `referencenet`: similar to `AnimateAnyone`\n        - `ip_adapter_image_proj.bin`: images clip emb project layer, refer to `IPAdapter`\n    - `musev_referencenet_pose`: based on `musev_referencenet`, fix `referencenet`and `controlnet_pose`, train `unet motion` and `IPAdapter`. `GPU memory consumption` $\\approx 12G$\n- `t2i\u002Fsd1.5`: text2image model, parameter are frozen when training motion module. Different `t2i` base_model has a significant impact.could be replaced with other t2i base.\n    - `majicmixRealv6Fp16`: example, download from [majicmixRealv6Fp16](https:\u002F\u002Fcivitai.com\u002Fmodels\u002F43331?modelVersionId=94640)\n    - `fantasticmix_v10`: example, download from [fantasticmix_v10](https:\u002F\u002Fcivitai.com\u002Fmodels\u002F22402?modelVersionId=26744)\n- `IP-Adapter\u002Fmodels`: download from [IPAdapter](https:\u002F\u002Fhuggingface.co\u002Fh94\u002FIP-Adapter\u002Ftree\u002Fmain)\n    - `image_encoder`: vision clip model.\n    - `ip-adapter_sd15.bin`: original IPAdapter model checkpoint.\n    - `ip-adapter-faceid_sd15.bin`: original IPAdapter model checkpoint.\n\n## Inference\n\n### Prepare model_path\nSkip this step when run example task with example inference command.\nSet model path and abbreviation in config, to use abbreviation in inference script.\n- T2I SD：ref to `musev\u002Fconfigs\u002Fmodel\u002FT2I_all_model.py`\n- Motion Unet: refer to `musev\u002Fconfigs\u002Fmodel\u002Fmotion_model.py`\n- Task: refer to `musev\u002Fconfigs\u002Ftasks\u002Fexample.yaml`\n\n### musev_referencenet\n#### text2video\n```bash\npython scripts\u002Finference\u002Ftext2video.py   --sd_model_name majicmixRealv6Fp16   --unet_model_name musev_referencenet --referencenet_model_name musev_referencenet --ip_adapter_model_name musev_referencenet   -test_data_path .\u002Fconfigs\u002Ftasks\u002Fexample.yaml  --output_dir .\u002Foutput  --n_batch 1  --target_datas yongen  --vision_clip_extractor_class_name ImageClipVisionFeatureExtractor --vision_clip_model_path .\u002Fcheckpoints\u002FIP-Adapter\u002Fmodels\u002Fimage_encoder  --time_size 12 --fps 12  \n```\n**common parameters**:\n- `test_data_path`: task_path in yaml extention\n- `target_datas`: sep is `,`, sample subtasks if `name` in `test_data_path` is in `target_datas`.\n- `sd_model_cfg_path`: T2I sd models path, model config path or model path.\n- `sd_model_name`: sd model name, which use to choose full model path in sd_model_cfg_path. multi model names with sep =`,`, or `all`\n- `unet_model_cfg_path`: motion unet model config path or model path。\n- `unet_model_name`: unet model name, use to get model path in `unet_model_cfg_path`, and init unet class instance in `musev\u002Fmodels\u002Funet_loader.py`. multi model names with sep=`,`, or `all`. If `unet_model_cfg_path` is model path, `unet_name` must be supported in `musev\u002Fmodels\u002Funet_loader.py`\n- `time_size`: num_frames per diffusion denoise generation。default=`12`.\n- `n_batch`: generation numbers of shot, $total\\_frames=n\\_batch * time\\_size + n\\_viscond$, default=`1`。\n- `context_frames`: context_frames num. If `time_size` > `context_frame`，`time_size` window is split into many sub-windows for parallel denoising\"。 default=`12`。\n\n**To generate long videos**, there two ways:\n1. `visual conditioned parallel denoise`: set `n_batch=1`, `time_size` = all frames you want.\n1. `traditional end-to-end`: set `time_size` = `context_frames` = frames of a shot (`12`), `context_overlap` = 0；\n\n\n**model parameters**：\nsupports `referencenet`, `IPAdapter`, `IPAdapterFaceID`, `Facein`.\n- referencenet_model_name: `referencenet` model name.\n- ImageClipVisionFeatureExtractor: `ImageEmbExtractor` name, extractor vision clip emb used in `IPAdapter`.\n- vision_clip_model_path: `ImageClipVisionFeatureExtractor` model path.\n- ip_adapter_model_name: from `IPAdapter`, it's `ImagePromptEmbProj`, used with `ImageEmbExtractor`。\n- ip_adapter_face_model_name: `IPAdapterFaceID`, from `IPAdapter` to keep faceid，should set `face_image_path`。\n\n**Some parameters that affect the motion range and generation results**：\n- `video_guidance_scale`: Similar to text2image, control influence between cond and uncond，default=`3.5`\n- `use_condition_image`:  Whether to use the given first frame for video generation, if not generate vision condition frames first. Default=`True`.\n- `redraw_condition_image`: Whether to redraw the given first frame image.\n- `video_negative_prompt`: Abbreviation of full `negative_prompt` in config path. default=`V2`.\n\n\n#### video2video\n`t2i` base_model has a significant impact. In this case, `fantasticmix_v10` performs better than `majicmixRealv6Fp16`.\n```bash\npython scripts\u002Finference\u002Fvideo2video.py --sd_model_name fantasticmix_v10  --unet_model_name musev_referencenet --referencenet_model_name   musev_referencenet --ip_adapter_model_name musev_referencenet    -test_data_path .\u002Fconfigs\u002Ftasks\u002Fexample.yaml    --vision_clip_extractor_class_name ImageClipVisionFeatureExtractor --vision_clip_model_path .\u002Fcheckpoints\u002FIP-Adapter\u002Fmodels\u002Fimage_encoder      --output_dir .\u002Foutput  --n_batch 1 --controlnet_name dwpose_body_hand  --which2video \"video_middle\"  --target_datas dance1 --fps 12 --time_size 12\n```\n**import parameters**\n\nMost of the parameters are same as `musev_text2video`. Special parameters of `video2video` are:\n1. need to set `video_path` as reference video in `test_data`. Now reference video supports `rgb video` and `controlnet_middle_video`。\n- `which2video`: whether `rgb` video influences initial noise, influence of `rgb` is stronger than of  controlnet condition.\n- `controlnet_name`：whether to use `controlnet condition`, such as `dwpose,depth`.\n- `video_is_middle`: `video_path` is `rgb video` or  `controlnet_middle_video`. Can be set for every `test_data` in test_data_path.\n- `video_has_condition`: whether condtion_images is aligned with the first frame of video_path. If Not, exrtact condition of `condition_images` firstly generate, and then align with concatation. set in `test_data`。\n\nall controlnet_names refer to [mmcm](https:\u002F\u002Fgithub.com\u002FTMElyralab\u002FMMCM\u002Fblob\u002Fmain\u002Fmmcm\u002Fvision\u002Ffeature_extractor\u002Fcontrolnet.py#L513)\n```python\n['pose', 'pose_body', 'pose_hand', 'pose_face', 'pose_hand_body', 'pose_hand_face', 'dwpose', 'dwpose_face', 'dwpose_hand', 'dwpose_body', 'dwpose_body_hand', 'canny', 'tile', 'hed', 'hed_scribble', 'depth', 'pidi', 'normal_bae', 'lineart', 'lineart_anime', 'zoe', 'sam', 'mobile_sam', 'leres', 'content', 'face_detector']\n```\n\n### musev_referencenet_pose\nOnly used for `pose2video`\ntrain based on `musev_referencenet`, fix `referencenet`, `pose-controlnet`, and `T2I`, train `motion` module and `IPAdapter`.\n\n`t2i` base_model has a significant impact. In this case, `fantasticmix_v10` performs better than `majicmixRealv6Fp16`.\n\n```bash\npython scripts\u002Finference\u002Fvideo2video.py --sd_model_name fantasticmix_v10  --unet_model_name musev_referencenet_pose --referencenet_model_name   musev_referencenet --ip_adapter_model_name musev_referencenet_pose    -test_data_path .\u002Fconfigs\u002Ftasks\u002Fexample.yaml    --vision_clip_extractor_class_name ImageClipVisionFeatureExtractor --vision_clip_model_path .\u002Fcheckpoints\u002FIP-Adapter\u002Fmodels\u002Fimage_encoder      --output_dir .\u002Foutput  --n_batch 1 --controlnet_name dwpose_body_hand  --which2video \"video_middle\"  --target_datas  dance1   --fps 12 --time_size 12\n```\n\n### musev\nOnly has motion module, no referencenet, requiring less gpu memory.\n#### text2video\n```bash\npython scripts\u002Finference\u002Ftext2video.py   --sd_model_name majicmixRealv6Fp16   --unet_model_name musev   -test_data_path .\u002Fconfigs\u002Ftasks\u002Fexample.yaml  --output_dir .\u002Foutput  --n_batch 1  --target_datas yongen  --time_size 12 --fps 12\n```\n#### video2video\n```bash\npython scripts\u002Finference\u002Fvideo2video.py --sd_model_name fantasticmix_v10  --unet_model_name musev    -test_data_path .\u002Fconfigs\u002Ftasks\u002Fexample.yaml --output_dir .\u002Foutput  --n_batch 1 --controlnet_name dwpose_body_hand  --which2video \"video_middle\"  --target_datas  dance1   --fps 12 --time_size 12\n```\n\n### Gradio demo\nMuseV provides gradio script to generate a GUI in a local machine to generate video conveniently. \n\n```bash\ncd scripts\u002Fgradio\npython app.py\n```\n\n\n# Acknowledgements\n\n1. MuseV has referred much to [TuneAVideo](https:\u002F\u002Fgithub.com\u002Fshowlab\u002FTune-A-Video), [diffusers](https:\u002F\u002Fgithub.com\u002Fhuggingface\u002Fdiffusers), [Moore-AnimateAnyone](https:\u002F\u002Fgithub.com\u002FMooreThreads\u002FMoore-AnimateAnyone\u002Ftree\u002Fmaster\u002Fsrc\u002Fpipelines), [animatediff](https:\u002F\u002Fgithub.com\u002Fguoyww\u002FAnimateDiff), [IP-Adapter](https:\u002F\u002Fgithub.com\u002Ftencent-ailab\u002FIP-Adapter), [AnimateAnyone](https:\u002F\u002Farxiv.org\u002Fabs\u002F2311.17117), [VideoFusion](https:\u002F\u002Farxiv.org\u002Fabs\u002F2303.08320), [insightface](https:\u002F\u002Fgithub.com\u002Fdeepinsight\u002Finsightface). \n2. MuseV has been built on `ucf101` and `webvid` datasets.\n\nThanks for open-sourcing!\n\n# Limitation\nThere are still many limitations, including\n\n1. Lack of generalization ability. Some visual condition image perform well, some perform bad. Some t2i pretraied model perform well, some perform bad.\n1. Limited types of video generation and limited motion range, partly because of limited types of training data. The released `MuseV` has been trained on approximately 60K human text-video pairs with resolution `512*320`. `MuseV` has greater motion range while lower video quality at lower resolution. `MuseV` tends to generate less motion range with high video quality. Trained on larger, higher resolution, higher quality text-video dataset may make `MuseV` better.\n1. Watermarks may appear because of `webvid`. A cleaner dataset without watermarks may solve this issue.\n1. Limited types of long video generation. Visual Conditioned Parallel Denoise can solve accumulated error of video generation, but the current method is only suitable for relatively fixed camera scenes.\n1. Undertrained referencenet and IP-Adapter, beacause of limited time and limited resources.\n1. Understructured code. `MuseV`  supports rich and dynamic features, but with complex and unrefacted codes. It takes time to familiarize.\n   \n\n\u003C!-- # Contribution 暂时不需要组织开源共建 -->\n# Citation\n```bib\n@article{musev,\n  title={MuseV: Infinite-length and High Fidelity Virtual Human Video Generation with Visual Conditioned Parallel Denoising},\n  author={Xia, Zhiqiang and Chen, Zhaokang and Wu, Bin and Li, Chao and Hung, Kwok-Wai and Zhan, Chao and He, Yingjie and Zhou, Wenjiang},\n  journal={arxiv},\n  year={2024}\n}\n```\n# Disclaimer\u002FLicense\n1. `code`: The code of MuseV is released under the MIT License. There is no limitation for both academic and commercial usage.\n1. `model`: The trained model are available for non-commercial research purposes only.\n1. `other opensource model`: Other open-source models used must comply with their license, such as `insightface`, `IP-Adapter`, `ft-mse-vae`, etc.\n1. The testdata are collected from internet, which are available for non-commercial research purposes only.\n1. `AIGC`: This project strives to impact the domain of AI-driven video generation positively. Users are granted the freedom to create videos using this tool, but they are expected to comply with local laws and utilize it responsibly. The developers do not assume any responsibility for potential misuse by users.\n","MuseV是一个基于扩散模型的虚拟人视频生成框架，能够生成无限长度且高保真的虚拟人视频。它采用了一种新颖的视觉条件并行去噪方案，支持从图像到视频、文本到图像再到视频以及视频到视频的转换，并兼容Stable Diffusion生态系统中的多种组件，如基础模型、LoRA和ControlNet等。此外，MuseV还引入了多参考图像技术，增强了生成内容的多样性和准确性。该项目非常适合需要创建高质量、长时长虚拟人物视频的应用场景，比如数字娱乐、在线教育或虚拟现实等领域。","2026-06-11 03:41:53","high_star"]