[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"project-71981":3},{"id":4,"name":5,"fullName":6,"owner":7,"repo":5,"description":8,"homepage":9,"htmlUrl":10,"language":11,"languages":10,"totalLinesOfCode":10,"stars":12,"forks":13,"watchers":14,"openIssues":15,"contributorsCount":16,"subscribersCount":16,"size":16,"stars1d":17,"stars7d":17,"stars30d":18,"stars90d":16,"forks30d":16,"starsTrendScore":18,"compositeScore":19,"rankGlobal":10,"rankLanguage":10,"license":20,"archived":21,"fork":21,"defaultBranch":22,"hasWiki":23,"hasPages":23,"topics":24,"createdAt":10,"pushedAt":10,"updatedAt":28,"readmeContent":29,"aiSummary":30,"trendingCount":16,"starSnapshotCount":16,"syncStatus":17,"lastSyncTime":31,"discoverSource":32},71981,"hallo","fudan-generative-vision\u002Fhallo","fudan-generative-vision","Hallo: Hierarchical Audio-Driven Visual Synthesis for Portrait Image Animation","https:\u002F\u002Ffudan-generative-vision.github.io\u002Fhallo\u002F",null,"Python",8646,1119,449,114,0,2,6,40.15,"MIT License",false,"main",true,[25,26,27],"face-animation","image-animation","video-animation","2026-06-12 02:02:56","\u003Ch1 align='center'>Hallo: Hierarchical Audio-Driven Visual Synthesis for Portrait Image Animation\u003C\u002Fh1>\n\n\u003Cdiv align='center'>\n    \u003Ca href='https:\u002F\u002Fgithub.com\u002Fxumingw' target='_blank'>Mingwang Xu\u003C\u002Fa>\u003Csup>1*\u003C\u002Fsup> \n    \u003Ca href='https:\u002F\u002Fgithub.com\u002Fcrystallee-ai' target='_blank'>Hui Li\u003C\u002Fa>\u003Csup>1*\u003C\u002Fsup> \n    \u003Ca href='https:\u002F\u002Fgithub.com\u002Fsubazinga' target='_blank'>Qingkun Su\u003C\u002Fa>\u003Csup>1*\u003C\u002Fsup> \n    \u003Ca href='https:\u002F\u002Fgithub.com\u002FNinoNeumann' target='_blank'>Hanlin Shang\u003C\u002Fa>\u003Csup>1\u003C\u002Fsup> \n    \u003Ca href='https:\u002F\u002Fgithub.com\u002FAricGamma' target='_blank'>Liwei Zhang\u003C\u002Fa>\u003Csup>1\u003C\u002Fsup> \n    \u003Ca href='https:\u002F\u002Fgithub.com\u002Fcnexah' target='_blank'>Ce Liu\u003C\u002Fa>\u003Csup>3\u003C\u002Fsup> \n\u003C\u002Fdiv>\n\u003Cdiv align='center'>\n    \u003Ca href='https:\u002F\u002Fjingdongwang2017.github.io\u002F' target='_blank'>Jingdong Wang\u003C\u002Fa>\u003Csup>2\u003C\u002Fsup> \n    \u003Ca href='https:\u002F\u002Fyoyo000.github.io\u002F' target='_blank'>Yao Yao\u003C\u002Fa>\u003Csup>4\u003C\u002Fsup> \n    \u003Ca href='https:\u002F\u002Fsites.google.com\u002Fsite\u002Fzhusiyucs\u002Fhome' target='_blank'>Siyu Zhu\u003C\u002Fa>\u003Csup>1\u003C\u002Fsup> \n\u003C\u002Fdiv>\n\n\u003Cdiv align='center'>\n    \u003Csup>1\u003C\u002Fsup>Fudan University  \u003Csup>2\u003C\u002Fsup>Baidu Inc  \u003Csup>3\u003C\u002Fsup>ETH Zurich  \u003Csup>4\u003C\u002Fsup>Nanjing University\n\u003C\u002Fdiv>\n\n\u003Cbr>\n\u003Cdiv align='center'>\n    \u003Ca href='https:\u002F\u002Fgithub.com\u002Ffudan-generative-vision\u002Fhallo'>\u003Cimg src='https:\u002F\u002Fimg.shields.io\u002Fgithub\u002Fstars\u002Ffudan-generative-vision\u002Fhallo?style=social'>\u003C\u002Fa>\n    \u003Ca href='https:\u002F\u002Ffudan-generative-vision.github.io\u002Fhallo\u002F#\u002F'>\u003Cimg src='https:\u002F\u002Fimg.shields.io\u002Fbadge\u002FProject-HomePage-Green'>\u003C\u002Fa>\n    \u003Ca href='https:\u002F\u002Farxiv.org\u002Fpdf\u002F2406.08801'>\u003Cimg src='https:\u002F\u002Fimg.shields.io\u002Fbadge\u002FPaper-Arxiv-red'>\u003C\u002Fa>\n    \u003Ca href='https:\u002F\u002Fhuggingface.co\u002Ffudan-generative-ai\u002Fhallo'>\u003Cimg src='https:\u002F\u002Fimg.shields.io\u002Fbadge\u002F%F0%9F%A4%97%20HuggingFace-Model-yellow'>\u003C\u002Fa>\n    \u003Ca href='https:\u002F\u002Fhuggingface.co\u002Fspaces\u002Ffffiloni\u002Ftts-hallo-talking-portrait'>\u003Cimg src='https:\u002F\u002Fimg.shields.io\u002Fbadge\u002F%F0%9F%A4%97%20HuggingFace-Demo-yellow'>\u003C\u002Fa>\n    \u003Ca href='https:\u002F\u002Fwww.modelscope.cn\u002Fmodels\u002Ffudan-generative-vision\u002FHallo\u002Fsummary'>\u003Cimg src='https:\u002F\u002Fimg.shields.io\u002Fbadge\u002FModelscope-Model-purple'>\u003C\u002Fa>\n    \u003Ca href='assets\u002Fwechat.jpeg'>\u003Cimg src='https:\u002F\u002Fbadges.aleen42.com\u002Fsrc\u002Fwechat.svg'>\u003C\u002Fa>\n\u003C\u002Fdiv>\n\n\u003Cbr>\n\n## 📸 Showcase\n\n\nhttps:\u002F\u002Fgithub.com\u002Ffudan-generative-vision\u002Fhallo\u002Fassets\u002F17402682\u002F9d1a0de4-3470-4d38-9e4f-412f517f834c\n\n### 🎬 Honoring Classic Films\n\n\u003Ctable class=\"center\">\n  \u003Ctr>\n    \u003Ctd style=\"text-align: center\">\u003Cb>Devil Wears Prada\u003C\u002Fb>\u003C\u002Ftd>\n    \u003Ctd style=\"text-align: center\">\u003Cb>Green Book\u003C\u002Fb>\u003C\u002Ftd>\n    \u003Ctd style=\"text-align: center\">\u003Cb>Infernal Affairs\u003C\u002Fb>\u003C\u002Ftd>\n  \u003C\u002Ftr>\n  \u003Ctr>\n    \u003Ctd style=\"text-align: center\">\u003Ca target=\"_blank\" href=\"https:\u002F\u002Fcdn.aondata.work\u002Fvideo\u002Fshort_movie\u002FDevil_Wears_Prada-480p.mp4\">\u003Cimg src=\"https:\u002F\u002Fcdn.aondata.work\u002Fimg\u002Fshort_movie\u002FDevil_Wears_Prada_GIF.gif\">\u003C\u002Fa>\u003C\u002Ftd>\n    \u003Ctd style=\"text-align: center\">\u003Ca target=\"_blank\" href=\"https:\u002F\u002Fcdn.aondata.work\u002Fvideo\u002Fshort_movie\u002FGreen_Book-480p.mp4\">\u003Cimg src=\"https:\u002F\u002Fcdn.aondata.work\u002Fimg\u002Fshort_movie\u002FGreen_Book_GIF.gif\">\u003C\u002Fa>\u003C\u002Ftd>\n    \u003Ctd style=\"text-align: center\">\u003Ca target=\"_blank\" href=\"https:\u002F\u002Fcdn.aondata.work\u002Fvideo\u002Fshort_movie\u002F无间道-480p.mp4\">\u003Cimg src=\"https:\u002F\u002Fcdn.aondata.work\u002Fimg\u002Fshort_movie\u002FInfernal_Affairs_GIF.gif\">\u003C\u002Fa>\u003C\u002Ftd>\n  \u003C\u002Ftr>\n  \u003Ctr>\n    \u003Ctd style=\"text-align: center\">\u003Cb>Patch Adams\u003C\u002Fb>\u003C\u002Ftd>\n    \u003Ctd style=\"text-align: center\">\u003Cb>Tough Love\u003C\u002Fb>\u003C\u002Ftd>\n    \u003Ctd style=\"text-align: center\">\u003Cb>Shawshank Redemption\u003C\u002Fb>\u003C\u002Ftd>\n  \u003C\u002Ftr>\n  \u003Ctr>\n    \u003Ctd style=\"text-align: center\">\u003Ca target=\"_blank\" href=\"https:\u002F\u002Fcdn.aondata.work\u002Fvideo\u002Fshort_movie\u002FPatch_Adams-480p.mp4\">\u003Cimg src=\"https:\u002F\u002Fcdn.aondata.work\u002Fimg\u002Fshort_movie\u002FPatch_Adams_GIF.gif\">\u003C\u002Fa>\u003C\u002Ftd>\n    \u003Ctd style=\"text-align: center\">\u003Ca target=\"_blank\" href=\"https:\u002F\u002Fcdn.aondata.work\u002Fvideo\u002Fshort_movie\u002FTough_Love-480p.mp4\">\u003Cimg src=\"https:\u002F\u002Fcdn.aondata.work\u002Fimg\u002Fshort_movie\u002FTough_Love_GIF.gif\">\u003C\u002Fa>\u003C\u002Ftd>\n    \u003Ctd style=\"text-align: center\">\u003Ca target=\"_blank\" href=\"https:\u002F\u002Fcdn.aondata.work\u002Fvideo\u002Fshort_movie\u002FShawshank-480p.mp4\">\u003Cimg src=\"https:\u002F\u002Fcdn.aondata.work\u002Fimg\u002Fshort_movie\u002FShawshank_GIF.gif\">\u003C\u002Fa>\u003C\u002Ftd>\n  \u003C\u002Ftr>\n\u003C\u002Ftable>\n\nExplore [more examples](https:\u002F\u002Ffudan-generative-vision.github.io\u002Fhallo).\n\n## 📰 News\n\n- **`2024\u002F06\u002F28`**: 🎉🎉🎉 We are proud to announce the release of our model training code. Try your own training data. Here is [tutorial](#training).\n- **`2024\u002F06\u002F21`**: 🚀🚀🚀 Cloned a Gradio demo on [🤗Huggingface space](https:\u002F\u002Fhuggingface.co\u002Fspaces\u002Ffudan-generative-ai\u002Fhallo).\n- **`2024\u002F06\u002F20`**: 🌟🌟🌟 Received numerous contributions from the community, including a [Windows version](https:\u002F\u002Fgithub.com\u002Fsdbds\u002Fhallo-for-windows), [ComfyUI](https:\u002F\u002Fgithub.com\u002FAIFSH\u002FComfyUI-Hallo), [WebUI](https:\u002F\u002Fgithub.com\u002Ffudan-generative-vision\u002Fhallo\u002Fpull\u002F51), and [Docker template](https:\u002F\u002Fgithub.com\u002Fashleykleynhans\u002Fhallo-docker).\n- **`2024\u002F06\u002F15`**: ✨✨✨ Released some images and audios for inference testing on [🤗Huggingface](https:\u002F\u002Fhuggingface.co\u002Fdatasets\u002Ffudan-generative-ai\u002Fhallo_inference_samples).\n- **`2024\u002F06\u002F15`**: 🎉🎉🎉 Launched the first version on 🫡[GitHub](https:\u002F\u002Fgithub.com\u002Ffudan-generative-vision\u002Fhallo).\n\n## 🤝 Community Resources\n\nExplore the resources developed by our community to enhance your experience with Hallo:\n\n- [TTS x Hallo Talking Portrait Generator](https:\u002F\u002Fhuggingface.co\u002Fspaces\u002Ffffiloni\u002Ftts-hallo-talking-portrait) - Check out this awesome Gradio demo by [@Sylvain Filoni](https:\u002F\u002Fhuggingface.co\u002Ffffiloni)! With this tool, you can conveniently prepare portrait image and audio for Hallo.\n- [Demo on Huggingface](https:\u002F\u002Fhuggingface.co\u002Fspaces\u002Fmultimodalart\u002Fhallo) - Check out this easy-to-use Gradio demo by [@multimodalart](https:\u002F\u002Fhuggingface.co\u002Fmultimodalart).\n- [hallo-webui](https:\u002F\u002Fgithub.com\u002Fdaswer123\u002Fhallo-webui) - Explore the WebUI created by [@daswer123](https:\u002F\u002Fgithub.com\u002Fdaswer123).\n- [hallo-for-windows](https:\u002F\u002Fgithub.com\u002Fsdbds\u002Fhallo-for-windows) - Utilize Hallo on Windows with the guide by [@sdbds](https:\u002F\u002Fgithub.com\u002Fsdbds).\n- [ComfyUI-Hallo](https:\u002F\u002Fgithub.com\u002FAIFSH\u002FComfyUI-Hallo) - Integrate Hallo with the ComfyUI tool by [@AIFSH](https:\u002F\u002Fgithub.com\u002FAIFSH).\n- [hallo-docker](https:\u002F\u002Fgithub.com\u002Fashleykleynhans\u002Fhallo-docker) - Docker image for Hallo by [@ashleykleynhans](https:\u002F\u002Fgithub.com\u002Fashleykleynhans).\n- [RunPod Template](https:\u002F\u002Frunpod.io\u002Fconsole\u002Fdeploy?template=aeyibwyvzy&ref=2xxro4syy) - Deploy Hallo to RunPod by [@ashleykleynhans](https:\u002F\u002Fgithub.com\u002Fashleykleynhans).\n- [JoyHallo](https:\u002F\u002Fjdh-algo.github.io\u002FJoyHallo\u002F) - JoyHallo extends the capabilities of Hallo, enabling it to support Mandarin\n\nThanks to all of them.\n\nJoin our community and explore these amazing resources to make the most out of Hallo. Enjoy and elevate their creative projects!\n\n## 🔧️ Framework\n\n![abstract](assets\u002Fframework_1.jpg)\n![framework](assets\u002Fframework_2.jpg)\n\n## ⚙️ Installation\n\n- System requirement: Ubuntu 20.04\u002FUbuntu 22.04, Cuda 12.1\n- Tested GPUs: A100\n\nCreate conda environment:\n\n```bash\n  conda create -n hallo python=3.10\n  conda activate hallo\n```\n\nInstall packages with `pip`\n\n```bash\n  pip install -r requirements.txt\n  pip install .\n```\n\nBesides, ffmpeg is also needed:\n```bash\n  apt-get install ffmpeg\n```\n\n## 🗝️️ Usage\n\nThe entry point for inference is `scripts\u002Finference.py`. Before testing your cases, two preparations need to be completed:\n\n1. [Download all required pretrained models](#download-pretrained-models).\n2. [Prepare source image and driving audio pairs](#prepare-inference-data).\n3. [Run inference](#run-inference).\n\n### 📥 Download Pretrained Models\n\nYou can easily get all pretrained models required by inference from our [HuggingFace repo](https:\u002F\u002Fhuggingface.co\u002Ffudan-generative-ai\u002Fhallo).\n\nClone the pretrained models into `${PROJECT_ROOT}\u002Fpretrained_models` directory by cmd below:\n\n```shell\ngit lfs install\ngit clone https:\u002F\u002Fhuggingface.co\u002Ffudan-generative-ai\u002Fhallo pretrained_models\n```\n\nOr you can download them separately from their source repo:\n\n- [hallo](https:\u002F\u002Fhuggingface.co\u002Ffudan-generative-ai\u002Fhallo\u002Ftree\u002Fmain\u002Fhallo): Our checkpoints consist of denoising UNet, face locator, image & audio proj.\n- [audio_separator](https:\u002F\u002Fhuggingface.co\u002Fhuangjackson\u002FKim_Vocal_2): Kim\\_Vocal\\_2 MDX-Net vocal removal model. (_Thanks to [KimberleyJensen](https:\u002F\u002Fgithub.com\u002FKimberleyJensen)_)\n- [insightface](https:\u002F\u002Fgithub.com\u002Fdeepinsight\u002Finsightface\u002Ftree\u002Fmaster\u002Fpython-package#model-zoo): 2D and 3D Face Analysis placed into `pretrained_models\u002Fface_analysis\u002Fmodels\u002F`. (_Thanks to deepinsight_)\n- [face landmarker](https:\u002F\u002Fstorage.googleapis.com\u002Fmediapipe-models\u002Fface_landmarker\u002Fface_landmarker\u002Ffloat16\u002F1\u002Fface_landmarker.task): Face detection & mesh model from [mediapipe](https:\u002F\u002Fai.google.dev\u002Fedge\u002Fmediapipe\u002Fsolutions\u002Fvision\u002Fface_landmarker#models) placed into `pretrained_models\u002Fface_analysis\u002Fmodels`.\n- [motion module](https:\u002F\u002Fgithub.com\u002Fguoyww\u002FAnimateDiff\u002Fblob\u002Fmain\u002FREADME.md#202309-animatediff-v2): motion module from [AnimateDiff](https:\u002F\u002Fgithub.com\u002Fguoyww\u002FAnimateDiff). (_Thanks to [guoyww](https:\u002F\u002Fgithub.com\u002Fguoyww)_).\n- [sd-vae-ft-mse](https:\u002F\u002Fhuggingface.co\u002Fstabilityai\u002Fsd-vae-ft-mse): Weights are intended to be used with the diffusers library. (_Thanks to [stablilityai](https:\u002F\u002Fhuggingface.co\u002Fstabilityai)_)\n- [StableDiffusion V1.5](https:\u002F\u002Fhuggingface.co\u002Frunwayml\u002Fstable-diffusion-v1-5): Initialized and fine-tuned from Stable-Diffusion-v1-2. (_Thanks to [runwayml](https:\u002F\u002Fhuggingface.co\u002Frunwayml)_)\n- [wav2vec](https:\u002F\u002Fhuggingface.co\u002Ffacebook\u002Fwav2vec2-base-960h): wav audio to vector model from [Facebook](https:\u002F\u002Fhuggingface.co\u002Ffacebook\u002Fwav2vec2-base-960h).\n\nFinally, these pretrained models should be organized as follows:\n\n```text\n.\u002Fpretrained_models\u002F\n|-- audio_separator\u002F\n|   |-- download_checks.json\n|   |-- mdx_model_data.json\n|   |-- vr_model_data.json\n|   `-- Kim_Vocal_2.onnx\n|-- face_analysis\u002F\n|   `-- models\u002F\n|       |-- face_landmarker_v2_with_blendshapes.task  # face landmarker model from mediapipe\n|       |-- 1k3d68.onnx\n|       |-- 2d106det.onnx\n|       |-- genderage.onnx\n|       |-- glintr100.onnx\n|       `-- scrfd_10g_bnkps.onnx\n|-- motion_module\u002F\n|   `-- mm_sd_v15_v2.ckpt\n|-- sd-vae-ft-mse\u002F\n|   |-- config.json\n|   `-- diffusion_pytorch_model.safetensors\n|-- stable-diffusion-v1-5\u002F\n|   `-- unet\u002F\n|       |-- config.json\n|       `-- diffusion_pytorch_model.safetensors\n`-- wav2vec\u002F\n    `-- wav2vec2-base-960h\u002F\n        |-- config.json\n        |-- feature_extractor_config.json\n        |-- model.safetensors\n        |-- preprocessor_config.json\n        |-- special_tokens_map.json\n        |-- tokenizer_config.json\n        `-- vocab.json\n```\n\n### 🛠️ Prepare Inference Data\n\nHallo has a few simple requirements for input data:\n\nFor the source image:\n\n1. It should be cropped into squares.\n2. The face should be the main focus, making up 50%-70% of the image.\n3. The face should be facing forward, with a rotation angle of less than 30° (no side profiles).\n\nFor the driving audio:\n\n1. It must be in WAV format.\n2. It must be in English since our training datasets are only in this language.\n3. Ensure the vocals are clear; background music is acceptable.\n\nWe have provided [some samples](examples\u002F) for your reference.\n\n### 🎮 Run Inference\n\nSimply to run the `scripts\u002Finference.py` and pass `source_image` and `driving_audio` as input:\n\n```bash\npython scripts\u002Finference.py --source_image examples\u002Freference_images\u002F1.jpg --driving_audio examples\u002Fdriving_audios\u002F1.wav\n```\n\nAnimation results will be saved as `${PROJECT_ROOT}\u002F.cache\u002Foutput.mp4` by default. You can pass `--output` to specify the output file name. You can find more examples for inference at [examples folder](https:\u002F\u002Fgithub.com\u002Ffudan-generative-vision\u002Fhallo\u002Ftree\u002Fmain\u002Fexamples).\n\nFor more options:\n\n```shell\nusage: inference.py [-h] [-c CONFIG] [--source_image SOURCE_IMAGE] [--driving_audio DRIVING_AUDIO] [--output OUTPUT] [--pose_weight POSE_WEIGHT]\n                    [--face_weight FACE_WEIGHT] [--lip_weight LIP_WEIGHT] [--face_expand_ratio FACE_EXPAND_RATIO]\n\noptions:\n  -h, --help            show this help message and exit\n  -c CONFIG, --config CONFIG\n  --source_image SOURCE_IMAGE\n                        source image\n  --driving_audio DRIVING_AUDIO\n                        driving audio\n  --output OUTPUT       output video file name\n  --pose_weight POSE_WEIGHT\n                        weight of pose\n  --face_weight FACE_WEIGHT\n                        weight of face\n  --lip_weight LIP_WEIGHT\n                        weight of lip\n  --face_expand_ratio FACE_EXPAND_RATIO\n                        face region\n```\n\n## Training\n\n### Prepare Data for Training\n\nThe training data, which utilizes some talking-face videos similar to the source images used for inference, also needs to meet the following requirements:\n\n1. It should be cropped into squares.\n2. The face should be the main focus, making up 50%-70% of the image.\n3. The face should be facing forward, with a rotation angle of less than 30° (no side profiles).\n\nOrganize your raw videos into the following directory structure:\n\n\n```text\ndataset_name\u002F\n|-- videos\u002F\n|   |-- 0001.mp4\n|   |-- 0002.mp4\n|   |-- 0003.mp4\n|   `-- 0004.mp4\n```\n\nYou can use any `dataset_name`, but ensure the `videos` directory is named as shown above.\n\nNext, process the videos with the following commands:\n\n```bash\npython -m scripts.data_preprocess --input_dir dataset_name\u002Fvideos --step 1\npython -m scripts.data_preprocess --input_dir dataset_name\u002Fvideos --step 2\n```\n\n**Note:** Execute steps 1 and 2 sequentially as they perform different tasks. Step 1 converts videos into frames, extracts audio from each video, and generates the necessary masks. Step 2 generates face embeddings using InsightFace and audio embeddings using Wav2Vec, and requires a GPU. For parallel processing, use the `-p` and `-r` arguments. The `-p` argument specifies the total number of instances to launch, dividing the data into `p` parts. The `-r` argument specifies which part the current process should handle. You need to manually launch multiple instances with different values for `-r`.\n\nGenerate the metadata JSON files with the following commands:\n\n```bash\npython scripts\u002Fextract_meta_info_stage1.py -r path\u002Fto\u002Fdataset -n dataset_name\npython scripts\u002Fextract_meta_info_stage2.py -r path\u002Fto\u002Fdataset -n dataset_name\n```\n\nReplace `path\u002Fto\u002Fdataset` with the path to the parent directory of `videos`, such as `dataset_name` in the example above. This will generate `dataset_name_stage1.json` and `dataset_name_stage2.json` in the `.\u002Fdata` directory.\n\n### Training\n\nUpdate the data meta path settings in the configuration YAML files, `configs\u002Ftrain\u002Fstage1.yaml` and `configs\u002Ftrain\u002Fstage2.yaml`:\n\n\n```yaml\n#stage1.yaml\ndata:\n  meta_paths:\n    - .\u002Fdata\u002Fdataset_name_stage1.json\n\n#stage2.yaml\ndata:\n  meta_paths:\n    - .\u002Fdata\u002Fdataset_name_stage2.json\n```\n\nStart training with the following command:\n\n```shell\naccelerate launch -m \\\n  --config_file accelerate_config.yaml \\\n  --machine_rank 0 \\\n  --main_process_ip 0.0.0.0 \\\n  --main_process_port 20055 \\\n  --num_machines 1 \\\n  --num_processes 8 \\\n  scripts.train_stage1 --config .\u002Fconfigs\u002Ftrain\u002Fstage1.yaml\n```\n\n#### Accelerate Usage Explanation\n\nThe `accelerate launch` command is used to start the training process with distributed settings.\n\n```shell\naccelerate launch [arguments] {training_script} --{training_script-argument-1} --{training_script-argument-2} ...\n```\n\n**Arguments for Accelerate:**\n\n- `-m, --module`: Interpret the launch script as a Python module.\n- `--config_file`: Configuration file for Hugging Face Accelerate.\n- `--machine_rank`: Rank of the current machine in a multi-node setup.\n- `--main_process_ip`: IP address of the master node.\n- `--main_process_port`: Port of the master node.\n- `--num_machines`: Total number of nodes participating in the training.\n- `--num_processes`: Total number of processes for training, matching the total number of GPUs across all machines.\n\n**Arguments for Training:**\n\n- `{training_script}`: The training script, such as `scripts.train_stage1` or `scripts.train_stage2`.\n- `--{training_script-argument-1}`: Arguments specific to the training script. Our training scripts accept one argument, `--config`, to specify the training configuration file.\n\nFor multi-node training, you need to manually run the command with different `machine_rank` on each node separately.\n\nFor more settings, refer to the [Accelerate documentation](https:\u002F\u002Fhuggingface.co\u002Fdocs\u002Faccelerate\u002Fen\u002Findex).\n\n## 📅️ Roadmap\n\n| Status | Milestone                                                                                             |    ETA     |\n| :----: | :---------------------------------------------------------------------------------------------------- | :--------: |\n|   ✅   | **[Inference source code meet everyone on GitHub](https:\u002F\u002Fgithub.com\u002Ffudan-generative-vision\u002Fhallo)** | 2024-06-15 |\n|   ✅   | **[Pretrained models on Huggingface](https:\u002F\u002Fhuggingface.co\u002Ffudan-generative-ai\u002Fhallo)**              | 2024-06-15 |\n| ✅ | **[Releasing data preparation and training scripts](#training)**                                                | 2024-06-28 |\n| 🚀 | **[Improving the model's performance on Mandarin Chinese]()**                                                    |    TBD     |\n\n\u003Cdetails>\n\u003Csummary>Other Enhancements\u003C\u002Fsummary>\n\n- [x] Enhancement: Test and ensure compatibility with Windows operating system. [#39](https:\u002F\u002Fgithub.com\u002Ffudan-generative-vision\u002Fhallo\u002Fissues\u002F39)\n- [x] Bug: Output video may lose several frames. [#41](https:\u002F\u002Fgithub.com\u002Ffudan-generative-vision\u002Fhallo\u002Fissues\u002F41)\n- [ ] Bug: Sound volume affecting inference results (audio normalization).\n- [ ] ~~Enhancement: Inference code logic optimization~~. This solution doesn't show significant performance improvements. Trying other approaches.\n\n\u003C\u002Fdetails>\n\n\n## 📝 Citation\n\nIf you find our work useful for your research, please consider citing the paper:\n\n```\n@misc{xu2024hallo,\n  title={Hallo: Hierarchical Audio-Driven Visual Synthesis for Portrait Image Animation},\n  author={Mingwang Xu and Hui Li and Qingkun Su and Hanlin Shang and Liwei Zhang and Ce Liu and Jingdong Wang and Yao Yao and Siyu zhu},\n  year={2024},\n  eprint={2406.08801},\n  archivePrefix={arXiv},\n  primaryClass={cs.CV}\n}\n```\n\n## 🌟 Opportunities Available\n\nMultiple research positions are open at the **Generative Vision Lab, Fudan University**! Include:\n\n- Research assistant\n- Postdoctoral researcher\n- PhD candidate\n- Master students\n\nInterested individuals are encouraged to contact us at [siyuzhu@fudan.edu.cn](mailto:\u002F\u002Fsiyuzhu@fudan.edu.cn) for further information.\n\n## ⚠️ Social Risks and Mitigations\n\nThe development of portrait image animation technologies driven by audio inputs poses social risks, such as the ethical implications of creating realistic portraits that could be misused for deepfakes. To mitigate these risks, it is crucial to establish ethical guidelines and responsible use practices. Privacy and consent concerns also arise from using individuals' images and voices. Addressing these involves transparent data usage policies, informed consent, and safeguarding privacy rights. By addressing these risks and implementing mitigations, the research aims to ensure the responsible and ethical development of this technology.\n\n## 🤗 Acknowledgements\n\nWe would like to thank the contributors to the [magic-animate](https:\u002F\u002Fgithub.com\u002Fmagic-research\u002Fmagic-animate), [AnimateDiff](https:\u002F\u002Fgithub.com\u002Fguoyww\u002FAnimateDiff), [ultimatevocalremovergui](https:\u002F\u002Fgithub.com\u002FAnjok07\u002Fultimatevocalremovergui), [AniPortrait](https:\u002F\u002Fgithub.com\u002FZejun-Yang\u002FAniPortrait) and [Moore-AnimateAnyone](https:\u002F\u002Fgithub.com\u002FMooreThreads\u002FMoore-AnimateAnyone) repositories, for their open research and exploration.\n\nIf we missed any open-source projects or related articles, we would like to complement the acknowledgement of this specific work immediately.\n\n## 👏 Community Contributors\n\nThank you to all the contributors who have helped to make this project better!\n\n\u003Ca href=\"https:\u002F\u002Fgithub.com\u002Ffudan-generative-vision\u002Fhallo\u002Fgraphs\u002Fcontributors\">\n  \u003Cimg src=\"https:\u002F\u002Fcontrib.rocks\u002Fimage?repo=fudan-generative-vision\u002Fhallo\" \u002F>\n\u003C\u002Fa>\n","Hallo是一个基于音频驱动的肖像图像动画生成项目，它能够根据输入的音频自动生成与之匹配的面部表情和口型变化。该项目利用了层级结构的视觉合成技术，使得生成的动画不仅自然流畅，而且能够很好地保持原始肖像的身份特征。采用Python语言开发，并且在GitHub上获得了广泛的关注。适用于需要将静态图片转换为动态视频的场景，如电影片段重现、虚拟人物创建等，尤其适合于那些希望以低成本方式实现高质量人脸动画效果的应用场合。","2026-06-11 03:39:47","high_star"]