[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"project-72176":3},{"id":4,"name":5,"fullName":6,"owner":7,"repo":5,"description":8,"homepage":9,"htmlUrl":10,"language":11,"languages":10,"totalLinesOfCode":10,"stars":12,"forks":13,"watchers":14,"openIssues":15,"contributorsCount":16,"subscribersCount":16,"size":16,"stars1d":16,"stars7d":16,"stars30d":17,"stars90d":16,"forks30d":16,"starsTrendScore":16,"compositeScore":18,"rankGlobal":10,"rankLanguage":10,"license":19,"archived":20,"fork":20,"defaultBranch":21,"hasWiki":22,"hasPages":22,"topics":23,"createdAt":10,"pushedAt":10,"updatedAt":27,"readmeContent":28,"aiSummary":29,"trendingCount":16,"starSnapshotCount":16,"syncStatus":30,"lastSyncTime":31,"discoverSource":32},72176,"champ","fudan-generative-vision\u002Fchamp","fudan-generative-vision","[ECCV 2024] Champ: Controllable and Consistent Human Image Animation with 3D Parametric Guidance","https:\u002F\u002Ffudan-generative-vision.github.io\u002Fchamp\u002F",null,"Python",4251,483,211,49,0,1,60.15,"MIT License",false,"master",true,[24,25,26],"human-animation","image-animatioln","video-generation","2026-06-12 04:01:03","\u003Ch1 align='Center'>Champ: Controllable and Consistent Human Image Animation with 3D Parametric Guidance\u003C\u002Fh1>\n\n\u003Cdiv align='Center'>\n    \u003Ca href='https:\u002F\u002Fgithub.com\u002FShenhaoZhu' target='_blank'>Shenhao Zhu\u003C\u002Fa>\u003Csup>*1\u003C\u002Fsup> \n    \u003Ca href='https:\u002F\u002Fgithub.com\u002FLeoooo333' target='_blank'>Junming Leo Chen\u003C\u002Fa>\u003Csup>*2\u003C\u002Fsup> \n    \u003Ca href='https:\u002F\u002Fgithub.com\u002Fdaizuozhuo' target='_blank'>Zuozhuo Dai\u003C\u002Fa>\u003Csup>3\u003C\u002Fsup> \n    \u003Ca href='https:\u002F\u002Fai3.fudan.edu.cn\u002Finfo\u002F1088\u002F1266.htm' target='_blank'>Yinghui Xu\u003C\u002Fa>\u003Csup>2\u003C\u002Fsup> \n    \u003Ca href='https:\u002F\u002Fcite.nju.edu.cn\u002FPeople\u002FFaculty\u002F20190621\u002Fi5054.html' target='_blank'>Xun Cao\u003C\u002Fa>\u003Csup>1\u003C\u002Fsup> \n    \u003Ca href='https:\u002F\u002Fyoyo000.github.io\u002F' target='_blank'>Yao Yao\u003C\u002Fa>\u003Csup>1\u003C\u002Fsup> \n    \u003Ca href='http:\u002F\u002Fzhuhao.cc\u002Fhome\u002F' target='_blank'>Hao Zhu\u003C\u002Fa>\u003Csup>+1\u003C\u002Fsup> \n    \u003Ca href='https:\u002F\u002Fsites.google.com\u002Fsite\u002Fzhusiyucs\u002Fhome' target='_blank'>Siyu Zhu\u003C\u002Fa>\u003Csup>+2\u003C\u002Fsup>\n\u003C\u002Fdiv>\n\u003Cdiv align='Center'>\n    \u003Csup>1\u003C\u002Fsup>Nanjing University \u003Csup>2\u003C\u002Fsup>Fudan University \u003Csup>3\u003C\u002Fsup>Alibaba Group\n\u003C\u002Fdiv>\n\u003Cdiv align='Center'>\n\u003Ci>\u003Cstrong>\u003Ca href='https:\u002F\u002Feccv2024.ecva.net' target='_blank'>ECCV 2024\u003C\u002Fa>\u003C\u002Fstrong>\u003C\u002Fi>\n\u003C\u002Fdiv>\n\n\n\u003Cdiv align='Center'>\n    \u003Ca href='https:\u002F\u002Ffudan-generative-vision.github.io\u002Fchamp\u002F#\u002F'>\u003Cimg src='https:\u002F\u002Fimg.shields.io\u002Fbadge\u002FProject-Page-Green'>\u003C\u002Fa>\n    \u003Ca href='https:\u002F\u002Farxiv.org\u002Fabs\u002F2403.14781'>\u003Cimg src='https:\u002F\u002Fimg.shields.io\u002Fbadge\u002FPaper-Arxiv-red'>\u003C\u002Fa>\n    \u003Ca href='https:\u002F\u002Fyoutu.be\u002F2XVsy9tQRAY'>\u003Cimg src='https:\u002F\u002Fbadges.aleen42.com\u002Fsrc\u002Fyoutube.svg'>\u003C\u002Fa>\n    \u003Ca href='assets\u002Fwechat.jpeg'>\u003Cimg src='https:\u002F\u002Fbadges.aleen42.com\u002Fsrc\u002Fwechat.svg'>\u003C\u002Fa>\n\u003C\u002Fdiv>\n\nhttps:\u002F\u002Fgithub.com\u002Ffudan-generative-vision\u002Fchamp\u002Fassets\u002F82803297\u002Fb4571be6-dfb0-4926-8440-3db229ebd4aa\n\n# Framework\n\n![framework](assets\u002Fframework.jpg)\n\n# News\n\n- **`2024\u002F05\u002F05`**:  🎉🎉🎉[Sample training data on HuggingFace](https:\u002F\u002Fhuggingface.co\u002Fdatasets\u002Ffudan-generative-ai\u002Fchamp_trainning_sample) released.\n- **`2024\u002F05\u002F02`**:  🌟🌟🌟Training source code released [#99](https:\u002F\u002Fgithub.com\u002Ffudan-generative-vision\u002Fchamp\u002Fpull\u002F99).\n- **`2024\u002F04\u002F28`**:  👏👏👏Smooth SMPLs in Blender method released [#96](https:\u002F\u002Fgithub.com\u002Ffudan-generative-vision\u002Fchamp\u002Fpull\u002F96).\n- **`2024\u002F04\u002F26`**:  🚁Great Blender Adds-on [CEB Studios\n](https:\u002F\u002Fwww.patreon.com\u002Fcebstudios\u002Fposts) for various SMPL process!\n- **`2024\u002F04\u002F12`**: ✨✨✨SMPL & Rendering scripts released! Champ your dance videos now💃🤸‍♂️🕺. See [docs](https:\u002F\u002Fgithub.com\u002Ffudan-generative-vision\u002Fchamp\u002Fblob\u002Fmaster\u002Fdocs\u002Fdata_process.md).\n  \n- **`2024\u002F03\u002F30`**: 🚀🚀🚀Amazing [ComfyUI Wrapper](https:\u002F\u002Fgithub.com\u002Fkijai\u002FComfyUI-champWrapper) by community. Here is the [video tutorial](https:\u002F\u002Fwww.youtube.com\u002Fwatch?app=desktop&v=cbElsTBv2-A). Thanks to [@kijai](https:\u002F\u002Fgithub.com\u002Fkijai)🥳\n  \n- **`2024\u002F03\u002F27`**: Cool Demo on [replicate](https:\u002F\u002Freplicate.com\u002Fcamenduru\u002Fchamp)🌟. Thanks to [@camenduru](https:\u002F\u002Fgithub.com\u002Fcamenduru)👏\n\n- **`2024\u002F03\u002F27`**: Visit our [roadmap🕒](#roadmap) to preview the future of Champ.\n\n# Installation\n\n- System requirement: Ubuntu20.04\u002FWindows 11, Cuda 12.1\n- Tested GPUs: A100, RTX3090\n\nCreate conda environment:\n\n```bash\n  conda create -n champ python=3.10\n  conda activate champ\n```\n\nInstall packages with `pip`\n\n```bash\n  pip install -r requirements.txt\n```\n\nInstall packages with [poetry](https:\u002F\u002Fpython-poetry.org\u002F)\n> If you want to run this project on a Windows device, we strongly recommend to use `poetry`.\n```shell\npoetry install --no-root\n```\n\n# Inference\n\nThe inference entrypoint script is `${PROJECT_ROOT}\u002Finference.py`. Before testing your cases, there are two preparations need to be completed:\n1. [Download all required pretrained models](#download-pretrained-models).\n2. [Prepare your guidance motions](#preparen-your-guidance-motions).\n2. [Run inference](#run-inference).\n\n## Download pretrained models\n\nYou can easily get all pretrained models required by inference from our [HuggingFace repo](https:\u002F\u002Fhuggingface.co\u002Ffudan-generative-ai\u002Fchamp).\n\nClone the the pretrained models into `${PROJECT_ROOT}\u002Fpretrained_models` directory by cmd below:\n```shell\ngit lfs install\ngit clone https:\u002F\u002Fhuggingface.co\u002Ffudan-generative-ai\u002Fchamp pretrained_models\n```\n\nOr you can download them separately from their source repo:\n   - [Champ ckpts](https:\u002F\u002Fhuggingface.co\u002Ffudan-generative-ai\u002Fchamp\u002Ftree\u002Fmain):  Consist of denoising UNet, guidance encoders, Reference UNet, and motion module.\n   - [StableDiffusion V1.5](https:\u002F\u002Fhuggingface.co\u002Frunwayml\u002Fstable-diffusion-v1-5): Initialized and fine-tuned from Stable-Diffusion-v1-2. (*Thanks to runwayml*)\n   - [sd-vae-ft-mse](https:\u002F\u002Fhuggingface.co\u002Fstabilityai\u002Fsd-vae-ft-mse): Weights are intended to be used with the diffusers library. (*Thanks to stablilityai*)\n   - [image_encoder](https:\u002F\u002Fhuggingface.co\u002Flambdalabs\u002Fsd-image-variations-diffusers\u002Ftree\u002Fmain\u002Fimage_encoder): Fine-tuned from CompVis\u002Fstable-diffusion-v1-4-original to accept CLIP image embedding rather than text embeddings. (*Thanks to lambdalabs*)\n\nFinally, these pretrained models should be organized as follows:\n\n```text\n.\u002Fpretrained_models\u002F\n|-- champ\n|   |-- denoising_unet.pth\n|   |-- guidance_encoder_depth.pth\n|   |-- guidance_encoder_dwpose.pth\n|   |-- guidance_encoder_normal.pth\n|   |-- guidance_encoder_semantic_map.pth\n|   |-- reference_unet.pth\n|   `-- motion_module.pth\n|-- image_encoder\n|   |-- config.json\n|   `-- pytorch_model.bin\n|-- sd-vae-ft-mse\n|   |-- config.json\n|   |-- diffusion_pytorch_model.bin\n|   `-- diffusion_pytorch_model.safetensors\n`-- stable-diffusion-v1-5\n    |-- feature_extractor\n    |   `-- preprocessor_config.json\n    |-- model_index.json\n    |-- unet\n    |   |-- config.json\n    |   `-- diffusion_pytorch_model.bin\n    `-- v1-inference.yaml\n```\n\n## Prepare your guidance motions\n\nGuidance motion data which is produced via SMPL & Rendering is necessary when performing inference.\n\nYou can download our pre-rendered samples on our [HuggingFace repo](https:\u002F\u002Fhuggingface.co\u002Fdatasets\u002Ffudan-generative-ai\u002Fchamp_motions_example) and place into `${PROJECT_ROOT}\u002Fexample_data` directory:\n```shell\ngit lfs install\ngit clone https:\u002F\u002Fhuggingface.co\u002Fdatasets\u002Ffudan-generative-ai\u002Fchamp_motions_example example_data\n```\n\nOr you can follow the [SMPL & Rendering doc](https:\u002F\u002Fgithub.com\u002Ffudan-generative-vision\u002Fchamp\u002Fblob\u002Fmaster\u002Fdocs\u002Fdata_process.md) to produce your own motion datas.\n\nFinally, the `${PROJECT_ROOT}\u002Fexample_data` will be like this:\n```\n.\u002Fexample_data\u002F\n|-- motions\u002F  # Directory includes motions per subfolder\n|   |-- motion-01\u002F  # A motion sample\n|   |   |-- depth\u002F  # Depth frame sequance\n|   |   |-- dwpose\u002F # Dwpose frame sequance\n|   |   |-- mask\u002F   # Mask frame sequance\n|   |   |-- normal\u002F # Normal map frame sequance\n|   |   `-- semantic_map\u002F # Semanic map frame sequance\n|   |-- motion-02\u002F\n|   |   |-- ...\n|   |   `-- ...\n|   `-- motion-N\u002F\n|       |-- ...\n|       `-- ...\n`-- ref_images\u002F # Reference image samples(Optional)\n    |-- ref-01.png\n    |-- ...\n    `-- ref-N.png\n```\n\n## Run inference\n\nNow we have all prepared models and motions in `${PROJECT_ROOT}\u002Fpretrained_models` and `${PROJECT_ROOT}\u002Fexample_data` separately. \n\nHere is the command for inference:\n\n```bash\n  python inference.py --config configs\u002Finference\u002Finference.yaml\n```\n\nIf using `poetry`, command is \n```shell\npoetry run python inference.py --config configs\u002Finference\u002Finference.yaml\n```\n\nAnimation results will be saved in `${PROJECT_ROOT}\u002Fresults` folder. You can change the reference image or the guidance motion by modifying `inference.yaml`.\n\nThe default motion-02 in `inference.yaml` has about 250 frames, requires ~20GB VRAM.\n\n**Note**: If your VRAM is insufficient, you can switch to a shorter motion sequence or cut out a segment from a long sequence. We provide a frame range selector in `inference.yaml`, which you can replace with a list of `[min_frame_index, max_frame_index]` to conveniently cut out a segment from the sequence.\n\n# Train the Model\n\nThe training process consists of two distinct stages. For more information, refer to the `Training Section` in the [paper on arXiv](https:\u002F\u002Farxiv.org\u002Fabs\u002F2403.14781).\n\n## Prepare Datasets\n\nPrepare your own training videos with human motion (or use [our sample training data on HuggingFace](https:\u002F\u002Fhuggingface.co\u002Fdatasets\u002Ffudan-generative-ai\u002Fchamp_trainning_sample)) and modify `data.video_folder` value in training config yaml.\n\nAll training videos need to be processed into SMPL & DWPose format. Refer to the [Data Process doc](https:\u002F\u002Fgithub.com\u002Ffudan-generative-vision\u002Fchamp\u002Fblob\u002Fmaster\u002Fdocs\u002Fdata_process.md).\n\nThe directory structure will be like this:\n```txt\n\u002Ftraining_data\u002F\n|-- video01\u002F          # A video data frame\n|   |-- depth\u002F        # Depth frame sequance\n|   |-- dwpose\u002F       # Dwpose frame sequance\n|   |-- mask\u002F         # Mask frame sequance\n|   |-- normal\u002F       # Normal map frame sequance\n|   `-- semantic_map\u002F # Semanic map frame sequance\n|-- video02\u002F\n|   |-- ...\n|   `-- ...\n`-- videoN\u002F\n|-- ...\n`-- ...\n```\n\nSelect another small batch of data as the validation set, and modify the `validation.ref_images` and `validation.guidance_folders` roots in training config yaml.\n\n## Run Training Scripts\n\nTo train the Champ model, use the following command:\n```shell\n# Run training script of stage1\naccelerate launch train_s1.py --config configs\u002Ftrain\u002Fstage1.yaml\n\n# Modify the `stage1_ckpt_dir` value in yaml and run training script of stage2\naccelerate launch train_s2.py --config configs\u002Ftrain\u002Fstage2.yaml\n```\n\n# Datasets\n\n| Type | HuggingFace |       ETA       |\n| :----: | :----------------------------------------------------------------------------------------- | :-------------: |\n|   Inference   | **[SMPL motion samples](https:\u002F\u002Fhuggingface.co\u002Fdatasets\u002Ffudan-generative-ai\u002Fchamp_motions_example)** | Thu Apr 18 2024 |\n|   Training | **[Sample datasets for Training](https:\u002F\u002Fhuggingface.co\u002Fdatasets\u002Ffudan-generative-ai\u002Fchamp_trainning_sample)** | Sun May 05 2024 |\n# Roadmap\n\n| Status | Milestone                                                                                  |       ETA       |\n| :----: | :----------------------------------------------------------------------------------------- | :-------------: |\n|   ✅   | **[Inference source code meet everyone on GitHub first time](https:\u002F\u002Fgithub.com\u002Ffudan-generative-vision\u002Fchamp)** | Sun Mar 24 2024 |\n|   ✅   | **[Model and test data on Huggingface](https:\u002F\u002Fhuggingface.co\u002Ffudan-generative-ai\u002Fchamp)** | Tue Mar 26 2024 |\n|   ✅   | **[Optimize dependencies and go well on Windows](https:\u002F\u002Fgithub.com\u002Ffudan-generative-vision\u002Fchamp?tab=readme-ov-file#installation)** | Sun Mar 31 2024 |\n|   ✅   | **[Data preprocessing code release](https:\u002F\u002Fgithub.com\u002Ffudan-generative-vision\u002Fchamp\u002Fblob\u002Fmaster\u002Fdocs\u002Fdata_process.md)**                                                    | Fri Apr 12 2024 |\n|   ✅   | **[Training code release](https:\u002F\u002Fgithub.com\u002Ffudan-generative-vision\u002Fchamp\u002Fpull\u002F99)**                                                  | Thu May 02 2024 |\n|   ✅   | **[Sample of training data release on HuggingFace](https:\u002F\u002Fhuggingface.co\u002Fdatasets\u002Ffudan-generative-ai\u002Fchamp_trainning_sample)**                                                  | Sun May 05 2024 |\n|   ✅  | **[Smoothing SMPL motion](https:\u002F\u002Fgithub.com\u002Ffudan-generative-vision\u002Fchamp\u002Fpull\u002F96)**                                                  | Sun Apr 28 2024 |\n|   🚀🚀🚀  | **[Gradio demo on HuggingFace]()**                                                  | TBD |\n\n# Citation\n\nIf you find our work useful for your research, please consider citing the paper:\n\n```\n@inproceedings{zhu2024champ,\n      title={Champ: Controllable and Consistent Human Image Animation with 3D Parametric Guidance},\n      author={Shenhao Zhu and Junming Leo Chen and Zuozhuo Dai and Yinghui Xu and Xun Cao and Yao Yao and Hao Zhu and Siyu Zhu},\n      booktitle={European Conference on Computer Vision (ECCV)},\n      year={2024}\n}\n```\n\n# Opportunities available\n\nMultiple research positions are open at the **Generative Vision Lab, Fudan University**! Include:\n\n- Research assistant\n- Postdoctoral researcher\n- PhD candidate\n- Master students\n\nInterested individuals are encouraged to contact us at [siyuzhu@fudan.edu.cn](mailto:\u002F\u002Fsiyuzhu@fudan.edu.cn) for further information.\n","Champ 是一个用于可控且一致的人体图像动画生成的项目，通过3D参数化指导实现高质量的人体动画。其核心功能包括基于SMPL模型的3D人体姿态估计和渲染，支持用户自定义输入以生成自然流畅的动作序列。技术上，Champ利用深度学习方法处理图像，并结合3D建模技术确保输出动画的真实性和连贯性。该项目适用于需要高质量人体动画的应用场景，如虚拟现实、游戏开发或影视特效制作等，为开发者提供了一个强大而灵活的工具来创建逼真的数字人物动画。",2,"2026-06-11 03:40:43","high_star"]