[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"project-72530":3},{"id":4,"name":5,"fullName":6,"owner":7,"repo":5,"description":8,"homepage":9,"htmlUrl":10,"language":11,"languages":10,"totalLinesOfCode":10,"stars":12,"forks":13,"watchers":14,"openIssues":15,"contributorsCount":16,"subscribersCount":16,"size":16,"stars1d":17,"stars7d":17,"stars30d":18,"stars90d":16,"forks30d":16,"starsTrendScore":18,"compositeScore":19,"rankGlobal":10,"rankLanguage":10,"license":20,"archived":21,"fork":21,"defaultBranch":22,"hasWiki":23,"hasPages":21,"topics":24,"createdAt":10,"pushedAt":10,"updatedAt":25,"readmeContent":26,"aiSummary":27,"trendingCount":16,"starSnapshotCount":16,"syncStatus":17,"lastSyncTime":28,"discoverSource":29},72530,"EasyAnimate","aigc-apps\u002FEasyAnimate","aigc-apps","📺 An End-to-End Solution for High-Resolution and Long Video Generation Based on Transformer Diffusion","",null,"Python",2268,184,27,93,0,2,6,28.8,"Apache License 2.0",false,"main",true,[],"2026-06-12 02:03:04","# 📷 EasyAnimate | An End-to-End Solution for High-Resolution and Long Video Generation\n😊 EasyAnimate is an end-to-end solution for generating high-resolution and long videos. We can train transformer based diffusion generators, train VAEs for processing long videos, and preprocess metadata. \n\n😊 We use DIT and transformer as a diffuser for video and image generation.\n\n😊 Welcome!\n\n[![Arxiv Page](https:\u002F\u002Fimg.shields.io\u002Fbadge\u002FArxiv-Page-red)](https:\u002F\u002Farxiv.org\u002Fabs\u002F2405.18991)\n[![Project Page](https:\u002F\u002Fimg.shields.io\u002Fbadge\u002FProject-Website-green)](https:\u002F\u002Feasyanimate.github.io\u002F)\n[![Modelscope Studio](https:\u002F\u002Fimg.shields.io\u002Fbadge\u002FModelscope-Studio-blue)](https:\u002F\u002Fmodelscope.cn\u002Fstudios\u002FPAI\u002FEasyAnimate\u002Fsummary)\n[![Hugging Face Spaces](https:\u002F\u002Fimg.shields.io\u002Fbadge\u002F%F0%9F%A4%97%20Hugging%20Face-Spaces-yellow)](https:\u002F\u002Fhuggingface.co\u002Fspaces\u002Falibaba-pai\u002FEasyAnimate)\n[![Discord Page](https:\u002F\u002Fimg.shields.io\u002Fbadge\u002FDiscord-Page-blue)](https:\u002F\u002Fdiscord.gg\u002FUzkpB4Bn)\n\nEnglish | [简体中文](.\u002FREADME_zh-CN.md) | [日本語](.\u002FREADME_ja-JP.md)\n\n# Table of Contents\n- [Table of Contents](#table-of-contents)\n- [Introduction](#introduction)\n- [Quick Start](#quick-start)\n- [Video Result](#video-result)\n- [How to use](#how-to-use)\n- [Model zoo](#model-zoo)\n- [TODO List](#todo-list)\n- [Contact Us](#contact-us)\n- [Reference](#reference)\n- [License](#license)\n\n# Introduction\nEasyAnimate is a pipeline based on the transformer architecture, designed for generating AI images and videos, and for training baseline models and Lora models for Diffusion Transformer. We support direct prediction from pre-trained EasyAnimate models, allowing for the generation of videos with various resolutions, approximately 6 seconds in length, at 8fps (EasyAnimateV5, 1 to 49 frames). Additionally, users can train their own baseline and Lora models for specific style transformations.\n\nWe will support quick pull-ups from different platforms, refer to [Quick Start](#quick-start).\n\n**New Features:**\n- EasyAnimate-V5.1 is now supported in diffusers. For more implementation details, please refer to the [PR](https:\u002F\u002Fgithub.com\u002Fhuggingface\u002Fdiffusers\u002Fpull\u002F10626). Relevant weights can be downloaded from [EasyAnimate-V5.1-diffusers](https:\u002F\u002Fhuggingface.co\u002Fcollections\u002Falibaba-pai\u002Feasyanimate-v51-diffusers-67c81d1d19b236e056675cce). For usage instructions, please refer to [Usage](https:\u002F\u002Fhuggingface.co\u002Falibaba-pai\u002FEasyAnimateV5.1-7b-zh-diffusers#a%E3%80%81text-to-video). [ 2025.03.06 ]\n- **Updated to version v5.1**, the Qwen2 VL is used as the text encoder, and Flow is used as the sampling method. It supports bilingual prediction in both Chinese and English. In addition to common controls such as Canny and Pose, it also supports trajectory control, camera control. [2025.01.21]\n- Use reward backpropagation to train Lora and optimize the video, aligning it better with human preferences, detailes in [here](scripts\u002FREADME_TRAIN_REWARD.md). EasyAnimateV5-7b is released now. [2024.11.27]\n- **Updated to v5**, supporting video generation up to 1024x1024, 49 frames, 6s, 8fps, with expanded model scale to 12B, incorporating the MMDIT structure, and enabling control models with diverse inputs; supports bilingual predictions in Chinese and English. [2024.11.08]\n- **Updated to v4**, allowing for video generation up to 1024x1024, 144 frames, 6s, 24fps; supports video generation from text, image, and video, with a single model handling resolutions from 512 to 1280; bilingual predictions in Chinese and English enabled. [2024.08.15]\n- **Updated to v3**, supporting video generation up to 960x960, 144 frames, 6s, 24fps, from text and image. [2024.07.01]\n- **ModelScope-Sora “Data Director” Creative Race** — The third Data-Juicer Big Model Data Challenge is now officially launched! Utilizing EasyAnimate as the base model, it explores the impact of data processing on model training. Visit the [competition website](https:\u002F\u002Ftianchi.aliyun.com\u002Fcompetition\u002Fentrance\u002F532219) for details. [2024.06.17]\n- **Updated to v2**, supporting video generation up to 768x768, 144 frames, 6s, 24fps. [2024.05.26]\n- **Code Created!** Now supporting Windows and Linux. [2024.04.12]\n\nFunction：\n- [Data Preprocessing](#data-preprocess)\n- [Train VAE](#vae-train)\n- [Train DiT](#dit-train)\n- [Video Generation](#video-gen)\n\nOur UI interface is as follows:\n![ui](https:\u002F\u002Fpai-aigc-photog.oss-cn-hangzhou.aliyuncs.com\u002Feasyanimate\u002Fasset\u002Fui_v3.jpg)\n\n# Quick Start\n### 1. Cloud usage: AliyunDSW\u002FDocker\n#### a. From AliyunDSW\nDSW has free GPU time, which can be applied once by a user and is valid for 3 months after applying.\n\nAliyun provide free GPU time in [Freetier](https:\u002F\u002Ffree.aliyun.com\u002F?product=9602825&crowd=enterprise&spm=5176.28055625.J_5831864660.1.e939154aRgha4e&scm=20140722.M_9974135.P_110.MO_1806-ID_9974135-MID_9974135-CID_30683-ST_8512-V_1), get it and use in Aliyun PAI-DSW to start EasyAnimate within 5min!\n\n[![DSW Notebook](https:\u002F\u002Fpai-aigc-photog.oss-cn-hangzhou.aliyuncs.com\u002Feasyanimate\u002Fasset\u002Fdsw.png)](https:\u002F\u002Fgallery.pai-ml.com\u002F#\u002Fpreview\u002FdeepLearning\u002Fcv\u002Feasyanimate_v5)\n\n#### b. From ComfyUI\nOur ComfyUI is as follows, please refer to [ComfyUI README](comfyui\u002FREADME.md) for details.\n![workflow graph](https:\u002F\u002Fpai-aigc-photog.oss-cn-hangzhou.aliyuncs.com\u002Feasyanimate\u002Fasset\u002Fv3\u002Fcomfyui_i2v.jpg)\n\n#### c. From docker\nIf you are using docker, please make sure that the graphics card driver and CUDA environment have been installed correctly in your machine.\n\nThen execute the following commands in this way:\n```\n# pull image\ndocker pull mybigpai-public-registry.cn-beijing.cr.aliyuncs.com\u002Feasycv\u002Ftorch_cuda:easyanimate\n\n# enter image\ndocker run -it -p 7860:7860 --network host --gpus all --security-opt seccomp:unconfined --shm-size 200g mybigpai-public-registry.cn-beijing.cr.aliyuncs.com\u002Feasycv\u002Ftorch_cuda:easyanimate\n\n# clone code\ngit clone https:\u002F\u002Fgithub.com\u002Faigc-apps\u002FEasyAnimate.git\n\n# enter EasyAnimate's dir\ncd EasyAnimate\n\n# download weights\nmkdir models\u002FDiffusion_Transformer\nmkdir models\u002FMotion_Module\nmkdir models\u002FPersonalized_Model\n\n# Please use the hugginface link or modelscope link to download the EasyAnimateV5.1 model.\n# https:\u002F\u002Fhuggingface.co\u002Falibaba-pai\u002FEasyAnimateV5.1-12b-zh-InP\n# https:\u002F\u002Fmodelscope.cn\u002Fmodels\u002FPAI\u002FEasyAnimateV5.1-12b-zh-InP\n\n# https:\u002F\u002Fhuggingface.co\u002Falibaba-pai\u002FEasyAnimateV5.1-12b-zh\n# https:\u002F\u002Fmodelscope.cn\u002Fmodels\u002FPAI\u002FEasyAnimateV5.1-12b-zh\n```\n\n### 2. Local install: Environment Check\u002FDownloading\u002FInstallation\n#### a. Environment Check\nWe have verified EasyAnimate execution on the following environment:\n\nThe detailed of Windows:\n- OS: Windows 10\n- python: python3.10 & python3.11\n- pytorch: torch2.2.0\n- CUDA: 11.8 & 12.1\n- CUDNN: 8+\n- GPU： Nvidia-3060 12G\n\nThe detailed of Linux:\n- OS: Ubuntu 20.04, CentOS\n- python: python3.10 & python3.11\n- pytorch: torch2.2.0\n- CUDA: 11.8 & 12.1\n- CUDNN: 8+\n- GPU：Nvidia-V100 16G & Nvidia-A10 24G & Nvidia-A100 40G & Nvidia-A100 80G\n\nWe need about 60GB available on disk (for saving weights), please check!\n\nThe video size for EasyAnimateV5.1-12B can be generated by different GPU Memory, including:\n| GPU memory | 384x672x25 | 384x672x49 | 576x1008x25 | 576x1008x49 | 768x1344x25 | 768x1344x49 |\n|------------|------------|------------|------------|------------|------------|------------|\n| 16GB | 🧡 | ⭕️ | ⭕️ | ⭕️ | ❌ | ❌ |\n| 24GB | 🧡 | 🧡 | 🧡 | 🧡 | 🧡 | ❌ | \n| 40GB | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | \n| 80GB | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | \n\nThe video size for EasyAnimateV5.1-7B can be generated by different GPU Memory, including:\n| GPU memory |384x672x25|384x672x49|576x1008x25|576x1008x49|768x1344x25|768x1344x49|\n|----------|----------|----------|----------|----------|----------|----------|\n| 16GB | 🧡 | 🧡 | ⭕️ | ⭕️ | ❌ | ❌ | \n| 24GB | ✅ | ✅ | ✅ | 🧡 | 🧡 | ❌ | \n| 40GB | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | \n| 80GB | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | \n\n✅ indicates it can run under \"model_cpu_offload\", 🧡 represents it can run under \"model_cpu_offload_and_qfloat8\", ⭕️ indicates it can run under \"sequential_cpu_offload\", ❌ means it can't run. Please note that running with sequential_cpu_offload will be slower.\n\nSome GPUs that do not support torch.bfloat16, such as 2080ti and V100, require changing the weight_dtype in app.py and predict files to torch.float16 in order to run.\n\nThe generation time for EasyAnimateV5.1-12B using different GPUs over 25 steps is as follows:\n\n| GPU       | 384x672x25       | 384x672x49       | 576x1008x25      | 576x1008x49      | 768x1344x25      | 768x1344x49     |\n|-----------|------------------|------------------|------------------|------------------|------------------|-----------------|\n| A10 24GB  | ~120s (4.8s\u002Fit)  | ~240s (9.6s\u002Fit)  | ~320s (12.7s\u002Fit) | ~750s (29.8s\u002Fit) | ❌               | ❌              |\n| A100 80GB | ~45s (1.75s\u002Fit)  | ~90s (3.7s\u002Fit)   | ~120s (4.7s\u002Fit)  | ~300s (11.4s\u002Fit) | ~265s (10.6s\u002Fit) | ~710s (28.3s\u002Fit) |\n\n\u003Cdetails>\n  \u003Csummary>(Obsolete) EasyAnimateV3:\u003C\u002Fsummary>\n  \nThe video size for EasyAnimateV3 can be generated by different GPU Memory, including:\n\n| GPU memory | 384x672x72 | 384x672x144 | 576x1008x72 | 576x1008x144 | 720x1280x72 | 720x1280x144 |\n|------------|------------|-------------|-------------|--------------|-------------|--------------|\n| 12GB       | ⭕️         | ⭕️          | ⭕️          | ⭕️           | ❌          | ❌           |\n| 16GB       | ✅         | ✅          | ⭕️          | ⭕️           | ⭕️          | ❌           |\n| 24GB       | ✅         | ✅          | ✅          | ✅           | ✅          | ❌           |\n| 40GB       | ✅         | ✅          | ✅          | ✅           | ✅          | ✅           |\n| 80GB       | ✅         | ✅          | ✅          | ✅           | ✅          | ✅           |\n\n(⭕️) indicates it can run with low_gpu_memory_mode=True, but at a slower speed, and ❌ means it can't run.\n\u003C\u002Fdetails>\n\n#### b. Weights\nWe'd better place the [weights](#model-zoo) along the specified path:\n\nEasyAnimateV5.1:\n```\n📦 models\u002F\n├── 📂 Diffusion_Transformer\u002F\n│   ├── 📂 EasyAnimateV5.1-12b-zh-InP\u002F\n│   └── 📂 EasyAnimateV5.1-12b-zh\u002F\n├── 📂 Personalized_Model\u002F\n│   └── your trained trainformer model \u002F your trained lora model (for UI load)\n```\n\n# Video Result\n\n### Image to Video with EasyAnimateV5.1-12b-zh-InP\n\u003Ctable border=\"0\" style=\"width: 100%; text-align: left; margin-top: 20px;\">\n  \u003Ctr>\n      \u003Ctd>\n          \u003Cvideo src=\"https:\u002F\u002Fgithub.com\u002Fuser-attachments\u002Fassets\u002F74a23109-f555-4026-a3d8-1ac27bb3884c\" width=\"100%\" controls autoplay loop>\u003C\u002Fvideo>\n      \u003C\u002Ftd>\n      \u003Ctd>\n          \u003Cvideo src=\"https:\u002F\u002Fgithub.com\u002Fuser-attachments\u002Fassets\u002Fab5aab27-fbd7-4f55-add9-29644125bde7\" width=\"100%\" controls autoplay loop>\u003C\u002Fvideo>\n      \u003C\u002Ftd>\n       \u003Ctd>\n          \u003Cvideo src=\"https:\u002F\u002Fgithub.com\u002Fuser-attachments\u002Fassets\u002F238043c2-cdbd-4288-9857-a273d96f021f\" width=\"100%\" controls autoplay loop>\u003C\u002Fvideo>\n     \u003C\u002Ftd>\n      \u003Ctd>\n          \u003Cvideo src=\"https:\u002F\u002Fgithub.com\u002Fuser-attachments\u002Fassets\u002F48881a0e-5513-4482-ae49-13a0ad7a2557\" width=\"100%\" controls autoplay loop>\u003C\u002Fvideo>\n     \u003C\u002Ftd>\n  \u003C\u002Ftr>\n\u003C\u002Ftable>\n\n\n\u003Ctable border=\"0\" style=\"width: 100%; text-align: left; margin-top: 20px;\">\n  \u003Ctr>\n      \u003Ctd>\n          \u003Cvideo src=\"https:\u002F\u002Fgithub.com\u002Fuser-attachments\u002Fassets\u002F3e7aba7f-6232-4f39-80a8-6cfae968f38c\" width=\"100%\" controls autoplay loop>\u003C\u002Fvideo>\n      \u003C\u002Ftd>\n      \u003Ctd>\n          \u003Cvideo src=\"https:\u002F\u002Fgithub.com\u002Fuser-attachments\u002Fassets\u002F986d9f77-8dc3-45fa-bc9d-8b26023fffbc\" width=\"100%\" controls autoplay loop>\u003C\u002Fvideo>\n      \u003C\u002Ftd>\n       \u003Ctd>\n          \u003Cvideo src=\"https:\u002F\u002Fgithub.com\u002Fuser-attachments\u002Fassets\u002F7f62795a-2b3b-4c14-aeb1-1230cb818067\" width=\"100%\" controls autoplay loop>\u003C\u002Fvideo>\n     \u003C\u002Ftd>\n      \u003Ctd>\n          \u003Cvideo src=\"https:\u002F\u002Fgithub.com\u002Fuser-attachments\u002Fassets\u002Fb581df84-ade1-4605-a7a8-fd735ce3e222\" width=\"100%\" controls autoplay loop>\u003C\u002Fvideo>\n     \u003C\u002Ftd>\n  \u003C\u002Ftr>\n\u003C\u002Ftable>\n\n\u003Ctable border=\"0\" style=\"width: 100%; text-align: left; margin-top: 20px;\">\n  \u003Ctr>\n      \u003Ctd>\n          \u003Cvideo src=\"https:\u002F\u002Fgithub.com\u002Fuser-attachments\u002Fassets\u002Feab1db91-1082-4de2-bb0a-d97fd25ceea1\" width=\"100%\" controls autoplay loop>\u003C\u002Fvideo>\n      \u003C\u002Ftd>\n      \u003Ctd>\n          \u003Cvideo src=\"https:\u002F\u002Fgithub.com\u002Fuser-attachments\u002Fassets\u002F3fda0e96-c1a8-4186-9c4c-043e11420f05\" width=\"100%\" controls autoplay loop>\u003C\u002Fvideo>\n      \u003C\u002Ftd>\n       \u003Ctd>\n          \u003Cvideo src=\"https:\u002F\u002Fgithub.com\u002Fuser-attachments\u002Fassets\u002F4b53145d-7e98-493a-83c9-4ea4f5b58289\" width=\"100%\" controls autoplay loop>\u003C\u002Fvideo>\n     \u003C\u002Ftd>\n      \u003Ctd>\n          \u003Cvideo src=\"https:\u002F\u002Fgithub.com\u002Fuser-attachments\u002Fassets\u002F75f7935f-17a8-4e20-b24c-b61479cf07fc\" width=\"100%\" controls autoplay loop>\u003C\u002Fvideo>\n     \u003C\u002Ftd>\n  \u003C\u002Ftr>\n\u003C\u002Ftable>\n\n### Text to Video with EasyAnimateV5.1-12b-zh\n\n\u003Ctable border=\"0\" style=\"width: 100%; text-align: left; margin-top: 20px;\">\n  \u003Ctr>\n      \u003Ctd>\n          \u003Cvideo src=\"https:\u002F\u002Fgithub.com\u002Fuser-attachments\u002Fassets\u002F8818dae8-e329-4b08-94fa-00d923f38fd2\" width=\"100%\" controls autoplay loop>\u003C\u002Fvideo>\n      \u003C\u002Ftd>\n      \u003Ctd>\n          \u003Cvideo src=\"https:\u002F\u002Fgithub.com\u002Fuser-attachments\u002Fassets\u002Fd3e483c3-c710-47d2-9fac-89f732f2260a\" width=\"100%\" controls autoplay loop>\u003C\u002Fvideo>\n      \u003C\u002Ftd>\n       \u003Ctd>\n          \u003Cvideo src=\"https:\u002F\u002Fgithub.com\u002Fuser-attachments\u002Fassets\u002F4dfa2067-d5d4-4741-a52c-97483de1050d\" width=\"100%\" controls autoplay loop>\u003C\u002Fvideo>\n     \u003C\u002Ftd>\n      \u003Ctd>\n          \u003Cvideo src=\"https:\u002F\u002Fgithub.com\u002Fuser-attachments\u002Fassets\u002Ffb44c2db-82c6-427e-9297-97dcce9a4948\" width=\"100%\" controls autoplay loop>\u003C\u002Fvideo>\n     \u003C\u002Ftd>\n  \u003C\u002Ftr>\n\u003C\u002Ftable>\n\n\u003Ctable border=\"0\" style=\"width: 100%; text-align: left; margin-top: 20px;\">\n  \u003Ctr>\n      \u003Ctd>\n          \u003Cvideo src=\"https:\u002F\u002Fgithub.com\u002Fuser-attachments\u002Fassets\u002Fdc6b8eaf-f21b-4576-a139-0e10438f20e4\" width=\"100%\" controls autoplay loop>\u003C\u002Fvideo>\n      \u003C\u002Ftd>\n      \u003Ctd>\n          \u003Cvideo src=\"https:\u002F\u002Fgithub.com\u002Fuser-attachments\u002Fassets\u002Fb3f8fd5b-c5c8-44ee-9b27-49105a08fbff\" width=\"100%\" controls autoplay loop>\u003C\u002Fvideo>\n      \u003C\u002Ftd>\n       \u003Ctd>\n          \u003Cvideo src=\"https:\u002F\u002Fgithub.com\u002Fuser-attachments\u002Fassets\u002Fa68ed61b-eed3-41d2-b208-5f039bf2788e\" width=\"100%\" controls autoplay loop>\u003C\u002Fvideo>\n     \u003C\u002Ftd>\n      \u003Ctd>\n          \u003Cvideo src=\"https:\u002F\u002Fgithub.com\u002Fuser-attachments\u002Fassets\u002F4e33f512-0126-4412-9ae8-236ff08bcd21\" width=\"100%\" controls autoplay loop>\u003C\u002Fvideo>\n     \u003C\u002Ftd>\n  \u003C\u002Ftr>\n\u003C\u002Ftable>\n\n### Control Video with EasyAnimateV5.1-12b-zh-Control\n\nTrajectory Control:\n\u003Ctable border=\"0\" style=\"width: 100%; text-align: left; margin-top: 20px;\">\n  \u003Ctr>\n      \u003Ctd>\n          \u003Cvideo src=\"https:\u002F\u002Fgithub.com\u002Fuser-attachments\u002Fassets\u002Fbf3b8970-ca7b-447f-8301-72dfe028055b\" width=\"100%\" controls autoplay loop>\u003C\u002Fvideo>\n      \u003C\u002Ftd>\n      \u003Ctd>\n          \u003Cvideo src=\"https:\u002F\u002Fgithub.com\u002Fuser-attachments\u002Fassets\u002F63a7057b-573e-4f73-9d7b-8f8001245af4\" width=\"100%\" controls autoplay loop>\u003C\u002Fvideo>\n      \u003C\u002Ftd>\n       \u003Ctd>\n          \u003Cvideo src=\"https:\u002F\u002Fgithub.com\u002Fuser-attachments\u002Fassets\u002F090ac2f3-1a76-45cf-abe5-4e326113389b\" width=\"100%\" controls autoplay loop>\u003C\u002Fvideo>\n     \u003C\u002Ftd>\n  \u003Ctr>\n\u003C\u002Ftable>\n\nGeneric Control Video (Canny, Pose, Depth, etc.):\n\u003Ctable border=\"0\" style=\"width: 100%; text-align: left; margin-top: 20px;\">\n  \u003Ctr>\n      \u003Ctd>\n          \u003Cvideo src=\"https:\u002F\u002Fgithub.com\u002Fuser-attachments\u002Fassets\u002F53002ce2-dd18-4d4f-8135-b6f68364cabd\" width=\"100%\" controls autoplay loop>\u003C\u002Fvideo>\n      \u003C\u002Ftd>\n      \u003Ctd>\n          \u003Cvideo src=\"https:\u002F\u002Fgithub.com\u002Fuser-attachments\u002Fassets\u002Ffce43c0b-81fa-4ab2-9ca7-78d786f520e6\" width=\"100%\" controls autoplay loop>\u003C\u002Fvideo>\n      \u003C\u002Ftd>\n       \u003Ctd>\n          \u003Cvideo src=\"https:\u002F\u002Fgithub.com\u002Fuser-attachments\u002Fassets\u002Fb208b92c-5add-4ece-a200-3dbbe47b93c3\" width=\"100%\" controls autoplay loop>\u003C\u002Fvideo>\n     \u003C\u002Ftd>\n  \u003Ctr>\n      \u003Ctd>\n          \u003Cvideo src=\"https:\u002F\u002Fgithub.com\u002Fuser-attachments\u002Fassets\u002F3aec95d5-d240-49fb-a9e9-914446c7a4cf\" width=\"100%\" controls autoplay loop>\u003C\u002Fvideo>\n      \u003C\u002Ftd>\n      \u003Ctd>\n          \u003Cvideo src=\"https:\u002F\u002Fgithub.com\u002Fuser-attachments\u002Fassets\u002F60fa063b-5c1f-485f-b663-09bd6669de3f\" width=\"100%\" controls autoplay loop>\u003C\u002Fvideo>\n      \u003C\u002Ftd>\n       \u003Ctd>\n          \u003Cvideo src=\"https:\u002F\u002Fgithub.com\u002Fuser-attachments\u002Fassets\u002F4adde728-8397-42f3-8a2a-23f7b39e9a1e\" width=\"100%\" controls autoplay loop>\u003C\u002Fvideo>\n     \u003C\u002Ftd>\n  \u003C\u002Ftr>\n\u003C\u002Ftable>\n\n### Camera Control with EasyAnimateV5.1-12b-zh-Control-Camera\n\n\u003Ctable border=\"0\" style=\"width: 100%; text-align: left; margin-top: 20px;\">\n  \u003Ctr>\n      \u003Ctd>\n          Pan Up\n      \u003C\u002Ftd>\n      \u003Ctd>\n          Pan Left\n      \u003C\u002Ftd>\n       \u003Ctd>\n          Pan Right\n     \u003C\u002Ftd>\n  \u003Ctr>\n      \u003Ctd>\n          \u003Cvideo src=\"https:\u002F\u002Fgithub.com\u002Fuser-attachments\u002Fassets\u002Fa88f81da-e263-4038-a5b3-77b26f79719e\" width=\"100%\" controls autoplay loop>\u003C\u002Fvideo>\n      \u003C\u002Ftd>\n      \u003Ctd>\n          \u003Cvideo src=\"https:\u002F\u002Fgithub.com\u002Fuser-attachments\u002Fassets\u002Fe346c59d-7bca-4253-97fb-8cbabc484afb\" width=\"100%\" controls autoplay loop>\u003C\u002Fvideo>\n      \u003C\u002Ftd>\n       \u003Ctd>\n          \u003Cvideo src=\"https:\u002F\u002Fgithub.com\u002Fuser-attachments\u002Fassets\u002F4de470d4-47b7-46e3-82d3-b714a2f6aef6\" width=\"100%\" controls autoplay loop>\u003C\u002Fvideo>\n     \u003C\u002Ftd>\n  \u003Ctr>\n      \u003Ctd>\n          Pan Down\n      \u003C\u002Ftd>\n      \u003Ctd>\n          Pan Up + Pan Left\n      \u003C\u002Ftd>\n       \u003Ctd>\n          Pan Up + Pan Right\n     \u003C\u002Ftd>\n  \u003Ctr>\n      \u003Ctd>\n          \u003Cvideo src=\"https:\u002F\u002Fgithub.com\u002Fuser-attachments\u002Fassets\u002F7a3fecc2-d41a-4de3-86cd-5e19aea34a0d\" width=\"100%\" controls autoplay loop>\u003C\u002Fvideo>\n      \u003C\u002Ftd>\n      \u003Ctd>\n          \u003Cvideo src=\"https:\u002F\u002Fgithub.com\u002Fuser-attachments\u002Fassets\u002Fcb281259-28b6-448e-a76f-643c3465672e\" width=\"100%\" controls autoplay loop>\u003C\u002Fvideo>\n      \u003C\u002Ftd>\n       \u003Ctd>\n          \u003Cvideo src=\"https:\u002F\u002Fgithub.com\u002Fuser-attachments\u002Fassets\u002F44faf5b6-d83c-4646-9436-971b2b9c7216\" width=\"100%\" controls autoplay loop>\u003C\u002Fvideo>\n     \u003C\u002Ftd>\n  \u003C\u002Ftr>\n\u003C\u002Ftable>\n\n# How to use\n\n\u003Ch3 id=\"video-gen\">1. Inference \u003C\u002Fh3>\n\n#### a. Memory-Saving Options\nSince EasyAnimateV5 and V5.1 have very large parameters, we need to consider memory-saving options to adapt to consumer-grade graphics cards. We provide GPU_memory_mode for each prediction file, allowing you to choose from model_cpu_offload, model_cpu_offload_and_qfloat8, or sequential_cpu_offload.\n\n- model_cpu_offload means the entire model will move to the CPU after use, saving some memory.\n- model_cpu_offload_and_qfloat8 means the entire model will move to the CPU after use and applies float8 quantization to the transformer model, saving more memory.\n- sequential_cpu_offload means each layer of the model moves to CPU after use, which is slower but saves a lot of memory.\n\nqfloat8 may reduce model performance but saves more memory. If memory is sufficient, it's recommended to use model_cpu_offload.\n\n#### b. Via ComfyUI\nFor more details, see the [ComfyUI README](comfyui\u002FREADME.md).\n\n#### c. Run Python Files\n- Step 1: Download the corresponding [weights](#model-zoo) and place them in the models folder.\n- Step 2: Use different files for predictions based on the weights and prediction goals.\n  - Text-to-Video:\n    - Modify the prompt, neg_prompt, guidance_scale, and seed in the predict_t2v.py file.\n    - Then run the predict_t2v.py file and wait for the results, which are stored in the samples\u002Feasyanimate-videos folder.\n  - Image-to-Video:\n    - Modify validation_image_start, validation_image_end, prompt, neg_prompt, guidance_scale, and seed in the predict_i2v.py file.\n    - validation_image_start is the starting image, and validation_image_end is the ending image of the video.\n    - Then run the predict_i2v.py file and wait for the results, which are stored in the samples\u002Feasyanimate-videos_i2v folder.\n  - Video-to-Video:\n    - Modify validation_video, validation_image_end, prompt, neg_prompt, guidance_scale, and seed in the predict_v2v.py file.\n    - validation_video is the reference video for video-to-video. You can run a demo with the following video: [Demo Video](https:\u002F\u002Fpai-aigc-photog.oss-cn-hangzhou.aliyuncs.com\u002Fcogvideox_fun\u002Fasset\u002Fv1\u002Fplay_guitar.mp4)\n    - Then run the predict_v2v.py file and wait for the results, which are stored in samples\u002Feasyanimate-videos_v2v folder.\n  - Generic Control Video (Canny, Pose, Depth, etc.):\n    - Modify control_video, validation_image_end, prompt, neg_prompt, guidance_scale, and seed in the predict_v2v_control.py file.\n    - control_video is the control video for video generation, extracted using Canny, Pose, Depth, etc. You can run a demo with the following video: [Demo Video](https:\u002F\u002Fpai-aigc-photog.oss-cn-hangzhou.aliyuncs.com\u002Fcogvideox_fun\u002Fasset\u002Fv1.1\u002Fpose.mp4)\n    - Then run the predict_v2v_control.py file and wait for the results, which are stored in samples\u002Feasyanimate-videos_v2v_control folder.\n  - Trajectory Control Video:\n    - Modify control_video, ref_image, validation_image_end, prompt, neg_prompt, guidance_scale, and seed in the predict_v2v_control.py file.\n    - control_video is the control video, and ref_image is the reference first frame image. You can run a demo with the following image and video: [Demo Image](https:\u002F\u002Fpai-aigc-photog.oss-cn-hangzhou.aliyuncs.com\u002Feasyanimate\u002Fasset\u002Fv5.1\u002Fdog.png), [Demo Video](https:\u002F\u002Fpai-aigc-photog.oss-cn-hangzhou.aliyuncs.com\u002Feasyanimate\u002Fasset\u002Fv5.1\u002Ftrajectory_demo.mp4)\n    - Then run the predict_v2v_control.py file and wait for the results, which are stored in samples\u002Feasyanimate-videos_v2v_control folder.\n    - Interaction via ComfyUI is recommended.\n  - Camera Control Video:\n    - Modify control_video, ref_image, validation_image_end, prompt, neg_prompt, guidance_scale, and seed in the predict_v2v_control.py file.\n    - control_camera_txt is the control file for camera control video, and ref_image is the reference first frame image. You can run a demo with the following image and control file: [Demo Image](https:\u002F\u002Fpai-aigc-photog.oss-cn-hangzhou.aliyuncs.com\u002Fcogvideox_fun\u002Fasset\u002Fv1\u002Ffirework.png), [Demo File (from CameraCtrl)](https:\u002F\u002Fpai-aigc-photog.oss-cn-hangzhou.aliyuncs.com\u002Feasyanimate\u002Fasset\u002Fv5.1\u002F0a3b5fb184936a83.txt)\n    - Then run the predict_v2v_control.py file and wait for the results, which are stored in samples\u002Feasyanimate-videos_v2v_control folder.\n    - Interaction via ComfyUI is recommended.\n- Step 3: To combine with other backbones and Lora trained by yourself, modify predict_t2v.py and lora_path accordingly in the predict_t2v.py file.\n\n#### d. Via WebUI Interface\n\nWebUI supports text-to-video, image-to-video, video-to-video, and control-based video generation (such as Canny, Pose, Depth, etc.).\n\n- Step 1: Download the corresponding [weights](#model-zoo) and place them in the models folder.\n- Step 2: Run the app.py file to enter the Gradio page.\n- Step 3: Choose the generation model from the page, fill in prompt, neg_prompt, guidance_scale, seed, etc., click generate, and wait for the results, which are stored in the sample folder.\n\n\n### 2. Model Training\nA complete EasyAnimate training pipeline should include data preprocessing, Video VAE training, and Video DiT training. Among these, Video VAE training is optional because we have already provided a pre-trained Video VAE.\n\n\u003Ch4 id=\"data-preprocess\">a. data preprocessing\u003C\u002Fh4>\n\nWe provide two simple demos:\n- Train a Lora model using image data. For more details, you can refer to the [wiki](https:\u002F\u002Fgithub.com\u002Faigc-apps\u002FEasyAnimate\u002Fwiki\u002FTraining-Lora).\n- Perform SFT model training using video data. For more details, you can refer to the [wiki](https:\u002F\u002Fgithub.com\u002Faigc-apps\u002FEasyAnimate\u002Fwiki\u002FTraining-SFT).\n\nA complete data preprocessing link for long video segmentation, cleaning, and description can refer to [README](.\u002Feasyanimate\u002Fvideo_caption\u002FREADME.md) in the video captions section. \n\nIf you want to train a text to image and video generation model. You need to arrange the dataset in this format.\n\n```\n📦 project\u002F\n├── 📂 datasets\u002F\n│   ├── 📂 internal_datasets\u002F\n│       ├── 📂 train\u002F\n│       │   ├── 📄 00000001.mp4\n│       │   ├── 📄 00000002.jpg\n│       │   └── 📄 .....\n│       └── 📄 json_of_internal_datasets.json\n```\n\nThe json_of_internal_datasets.json is a standard JSON file. The file_path in the json can to be set as relative path, as shown in below:\n```json\n[\n    {\n      \"file_path\": \"train\u002F00000001.mp4\",\n      \"text\": \"A group of young men in suits and sunglasses are walking down a city street.\",\n      \"type\": \"video\"\n    },\n    {\n      \"file_path\": \"train\u002F00000002.jpg\",\n      \"text\": \"A group of young men in suits and sunglasses are walking down a city street.\",\n      \"type\": \"image\"\n    },\n    .....\n]\n```\n\nYou can also set the path as absolute path as follow:\n```json\n[\n    {\n      \"file_path\": \"\u002Fmnt\u002Fdata\u002Fvideos\u002F00000001.mp4\",\n      \"text\": \"A group of young men in suits and sunglasses are walking down a city street.\",\n      \"type\": \"video\"\n    },\n    {\n      \"file_path\": \"\u002Fmnt\u002Fdata\u002Ftrain\u002F00000001.jpg\",\n      \"text\": \"A group of young men in suits and sunglasses are walking down a city street.\",\n      \"type\": \"image\"\n    },\n    .....\n]\n```\n\n\u003Ch4 id=\"vae-train\">b. Video VAE training (optional)\u003C\u002Fh4>\n\nVideo VAE training is an optional option as we have already provided pre trained Video VAEs.\nIf you want to train video vae, you can refer to [README](easyanimate\u002Fvae\u002FREADME.md) in the video vae section.\n\n\u003Ch4 id=\"dit-train\">c. Video DiT training \u003C\u002Fh4>\n \nIf the data format is relative path during data preprocessing, please set ```scripts\u002Ftrain.sh``` as follow.\n```\nexport DATASET_NAME=\"datasets\u002Finternal_datasets\u002F\"\nexport DATASET_META_NAME=\"datasets\u002Finternal_datasets\u002Fjson_of_internal_datasets.json\"\n```\n\nIf the data format is absolute path during data preprocessing, please set ```scripts\u002Ftrain.sh``` as follow.\n```\nexport DATASET_NAME=\"\"\nexport DATASET_META_NAME=\"\u002Fmnt\u002Fdata\u002Fjson_of_internal_datasets.json\"\n```\n\nThen, we run scripts\u002Ftrain.sh.\n```sh\nsh scripts\u002Ftrain.sh\n```\n\nFor details on setting some parameters, please refer to [Readme Train](scripts\u002FREADME_TRAIN.md) and [Readme Lora](scripts\u002FREADME_TRAIN_LORA.md). \n\n\u003Cdetails>\n  \u003Csummary>(Obsolete) EasyAnimateV1:\u003C\u002Fsummary>\n  If you want to train EasyAnimateV1. Please switch to the git branch v1.\n\u003C\u002Fdetails>\n\n\n# Model zoo\n\nEasyAnimateV5.1:\n\n7B:\n| Name | Type | Storage Space | Hugging Face | Model Scope | Description |\n|--|--|--|--|--|--|\n| EasyAnimateV5.1-7b-zh-InP | EasyAnimateV5.1 | 30 GB | [🤗Link](https:\u002F\u002Fhuggingface.co\u002Falibaba-pai\u002FEasyAnimateV5.1-7b-zh-InP) | [😄Link](https:\u002F\u002Fmodelscope.cn\u002Fmodels\u002FPAI\u002FEasyAnimateV5.1-7b-zh-InP) | Official image-to-video weights. Supports video prediction at multiple resolutions (512, 768, 1024), trained with 49 frames at 8 frames per second, and supports for multilingual prediction. |\n| EasyAnimateV5.1-7b-zh-Control | EasyAnimateV5.1 | 30 GB | [🤗Link](https:\u002F\u002Fhuggingface.co\u002Falibaba-pai\u002FEasyAnimateV5.1-7b-zh-Control) | [😄Link](https:\u002F\u002Fmodelscope.cn\u002Fmodels\u002FPAI\u002FEasyAnimateV5.1-7b-zh-Control) | Official video control weights, supporting various control conditions such as Canny, Depth, Pose, MLSD, and trajectory control. Supports video prediction at multiple resolutions (512, 768, 1024), trained with 49 frames at 8 frames per second, and supports for multilingual prediction. |\n| EasyAnimateV5.1-7b-zh-Control-Camera | EasyAnimateV5.1 | 30 GB | [🤗Link](https:\u002F\u002Fhuggingface.co\u002Falibaba-pai\u002FEasyAnimateV5.1-7b-zh-Control-Camera) | [😄Link](https:\u002F\u002Fmodelscope.cn\u002Fmodels\u002FPAI\u002FEasyAnimateV5.1-7b-zh-Control-Camera) | Official video camera control weights, supporting direction generation control by inputting camera motion trajectories. Supports video prediction at multiple resolutions (512, 768, 1024), trained with 49 frames at 8 frames per second, and supports for multilingual prediction. |\n| EasyAnimateV5.1-7b-zh | EasyAnimateV5.1 | 30 GB | [🤗Link](https:\u002F\u002Fhuggingface.co\u002Falibaba-pai\u002FEasyAnimateV5.1-7b-zh) | [😄Link](https:\u002F\u002Fmodelscope.cn\u002Fmodels\u002FPAI\u002FEasyAnimateV5.1-7b-zh) | Official text-to-video weights. Supports video prediction at multiple resolutions (512, 768, 1024), trained with 49 frames at 8 frames per second, and supports for multilingual prediction. |\n\n12B:\n| Name | Type | Storage Space | Hugging Face | Model Scope | Description |\n|--|--|--|--|--|--|\n| EasyAnimateV5.1-12b-zh-InP | EasyAnimateV5.1 | 39 GB | [🤗Link](https:\u002F\u002Fhuggingface.co\u002Falibaba-pai\u002FEasyAnimateV5.1-12b-zh-InP) | [😄Link](https:\u002F\u002Fmodelscope.cn\u002Fmodels\u002FPAI\u002FEasyAnimateV5.1-12b-zh-InP) | Official image-to-video weights. Supports video prediction at multiple resolutions (512, 768, 1024), trained with 49 frames at 8 frames per second, and supports for multilingual prediction. |\n| EasyAnimateV5.1-12b-zh-Control | EasyAnimateV5.1 | 39 GB | [🤗Link](https:\u002F\u002Fhuggingface.co\u002Falibaba-pai\u002FEasyAnimateV5.1-12b-zh-Control) | [😄Link](https:\u002F\u002Fmodelscope.cn\u002Fmodels\u002FPAI\u002FEasyAnimateV5.1-12b-zh-Control) | Official video control weights, supporting various control conditions such as Canny, Depth, Pose, MLSD, and trajectory control. Supports video prediction at multiple resolutions (512, 768, 1024), trained with 49 frames at 8 frames per second, and supports for multilingual prediction. |\n| EasyAnimateV5.1-12b-zh-Control-Camera | EasyAnimateV5.1 | 39 GB | [🤗Link](https:\u002F\u002Fhuggingface.co\u002Falibaba-pai\u002FEasyAnimateV5.1-12b-zh-Control-Camera) | [😄Link](https:\u002F\u002Fmodelscope.cn\u002Fmodels\u002FPAI\u002FEasyAnimateV5.1-12b-zh-Control-Camera) | Official video camera control weights, supporting direction generation control by inputting camera motion trajectories. Supports video prediction at multiple resolutions (512, 768, 1024), trained with 49 frames at 8 frames per second, and supports for multilingual prediction. |\n| EasyAnimateV5.1-12b-zh | EasyAnimateV5.1 | 39 GB | [🤗Link](https:\u002F\u002Fhuggingface.co\u002Falibaba-pai\u002FEasyAnimateV5.1-12b-zh) | [😄Link](https:\u002F\u002Fmodelscope.cn\u002Fmodels\u002FPAI\u002FEasyAnimateV5.1-12b-zh) | Official text-to-video weights. Supports video prediction at multiple resolutions (512, 768, 1024), trained with 49 frames at 8 frames per second, and supports for multilingual prediction. |\n\n\u003Cdetails>\n  \u003Csummary>(Obsolete) EasyAnimateV5:\u003C\u002Fsummary>\n\n7B:\n| Name | Type | Storage Space | Hugging Face | Model Scope | Description |\n|--|--|--|--|--|--|\n| EasyAnimateV5-7b-zh-InP | EasyAnimateV5 | 22 GB | [🤗Link](https:\u002F\u002Fhuggingface.co\u002Falibaba-pai\u002FEasyAnimateV5-7b-zh-InP) | [😄Link](https:\u002F\u002Fmodelscope.cn\u002Fmodels\u002FPAI\u002FEasyAnimateV5-7b-zh-InP) | Official 7B image-to-video weights. Supports video prediction at multiple resolutions (512, 768, 1024), trained with 49 frames at 8 frames per second, and supports bilingual prediction in Chinese and English. |\n| EasyAnimateV5-7b-zh | EasyAnimateV5 | 22 GB | [🤗Link](https:\u002F\u002Fhuggingface.co\u002Falibaba-pai\u002FEasyAnimateV5-7b-zh) | [😄Link](https:\u002F\u002Fmodelscope.cn\u002Fmodels\u002FPAI\u002FEasyAnimateV5-7b-zh) | Official 7B text-to-video weights. Supports video prediction at multiple resolutions (512, 768, 1024), trained with 49 frames at 8 frames per second, and supports bilingual prediction in Chinese and English. |\n| EasyAnimateV5-Reward-LoRAs | EasyAnimateV5 | - | [🤗Link](https:\u002F\u002Fhuggingface.co\u002Falibaba-pai\u002FEasyAnimateV5-Reward-LoRAs) | [😄Link](https:\u002F\u002Fmodelscope.cn\u002Fmodels\u002FPAI\u002FEasyAnimateV5-Reward-LoRAs) | The official reward backpropagation technology model optimizes the videos generated by EasyAnimateV5-12b to better match human preferences. ｜\n\n12B:\n| Name | Type | Storage Space | Hugging Face | Model Scope | Description |\n|--|--|--|--|--|--|\n| EasyAnimateV5-12b-zh-InP | EasyAnimateV5 | 34 GB | [🤗Link](https:\u002F\u002Fhuggingface.co\u002Falibaba-pai\u002FEasyAnimateV5-12b-zh-InP) | [😄Link](https:\u002F\u002Fmodelscope.cn\u002Fmodels\u002FPAI\u002FEasyAnimateV5-12b-zh-InP) | Official image-to-video weights. Supports video prediction at multiple resolutions (512, 768, 1024), trained with 49 frames at 8 frames per second, and supports bilingual prediction in Chinese and English. |\n| EasyAnimateV5-12b-zh-Control | EasyAnimateV5 | 34 GB | [🤗Link](https:\u002F\u002Fhuggingface.co\u002Falibaba-pai\u002FEasyAnimateV5-12b-zh-Control) | [😄Link](https:\u002F\u002Fmodelscope.cn\u002Fmodels\u002FPAI\u002FEasyAnimateV5-12b-zh-Control) | Official video control weights, supporting various control conditions such as Canny, Depth, Pose, MLSD, etc. Supports video prediction at multiple resolutions (512, 768, 1024) and is trained with 49 frames at 8 frames per second. Bilingual prediction in Chinese and English is supported. |\n| EasyAnimateV5-12b-zh | EasyAnimateV5 | 34 GB | [🤗Link](https:\u002F\u002Fhuggingface.co\u002Falibaba-pai\u002FEasyAnimateV5-12b-zh) | [😄Link](https:\u002F\u002Fmodelscope.cn\u002Fmodels\u002FPAI\u002FEasyAnimateV5-12b-zh) | Official text-to-video weights. Supports video prediction at multiple resolutions (512, 768, 1024), trained with 49 frames at 8 frames per second, and supports bilingual prediction in Chinese and English. |\n| EasyAnimateV5-Reward-LoRAs | EasyAnimateV5 | - | [🤗Link](https:\u002F\u002Fhuggingface.co\u002Falibaba-pai\u002FEasyAnimateV5-Reward-LoRAs) | [😄Link](https:\u002F\u002Fmodelscope.cn\u002Fmodels\u002FPAI\u002FEasyAnimateV5-Reward-LoRAs) | The official reward backpropagation technology model optimizes the videos generated by EasyAnimateV5-12b to better match human preferences. ｜\n\u003C\u002Fdetails>\n\n\u003Cdetails>\n  \u003Csummary>(Obsolete) EasyAnimateV4:\u003C\u002Fsummary>\n\n| Name | Type | Storage Space | Hugging Face | Model Scope | Description |\n|--|--|--|--|--|--|\n| EasyAnimateV4-XL-2-InP | EasyAnimateV4 | Before extraction: 8.9 GB \\\u002F After extraction: 14.0 GB |[🤗Link](https:\u002F\u002Fhuggingface.co\u002Falibaba-pai\u002FEasyAnimateV4-XL-2-InP)| [😄Link](https:\u002F\u002Fmodelscope.cn\u002Fmodels\u002FPAI\u002FEasyAnimateV4-XL-2-InP)| | Our official graph-generated video model is capable of predicting videos at multiple resolutions (512, 768, 1024, 1280) and has been trained on 144 frames at a rate of 24 frames per second. |\n\u003C\u002Fdetails>\n\n\u003Cdetails>\n  \u003Csummary>(Obsolete) EasyAnimateV3:\u003C\u002Fsummary>\n\n| Name | Type | Storage Space | Hugging Face | Model Scope | Description |\n|--|--|--|--|--|--|\n| EasyAnimateV3-XL-2-InP-512x512 | EasyAnimateV3 | 18.2GB | [🤗Link](https:\u002F\u002Fhuggingface.co\u002Falibaba-pai\u002FEasyAnimateV3-XL-2-InP-512x512)| [😄Link](https:\u002F\u002Fmodelscope.cn\u002Fmodels\u002FPAI\u002FEasyAnimateV3-XL-2-InP-512x512) | EasyAnimateV3 official weights for 512x512 text and image to video resolution. Training with 144 frames and fps 24 |\n| EasyAnimateV3-XL-2-InP-768x768 | EasyAnimateV3 | 18.2GB | [🤗Link](https:\u002F\u002Fhuggingface.co\u002Falibaba-pai\u002FEasyAnimateV3-XL-2-InP-768x768) | [😄Link](https:\u002F\u002Fmodelscope.cn\u002Fmodels\u002FPAI\u002FEasyAnimateV3-XL-2-InP-768x768) | EasyAnimateV3 official weights for 768x768 text and image to video resolution. Training with 144 frames and fps 24 |\n| EasyAnimateV3-XL-2-InP-960x960 | EasyAnimateV3 | 18.2GB | [🤗Link](https:\u002F\u002Fhuggingface.co\u002Falibaba-pai\u002FEasyAnimateV3-XL-2-InP-960x960) | [😄Link](https:\u002F\u002Fmodelscope.cn\u002Fmodels\u002FPAI\u002FEasyAnimateV3-XL-2-InP-960x960) | EasyAnimateV3 official weights for 960x960 text and  image to video resolution. Training with 144 frames and fps 24 |\n\u003C\u002Fdetails>\n\n\u003Cdetails>\n  \u003Csummary>(Obsolete) EasyAnimateV2:\u003C\u002Fsummary>\n\n| Name | Type | Storage Space | Url | Hugging Face | Model Scope | Description |\n|--|--|--|--|--|--|--|\n| EasyAnimateV2-XL-2-512x512 | EasyAnimateV2 | 16.2GB | - | [🤗Link](https:\u002F\u002Fhuggingface.co\u002Falibaba-pai\u002FEasyAnimateV2-XL-2-512x512)| [😄Link](https:\u002F\u002Fmodelscope.cn\u002Fmodels\u002FPAI\u002FEasyAnimateV2-XL-2-512x512)| EasyAnimateV2 official weights for 512x512 resolution. Training with 144 frames and fps 24 |\n| EasyAnimateV2-XL-2-768x768 | EasyAnimateV2 | 16.2GB | - | [🤗Link](https:\u002F\u002Fhuggingface.co\u002Falibaba-pai\u002FEasyAnimateV2-XL-2-768x768) | [😄Link](https:\u002F\u002Fmodelscope.cn\u002Fmodels\u002FPAI\u002FEasyAnimateV2-XL-2-768x768)| EasyAnimateV2 official weights for 768x768 resolution. Training with 144 frames and fps 24 |\n| easyanimatev2_minimalism_lora.safetensors | Lora of Pixart | 485.1MB | [Download](https:\u002F\u002Fpai-aigc-photog.oss-cn-hangzhou.aliyuncs.com\u002Feasyanimate\u002FPersonalized_Model\u002Feasyanimatev2_minimalism_lora.safetensors)| - | - | A lora training with a specifial type images. Images can be downloaded from [Url](https:\u002F\u002Fpai-aigc-photog.oss-cn-hangzhou.aliyuncs.com\u002Feasyanimate\u002Fasset\u002Fv2\u002FMinimalism.zip). |\n\u003C\u002Fdetails>\n\n\u003Cdetails>\n  \u003Csummary>(Obsolete) EasyAnimateV1:\u003C\u002Fsummary>\n\n### 1、Motion Weights\n| Name | Type | Storage Space | Url | Description |\n|--|--|--|--|--| \n| easyanimate_v1_mm.safetensors | Motion Module | 4.1GB | [download](https:\u002F\u002Fpai-aigc-photog.oss-cn-hangzhou.aliyuncs.com\u002Feasyanimate\u002FMotion_Module\u002Feasyanimate_v1_mm.safetensors) | Training with 80 frames and fps 12 |\n\n### 2、Other Weights\n| Name | Type | Storage Space | Url | Description |\n|--|--|--|--|--| \n| PixArt-XL-2-512x512.tar | Pixart | 11.4GB | [download](https:\u002F\u002Fpai-aigc-photog.oss-cn-hangzhou.aliyuncs.com\u002Feasyanimate\u002FDiffusion_Transformer\u002FPixArt-XL-2-512x512.tar)| Pixart-Alpha official weights |\n| easyanimate_portrait.safetensors | Checkpoint of Pixart | 2.3GB | [download](https:\u002F\u002Fpai-aigc-photog.oss-cn-hangzhou.aliyuncs.com\u002Feasyanimate\u002FPersonalized_Model\u002Feasyanimate_portrait.safetensors) | Training with internal portrait datasets |\n| easyanimate_portrait_lora.safetensors | Lora of Pixart | 654.0MB | [download](https:\u002F\u002Fpai-aigc-photog.oss-cn-hangzhou.aliyuncs.com\u002Feasyanimate\u002FPersonalized_Model\u002Feasyanimate_portrait_lora.safetensors)| Training with internal portrait datasets |\n\u003C\u002Fdetails>\n\n# TODO List\n- Support model with larger params.\n\n# Contact Us\n1. Use Dingding to search group 77450006752 or Scan to join\n2. You need to scan the image to join the WeChat group or if it is expired, add this student as a friend first to invite you.\n\n\u003Cimg src=\"https:\u002F\u002Fpai-aigc-photog.oss-cn-hangzhou.aliyuncs.com\u002Feasyanimate\u002Fasset\u002Fgroup\u002Fdd.png\" alt=\"ding group\" width=\"30%\"\u002F>\n\u003Cimg src=\"https:\u002F\u002Fpai-aigc-photog.oss-cn-hangzhou.aliyuncs.com\u002Feasyanimate\u002Fasset\u002Fgroup\u002Fwechat.jpg\" alt=\"Wechat group\" width=\"30%\"\u002F>\n\u003Cimg src=\"https:\u002F\u002Fpai-aigc-photog.oss-cn-hangzhou.aliyuncs.com\u002Feasyanimate\u002Fasset\u002Fgroup\u002Fperson.jpg\" alt=\"Person\" width=\"30%\"\u002F>\n\n\n# Reference\n- CogVideo: https:\u002F\u002Fgithub.com\u002FTHUDM\u002FCogVideo\u002F\n- Flux: https:\u002F\u002Fgithub.com\u002Fblack-forest-labs\u002Fflux\n- magvit: https:\u002F\u002Fgithub.com\u002Fgoogle-research\u002Fmagvit\n- PixArt: https:\u002F\u002Fgithub.com\u002FPixArt-alpha\u002FPixArt-alpha\n- Open-Sora-Plan: https:\u002F\u002Fgithub.com\u002FPKU-YuanGroup\u002FOpen-Sora-Plan\n- Open-Sora: https:\u002F\u002Fgithub.com\u002Fhpcaitech\u002FOpen-Sora\n- Animatediff: https:\u002F\u002Fgithub.com\u002Fguoyww\u002FAnimateDiff\n- HunYuan DiT: https:\u002F\u002Fgithub.com\u002Ftencent\u002FHunyuanDiT\n- ComfyUI-KJNodes: https:\u002F\u002Fgithub.com\u002Fkijai\u002FComfyUI-KJNodes\n- ComfyUI-EasyAnimateWrapper: https:\u002F\u002Fgithub.com\u002Fkijai\u002FComfyUI-EasyAnimateWrapper\n- ComfyUI-CameraCtrl-Wrapper: https:\u002F\u002Fgithub.com\u002Fchaojie\u002FComfyUI-CameraCtrl-Wrapper\n- CameraCtrl: https:\u002F\u002Fgithub.com\u002Fhehao13\u002FCameraCtrl\n- DragAnything: https:\u002F\u002Fgithub.com\u002Fshowlab\u002FDragAnything\n\n# License\nThis project is licensed under the [Apache License (Version 2.0)](https:\u002F\u002Fgithub.com\u002Fmodelscope\u002Fmodelscope\u002Fblob\u002Fmaster\u002FLICENSE).\n","EasyAnimate 是一个基于Transformer架构的端到端解决方案，用于生成高分辨率和长视频。该项目利用DIT和Transformer作为扩散模型来处理视频和图像生成，并通过训练VAE来处理长视频数据，同时支持元数据预处理。其核心功能包括直接从预训练模型预测生成视频、用户自定义风格转换模型训练等。特别适合需要高质量AI生成内容的应用场景，如影视制作、广告创意设计以及多媒体内容创作等领域。此外，项目提供了丰富的资源链接，便于开发者快速上手使用。","2026-06-11 03:42:26","high_star"]