[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"project-70948":3},{"id":4,"name":5,"fullName":6,"owner":7,"repo":5,"description":8,"homepage":9,"htmlUrl":10,"language":11,"languages":10,"totalLinesOfCode":10,"stars":12,"forks":13,"watchers":14,"openIssues":15,"contributorsCount":16,"subscribersCount":16,"size":16,"stars1d":17,"stars7d":18,"stars30d":19,"stars90d":16,"forks30d":16,"starsTrendScore":20,"compositeScore":21,"rankGlobal":10,"rankLanguage":10,"license":22,"archived":23,"fork":23,"defaultBranch":24,"hasWiki":25,"hasPages":23,"topics":26,"createdAt":10,"pushedAt":10,"updatedAt":33,"readmeContent":34,"aiSummary":35,"trendingCount":16,"starSnapshotCount":16,"syncStatus":36,"lastSyncTime":37,"discoverSource":38},70948,"CogVideo","zai-org\u002FCogVideo","zai-org","text and image to video generation: CogVideoX (2024) and CogVideo (ICLR 2023)","",null,"Python",12775,1305,137,104,0,11,23,58,33,106.65,"Apache License 2.0",false,"main",true,[27,28,29,30,31,32],"cogvideox","image-to-video","llm","sora","text-to-video","video-generation","2026-06-12 04:00:58","# CogVideo & CogVideoX\n\n[中文阅读](.\u002FREADME_zh.md)\n\n[日本語で読む](.\u002FREADME_ja.md)\n\n\u003Cdiv align=\"center\">\n\u003Cimg src=resources\u002Flogo.svg width=\"50%\"\u002F>\n\u003C\u002Fdiv>\n\u003Cp align=\"center\">\nExperience the CogVideoX-5B model online at \u003Ca href=\"https:\u002F\u002Fhuggingface.co\u002Fspaces\u002FTHUDM\u002FCogVideoX-5B\" target=\"_blank\"> 🤗 Huggingface Space\u003C\u002Fa> or \u003Ca href=\"https:\u002F\u002Fmodelscope.cn\u002Fstudios\u002FZhipuAI\u002FCogVideoX-5b-demo\" target=\"_blank\"> 🤖 ModelScope Space\u003C\u002Fa>\n\u003C\u002Fp>\n\u003Cp align=\"center\">\n📚 View the \u003Ca href=\"https:\u002F\u002Farxiv.org\u002Fabs\u002F2408.06072\" target=\"_blank\">paper\u003C\u002Fa> and \u003Ca href=\"https:\u002F\u002Fzhipu-ai.feishu.cn\u002Fwiki\u002FDHCjw1TrJiTyeukfc9RceoSRnCh\" target=\"_blank\">user guide\u003C\u002Fa>\n\u003C\u002Fp>\n\u003Cp align=\"center\">\n    👋 Join our \u003Ca href=\"resources\u002FWECHAT.md\" target=\"_blank\">WeChat\u003C\u002Fa> and \u003Ca href=\"https:\u002F\u002Fdiscord.gg\u002FdCGfUsagrD\" target=\"_blank\">Discord\u003C\u002Fa>\n\u003C\u002Fp>\n\u003Cp align=\"center\">\n📍 Visit \u003Ca href=\"https:\u002F\u002Fchatglm.cn\u002Fvideo?lang=en?fr=osm_cogvideo\">QingYing\u003C\u002Fa> and \u003Ca href=\"https:\u002F\u002Fopen.bigmodel.cn\u002F?utm_campaign=open&_channel_track_key=OWTVNma9\">API Platform\u003C\u002Fa> to experience larger-scale commercial video generation models.\n\u003C\u002Fp>\n\n## Project Updates\n\n- 🔥🔥 **News**: ```2025\u002F03\u002F24```: We have launched [CogKit](https:\u002F\u002Fgithub.com\u002FTHUDM\u002FCogKit), a fine-tuning and inference framework for the **CogView4** and **CogVideoX** series. This toolkit allows you to fully explore and utilize our multimodal generation models.\n- 🔥 **News**: ```2025\u002F02\u002F28```: DDIM Inverse is now supported in `CogVideoX-5B` and `CogVideoX1.5-5B`. Check [here](inference\u002Fddim_inversion.py).\n- 🔥 **News**: ```2025\u002F01\u002F08```: We have updated the code for `Lora` fine-tuning based on the `diffusers` version model, which uses less GPU memory. For more details, please see [here](finetune\u002FREADME.md).\n- 🔥 **News**: ```2024\u002F11\u002F15```: We released the `CogVideoX1.5` model in the diffusers version. Only minor parameter adjustments are needed to continue using previous code.\n- 🔥 **News**: ```2024\u002F11\u002F08```: We have released the CogVideoX1.5 model. CogVideoX1.5 is an upgraded version of the open-source model CogVideoX.\nThe CogVideoX1.5-5B series supports 10-second videos with higher resolution, and CogVideoX1.5-5B-I2V supports video generation at any resolution.\nThe SAT code has already been updated, while the diffusers version is still under adaptation. Download the SAT version code [here](https:\u002F\u002Fhuggingface.co\u002FTHUDM\u002FCogVideoX1.5-5B-SAT).\n- 🔥 **News**: ```2024\u002F10\u002F13```: A more cost-effective fine-tuning framework for `CogVideoX-5B` that works with a single\n  4090 GPU, [cogvideox-factory](https:\u002F\u002Fgithub.com\u002Fa-r-r-o-w\u002Fcogvideox-factory), has been released. It supports\n  fine-tuning with multiple resolutions. Feel free to use it!\n- 🔥 **News**: ```2024\u002F10\u002F10```: We have updated our technical report. Please\n  click [here](https:\u002F\u002Farxiv.org\u002Fpdf\u002F2408.06072) to view it. More training details and a demo have been added. To see\n  the demo, click [here](https:\u002F\u002Fyzy-thu.github.io\u002FCogVideoX-demo\u002F).- 🔥 **News**: ```2024\u002F10\u002F09```: We have publicly\n  released the [technical documentation](https:\u002F\u002Fzhipu-ai.feishu.cn\u002Fwiki\u002FDHCjw1TrJiTyeukfc9RceoSRnCh) for CogVideoX\n  fine-tuning on Feishu, further increasing distribution flexibility. All examples in the public documentation can be\n  fully reproduced.\n- 🔥 **News**: ```2024\u002F9\u002F19```: We have open-sourced the CogVideoX series image-to-video model **CogVideoX-5B-I2V**.\n  This model can take an image as a background input and generate a video combined with prompt words, offering greater\n  controllability. With this, the CogVideoX series models now support three tasks: text-to-video generation, video\n  continuation, and image-to-video generation. Welcome to try it online\n  at [Experience](https:\u002F\u002Fhuggingface.co\u002Fspaces\u002FTHUDM\u002FCogVideoX-5B-Space).\n- 🔥 ```2024\u002F9\u002F19```: The Caption\n  model [CogVLM2-Caption](https:\u002F\u002Fhuggingface.co\u002FTHUDM\u002Fcogvlm2-llama3-caption), used in the training process of\n  CogVideoX to convert video data into text descriptions, has been open-sourced. Welcome to download and use it.\n- 🔥 ```2024\u002F8\u002F27```: We have open-sourced a larger model in the CogVideoX series, **CogVideoX-5B**. We have\n  significantly optimized the model's inference performance, greatly lowering the inference threshold.\n  You can run **CogVideoX-2B** on older GPUs like `GTX 1080TI`, and **CogVideoX-5B** on desktop GPUs like `RTX 3060`. Please strictly\n  follow the [requirements](requirements.txt) to update and install dependencies, and refer\n  to [cli_demo](inference\u002Fcli_demo.py) for inference code. Additionally, the open-source license for\n  the **CogVideoX-2B** model has been changed to the **Apache 2.0 License**.\n- 🔥 ```2024\u002F8\u002F6```: We have open-sourced **3D Causal VAE**, used for **CogVideoX-2B**, which can reconstruct videos with\n  almost no loss.\n- 🔥 ```2024\u002F8\u002F6```: We have open-sourced the first model of the CogVideoX series video generation models, **CogVideoX-2B\n  **.\n- 🌱 **Source**: ```2022\u002F5\u002F19```: We have open-sourced the CogVideo video generation model (now you can see it in\n  the `CogVideo` branch). This is the first open-source large Transformer-based text-to-video generation model. You can\n  access the [ICLR'23 paper](https:\u002F\u002Farxiv.org\u002Fabs\u002F2205.15868) for technical details.\n\n## Table of Contents\n\nJump to a specific section:\n\n- [Quick Start](#quick-start)\n  - [Prompt Optimization](#prompt-optimization)\n  - [SAT](#sat)\n  - [Diffusers](#diffusers)\n- [Gallery](#gallery)\n  - [CogVideoX-5B](#cogvideox-5b)\n  - [CogVideoX-2B](#cogvideox-2b)\n- [Model Introduction](#model-introduction)\n- [Friendly Links](#friendly-links)\n- [Project Structure](#project-structure)\n  - [Quick Start with Colab](#quick-start-with-colab)\n  - [Inference](#inference)\n  - [finetune](#finetune)\n  - [sat](#sat-1)\n  - [Tools](#tools)\n- [CogVideo(ICLR'23)](#cogvideoiclr23)\n- [Citation](#citation)\n- [Model-License](#model-license)\n\n## Quick Start\n\n### Prompt Optimization\n\nBefore running the model, please refer to [this guide](inference\u002Fconvert_demo.py) to see how we use large models like\nGLM-4 (or other comparable products, such as GPT-4) to optimize the model. This is crucial because the model is trained\nwith long prompts, and a good prompt directly impacts the quality of the video generation.\n\n### SAT\n\n**Please make sure your Python version is between 3.10 and 3.12, inclusive of both 3.10 and 3.12.**\n\nFollow instructions in [sat_demo](sat\u002FREADME.md): Contains the inference code and fine-tuning code of SAT weights. It is\nrecommended to improve based on the CogVideoX model structure. Innovative researchers use this code to better perform\nrapid stacking and development.\n\n### Diffusers\n\n**Please make sure your Python version is between 3.10 and 3.12, inclusive of both 3.10 and 3.12.**\n\n```\npip install -r requirements.txt\n```\n\nThen follow [diffusers_demo](inference\u002Fcli_demo.py): A more detailed explanation of the inference code, mentioning the\nsignificance of common parameters.\n\nFor more details on quantized inference, please refer\nto [diffusers-torchao](https:\u002F\u002Fgithub.com\u002Fsayakpaul\u002Fdiffusers-torchao\u002F). With Diffusers and TorchAO, quantized inference\nis also possible leading to memory-efficient inference as well as speedup in some cases when compiled. A full list of\nmemory and time benchmarks with various settings on A100 and H100 has been published\nat [diffusers-torchao](https:\u002F\u002Fgithub.com\u002Fsayakpaul\u002Fdiffusers-torchao).\n\n## Gallery\n\n### CogVideoX-5B\n\n\u003Ctable border=\"0\" style=\"width: 100%; text-align: left; margin-top: 20px;\">\n  \u003Ctr>\n      \u003Ctd>\n          \u003Cvideo src=\"https:\u002F\u002Fgithub.com\u002Fuser-attachments\u002Fassets\u002Fcf5953ea-96d3-48fd-9907-c4708752c714\" width=\"100%\" controls autoplay loop>\u003C\u002Fvideo>\n      \u003C\u002Ftd>\n      \u003Ctd>\n          \u003Cvideo src=\"https:\u002F\u002Fgithub.com\u002Fuser-attachments\u002Fassets\u002Ffe0a78e6-b669-4800-8cf0-b5f9b5145b52\" width=\"100%\" controls autoplay loop>\u003C\u002Fvideo>\n      \u003C\u002Ftd>\n       \u003Ctd>\n          \u003Cvideo src=\"https:\u002F\u002Fgithub.com\u002Fuser-attachments\u002Fassets\u002Fc182f606-8f8c-421d-b414-8487070fcfcb\" width=\"100%\" controls autoplay loop>\u003C\u002Fvideo>\n     \u003C\u002Ftd>\n      \u003Ctd>\n          \u003Cvideo src=\"https:\u002F\u002Fgithub.com\u002Fuser-attachments\u002Fassets\u002F7db2bbce-194d-434d-a605-350254b6c298\" width=\"100%\" controls autoplay loop>\u003C\u002Fvideo>\n     \u003C\u002Ftd>\n  \u003C\u002Ftr>\n  \u003Ctr>\n      \u003Ctd>\n          \u003Cvideo src=\"https:\u002F\u002Fgithub.com\u002Fuser-attachments\u002Fassets\u002F62b01046-8cab-44cc-bd45-4d965bb615ec\" width=\"100%\" controls autoplay loop>\u003C\u002Fvideo>\n      \u003C\u002Ftd>\n      \u003Ctd>\n          \u003Cvideo src=\"https:\u002F\u002Fgithub.com\u002Fuser-attachments\u002Fassets\u002Fd78e552a-4b3f-4b81-ac3f-3898079554f6\" width=\"100%\" controls autoplay loop>\u003C\u002Fvideo>\n      \u003C\u002Ftd>\n       \u003Ctd>\n          \u003Cvideo src=\"https:\u002F\u002Fgithub.com\u002Fuser-attachments\u002Fassets\u002F30894f12-c741-44a2-9e6e-ddcacc231e5b\" width=\"100%\" controls autoplay loop>\u003C\u002Fvideo>\n     \u003C\u002Ftd>\n      \u003Ctd>\n          \u003Cvideo src=\"https:\u002F\u002Fgithub.com\u002Fuser-attachments\u002Fassets\u002F926575ca-7150-435b-a0ff-4900a963297b\" width=\"100%\" controls autoplay loop>\u003C\u002Fvideo>\n     \u003C\u002Ftd>\n  \u003C\u002Ftr>\n\u003C\u002Ftable>\n\n### CogVideoX-2B\n\n\u003Ctable border=\"0\" style=\"width: 100%; text-align: left; margin-top: 20px;\">\n  \u003Ctr>\n      \u003Ctd>\n          \u003Cvideo src=\"https:\u002F\u002Fgithub.com\u002Fuser-attachments\u002Fassets\u002Fea3af39a-3160-4999-90ec-2f7863c5b0e9\" width=\"100%\" controls autoplay loop>\u003C\u002Fvideo>\n      \u003C\u002Ftd>\n      \u003Ctd>\n          \u003Cvideo src=\"https:\u002F\u002Fgithub.com\u002Fuser-attachments\u002Fassets\u002F9de41efd-d4d1-4095-aeda-246dd834e91d\" width=\"100%\" controls autoplay loop>\u003C\u002Fvideo>\n      \u003C\u002Ftd>\n       \u003Ctd>\n          \u003Cvideo src=\"https:\u002F\u002Fgithub.com\u002Fuser-attachments\u002Fassets\u002F941d6661-6a8d-4a1b-b912-59606f0b2841\" width=\"100%\" controls autoplay loop>\u003C\u002Fvideo>\n     \u003C\u002Ftd>\n      \u003Ctd>\n          \u003Cvideo src=\"https:\u002F\u002Fgithub.com\u002Fuser-attachments\u002Fassets\u002F938529c4-91ae-4f60-b96b-3c3947fa63cb\" width=\"100%\" controls autoplay loop>\u003C\u002Fvideo>\n     \u003C\u002Ftd>\n  \u003C\u002Ftr>\n\u003C\u002Ftable>\n\nTo view the corresponding prompt words for the gallery, please click [here](resources\u002Fgalary_prompt.md)\n\n## Model Introduction\n\nCogVideoX is an open-source version of the video generation model originating\nfrom [QingYing](https:\u002F\u002Fchatglm.cn\u002Fvideo?lang=en?fr=osm_cogvideo). The table below displays the list of video generation\nmodels we currently offer, along with their foundational information.\n\n\u003Ctable style=\"border-collapse: collapse; width: 100%;\">\n  \u003Ctr>\n    \u003Cth style=\"text-align: center;\">Model Name\u003C\u002Fth>\n    \u003Cth style=\"text-align: center;\">CogVideoX1.5-5B (Latest)\u003C\u002Fth>\n    \u003Cth style=\"text-align: center;\">CogVideoX1.5-5B-I2V (Latest)\u003C\u002Fth>\n    \u003Cth style=\"text-align: center;\">CogVideoX-2B\u003C\u002Fth>\n    \u003Cth style=\"text-align: center;\">CogVideoX-5B\u003C\u002Fth>\n    \u003Cth style=\"text-align: center;\">CogVideoX-5B-I2V\u003C\u002Fth>\n  \u003C\u002Ftr>\n  \u003Ctr>\n    \u003Ctd style=\"text-align: center;\">Release Date\u003C\u002Ftd>\n    \u003Cth style=\"text-align: center;\">November 8, 2024\u003C\u002Fth>\n    \u003Cth style=\"text-align: center;\">November 8, 2024\u003C\u002Fth>\n    \u003Cth style=\"text-align: center;\">August 6, 2024\u003C\u002Fth>\n    \u003Cth style=\"text-align: center;\">August 27, 2024\u003C\u002Fth>\n    \u003Cth style=\"text-align: center;\">September 19, 2024\u003C\u002Fth>\n  \u003C\u002Ftr>\n  \u003Ctr>\n    \u003Ctd style=\"text-align: center;\">Video Resolution\u003C\u002Ftd>\n    \u003Ctd colspan=\"1\" style=\"text-align: center;\">1360 * 768\u003C\u002Ftd>\n    \u003Ctd colspan=\"1\" style=\"text-align: center;\"> Min(W, H) = 768 \u003Cbr> 768 ≤ Max(W, H) ≤ 1360 \u003Cbr> Max(W, H) % 16 = 0 \u003C\u002Ftd>\n    \u003Ctd colspan=\"3\" style=\"text-align: center;\">720 * 480\u003C\u002Ftd>\n  \u003C\u002Ftr>\n  \u003Ctr>\n    \u003Ctd style=\"text-align: center;\">Number of Frames\u003C\u002Ftd>\n    \u003Ctd colspan=\"2\" style=\"text-align: center;\">Should be \u003Cb>16N + 1\u003C\u002Fb> where N \u003C= 10 (default 81)\u003C\u002Ftd>\n    \u003Ctd colspan=\"3\" style=\"text-align: center;\">Should be \u003Cb>8N + 1\u003C\u002Fb> where N \u003C= 6 (default 49)\u003C\u002Ftd>\n  \u003C\u002Ftr>\n  \u003Ctr>\n    \u003Ctd style=\"text-align: center;\">Inference Precision\u003C\u002Ftd>\n    \u003Ctd colspan=\"2\" style=\"text-align: center;\">\u003Cb>BF16 (Recommended)\u003C\u002Fb>, FP16, FP32, FP8*, INT8, Not supported: INT4\u003C\u002Ftd>\n    \u003Ctd style=\"text-align: center;\">\u003Cb>FP16*(Recommended)\u003C\u002Fb>, BF16, FP32, FP8*, INT8, Not supported: INT4\u003C\u002Ftd>\n    \u003Ctd colspan=\"2\" style=\"text-align: center;\">\u003Cb>BF16 (Recommended)\u003C\u002Fb>, FP16, FP32, FP8*, INT8, Not supported: INT4\u003C\u002Ftd>\n  \u003C\u002Ftr>\n  \u003Ctr>\n    \u003Ctd style=\"text-align: center;\">Single GPU Memory Usage\u003Cbr>\u003C\u002Ftd>\n    \u003Ctd colspan=\"2\" style=\"text-align: center;\">\u003Ca href=\"https:\u002F\u002Fgithub.com\u002FTHUDM\u002FSwissArmyTransformer\">SAT\u003C\u002Fa> BF16: 76GB \u003Cbr>\u003Cb>diffusers BF16: from 10GB*\u003C\u002Fb>\u003Cbr>\u003Cb>diffusers INT8(torchao): from 7GB*\u003C\u002Fb>\u003C\u002Ftd>\n    \u003Ctd style=\"text-align: center;\">\u003Ca href=\"https:\u002F\u002Fgithub.com\u002FTHUDM\u002FSwissArmyTransformer\">SAT\u003C\u002Fa> FP16: 18GB \u003Cbr>\u003Cb>diffusers FP16: 4GB minimum* \u003C\u002Fb>\u003Cbr>\u003Cb>diffusers INT8 (torchao): 3.6GB minimum*\u003C\u002Fb>\u003C\u002Ftd>\n    \u003Ctd colspan=\"2\" style=\"text-align: center;\">\u003Ca href=\"https:\u002F\u002Fgithub.com\u002FTHUDM\u002FSwissArmyTransformer\">SAT\u003C\u002Fa> BF16: 26GB \u003Cbr>\u003Cb>diffusers BF16 : 5GB minimum* \u003C\u002Fb>\u003Cbr>\u003Cb>diffusers INT8 (torchao): 4.4GB minimum* \u003C\u002Fb>\u003C\u002Ftd>\n  \u003C\u002Ftr>\n  \u003Ctr>\n    \u003Ctd style=\"text-align: center;\">Multi-GPU Memory Usage\u003C\u002Ftd>\n    \u003Ctd colspan=\"2\" style=\"text-align: center;\">\u003Cb>BF16: 24GB* using diffusers\u003C\u002Fb>\u003Cbr>\u003C\u002Ftd>\n    \u003Ctd style=\"text-align: center;\">\u003Cb>FP16: 10GB* using diffusers\u003C\u002Fb>\u003Cbr>\u003C\u002Ftd>\n    \u003Ctd colspan=\"2\" style=\"text-align: center;\">\u003Cb>BF16: 15GB* using diffusers\u003C\u002Fb>\u003Cbr>\u003C\u002Ftd>\n  \u003C\u002Ftr>\n  \u003Ctr>\n    \u003Ctd style=\"text-align: center;\">Inference Speed\u003Cbr>(Step = 50, FP\u002FBF16)\u003C\u002Ftd>\n    \u003Ctd colspan=\"2\" style=\"text-align: center;\">Single A100: ~1000 seconds (5-second video)\u003Cbr>Single H100: ~550 seconds (5-second video)\u003C\u002Ftd>\n    \u003Ctd style=\"text-align: center;\">Single A100: ~90 seconds\u003Cbr>Single H100: ~45 seconds\u003C\u002Ftd>\n    \u003Ctd colspan=\"2\" style=\"text-align: center;\">Single A100: ~180 seconds\u003Cbr>Single H100: ~90 seconds\u003C\u002Ftd>\n  \u003C\u002Ftr>\n  \u003Ctr>\n    \u003Ctd style=\"text-align: center;\">Prompt Language\u003C\u002Ftd>\n    \u003Ctd colspan=\"5\" style=\"text-align: center;\">English*\u003C\u002Ftd>\n  \u003C\u002Ftr>\n  \u003Ctr>\n    \u003Ctd style=\"text-align: center;\">Prompt Token Limit\u003C\u002Ftd>\n    \u003Ctd colspan=\"2\" style=\"text-align: center;\">224 Tokens\u003C\u002Ftd>\n    \u003Ctd colspan=\"3\" style=\"text-align: center;\">226 Tokens\u003C\u002Ftd>\n  \u003C\u002Ftr>\n  \u003Ctr>\n    \u003Ctd style=\"text-align: center;\">Video Length\u003C\u002Ftd>\n    \u003Ctd colspan=\"2\" style=\"text-align: center;\">5 seconds or 10 seconds\u003C\u002Ftd>\n    \u003Ctd colspan=\"3\" style=\"text-align: center;\">6 seconds\u003C\u002Ftd>\n  \u003C\u002Ftr>\n  \u003Ctr>\n    \u003Ctd style=\"text-align: center;\">Frame Rate\u003C\u002Ftd>\n    \u003Ctd colspan=\"2\" style=\"text-align: center;\">16 frames \u002F second \u003C\u002Ftd>\n    \u003Ctd colspan=\"3\" style=\"text-align: center;\">8 frames \u002F second \u003C\u002Ftd>\n  \u003C\u002Ftr>\n  \u003Ctr>\n    \u003Ctd style=\"text-align: center;\">Position Encoding\u003C\u002Ftd>\n    \u003Ctd colspan=\"2\" style=\"text-align: center;\">3d_rope_pos_embed\u003C\u002Ftd>\n    \u003Ctd style=\"text-align: center;\">3d_sincos_pos_embed\u003C\u002Ftd>\n    \u003Ctd style=\"text-align: center;\">3d_rope_pos_embed\u003C\u002Ftd>\n    \u003Ctd style=\"text-align: center;\">3d_rope_pos_embed + learnable_pos_embed\u003C\u002Ftd>\n  \u003C\u002Ftr>\n  \u003Ctr>\n    \u003Ctd style=\"text-align: center;\">Download Link (Diffusers)\u003C\u002Ftd>\n    \u003Ctd style=\"text-align: center;\">\u003Ca href=\"https:\u002F\u002Fhuggingface.co\u002FTHUDM\u002FCogVideoX1.5-5B\">🤗 HuggingFace\u003C\u002Fa>\u003Cbr>\u003Ca href=\"https:\u002F\u002Fmodelscope.cn\u002Fmodels\u002FZhipuAI\u002FCogVideoX1.5-5B\">🤖 ModelScope\u003C\u002Fa>\u003Cbr>\u003Ca href=\"https:\u002F\u002Fwisemodel.cn\u002Fmodels\u002FZhipuAI\u002FCogVideoX1.5-5B\">🟣 WiseModel\u003C\u002Fa>\u003C\u002Ftd>\n    \u003Ctd style=\"text-align: center;\">\u003Ca href=\"https:\u002F\u002Fhuggingface.co\u002FTHUDM\u002FCogVideoX1.5-5B-I2V\">🤗 HuggingFace\u003C\u002Fa>\u003Cbr>\u003Ca href=\"https:\u002F\u002Fmodelscope.cn\u002Fmodels\u002FZhipuAI\u002FCogVideoX1.5-5B-I2V\">🤖 ModelScope\u003C\u002Fa>\u003Cbr>\u003Ca href=\"https:\u002F\u002Fwisemodel.cn\u002Fmodels\u002FZhipuAI\u002FCogVideoX1.5-5B-I2V\">🟣 WiseModel\u003C\u002Fa>\u003C\u002Ftd>\n    \u003Ctd style=\"text-align: center;\">\u003Ca href=\"https:\u002F\u002Fhuggingface.co\u002FTHUDM\u002FCogVideoX-2b\">🤗 HuggingFace\u003C\u002Fa>\u003Cbr>\u003Ca href=\"https:\u002F\u002Fmodelscope.cn\u002Fmodels\u002FZhipuAI\u002FCogVideoX-2b\">🤖 ModelScope\u003C\u002Fa>\u003Cbr>\u003Ca href=\"https:\u002F\u002Fwisemodel.cn\u002Fmodels\u002FZhipuAI\u002FCogVideoX-2b\">🟣 WiseModel\u003C\u002Fa>\u003C\u002Ftd>\n    \u003Ctd style=\"text-align: center;\">\u003Ca href=\"https:\u002F\u002Fhuggingface.co\u002FTHUDM\u002FCogVideoX-5b\">🤗 HuggingFace\u003C\u002Fa>\u003Cbr>\u003Ca href=\"https:\u002F\u002Fmodelscope.cn\u002Fmodels\u002FZhipuAI\u002FCogVideoX-5b\">🤖 ModelScope\u003C\u002Fa>\u003Cbr>\u003Ca href=\"https:\u002F\u002Fwisemodel.cn\u002Fmodels\u002FZhipuAI\u002FCogVideoX-5b\">🟣 WiseModel\u003C\u002Fa>\u003C\u002Ftd>\n    \u003Ctd style=\"text-align: center;\">\u003Ca href=\"https:\u002F\u002Fhuggingface.co\u002FTHUDM\u002FCogVideoX-5b-I2V\">🤗 HuggingFace\u003C\u002Fa>\u003Cbr>\u003Ca href=\"https:\u002F\u002Fmodelscope.cn\u002Fmodels\u002FZhipuAI\u002FCogVideoX-5b-I2V\">🤖 ModelScope\u003C\u002Fa>\u003Cbr>\u003Ca href=\"https:\u002F\u002Fwisemodel.cn\u002Fmodels\u002FZhipuAI\u002FCogVideoX-5b-I2V\">🟣 WiseModel\u003C\u002Fa>\u003C\u002Ftd>\n  \u003C\u002Ftr>\n  \u003Ctr>\n    \u003Ctd style=\"text-align: center;\">Download Link (SAT)\u003C\u002Ftd>\n    \u003Ctd colspan=\"2\" style=\"text-align: center;\">\u003Ca href=\"https:\u002F\u002Fhuggingface.co\u002FTHUDM\u002FCogVideoX1.5-5b-SAT\">🤗 HuggingFace\u003C\u002Fa>\u003Cbr>\u003Ca href=\"https:\u002F\u002Fmodelscope.cn\u002Fmodels\u002FZhipuAI\u002FCogVideoX1.5-5b-SAT\">🤖 ModelScope\u003C\u002Fa>\u003Cbr>\u003Ca href=\"https:\u002F\u002Fwisemodel.cn\u002Fmodels\u002FZhipuAI\u002FCogVideoX1.5-5b-SAT\">🟣 WiseModel\u003C\u002Fa>\u003C\u002Ftd>\n    \u003Ctd colspan=\"3\" style=\"text-align: center;\">\u003Ca href=\".\u002Fsat\u002FREADME_zh.md\">SAT\u003C\u002Fa>\u003C\u002Ftd>\n  \u003C\u002Ftr>\n\u003C\u002Ftable>\n\n**Data Explanation**\n\n+ While testing using the diffusers library, all optimizations included in the diffusers library were enabled. This\n  scheme has not been tested for actual memory usage on devices outside of **NVIDIA A100 \u002F H100** architectures.\n  Generally, this scheme can be adapted to all **NVIDIA Ampere architecture** and above devices. If optimizations are\n  disabled, memory consumption will multiply, with peak memory usage being about 3 times the value in the table.\n  However, speed will increase by about 3-4 times. You can selectively disable some optimizations, including:\n\n```\npipe.enable_sequential_cpu_offload()\npipe.vae.enable_slicing()\npipe.vae.enable_tiling()\n```\n\n+ For multi-GPU inference, the `enable_sequential_cpu_offload()` optimization needs to be disabled.\n+ Using INT8 models will slow down inference, which is done to accommodate lower-memory GPUs while maintaining minimal\n  video quality loss, though inference speed will significantly decrease.\n+ The CogVideoX-2B model was trained in `FP16` precision, and all CogVideoX-5B models were trained in `BF16` precision.\n  We recommend using the precision in which the model was trained for inference.\n+ [PytorchAO](https:\u002F\u002Fgithub.com\u002Fpytorch\u002Fao) and [Optimum-quanto](https:\u002F\u002Fgithub.com\u002Fhuggingface\u002Foptimum-quanto\u002F) can be\n  used to quantize the text encoder, transformer, and VAE modules to reduce the memory requirements of CogVideoX. This\n  allows the model to run on free T4 Colabs or GPUs with smaller memory! Also, note that TorchAO quantization is fully\n  compatible with `torch.compile`, which can significantly improve inference speed. FP8 precision must be used on\n  devices with NVIDIA H100 and above, requiring source installation of `torch`, `torchao` Python packages. CUDA 12.4 is recommended.\n+ The inference speed tests also used the above memory optimization scheme. Without memory optimization, inference speed\n  increases by about 10%. Only the `diffusers` version of the model supports quantization.\n+ The model only supports English input; other languages can be translated into English for use via large model\n  refinement.\n\n\n## Friendly Links\n\nWe highly welcome contributions from the community and actively contribute to the open-source community. The following\nworks have already been adapted for CogVideoX, and we invite everyone to use them:\n\n+ [LeMiCa](https:\u002F\u002Funicomai.github.io\u002FLeMiCa\u002F): a diffusion model inference acceleration solution developed by China Unicom Data Science and Artificial Intelligence Research Institute. By leveraging cache-based techniques and global denoising path optimization, LeMiCa provides efficient inference support for CogVideoX, achieving nearly 2.5x lossless acceleration while maintaining visual consistency and quality.\n+ [RIFLEx-CogVideoX](https:\u002F\u002Fgithub.com\u002Fthu-ml\u002FRIFLEx):\n  RIFLEx extends the video with just one line of code: `freq[k-1]=(2np.pi)\u002F(Ls)`. The framework not only supports training-free inference, but also offers models fine-tuned based on CogVideoX. By fine-tuning the model for just 1,000 steps on original-length videos, RIFLEx significantly enhances its length extrapolation capability.\n+ [CogVideoX-Fun](https:\u002F\u002Fgithub.com\u002Faigc-apps\u002FCogVideoX-Fun): CogVideoX-Fun is a modified pipeline based on the\n  CogVideoX architecture, supporting flexible resolutions and multiple launch methods.\n+ [CogStudio](https:\u002F\u002Fgithub.com\u002Fpinokiofactory\u002Fcogstudio): A separate repository for CogVideo's Gradio Web UI, which\n  supports more functional Web UIs.\n+ [Xorbits Inference](https:\u002F\u002Fgithub.com\u002Fxorbitsai\u002Finference): A powerful and comprehensive distributed inference\n  framework, allowing you to easily deploy your own models or the latest cutting-edge open-source models with just one\n  click.\n+ [ComfyUI-CogVideoXWrapper](https:\u002F\u002Fgithub.com\u002Fkijai\u002FComfyUI-CogVideoXWrapper) Use the ComfyUI framework to integrate\n  CogVideoX into your workflow.\n+ [VideoSys](https:\u002F\u002Fgithub.com\u002FNUS-HPC-AI-Lab\u002FVideoSys): VideoSys provides a user-friendly, high-performance\n  infrastructure for video generation, with full pipeline support and continuous integration of the latest models and\n  techniques.\n+ [AutoDL Space](https:\u002F\u002Fwww.codewithgpu.com\u002Fi\u002FTHUDM\u002FCogVideo\u002FCogVideoX-5b-demo): A one-click deployment Huggingface\n  Space image provided by community members.\n+ [Interior Design Fine-Tuning Model](https:\u002F\u002Fhuggingface.co\u002Fcollections\u002Fbertjiazheng\u002Fkoolcogvideox-66e4762f53287b7f39f8f3ba):\n  is a fine-tuned model based on CogVideoX, specifically designed for interior design.\n+ [xDiT](https:\u002F\u002Fgithub.com\u002Fxdit-project\u002FxDiT): xDiT is a scalable inference engine for Diffusion Transformers (DiTs)\n  on multiple GPU Clusters. xDiT supports real-time image and video generations services.\n  [cogvideox-factory](https:\u002F\u002Fgithub.com\u002Fa-r-r-o-w\u002Fcogvideox-factory): A cost-effective\n  fine-tuning framework for CogVideoX, compatible with the `diffusers` version model. Supports more resolutions, and\n  fine-tuning CogVideoX-5B can be done with a single 4090 GPU.\n+ [CogVideoX-Interpolation](https:\u002F\u002Fgithub.com\u002Ffeizc\u002FCogvideX-Interpolation): A pipeline based on the modified CogVideoX\n  structure, aimed at providing greater flexibility for keyframe interpolation generation.\n+ [DiffSynth-Studio](https:\u002F\u002Fgithub.com\u002Fmodelscope\u002FDiffSynth-Studio): DiffSynth Studio is a diffusion engine. It has\n  restructured the architecture, including text encoders, UNet, VAE, etc., enhancing computational performance while\n  maintaining compatibility with open-source community models. The framework has been adapted for CogVideoX.\n+ [CogVideoX-Controlnet](https:\u002F\u002Fgithub.com\u002FTheDenk\u002Fcogvideox-controlnet): A simple ControlNet module code that includes the CogVideoX model.\n+ [VideoTuna](https:\u002F\u002Fgithub.com\u002FVideoVerses\u002FVideoTuna): VideoTuna is the first repo that integrates multiple AI video generation models for text-to-video, image-to-video, text-to-image generation.\n+ [ConsisID](https:\u002F\u002Fgithub.com\u002FPKU-YuanGroup\u002FConsisID): An identity-preserving text-to-video generation model, bases on CogVideoX-5B, which keep the face consistent in the generated video by frequency decomposition.\n+ [A Step by Step Tutorial](https:\u002F\u002Fwww.youtube.com\u002Fwatch?v=5UCkMzP2VLE&ab_channel=SECourses): A step-by-step guide on installing and optimizing the CogVideoX1.5-5B-I2V model in Windows and cloud environments. Special thanks to the [FurkanGozukara](https:\u002F\u002Fgithub.com\u002FFurkanGozukara) for his effort and support!\n\n## Project Structure\n\nThis open-source repository will guide developers to quickly get started with the basic usage and fine-tuning examples\nof the **CogVideoX** open-source model.\n\n### Quick Start with Colab\n\nHere provide three projects that can be run directly on free Colab T4 instances:\n\n+ [CogVideoX-5B-T2V-Colab.ipynb](https:\u002F\u002Fcolab.research.google.com\u002Fdrive\u002F1pCe5s0bC_xuXbBlpvIH1z0kfdTLQPzCS?usp=sharing):\n  CogVideoX-5B Text-to-Video Colab code.\n+ [CogVideoX-5B-T2V-Int8-Colab.ipynb](https:\u002F\u002Fcolab.research.google.com\u002Fdrive\u002F1DUffhcjrU-uz7_cpuJO3E_D4BaJT7OPa?usp=sharing):\n  CogVideoX-5B Quantized Text-to-Video Inference Colab code, which takes about 30 minutes per run.\n+ [CogVideoX-5B-I2V-Colab.ipynb](https:\u002F\u002Fcolab.research.google.com\u002Fdrive\u002F17CqYCqSwz39nZAX2YyonDxosVKUZGzcX?usp=sharing):\n  CogVideoX-5B Image-to-Video Colab code.\n+ [CogVideoX-5B-V2V-Colab.ipynb](https:\u002F\u002Fcolab.research.google.com\u002Fdrive\u002F1comfGAUJnChl5NwPuO8Ox5_6WCy4kbNN?usp=sharing):\n  CogVideoX-5B Video-to-Video Colab code.\n\n### Inference\n\n+ [dcli_demo](inference\u002Fcli_demo.py): A more detailed inference code explanation, including the significance of\n  common parameters. All of this is covered here.\n+ [cli_demo_quantization](inference\u002Fcli_demo_quantization.py):\n  Quantized model inference code that can run on devices with lower memory. You can also modify this code to support\n  running CogVideoX models in FP8 precision.\n+ [diffusers_vae_demo](inference\u002Fcli_vae_demo.py): Code for running VAE inference separately.\n+ [space demo](inference\u002Fgradio_composite_demo): The same GUI code as used in the Huggingface Space, with frame\n  interpolation and super-resolution tools integrated.\n\n\u003Cdiv style=\"text-align: center;\">\n    \u003Cimg src=\"resources\u002Fweb_demo.png\" style=\"width: 100%; height: auto;\" \u002F>\n\u003C\u002Fdiv>\n\n+ [convert_demo](inference\u002Fconvert_demo.py): How to convert user input into long-form input suitable for CogVideoX.\n  Since CogVideoX is trained on long texts, we need to transform the input text distribution to match the training data\n  using an LLM. The script defaults to using GLM-4, but it can be replaced with GPT, Gemini, or any other large language\n  model.\n+ [gradio_web_demo](inference\u002Fgradio_composite_demo): A simple Gradio web application demonstrating how to use the\n  CogVideoX-2B \u002F 5B model to generate videos. Similar to our Huggingface Space, you can use this script to run a simple\n  web application for video generation.\n\n### finetune\n\n+ [finetune_demo](finetune\u002FREADME.md): Fine-tuning scheme and details of the diffusers version of the CogVideoX model.\n\n### sat\n\n+ [sat_demo](sat\u002FREADME.md): Contains the inference code and fine-tuning code of SAT weights. It is recommended to\n  improve based on the CogVideoX model structure. Innovative researchers use this code to better perform rapid stacking\n  and development.\n\n### Tools\n\nThis folder contains some tools for model conversion \u002F caption generation, etc.\n\n+ [convert_weight_sat2hf](tools\u002Fconvert_weight_sat2hf.py): Converts SAT model weights to Huggingface model weights.\n+ [caption_demo](tools\u002Fcaption\u002FREADME.md): Caption tool, a model that understands videos and outputs descriptions in\n  text.\n+ [export_sat_lora_weight](tools\u002Fexport_sat_lora_weight.py): SAT fine-tuning model export tool, exports the SAT Lora\n  Adapter in diffusers format.\n+ [load_cogvideox_lora](tools\u002Fload_cogvideox_lora.py): Tool code for loading the diffusers version of fine-tuned Lora\n  Adapter.\n+ [llm_flux_cogvideox](tools\u002Fllm_flux_cogvideox\u002Fllm_flux_cogvideox.py): Automatically generate videos using an\n  open-source local large language model + Flux + CogVideoX.\n+ [parallel_inference_xdit](tools\u002Fparallel_inference\u002Fparallel_inference_xdit.py):\n  Supported by [xDiT](https:\u002F\u002Fgithub.com\u002Fxdit-project\u002FxDiT), parallelize the\n  video generation process on multiple GPUs.\n\n## CogVideo(ICLR'23)\n\nThe official repo for the\npaper: [CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformers](https:\u002F\u002Farxiv.org\u002Fabs\u002F2205.15868)\nis on the [CogVideo branch](https:\u002F\u002Fgithub.com\u002FTHUDM\u002FCogVideo\u002Ftree\u002FCogVideo)\n\n**CogVideo is able to generate relatively high-frame-rate videos.**\nA 4-second clip of 32 frames is shown below.\n\n![High-frame-rate sample](https:\u002F\u002Fraw.githubusercontent.com\u002FTHUDM\u002FCogVideo\u002FCogVideo\u002Fassets\u002Fappendix-sample-highframerate.png)\n\n![Intro images](https:\u002F\u002Fraw.githubusercontent.com\u002FTHUDM\u002FCogVideo\u002FCogVideo\u002Fassets\u002Fintro-image.png)\n\u003Cdiv align=\"center\">\n  \u003Cvideo src=\"https:\u002F\u002Fgithub.com\u002Fuser-attachments\u002Fassets\u002F2fa19651-e925-4a2a-b8d6-b3f216d490ba\" width=\"80%\" controls autoplay>\u003C\u002Fvideo>\n\u003C\u002Fdiv>\n\n\nThe demo for CogVideo is at [https:\u002F\u002Fmodels.aminer.cn\u002Fcogvideo](https:\u002F\u002Fmodels.aminer.cn\u002Fcogvideo\u002F), where you can get\nhands-on practice on text-to-video generation. *The original input is in Chinese.*\n\n## Citation\n\n🌟 If you find our work helpful, please leave us a star and cite our paper.\n\n```\n@article{yang2024cogvideox,\n  title={CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer},\n  author={Yang, Zhuoyi and Teng, Jiayan and Zheng, Wendi and Ding, Ming and Huang, Shiyu and Xu, Jiazheng and Yang, Yuanming and Hong, Wenyi and Zhang, Xiaohan and Feng, Guanyu and others},\n  journal={arXiv preprint arXiv:2408.06072},\n  year={2024}\n}\n@article{hong2022cogvideo,\n  title={CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformers},\n  author={Hong, Wenyi and Ding, Ming and Zheng, Wendi and Liu, Xinghan and Tang, Jie},\n  journal={arXiv preprint arXiv:2205.15868},\n  year={2022}\n}\n```\n\n## Model-License\n\nThe code in this repository is released under the [Apache 2.0 License](LICENSE).\n\nThe CogVideoX-2B model (including its corresponding Transformers module and VAE module) is released under\nthe [Apache 2.0 License](LICENSE).\n\nThe CogVideoX-5B model (Transformers module, include I2V and T2V) is released under\nthe [CogVideoX LICENSE](https:\u002F\u002Fhuggingface.co\u002FTHUDM\u002FCogVideoX-5b\u002Fblob\u002Fmain\u002FLICENSE).\n","CogVideo & CogVideoX 是一个用于文本和图像生成视频的项目。其核心功能包括将文本或图像转换成高质量的视频，支持多种分辨率和时长。该项目基于Python开发，采用了先进的多模态生成模型和技术，如LLM、Sora等。用户可以通过Huggingface Space或ModelScope Space在线体验CogVideoX-5B模型，并且提供了详细的论文和技术文档供参考。此外，项目还支持通过DDIM Inverse进行反向扩散，以及使用Lora进行微调以减少GPU内存占用。适合需要从文本或图像生成视频的应用场景，例如内容创作、广告制作和教育等领域。",2,"2026-06-11 03:35:05","high_star"]