[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"project-84143":3},{"id":4,"name":5,"fullName":6,"owner":7,"repo":5,"description":8,"homepage":9,"htmlUrl":10,"language":11,"languages":10,"totalLinesOfCode":10,"stars":12,"forks":13,"watchers":14,"openIssues":15,"contributorsCount":13,"subscribersCount":13,"size":13,"stars1d":15,"stars7d":16,"stars30d":16,"stars90d":13,"forks30d":13,"starsTrendScore":17,"compositeScore":13,"rankGlobal":10,"rankLanguage":10,"license":10,"archived":18,"fork":18,"defaultBranch":19,"hasWiki":20,"hasPages":18,"topics":21,"createdAt":10,"pushedAt":10,"updatedAt":22,"readmeContent":23,"aiSummary":10,"trendingCount":13,"starSnapshotCount":13,"syncStatus":24,"lastSyncTime":25,"discoverSource":26},84143,"LoomVideo","MSALab-PKU\u002FLoomVideo","MSALab-PKU","Official implementation of  LoomVideo: Unifying Multimodal Inputs into Video Generation and Editing","https:\u002F\u002Fmsalab-pku.github.io\u002Fprojects\u002FLoomVideo\u002Findex.html",null,"Python",56,0,53,1,3,5,false,"main",true,[],"2026-06-12 02:04:38","\u003Cdiv align=\"center\">\n\n# LoomVideo: Unifying Multimodal Inputs into \u003Cbr> Video Generation and Editing\n\n\u003Ch3>Peking University · Alibaba Group\u003C\u002Fh3>\n\n\u003Ca href=\"https:\u002F\u002Farxiv.org\u002Fabs\u002F2606.06042\" target=\"_blank\">\u003Cimg src=\"https:\u002F\u002Fimg.shields.io\u002Fbadge\u002FPaper-b5212f.svg?logo=arxiv\" height=\"22px\">\u003C\u002Fa>\n\u003Ca href=\"https:\u002F\u002Fhuggingface.co\u002FMSALab\u002FLoomVideo\" target=\"_blank\">\u003Cimg src=\"https:\u002F\u002Fimg.shields.io\u002Fbadge\u002F%F0%9F%A4%97%20Model-d96902.svg\" height=\"22px\">\u003C\u002Fa>\n\u003Ca href=\"https:\u002F\u002Fmsalab-pku.github.io\u002Fprojects\u002FLoomVideo\u002Findex.html\" target=\"_blank\">\u003Cimg src=\"https:\u002F\u002Fimg.shields.io\u002Fbadge\u002FProject%20Page-333399.svg?logo=homepage\" height=\"22px\">\u003C\u002Fa>\n\n\u003C\u002Fdiv>\n\n# 🔥 News\n\n- [2026-06-06] We release the post-pretrained [model weights](https:\u002F\u002Fhuggingface.co\u002FMSALab\u002FLoomVideo) of LoomVideo!\n- [2026-06-05] We release LoomVideo [paper](https:\u002F\u002Farxiv.org\u002Fabs\u002F2606.06042) on Arxiv!\n- [2026-06-02] We release the [codebase](https:\u002F\u002Fgithub.com\u002FMSALab-PKU\u002FLoomVideo) and [model weights](https:\u002F\u002Fhuggingface.co\u002FMSALab\u002FLoomVideo) of LoomVideo!\n- [2026-06-02] We release the [project page](https:\u002F\u002Fmsalab-pku.github.io\u002Fprojects\u002FLoomVideo\u002Findex.html) of LoomVideo!\n\n# 📌 TL;DR\n\n**The Problem:** Existing unified video generation & editing models are massive (13B+) and rely on token concatenation for source conditioning — doubling sequence length and quadrupling attention cost.\n\n**The Method:** We present **LoomVideo**, a compact **5B-parameter** unified architecture built on MLLM + DiT that introduces three key designs:\n- **Deepstack Injection** — extracts features from every MLLM layer and injects them into corresponding DiT layers via cross-attention, enabling rich multi-granular semantic guidance.\n- **Scale-and-Add Conditioning** — a zero-overhead approach that scales the clean source latent by the current timestep and directly adds it to the noised target, completely bypassing token concatenation.\n- **Negative Temporal RoPE** — assigns negative temporal indices to reference images, seamlessly integrating multi-image conditions without architectural modification.\n\n**The Result:** Our 5B model achieves state-of-the-art or highly competitive performance across comprehensive benchmarks, with at least **5.41×** inference speedup over models of similar capabilities — demonstrating that efficiency and quality can coexist.\n\n\u003Cp align=\"center\">\n  \u003Cimg src=\"assets\u002Farchitecture.png\" width=\"90%\">\n\u003C\u002Fp>\n\n\n# 🎯 Supported Tasks\n\nLoomVideo supports **four** unified video generation and editing tasks within a single model:\n\n| Task                          | Input                      | Output  | Description                                             |\n| :---------------------------- | :------------------------- | :------ | :------------------------------------------------------ |\n| **Text-to-Video**             | Text 📝                     | Video 🎬 | Generate a video from a text prompt                     |\n| **Instruction Editing**       | Video 🎬 + Text 📝           | Video 🎬 | Edit a video following text instructions                |\n| **Instruction-Image Editing** | Video 🎬 + Image 🖼 + Text 📝 | Video 🎬 | Edit a video with a reference image as guidance         |\n| **Multi-Image-to-Video**      | Images 🖼 + Text 📝          | Video 🎬 | Compose multiple reference images into a coherent video |\n\n### 🎬 Text-to-Video\n\n\u003Cp align=\"center\">\n  \u003Cimg src=\"assets\u002Fresults_1\u002Ft2v_demo.gif\" width=\"480\"\u002F>\n\u003C\u002Fp>\n\n> **Prompt:** *Snow rocky mountains peaks canyon. Snow blanketed rocky mountains surround and shadow deep canyons. The canyons twist and bend through the high elevated mountain peaks.*\n\n\u003Cp align=\"center\">\n  \u003Cimg src=\"assets\u002Fresults_2\u002Ft2v_demo.gif\" width=\"480\"\u002F>\n\u003C\u002Fp>\n\n> **Prompt:** *Vampire makeup face of beautiful girl, red contact lenses.*\n\n### ✂️ Instruction Editing\n\n\u003Ctable align=\"center\">\n  \u003Ctr>\n    \u003Ctd align=\"center\" valign=\"middle\">\u003Cimg src=\"assets\u002Fresults_1\u002Fedit_input.gif\" height=\"180\"\u002F>\u003C\u002Ftd>\n    \u003Ctd align=\"center\" valign=\"middle\">\u003Cb>\u003Cfont size=\"5\">→\u003C\u002Ffont>\u003C\u002Fb>\u003C\u002Ftd>\n    \u003Ctd align=\"center\" valign=\"middle\">\u003Cimg src=\"assets\u002Fresults_1\u002Fedit_demo.gif\" height=\"180\"\u002F>\u003C\u002Ftd>\n  \u003C\u002Ftr>\n\u003C\u002Ftable>\n\n> **Prompt:** *Apply the Impressionist aesthetic to this video, ensuring seamless temporal consistency across all frames. The result should emulate the fluid brushstroke techniques and atmospheric focus of 19th-century Impressionist art, with each frame retaining the original motion, character actions, and camera movements.*\n\n\u003Ctable align=\"center\">\n  \u003Ctr>\n    \u003Ctd align=\"center\" valign=\"middle\">\u003Cimg src=\"assets\u002Fresults_2\u002Fedit_input.gif\" height=\"180\"\u002F>\u003C\u002Ftd>\n    \u003Ctd align=\"center\" valign=\"middle\">\u003Cb>\u003Cfont size=\"5\">→\u003C\u002Ffont>\u003C\u002Fb>\u003C\u002Ftd>\n    \u003Ctd align=\"center\" valign=\"middle\">\u003Cimg src=\"assets\u002Fresults_2\u002Fedit_demo.gif\" height=\"180\"\u002F>\u003C\u002Ftd>\n  \u003C\u002Ftr>\n\u003C\u002Ftable>\n\n> **Prompt:** *Replace the tree with a golden-leaved tree that shimmers softly, ensuring it maintains the same position and pose within the video scene.*\n\n### 🖼️ Instruction-Image Editing\n\n\u003Ctable align=\"center\">\n  \u003Ctr>\n    \u003Ctd align=\"center\" valign=\"middle\">\u003Cimg src=\"assets\u002Fresults_1\u002Fref_edit_input.gif\" height=\"180\"\u002F>\u003C\u002Ftd>\n    \u003Ctd align=\"center\" valign=\"middle\">\u003Cimg src=\"assets\u002Fresults_1\u002Fref_edit_reference.jpg\" height=\"100\"\u002F>\u003C\u002Ftd>\n    \u003Ctd align=\"center\" valign=\"middle\">\u003Cb>\u003Cfont size=\"5\">→\u003C\u002Ffont>\u003C\u002Fb>\u003C\u002Ftd>\n    \u003Ctd align=\"center\" valign=\"middle\">\u003Cimg src=\"assets\u002Fresults_1\u002Fref_edit_demo.gif\" height=\"180\"\u002F>\u003C\u002Ftd>\n  \u003C\u002Ftr>\n\u003C\u002Ftable>\n\n> **Prompt:** *Replace the green t-shirt of the man with the suit in the image.*\n\n\u003Ctable align=\"center\">\n  \u003Ctr>\n    \u003Ctd align=\"center\" valign=\"middle\">\u003Cimg src=\"assets\u002Fresults_2\u002Fref_edit_input.gif\" height=\"180\"\u002F>\u003C\u002Ftd>\n    \u003Ctd align=\"center\" valign=\"middle\">\u003Cimg src=\"assets\u002Fresults_2\u002Fref_edit_reference.jpg\" height=\"100\"\u002F>\u003C\u002Ftd>\n    \u003Ctd align=\"center\" valign=\"middle\">\u003Cb>\u003Cfont size=\"5\">→\u003C\u002Ffont>\u003C\u002Fb>\u003C\u002Ftd>\n    \u003Ctd align=\"center\" valign=\"middle\">\u003Cimg src=\"assets\u002Fresults_2\u002Fref_edit_demo.gif\" height=\"180\"\u002F>\u003C\u002Ftd>\n  \u003C\u002Ftr>\n\u003C\u002Ftable>\n\n> **Prompt:** *Replace the background with a Chinese ink painting, featuring a large golden mountain peak rising above swirling clouds, ensuring it appears in the same position and pose within the video scene.*\n\n### 🎞️ Multi-Image-to-Video\n\n\u003Ctable align=\"center\">\n  \u003Ctr>\n    \u003Ctd align=\"center\" valign=\"middle\">\u003Cimg src=\"assets\u002Fresults_1\u002Fmi2v_input_1.jpg\" height=\"140\"\u002F> \u003Cimg src=\"assets\u002Fresults_1\u002Fmi2v_input_2.jpg\" height=\"140\"\u002F> \u003Cimg src=\"assets\u002Fresults_1\u002Fmi2v_input_3.jpg\" height=\"140\"\u002F>\u003C\u002Ftd>\n    \u003Ctd align=\"center\" valign=\"middle\">\u003Cb>\u003Cfont size=\"5\">→\u003C\u002Ffont>\u003C\u002Fb>\u003C\u002Ftd>\n    \u003Ctd align=\"center\" valign=\"middle\">\u003Cimg src=\"assets\u002Fresults_1\u002Fmi2v_demo.gif\" height=\"180\"\u002F>\u003C\u002Ftd>\n  \u003C\u002Ftr>\n\u003C\u002Ftable>\n\n> **Prompt:** *The girl (@Image 2), wearing the denim jacket (@Image 3), black inner top, and black shorts, wearing sunglasses and carrying the handbag, walks down the street (@Image 1). Then, the girl (@Image 2) stops walking and turns her head to look to one side, followed by the girl (@Image 2) crossing her arms over her chest and striking a confident pose.*\n\n\u003Ctable align=\"center\">\n  \u003Ctr>\n    \u003Ctd align=\"center\" valign=\"middle\">\u003Cimg src=\"assets\u002Fresults_2\u002Fmi2v_input_1.jpg\" height=\"140\"\u002F> \u003Cimg src=\"assets\u002Fresults_2\u002Fmi2v_input_2.jpg\" height=\"140\"\u002F>\u003C\u002Ftd>\n    \u003Ctd align=\"center\" valign=\"middle\">\u003Cb>\u003Cfont size=\"5\">→\u003C\u002Ffont>\u003C\u002Fb>\u003C\u002Ftd>\n    \u003Ctd align=\"center\" valign=\"middle\">\u003Cimg src=\"assets\u002Fresults_2\u002Fmi2v_demo.gif\" height=\"180\"\u002F>\u003C\u002Ftd>\n  \u003C\u002Ftr>\n\u003C\u002Ftable>\n\n> **Prompt:** *The man wearing a Polo shirt (@Image 2), black casual pants, white sneakers, sunglasses, and a watch, striding forward on the lawn (@Image 1) with one hand in his pocket.*\n\n\n# 🔧 Preparation\n\n## Step 1: Clone the Repository\n\n```bash\ngit clone https:\u002F\u002Fgithub.com\u002FMSALab-PKU\u002FLoomVideo\ncd LoomVideo\n```\n\n## Step 2: Install Dependencies\n\nWe recommend using [uv](https:\u002F\u002Fgithub.com\u002Fastral-sh\u002Fuv) for a fast and fully reproducible environment setup.\n\n```bash\nuv sync\nsource .venv\u002Fbin\u002Factivate\n\n# (Optional) Include evaluation dependencies\nuv sync --extra eval\n```\n\nAdditionally, install [Flash Attention](https:\u002F\u002Fgithub.com\u002FDao-AILab\u002Fflash-attention) for faster inference and reduced GPU memory consumption. (for reference, our environment uses v2.7.4)\n\n## Step 3: Download Model Weights\n\nDownload the pretrained LoomVideo checkpoint from [Hugging Face](https:\u002F\u002Fhuggingface.co\u002FMSALab\u002FLoomVideo) and place it under `checkpoints\u002FLoomVideo\u002F`:\n\n```\ncheckpoints\u002FLoomVideo\u002F\n└── gen_model.pth\n```\n\nWe provide a helper script to download the weights automatically:\n\n```bash\npython hf_download.py\n```\n\nYou can also specify a custom path via the `--ckpt_path` argument at inference time.\n\n\n# 🎬 Inference\nLoomVideo provides a unified inference script that supports **four generation tasks** through a single entry point. Each task is selected via the `--task` flag.\n\n### 1. Text-to-Video \u002F Text-to-Image (`t2v`)\n\nGenerate a video from a text description. Default resolution is **480×832** at **81 frames**. When `--num_frames` is set to `1`, the pipeline automatically switches to **image generation** mode and saves the output as a `.jpg` file.\n\n**Required:** `--prompt`\n\n```bash\nNUM_GPUS=1\n\naccelerate launch --num_processes=${NUM_GPUS} \\\n    scripts\u002Finference\u002Fgenerate.py \\\n    --config_path configs\u002Finference\u002Fgeneration.yaml \\\n    --ckpt_path checkpoints\u002FLoomVideo \\\n    --task t2v \\\n    --prompt \"Vampire makeup face of beautiful girl, red contact lenses.\" \\\n    --height 480 \\\n    --width 832 \\\n    --num_frames 97 \\\n    --num_inference_steps 50 \\\n    --seed 0 \\\n    --output_path outputs\u002Ft2v_demo.mp4\n```\n\n### 2. Instruction Editing (`edit`)\n\nEdit an existing image or video based on a text instruction. The source can be either an image file (`.jpg`, `.png`, etc.) or a video file (`.mp4`). Resolution and frame count are automatically inferred from the source when not specified.\n\n**Required:** `--prompt` `--source_video_path`\n\n```bash\nNUM_GPUS=1\n\naccelerate launch --num_processes=${NUM_GPUS} \\\n    scripts\u002Finference\u002Fgenerate.py \\\n    --config_path configs\u002Finference\u002Fgeneration.yaml \\\n    --ckpt_path checkpoints\u002FLoomVideo \\\n    --task edit \\\n    --prompt \"Apply the Impressionist aesthetic to this video, ensuring seamless temporal consistency across all frames. The result should emulate the fluid brushstroke techniques and atmospheric focus of 19th-century Impressionist art, with each frame retaining the original motion, character actions, and camera movements.\" \\\n    --source_video_path assets\u002Fdemo\u002Fedit_input.mp4 \\\n    --num_inference_steps 50 \\\n    --seed 0 \\\n    --output_path outputs\u002Fedit_demo.mp4\n```\n\n### 3. Instruction-Image Editing (`ref_edit`)\n\nEdit a source video with guidance from one or more reference images along with a text instruction.\n\n**Required:** `--prompt` `--source_video_path` `--ref_image_paths`\n\n```bash\nNUM_GPUS=1\n\naccelerate launch --num_processes=${NUM_GPUS} \\\n    scripts\u002Finference\u002Fgenerate.py \\\n    --config_path configs\u002Finference\u002Fgeneration.yaml \\\n    --ckpt_path checkpoints\u002FLoomVideo \\\n    --task ref_edit \\\n    --prompt \"Replace the green t-shirt of the man with the suit in the image.\" \\\n    --source_video_path assets\u002Fdemo\u002Fref_edit_input.mp4 \\\n    --ref_image_paths assets\u002Fdemo\u002Fref_edit_reference.jpg \\\n    --num_inference_steps 50 \\\n    --seed 0 \\\n    --output_path outputs\u002Fref_edit_demo.mp4\n```\n\n### 4. Multi-Image-to-Video (`mi2v`)\n\nGenerate a video conditioned on multiple reference images and a text prompt. We recommend using `@Image N` in the prompt to reference specific input images.\n\n**Required:** `--prompt` `--ref_image_paths`\n\n```bash\nNUM_GPUS=1\n\naccelerate launch --num_processes=${NUM_GPUS} \\\n    scripts\u002Finference\u002Fgenerate.py \\\n    --config_path configs\u002Finference\u002Fgeneration.yaml \\\n    --ckpt_path checkpoints\u002FLoomVideo \\\n    --task mi2v \\\n    --prompt \"The man wearing a Polo shirt (@Image 2), black casual pants, white sneakers, sunglasses, and a watch, striding forward on the lawn (@Image 1) with one hand in his pocket.\" \\\n    --ref_image_paths assets\u002Fdemo\u002Fmi2v_input_1.jpg assets\u002Fdemo\u002Fmi2v_input_2.jpg \\\n    --num_frames 97 \\\n    --num_inference_steps 50 \\\n    --seed 0 \\\n    --output_path outputs\u002Fmi2v_demo.mp4\n```\n\n\n## Additional Arguments\n\nThe following arguments can be appended to any task command for further customization:\n\n### Generation Control\n\n\u003Ctable>\n  \u003Cthead>\n    \u003Ctr>\u003Cth>Argument\u003C\u002Fth>\u003Cth>Type\u003C\u002Fth>\u003Cth>Default\u003C\u002Fth>\u003Cth>Description\u003C\u002Fth>\u003C\u002Ftr>\n  \u003C\u002Fthead>\n  \u003Ctbody>\n    \u003Ctr>\u003Ctd nowrap>\u003Ccode>--num_inference_steps\u003C\u002Fcode>\u003C\u002Ftd>\u003Ctd>int\u003C\u002Ftd>\u003Ctd>\u003Ccode>50\u003C\u002Fcode>\u003C\u002Ftd>\u003Ctd>Number of denoising steps.\u003C\u002Ftd>\u003C\u002Ftr>\n    \u003Ctr>\u003Ctd nowrap>\u003Ccode>--guidance_scale\u003C\u002Fcode>\u003C\u002Ftd>\u003Ctd>float\u003C\u002Ftd>\u003Ctd>\u003Ccode>5.0\u003C\u002Fcode> \u002F \u003Ccode>2.5\u003C\u002Fcode>\u003C\u002Ftd>\u003Ctd>Text CFG scale. \u003Ccode>5.0\u003C\u002Fcode> for t2v\u002Fmi2v, \u003Ccode>2.5\u003C\u002Fcode> for edit\u002Fref_edit.\u003C\u002Ftd>\u003C\u002Ftr>\n    \u003Ctr>\u003Ctd nowrap>\u003Ccode>--guidance_scale_visual\u003C\u002Fcode>\u003C\u002Ftd>\u003Ctd>float\u003C\u002Ftd>\u003Ctd>\u003Ccode>1.5\u003C\u002Fcode>\u003C\u002Ftd>\u003Ctd>Visual CFG scale for source\u002Freference conditioning.\u003C\u002Ftd>\u003C\u002Ftr>\n    \u003Ctr>\u003Ctd nowrap>\u003Ccode>--negative_prompt\u003C\u002Fcode>\u003C\u002Ftd>\u003Ctd>str\u003C\u002Ftd>\u003Ctd>\u003Cem>(from config)\u003C\u002Fem>\u003C\u002Ftd>\u003Ctd>Negative prompt for quality improvement.\u003C\u002Ftd>\u003C\u002Ftr>\n    \u003Ctr>\u003Ctd nowrap>\u003Ccode>--seed\u003C\u002Fcode>\u003C\u002Ftd>\u003Ctd>int\u003C\u002Ftd>\u003Ctd>\u003Ccode>0\u003C\u002Fcode>\u003C\u002Ftd>\u003Ctd>Random seed. Set to \u003Ccode>-1\u003C\u002Fcode> for random generation.\u003C\u002Ftd>\u003C\u002Ftr>\n  \u003C\u002Ftbody>\n\u003C\u002Ftable>\n\n### Resolution & Frames\n\n\u003Ctable>\n  \u003Cthead>\n    \u003Ctr>\u003Cth>Argument\u003C\u002Fth>\u003Cth>Type\u003C\u002Fth>\u003Cth>Default\u003C\u002Fth>\u003Cth>Description\u003C\u002Fth>\u003C\u002Ftr>\n  \u003C\u002Fthead>\n  \u003Ctbody>\n    \u003Ctr>\u003Ctd nowrap>\u003Ccode>--height\u003C\u002Fcode>\u003C\u002Ftd>\u003Ctd>int\u003C\u002Ftd>\u003Ctd>\u003Cem>auto\u003C\u002Fem>\u003C\u002Ftd>\u003Ctd>Output height. \u003Ccode>480\u003C\u002Fcode> for t2v; inferred from source for edit.\u003C\u002Ftd>\u003C\u002Ftr>\n    \u003Ctr>\u003Ctd nowrap>\u003Ccode>--width\u003C\u002Fcode>\u003C\u002Ftd>\u003Ctd>int\u003C\u002Ftd>\u003Ctd>\u003Cem>auto\u003C\u002Fem>\u003C\u002Ftd>\u003Ctd>Output width. \u003Ccode>832\u003C\u002Fcode> for t2v; inferred from source for edit.\u003C\u002Ftd>\u003C\u002Ftr>\n    \u003Ctr>\u003Ctd nowrap>\u003Ccode>--num_frames\u003C\u002Fcode>\u003C\u002Ftd>\u003Ctd>int\u003C\u002Ftd>\u003Ctd>\u003Cem>auto\u003C\u002Fem>\u003C\u002Ftd>\u003Ctd>Output frames. \u003Ccode>81\u003C\u002Fcode> for t2v\u002Fmi2v; inferred for edit.\u003C\u002Ftd>\u003C\u002Ftr>\n    \u003Ctr>\u003Ctd nowrap>\u003Ccode>--fps\u003C\u002Fcode>\u003C\u002Ftd>\u003Ctd>int\u003C\u002Ftd>\u003Ctd>\u003Ccode>24\u003C\u002Fcode>\u003C\u002Ftd>\u003Ctd>Output video FPS.\u003C\u002Ftd>\u003C\u002Ftr>\n  \u003C\u002Ftbody>\n\u003C\u002Ftable>\n\n\n# 📦 Data Preparation\n\nSince our training relies heavily on proprietary datasets, we are unable to release the original data directly. However, we provide a **flexible data organization framework** that makes it easy to plug in your own data or publicly available datasets.\n\n## Open-Source Datasets\n\nBelow are the open-source datasets used in our training. You can download them or substitute with your own data:\n\n| Category                 | Dataset                                                                                                                                                                                                                                                                                                                                                                                                                              |\n| ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |\n| Video Generation         | [Koala-36M](https:\u002F\u002Fhuggingface.co\u002Fdatasets\u002FKoala-36M\u002FKoala-36M-v1), [OpenVid-1M](https:\u002F\u002Fhuggingface.co\u002Fdatasets\u002Fnkp37\u002FOpenVid-1M)                                                                                                                                                                                                                                                                                                  |\n| Image Editing            | [CrispEdit-2M](https:\u002F\u002Fhuggingface.co\u002Fdatasets\u002FWeiChow\u002FCrispEdit-2M), [OmniGen-2-Edit](https:\u002F\u002Fhuggingface.co\u002FOmniGen2), [GPT-Image-Edit-1.5M](https:\u002F\u002Fhuggingface.co\u002Fdatasets\u002FUCSC-VLAA\u002FGPT-Image-Edit-1.5M), [NHR-Edit](https:\u002F\u002Fhuggingface.co\u002Fdatasets\u002Fiitolstykh\u002FNHR-Edit), [Pico-Banana](https:\u002F\u002Fgithub.com\u002Fapple\u002Fpico-banana-400k), [ShareGPT-4o-Image](https:\u002F\u002Fhuggingface.co\u002Fdatasets\u002FFreedomIntelligence\u002FShareGPT-4o-Image) |\n| Video Editing            | [KIWI-Edit](https:\u002F\u002Fhuggingface.co\u002Fdatasets\u002Flinyq\u002Fkiwi_edit_training_data)                                                                                                                                                                                                                                                                                                                                                           |\n| Video Ref Editing \u002F MI2V | [RefVIE](https:\u002F\u002Fhuggingface.co\u002Fdatasets\u002Flinyq\u002Fkiwi_edit_training_data), [Phantom-Data](https:\u002F\u002Fhuggingface.co\u002Fdatasets\u002FZhuoweiChen\u002FPhantom-data-Koala36M)                                                                                                                                                                                                                                                                           |\n\n## Organize Data as Single JSON Files\n\nEach data sample should be stored as an **individual JSON file**, placed in a single directory (e.g., `single_jsons\u002F`), and named sequentially starting from `0.json`:\n\n```\nyour_dataset\u002F\n└── single_jsons\u002F\n    ├── 0.json\n    ├── 1.json\n    ├── 2.json\n    ├── ...\n```\n\n## JSON Format for Each Task\n\nEach task type expects a specific set of keys in its JSON file. Below are the templates — fill in according to your data:\n\n**Text-to-Video** (`process_t2v_data`):\n```json\n{\n    \"text\": \"A caption describing the video content.\",\n    \"path\": \"relative\u002Fpath\u002Fto\u002Fvideo.mp4\"\n}\n```\n\n**Text-to-Image** (`process_t2i_data`):\n```json\n{\n    \"caption\": \"A caption describing the image content.\",\n    \"image_path\": \"relative\u002Fpath\u002Fto\u002Fimage.jpg\"\n}\n```\n\n**Video Editing** (`process_video_edit_data`):\n```json\n{\n    \"source_video_path\": \"relative\u002Fpath\u002Fto\u002Fsource_video.mp4\",\n    \"instruction\": \"The editing instruction.\",\n    \"target_video_path\": \"relative\u002Fpath\u002Fto\u002Ftarget_video.mp4\"\n}\n```\n\n**Image Editing** (`process_image_edit_data`):\n```json\n{\n    \"source_image_path\": \"relative\u002Fpath\u002Fto\u002Fsource_image.jpg\",\n    \"instruction\": \"The editing instruction.\",\n    \"target_image_path\": \"relative\u002Fpath\u002Fto\u002Ftarget_image.jpg\"\n}\n```\n\n**Multi-Image-to-Video** (`process_t2v_data_withref`):\n```json\n{\n    \"instruction\": \"A prompt describing the video to generate with reference images.\",\n    \"reference_image_paths\": [\n        \"relative\u002Fpath\u002Fto\u002Fref1.jpg\",\n        \"relative\u002Fpath\u002Fto\u002Fref2.jpg\"\n    ],\n    \"target_video_path\": \"relative\u002Fpath\u002Fto\u002Ftarget_video.mp4\"\n}\n```\n\n**Reference-Guided Video Editing** (`process_video_edit_data_withref`):\n```json\n{\n    \"source_video_path\": \"relative\u002Fpath\u002Fto\u002Fsource_video.mp4\",\n    \"reference_image_paths\": [\n        \"relative\u002Fpath\u002Fto\u002Fref1.jpg\"\n    ],\n    \"instruction\": \"The editing instruction with reference guidance.\",\n    \"target_video_path\": \"relative\u002Fpath\u002Fto\u002Ftarget_video.mp4\"\n}\n```\n\n> 💡 All paths in JSON files are **relative** to the `data_root` specified in the dataset config.\n\n## Custom Process Functions (Optional)\n\nYou may also organize your JSON files in any format you prefer, as long as you implement a corresponding `process_*` function. We provide several reference implementations in `src\u002Fdataset\u002Fprocessors.py`. Each process function takes `(dataset_info, data_info)` and returns a list of segments describing the data flow. See the existing functions for examples.\n\n## Dataset Config\n\nCreate a YAML config file to register your datasets. See `configs\u002Fdataset\u002Ftrain_demo.yaml` as a reference. The config is organized into `train`, `val`, and `eval` sections, each containing dataset entries with the following arguments:\n\n| Argument            | Description                                                                                  |\n| ------------------- | -------------------------------------------------------------------------------------------- |\n| `task_weight`       | Controls the sampling probability of this task group relative to others during training.     |\n| `process_func_name` | Name of the processing function in `src\u002Fdataset\u002Fprocessors.py` that parses each JSON sample. |\n| `data_root`         | Base directory for resolving relative paths in JSON files.                                   |\n| `data_json_dir`     | Directory containing the JSON files (`0.json`, `1.json`, ...).                               |\n| `num_samples`       | Total number of samples in the directory.                                                    |\n| `sample_weight`     | Sampling weight of this dataset within its task group.                                       |\n\n\n# 🏋️ Training\n\n## Training Config\n\nThe training behavior is fully controlled by a YAML config file (e.g., `configs\u002Ftrain\u002Fstage3.yaml`).\n\n**Key arguments:**\n\n| Argument                 | Description                                                    |\n| ------------------------ | -------------------------------------------------------------- |\n| `log_dir`                | Directory for saving logs, checkpoints, and generated samples. |\n| `dataset_config_path`    | Path to the dataset config YAML file.                          |\n| `train_steps`            | Total number of training iterations.                           |\n| `checkpointing_interval` | Save a checkpoint every N steps.                               |\n| `validation_interval`    | Run validation every N steps.                                  |\n| `evaluation_interval`    | Run evaluation benchmarks every N steps.                       |\n\n**Model settings:**\n\n| Argument                            | Description                                                                      |\n| ----------------------------------- | -------------------------------------------------------------------------------- |\n| `model.trainable_modules.gen_model` | Which modules to train. `\"all\"` trains the full generation model.                |\n| `model.gradient_checkpointing`      | Enable gradient checkpointing to reduce GPU memory usage.                        |\n| `model.und.pretrained_model_path`   | Path to the pretrained understanding backbone.                                   |\n| `model.gen.pretrained_model_path`   | Path to the pretrained generation backbone.                                      |\n| `model.pretrained_ckpt_path`        | *(Optional)* Load weights from a previous training stage for continued training. |\n\n**Data settings:**\n\n| Argument                        | Description                                                            |\n| ------------------------------- | ---------------------------------------------------------------------- |\n| `data.train.resolution_buckets` | List of resolution buckets for dynamic batching.                       |\n| `data.train.num_frames`         | Number of frames per training sample.                                  |\n| `data.train.fps`                | Video FPS for frame sampling.                                          |\n| `data.train.all_dropout_rate`   | Probability of dropping all conditions (for unconditional training).   |\n| `data.train.text_dropout_rate`  | Probability of dropping text condition (for classifier-free guidance). |\n\n## Launch Training\n\nOnce the data and configs are ready, you can simply start training with:\n\n```bash\nNUM_GPUS=8\n\naccelerate launch --num_processes=${NUM_GPUS} \\\n    -m scripts.train.train \\\n    --config_path path\u002Fto\u002Fyour\u002Fconfig.yaml\n```\n\n> 💡 All training outputs — including checkpoints, EMA weights, logs, and generated samples — are saved under the `log_dir` directory specified in the config.\n\n\n# 📊 Evaluation\n\n## Environment Setup\n\n### Step 1: Prepare Benchmark Data\n\nWe evaluate on the following benchmarks. Download each dataset and organize it into the same **single JSON** format used for training data (see [Data Preparation](#-data-preparation)):\n\n| Benchmark                                                                  | Category                  | Samples |\n| -------------------------------------------------------------------------- | ------------------------- | ------- |\n| [GenEval](https:\u002F\u002Fgithub.com\u002Fdjghosh13\u002Fgeneval)                            | Image Generation          | 553     |\n| [ImgEdit-Bench](https:\u002F\u002Fgithub.com\u002Fpku-yuangroup\u002Fimgedit)                  | Image Editing             | 737     |\n| [VBench](https:\u002F\u002Fgithub.com\u002FVchitect\u002FVBench)                               | Video Generation          | 165     |\n| [OpenVE-Bench](https:\u002F\u002Fhuggingface.co\u002Fdatasets\u002FLewandofski\u002FOpenVE-Bench)   | Video Editing             | 431     |\n| [RefVIE-Bench](https:\u002F\u002Fhuggingface.co\u002Fdatasets\u002Flinyq\u002FRefVIE-Bench)         | Reference Video Editing   | 120     |\n| [Intelligent-VBench-MI2V](https:\u002F\u002Fgithub.com\u002FTencent-Hunyuan\u002FOmniWeaving)  | Multi-Image-to-Video      | 320     |\n| [Intelligent-VBench-TIV2V](https:\u002F\u002Fgithub.com\u002FTencent-Hunyuan\u002FOmniWeaving) | Text-Image-Video-to-Video | 210     |\n\n> 💡 For **Intelligent-VBench**, we split the original benchmark into two subsets based on task type — **MI2V** and **TIV2V**. Their JSON files should be placed in separate directories.\n\nAfter downloading, update the `data_root` and `data_json_dir` paths in `configs\u002Fdataset\u002Fbenchmarks.yaml` to point to your local directories.\n\n### Step 2: Install Evaluation Dependencies\n\n**VBench:**\n\n```bash\nmkdir -p libs && cd libs\ngit clone https:\u002F\u002Fgithub.com\u002FVchitect\u002FVBench.git\n```\n\nAdd the following to `libs\u002FVBench\u002Fvbench\u002F__init__.py`:\n\n```python\nimport sys, os\nlocal_lib_path = os.path.abspath(\"libs\u002FVBench\")\nif local_lib_path not in sys.path:\n    sys.path.append(local_lib_path)\n```\n\nIf you encounter a NumPy 2.0 compatibility error (`np.sctypes was removed`), modify lines 45–47 of `[YOUR_PYTHON_LIBS]\u002Fimgaug\u002Fimgaug.py`:\n\n```python\n# Replace:\n# NP_FLOAT_TYPES = set(np.sctypes[\"float\"])\n# NP_INT_TYPES = set(np.sctypes[\"int\"])\n# NP_UINT_TYPES = set(np.sctypes[\"uint\"])\n\n# With:\nNP_FLOAT_TYPES = {np.float16, np.float32, np.float64, np.longdouble}\nNP_INT_TYPES = {np.int8, np.int16, np.int32, np.int64, np.longlong}\nNP_UINT_TYPES = {np.uint8, np.uint16, np.uint32, np.uint64, np.ulonglong}\n```\n\nTo save disk space, remove unnecessary files:\n\n```bash\nrm -rf libs\u002FVBench\u002FVBench-2.0 libs\u002FVBench\u002F.git libs\u002FVBench\u002Fasset libs\u002FVBench\u002Fvbench2_beta_trustworthiness\n```\n\n**GenEval:**\n\n```bash\ncd libs\ngit clone https:\u002F\u002Fgithub.com\u002Fdjghosh13\u002Fgeneval.git\ncd geneval\n.\u002Fevaluation\u002Fdownload_models.sh \"..\u002F..\u002Fcheckpoints\u002F\"\n\ncd ..\npip install mmcv-full\ngit clone https:\u002F\u002Fgithub.com\u002Fopen-mmlab\u002Fmmdetection.git\ncd mmdetection && git checkout 2.x\npip install -v -e . --no-build-isolation\n```\n\nThe GenEval model paths are configured in `configs\u002Fevaluation\u002Fevaluation.yaml` under `model.evaluation.geneval`:\n\n```yaml\nmodel:\n  evaluation:\n    geneval:\n      model_path: checkpoints\u002Fevaluation\u002Fmask2former_swin-s-p4-w7-224_lsj_8x2_50e_coco.pth\n      model_config_path: libs\u002Fmmdetection\u002Fconfigs\u002Fmask2former\u002Fmask2former_swin-s-p4-w7-224_lsj_8x2_50e_coco.py\n      clip_path: checkpoints\u002Fevaluation\u002FViT-L-14.pt\n```\n\n### Step 3: Configure API Keys\n\nSome benchmarks (OpenVE-Bench, RefVIE-Bench, ImgEdit-Bench, Intelligent-VBench) require LLM API calls for metric computation. Configure your API keys in `configs\u002Fevaluation\u002Fevaluation.yaml` under `model.evaluation`:\n\n```yaml\nmodel:\n  evaluation:\n    # For OpenVE-Bench, RefVIE-Bench, Intelligent-VBench\n    gemini:\n      api_key: \"YOUR_GEMINI_API_KEY\"\n      base_url: \"YOUR_GEMINI_BASE_URL\"\n      model: \"gemini-2.5-pro-06-17\"\n    # For ImgEdit-Bench\n    openai:\n      api_key: \"YOUR_OPENAI_API_KEY\"\n      base_url: \"YOUR_OPENAI_BASE_URL\"\n      model: \"gpt-4.1\"\n```\n\n\n## Run Evaluation\n\nOnce the environment is set up, you can simply run evaluation with:\n\n```bash\nNUM_GPUS=8\n\naccelerate launch --num_processes=${NUM_GPUS} \\\n    -m scripts.evaluation.evaluate \\\n    --config configs\u002Fevaluation\u002Fevaluation.yaml \\\n    --checkpoint_dir checkpoints\u002FLoomVideo \\\n    --generation_configs configs\u002Fdataset\u002Fbenchmarks.yaml \\\n    --output_dir results\u002Fevaluation \\\n    --calculate_metrics\n```\n\n\n# 📧 Contact\n\nJianzong Wu (吴健宗): jzwu@stu.pku.edu.cn\n\n\n# 📄 Citation\n\nIf you find our work helpful, please consider giving us a ⭐ on this repo and citing our paper as follows:\n\n```bibtex\n@article{wu2026loomvideo,\n  title={LoomVideo: Unifying Multimodal Inputs into Video Generation and Editing},\n  author={Wu, Jianzong and Lian, Hao and Yang, Jiongfan and Hao, Dachao and Tian, Ye and Tong, Yunhai and Zhu, Jingyuan and Chen, Biaolong and Qi, Qiaosong and Zhang, Aixi and He, Wanggui and Liu, Mushui and Huang, Pipei and Jiang, Hao},\n  journal={arXiv preprint arXiv:2606.06042},\n  year={2026}\n}\n```",2,"2026-06-11 04:12:22","CREATED_QUERY"]