CogVideo is an open-source project from THUDM (now known as zai-org), containing two main model series: CogVideo (ICLR 2023) and CogVideoX (since 2024). CogVideo is the first open-source large-scale text-to-video generation model based on Transformer; CogVideoX is an upgraded version that supports text-to-video, video continuation, and image-to-video generation.
In terms of use, CogVideo is used to generate videos based on text prompts or images, suitable for research, development, and content creation. The models are trained on long text prompts, so the official recommendation is to use large language models (such as GLM-4, GPT-4) to optimize input prompts to improve generation quality.
Main capabilities include: text-to-video generation (CogVideoX-5B-T2V, CogVideoX-2B, etc.); image-to-video generation (CogVideoX-5B-I2V, etc.), which uses an image as the background combined with prompt words to generate a video, improving controllability; video continuation; and extended functions such as DDIM Inverse, Lora fine-tuning, and quantized inference. The CogVideoX1.5-5B series supports 10-second videos and higher resolution, and CogVideoX1.5-5B-I2V supports image-to-video generation at arbitrary resolution. The project has also open-sourced the video captioning model CogVLM2-Caption, used to convert video data into text descriptions; and 3D Causal VAE, used for nearly lossless video reconstruction.
Usage: The project provides various inference and fine-tuning code. Developers can refer to sat_demo (SAT weight inference and fine-tuning) and diffusers_demo (inference code based on the diffusers library, including parameter descriptions). The official project provides notebooks that can be run directly on the free Colab T4, such as CogVideoX-5B-T2V-Colab.ipynb, the quantized inference version CogVideoX-5B-T2V-Int8-Colab.ipynb, image-to-video CogVideoX-5B-I2V-Colab.ipynb, and video-to-video CogVideoX-5B-V2V-Colab.ipynb. There are also scripts such as cli_demo, cli_demo_quantization, gradio_web_demo, and finetune_demo. Users need Python 3.10 to 3.12. CogVideoX supports quantizing the text encoder, Transformer, and VAE through PytorchAO and Optimum-quanto to reduce VRAM, TorchAO quantization is compatible with torch.compile; FP8 precision requires NVIDIA H100 or above devices.
Applicable scenarios include rapid iterative development by researchers, developers integrating video generation into workflows, and content creation. The community already has multiple adaptation projects, such as CogStudio (Gradio Web UI), ComfyUI-CogVideoXWrapper, VideoSys, xDiT multi-GPU inference, cogvideox-factory single 4090 fine-tuning framework, CogVideoX-Fun, ConsisID identity-preserving generation, CogVideoX-Interpolation keyframe interpolation, etc.
In terms of limitations, the model only supports English input, and other languages need to be translated into English; CogVideoX-2B is trained with FP16, and CogVideoX-5B is trained with BF16, so it is recommended to use the corresponding precision for inference; quantized inference will reduce speed; some optimizations have not been tested on non-NVIDIA Ampere and above architectures, and disabling optimizations will consume about 3 times the VRAM but improve speed by about 3 to 4 times; multi-GPU inference requires disabling enable_sequential_cpu_offload(). The code is released under the Apache 2.0 license, and the CogVideoX-2B model license has been changed to Apache 2.0.
