SuperAPI AI Directory
Partner with us中文AI newsFavorites
All toolsAI videoOpen-Sora

Open-Sora

Open-Sora is an open-source video generation project initiated by hpcaitech, with the goal of making high-quality video generation efficient and accessible to everyone. The project follows open-source principles, publicly disclosing models, tools, and implementation details, and provides a complete pipeline from video data preprocessing and accelerated training to inference generation. Multiple versions have been released so far: Open-Sora 1.0 (2024.03.18), 1.1 (2024.04.25), 1.2 (2024.06.17), 1.3 (1B, 2025.02.20), and Open-Sora 2.0 (11B, 2025.03.12). According to official statements, the 11B model of Open-Sora 2.0 performs comparably to 11B HunyuanVideo and 30B Step-Video on VBench and human preference evaluations, and is an open-source solution with publicly available checkpoints and training code. The model supports text-to-video and image-to-video generation, and the 11B model supports 256px and 768px resolutions. The project also provides step-by-step guidance for model training and fine-tuning, video autoencoder training and evaluation, and offers a Gradio demo.

Open website

About this tool

Purpose

Open-Sora is an open-source initiative dedicated to efficiently generating high-quality video, maintained by hpcaitech. Its officially stated goal is to make the model, tools, and all details available to everyone, lowering the barrier to using advanced video generation technology through open source, and providing a simplified, user-friendly platform for the complex process of video generation, thereby promoting innovation, creativity, and inclusivity in the field of content creation.

Main Capabilities

  • Video generation: Supports text-to-video and image-to-video. Official statements explain that its model is optimized for image-to-video, but it can also be used for text-to-video; to generate high-quality video, the project uses the flux text-to-image model to build a text-to-image-to-video pipeline.
  • Resolution and parameters: The Open-Sora 2.0 (11B) model supports 256px and 768px resolutions, with T2V and I2V supported by the same model. The generation aspect ratio can be adjusted via --aspect_ratio, with candidate values including 16:9, 9:16, 1:1, 2.39:1; the generation length can be adjusted via --num_frames, and the candidate values should be 4k+1 and less than 129.
  • Motion score: During training, the project adds a motion score to the text prompt; during inference, a command can be used to generate video with a motion score, with a default score of 4. The project also provides a dynamic motion score evaluator, which can evaluate the motion score of a video after setting an OpenAI API key.
  • Prompt optimization: The project uses ChatGPT to optimize prompts, and this feature is available for both text-to-video and image-to-video.
  • Reproducibility: Results can be made reproducible by setting a random seed; --num-sample k can be used to generate k samples for each prompt.
  • Training and fine-tuning: Official guidance is provided for step-by-step training or fine-tuning of your own model, and step-by-step training and evaluation of video autoencoders.
  • Version iteration: Open-Sora 1.1 supports 2s~15s, 144p to 720p, arbitrary aspect ratio text-to-image, text-to-video, image-to-video, video-to-video, and infinite-length generation, and released a complete video processing pipeline; Open-Sora 1.2 includes 3D-VAE, rectified flow, score condition; Open-Sora 1.3 It adopts shift-window attention, a unified spatiotemporal VAE, etc.

Usage

  • Installation and deployment: The project provides checkpoint downloads (Hugging Face and ModelScope, including the 11B model). Flash attention 3 can optionally be installed for acceleration. The inference methods supported by the 11B model include direct text-to-video generation, as well as a text-to-image then image-to-video pipeline.
  • Inference commands: It provides text-to-video commands at 256x256 and 768x768 resolutions, direct text-to-video commands, image-to-video commands with a reference image and prompt, generation commands with a motion score (default 4), motion score evaluation commands, prompt optimization commands, random seed setting commands, and the --num-sample k multi-sampling command.
  • Computational efficiency: The official tests of text-to-video computational efficiency were conducted on H100/H800 GPUs. 256x256 uses colossalai's tensor parallelism with --offload True enabled; 768x768 uses colossalai's sequence parallelism; both use 50 steps. Results are presented in the format "total time (s)/peak GPU memory (GB)" (the specific values are not listed in the provided materials).
  • Cost optimization: The project mentions a 50% reduction in video generation model development costs (2024.12.23 blog), provides an open-source solution and H200 GPU vouchers; Open-Sora 2.0's training code and checkpoints are fully open source, with a training cost of $200,000; the 2024.03.04 blog mentions a 46% reduction in training costs.

Applicable scenarios

It is suitable for individuals and teams who wish to research, reproduce, or develop video generation based on open-source solutions, including training and fine-tuning of video generation models, research on video autoencoders, exploration of applications for text-to-video and image-to-video generation, and innovative practices in the field of content creation. The project maintains different branches by version (the latest is main, and older versions include v1.0, v1.1, v1.2, v1.3).

Limitations and notes

  • The project is under active development, and different versions are maintained in different branches.
  • The comparison data for Open-Sora 2.0 is officially published: on VBench, compared with Open-Sora 1.2, it narrows the gap with OpenAI Sora from 4.52% to 0.69%; the human preference results are comparable to HunyuanVideo 11B and Step-Video 30B.
  • Demo videos are shown in compressed GIF format; original-quality samples and corresponding prompts require visiting their Gallery; prompts are cropped when displayed.
  • The specific installation commands, complete command examples, and specific efficiency values are not fully provided in the supplied materials and need to be referenced from the project repository and related reports.

Related tools

Browse category