SuperAPI AI Directory
Partner with us中文AI newsFavorites
All toolsAI videoVideoTuna

VideoTuna

VideoTuna is a codebase for applications such as text-to-video, and is said to be the first repository that integrates multiple AI video generation models for inference and fine-tuning, including text-to-video (T2V), image-to-video (I2V), text-to-image (T2I), and video-to-video (V2V). It is also the first repository to provide a complete video generation pipeline from fine-tuning, pretraining, and continual training to post-training (alignment). It supports inference and fine-tuning for multiple pretrained video generation models, and includes capabilities such as continual training, human preference alignment, and video post-processing enhancement. The project's latest updates include support for Wan2.1, Step Video inference and HunyuanVideo T2V fine-tuning, Flux-T2I fine-tuning, Hunyuan Video and Mochi inference, and more. Usage requires preparing the environment, downloading model checkpoints, and running inference and training commands on GPUs.

Open website

About this tool

VideoTuna is a codebase for text-to-video applications, maintained by VideoVerses. According to the materials, it is the first repository to integrate multiple AI video generation models, covering text-to-video (T2V), image-to-video (I2V), text-to-image (T2I), and video-to-video (V2V) generation, for model inference and fine-tuning; it is also the first repository to provide a complete video generation pipeline from fine-tuning, pretraining, and continual training to post-training (alignment).

Main capabilities:

  • Unified framework: inference and fine-tuning for multiple of the latest pretrained video generation models.
  • Continual training: continuously improve models with new data.
  • Fine-tuning: adapt pretrained models to specific domains.
  • Human preference alignment: use RLHF to align with human preferences.
  • Post-processing: enhance and correct videos through video-to-video enhancement models.

Support status (from the project update log):

  • 2025-04-22: Support for Wan2.1 and Step Video inference, and support for HunyuanVideo T2V fine-tuning with a unified codebase architecture.
  • 2025-02-03: Support for automatic code formatting via PR#27.
  • 2025-02-01: Migration to Poetry to simplify dependency and script management (PR#25).
  • 2025-01-20: Support for Flux-T2I fine-tuning.
  • 2025-01-01: Release of VideoVAE+ training in the VideoVAEPlus repository; support for Hunyuan Video and Mochi inference.
  • 2024-12-24: Release of VideoVAE+, which is said to have better video reconstruction performance than NVIDIA's Cosmos-Tokenizer.
  • 2024-12-01: Support for CogVideoX-1.5-T2V&I2V inference and video-to-video enhancement from ModelScope; support for CogVideoX fine-tuning.
  • 2024-11-01: Release of VideoTuna v0.1.0, with initial support for inference of VideoCrafter1-T2V&I2V, VideoCrafter2-T2V, DynamiCrafter-I2V, OpenSora-T2V, CogVideoX-1-2B-T2V, CogVideoX-1-T2V, Flux-T2I, as well as VideoCTraining/fine-tuning for rafter, DynamiCrafter, and Open-Sora.

How to use:

  1. Prepare the environment: the project description mentions it takes about 3 minutes; optionally install flash-attn (the Hunyuan model uses it to reduce memory usage and speed up inference; if not installed, the model runs in normal mode); optional video-to-video enhancement (if the command gets stuck, terminating and rerunning it can resolve the issue). After installing Poetry, on MacOS Apple Silicon chips you need to use docker compose, because some dependencies do not support arm64 (such as bitsandbytes, decord, xformers). To ensure project file permissions, you need to set the corresponding environment variables, then install the dependencies. Note that installing swissarmytransformer may hang; retrying should succeed. You can add dependencies, check dependencies, run Poetry commands, and start a terminal.
  2. Prepare checkpoints: you need to download the model checkpoints according to docs/checkpoints.md, and after downloading, place them according to the checkpoint structure.
  3. Inference: run the corresponding command, and it will automatically execute T2V/T2I based on the prompts in inputs/t2v/prompts.txt, and execute I2V based on the images and prompts in inputs/i2v/576x1024.
  4. Fine-tune the T2V model: use the provided toydataset or build your own dataset according to docs/datasets.md. All training commands were tested on an H800 80G GPU. VBench evaluation is supported to evaluate T2V generation performance; see eval/README.md for details. Git hooks are handled through the pre-commit library; you can install the hooks to check formatting, code style, and types on commit, or run the hooks directly without committing.

Applicable scenarios: suitable for developers and researchers who need a unified codebase for inference, fine-tuning, continued training, alignment, and post-processing of multiple video generation models.

Limitations:

  • Usage requires preparing the environment and downloading model checkpoints; training commands were tested on an H800 80G GPU.
  • MacOS Apple Silicon chips require docker compose because some dependencies do not support arm64.
  • Installing swissarmytransformer may hang, and the command line may get stuck and require retrying.
  • Licenseis CC-BY-NC-ND; if authorization is needed, contact the project lead.

Related tools

Browse category