ViMax - HKU's Open-Source Multi-Agent Video Generation Framework

What is ViMax

ViMax is an end-to-end multi-agentvideo generationframework launched by the HKU Data Science Laboratory, supporting the automatic transformation of ideas, scripts, or novels into complete videos. The framework integrates the functions of director, screenwriter, producer, and video generator, supports modes such as Idea2Video, Novel2Video, Script2Video, and AutoCameo, and can generate minute-long videos while maintaining consistency of characters and scenes. Through technologies such as intelligent storyboarding, multi-camera simulation, and automated consistency detection, ViMax achieves an efficient automated process from idea to finished film, greatly simplifying video creation, lowering the technical threshold, and providing creators with a powerful tool.

ViMax's Main Features

Idea2Video: Transform simple creative concepts into complete video stories, suitable for preliminary ideas without a detailed script.
Novel2Video: Automatically adapt long novels into episodic video content, suitable for film and television adaptation of literary works.
Script2Video: Generate videos based on detailed scripts, suitable for creators who already have mature scripts.
AutoCameo: After users upload photos, generate videos containing their own image, achieving a personalized interactive experience.

ViMax's Technical Principles

ViMax uses a multi-agent collaborative architecture to decompose the video generation task into multiple modules, completed by different agents with divided labor:

Input Parsing: Extract key information from the idea or script, such as characters, scenes, and style.
Script Understanding and Storyboard Design: Based on the extracted information, generate a detailed storyboard script, planning shooting angles and narrative rhythm.
Visual Asset Planning: Intelligently select reference images and design appropriate scene layouts and styles for each shot.
Consistency Check: Use MLLM/VLM models to detect the consistency of generated images, ensuring that characters and scenes remain coherent throughout the video.
Parallel Generation and Composition: Efficiently generate shots through parallel processing technology, and stitch the shots into a complete video.

ViMax's Project Address

GitHub Repository：https://github.com/HKUDS/ViMax

ViMax's Application Scenarios

Short Video Production: Creators quickly transform ideas into short videos for use on social media platforms (such as Douyin, Bilibili, etc.).
Educational Videos: Transform complex teaching content into vivid videos, helping students better understand and remember.
Interactive Videos: Through the AutoCameo feature, users integrate their own image into videos, increasing interactivity and fun.
Novel Visualization: Adapt long novels into video content, providing new forms of dissemination for literary works.
Personal Story Videos: Users transform their own stories or ideas into videos for personal commemoration or sharing.