What is GenMAC
GenMAC is an iterative multi-agent collaboration framework launched by the University of Hong Kong, Tsinghua University, and Microsoft Research, used to solve the problem of complex scene generation in text-to-video generation. Based on decomposing the task into three stages: design, generation, and redesign, it establishes an iterative loop between generation and redesign to gradually verify and optimize video content. The redesign stage in the framework is further subdivided into four subtasks: verification, suggestion, correction, and structured output, which are executed sequentially by specialized agents, using an adaptive self-routing mechanism to select the agent suitable for the current scene, achieving more accurate video generation.
GenMAC's main features
- Compositional text-to-video generation: Generate videos based on complex compositional text prompts, handling scenes including multiple objects, attribute binding, temporal dynamics, and inter-object interactions.
- Iterative workflow: Use an iterative method, including three stages: Design, Generation, and Redesign, and the iterative loop between them, to gradually refine the video content.
- Multi-agent collaboration: The framework uses multiple specialized MLLM (multimodal large language model) agents, each responsible for handling specific subtasks, achieving collective intelligence.
- Task decomposition: The redesign stage is decomposed into four subtasks: verification, suggestion, correction, and structured output, executed sequentially by different agents.
- Adaptive self-routing mechanism: GenMAC designs an adaptive self-routing mechanism to adapt to different generation scenarios, selecting the most suitable agent from a set of specialized agents for correction.
- Improve scene accuracy and text alignment: Use multi-agent collaboration and iterative refinement to improve the accuracy of video scenes and their alignment with text prompts.
GenMAC's technical principles
- Task decomposition and role specialization: Decompose complex video generation tasks into simpler subtasks, assigning a specialized agent to each subtask, with each agent having specific roles and responsibilities.
- Iterative loop: Set up an iterative loop between the generation and redesign stages, where the model gradually verifies and corrects the generated video to better match the text prompt.
- Agent collaboration：
- Verification agent: Checks the alignment of video content with the text prompt.
- Suggestion agent: Proposes correction suggestions based on verification results and selects a suitable correction agent.
- Correction agent: Adjusts the video design according to suggestions, such as layout and guidance scale.
- Structured output agent: Converts correction results into a structured format to provide input for the next iteration of generation.
- Adaptive self-routing: Adaptively selects the most suitable correction agent based on different generation needs and scenarios, addressing issues such as consistency, temporal dynamics, and spatial dynamics.
- Cross-stage information flow: Between the design, generation, and redesign stages, information (such as layout, guidance scale, and text prompts) is continuously updated and passed on, achieving more accurate video generation.
GenMAC's project address
- Project website：karine-h.github.io/GenMAC
- GitHub repository：https://github.com/Karine-Huang/GenMAC
- arXiv technical paper：https://arxiv.org/pdf/2412.04440
GenMAC's application scenarios
- Film and video production: Generate video clips based on the text descriptions of scripts or storyboards, helping directors and producers preview scenes.
- Game development: In game design, generate concept videos of game environments and dynamic scenes to assist game designers in their creative work.
- Advertising and marketing: Generate video advertisements based on advertising copy, quickly transforming creative copy into visual content and improving the efficiency of advertising production.
- Education and training: Create educational videos, presenting complex theories or historical events in video form to enhance the learning experience.
- News and media: Automatically generate news videos based on press releases or reports, improving the efficiency and responsiveness of news production.