What is Mora
Mora is a multi-agent (AI Agents) framework launched by researchers from Microsoft and Lehigh University, specifically for general video generation tasks, with the goal of simulating and extending OpenAI'sSoravideo generation model. The core idea of the framework is to generate high-quality video content through the collaboration of multiple visual agents. Mora decomposes the video generation process into multiple subtasks and assigns a specialized agent to each subtask, thereby achieving various video generation functions.
According to the experimental results in the paper, Mora performs excellently in generating high-resolution (1024×576) videos with a duration of 12 seconds, containing a total of 75 frames. However, when it comes to scenes with a large amount of object motion, there is a significant performance gap between Mora and Sora. In addition, attempting to generate videos longer than 12 seconds leads to a significant decline in video quality.
Main features of Mora
- Text-to-video generation: Mora can automatically generate corresponding video content based on the text description provided by the user, and can be applied to video creation ranging from simple scene descriptions to complex storylines.
- Image-to-video generation: In addition to generating videos directly from text, Mora can also combine the initial image and text prompt provided by the user to generate a matching video sequence, enhancing the richness and detail of the content.
- Extended video generation: Mora can not only generate videos from scratch, but also extend and edit existing video content, adding new elements or extending the duration of the video.
- Video-to-video editing: Mora has advanced editing capabilities and can edit videos according to the user's text instructions, such as changing scenes, adjusting object attributes, or adding new elements.
- Connecting videos: Mora can seamlessly connect two or more video clips, creating smooth transition effects, suitable for making video compilations or edits.
- Simulating the digital world: Mora can also create and simulate the digital world, and can create video sequences in the style of the digital world based on text descriptions, such as game scenes or virtual environments.
Mora's official website entry
- GitHub address:https://github.com/lichao-sun/Mora(Source code and model to be open-sourced)
- arXiv research paper:http://arxiv.org/abs/2403.13248
How Mora works
Mora's working principle is based on a multi-agent framework that completes video generation tasks by coordinating multiple specialized AI agents. Each agent is responsible for handling a specific subtask, and these subtasks together constitute the complete video generation process.
The following are the detailed steps of Mora's workflow:
- Task decomposition: Mora decomposes complex video generation tasks into multiple subtasks, each of which is handled by a specialized agent.
- Agent role definition: Mora defines the following five basic role agents:
- Prompt selection and generation agent: Uses large language models (such as GPT-4 or Llama) to optimize and select text prompts to improve the relevance and quality of generated images.
- Text-to-image generation agent: Converts text prompts into high-quality initial images.
- Image-to-image generation agent: Modify the given source image according to the text instruction.
- Image-to-Video Generation Agent: Convert a static image into a dynamic video sequence.
- Video Concatenation Agent: Create a smoothly transitioning video based on two input videos.
- Workflow: Mora automatically organizes agents to execute subtasks in a specific order according to task requirements. For example, a text-to-video generation task may include the following steps:
- First,Prompt Selection and Generation Agentprocesses the text prompt.
- Next,Text-to-Image Generation Agentgenerates an initial image based on the optimized text prompt.
- Then,Image-to-Video Generation Agentconverts the initial image into a video sequence.
- Finally, if needed,Video Concatenation Agentcan concatenate multiple video clips into a coherent video.
- Multi-Agent Collaboration: Agents communicate and collaborate through predefined interfaces and protocols to ensure the coherence and consistency of the entire video generation process.
- Generation and Evaluation: After each agent completes its subtask, it passes the result to the next agent until the entire video generation process is completed. The generated video can be evaluated for quality according to predefined evaluation criteria.
- Iteration and Optimization: The Mora framework allows improving the quality of video generation through iteration and optimization. Agents can adjust their parameters based on feedback to improve the quality of the generated video and its consistency with the text prompt.