What is Movie Gen
Movie Gen is Meta'sAI video generation tool, capable of generating and editing videos based on text prompts, and adding synchronized audio to videos. The technology includes creating HD videos up to 16 seconds long, adding audio to existing videos, editing videos, and creating customized videos based on photos. Movie Gen's video generation model has 30B parameters and can generate videos at 16 frames per second. The audio generation model has 13B parameters and can generate high-quality audio up to 45 seconds long, including ambient sounds, sound effects, and instrumental background music. The tool is currently limited to internal company employees and a small number of partners, and Meta plans to integrate it into existing applications in the future to increase user engagement.
Main features of Movie Gen
- Video generation: Users input text prompts, and Movie Gen generates HD videos up to 16 seconds long. The videos are automatically created based on text descriptions and adapt to different aspect ratios and resolutions.
- Personalized video generation: Users upload photos, and Movie Gen can generate videos containing the person's features while maintaining consistency of the person's identity and actions.
- Precise video editing: Movie Gen supports users in precisely editing videos through text instructions, including adding, deleting, or replacing elements in the video, as well as making background or style changes.
- Audio generation: Movie Gen can generate synchronized high-quality audio for videos, including ambient sound effects, Foley sounds, and instrumental background music. The audio matches the video content and is generated based on text prompts.
Technical principles of Movie Gen
- Model architecture: Movie Gen includes two main foundation models, Movie Gen Video (video generation) and Movie Gen Audio (audio generation).
- Pre-training: The models are pre-trained on large amounts of internet image, video, and audio data to learn various concepts of the visual and auditory world.
- Flow Matching: The generative models are trained using the Flow Matching training objective, a method of iteratively updating samples, used in generating data.
- Variational Autoencoder (VAE): Variational autoencoders are used to compress videos and images into a learned latent space, improving training and inference efficiency.
- Text embedding: A pre-trained text encoder is used to convert input text into text embeddings, which serve as conditional input for the model.
- Multimodal input: The model can process multiple input types, including text, images, and videos, and generate corresponding outputs.
Movie Gen project address
- Project official website：ai.meta.com/blog/movie-gen
- Technical paper：https://ai.meta.com/static-resource/movie-gen-research-paper
Application scenarios of Movie Gen
- Content creation: Create unique video content for social media, blogs, or video sharing platforms.
- Film and video production: Generate concept videos during the pre-production stage, or perform rapid editing and audio mixing during post-production.
- Advertising and marketing: Quickly generate engaging advertising videos to attract potential customers' attention.
- Education and training: Create educational content, such as simulated experiments, historical reenactments, or language learning materials.
- Game development: Generate videos and audio for in-game video content, trailers, or promotional materials.
- News and reporting: Quickly generate visual content for news stories to increase the appeal of reporting.