What is Video Alchemist
Video Alchemist is a new type ofvideo generationmodel launched by Snap and others, with multi-subject, open-set personalization capabilities. It can generate videos based on text prompts and reference images without requiring optimization at test time. The model is based on a Diffusion Transformer module and incorporates reference image embeddings and subject-level text prompts into the video generation process through dual cross-attention layers. Video Alchemist also introduces an automatic data construction pipeline and various data augmentation techniques to enhance the model's focus on subject identity and avoid the "copy-paste effect". To evaluate its performance, a new video personalization benchmark, MSRVTT-Personalization, was also proposed.
Main features of Video Alchemist
- Personalized video generation: It has built-in multi-subject, open-set personalization capabilities, and can perform personalized generation of foreground objects and backgrounds simultaneously without requiring optimization at test time.
- Conditional generation based on text prompts and reference images: Given a text prompt and a set of reference images to conceptualize the entity words in the prompt, Video Alchemist can generate corresponding videos based on the text and reference images.
- Application of the Diffusion Transformer module: The model is built on a new Diffusion Transformer module, which fuses each conditional reference image and its corresponding subject-level text prompt through an additional cross-attention layer, enabling generation under multi-subject conditions and binding each subject's textual description with its image representation.
Technical principles of Video Alchemist
- Multi-subject open-set personalization: Video Alchemist has built-in multi-subject, open-set personalization capabilities, and can perform personalized generation of foreground objects and backgrounds simultaneously without requiring optimization at test time. It can handle various novel subject and background concepts without the need for separate optimization for each new subject or background.
- Diffusion Transformer module: Video Alchemist is built on a new Diffusion Transformer module, which fuses each conditional reference image and its corresponding subject-level text prompt through additional cross-attention layers. Specifically, the model achieves multi-subject conditional generation through the following steps:
- Input processing: Given a text prompt and a set of reference images, the model first encodes these inputs.
- Cross-attention layers: Through dual cross-attention layers, reference image embeddings and subject-level text prompts are incorporated into the video generation process, so that the generated video can naturally preserve subject identity and background fidelity.
- Subject-level fusion: A subject-level fusion mechanism is introduced to bind each subject's textual description with its image representation, ensuring the accuracy and consistency of subjects in the generated video.
- Automatic data construction pipeline and image augmentation: To address the difficulty of collecting paired reference image and video datasets, Video Alchemist designed a new automatic data construction pipeline and introduced extensive image augmentation techniques to enhance the model's focus on subject identity and avoid the "copy-paste effect":
- Data collection: Collect subject images from multiple frames and perform data augmentation processing.
- Image augmentation: Through various data augmentation techniques such as rotation, scaling, color adjustment, etc., enhance the model's generalization ability and reduce overfitting.
- MSRVTT-Personalization benchmark: To evaluate the performance of Video Alchemist, a new video personalization benchmark, MSRVTT-Personalization, is introduced. It accurately evaluates subject fidelity and supports multiple personalization scenarios, including conditioning modes based on facial crops, single or multiple arbitrary subjects, and combinations of foreground objects and backgrounds.
Video Alchemist's project address
- Project website：https://snap-research.github.io/open-set-video-personalization
- arXiv technical paper：https://arxiv.org/pdf/2501.06187
Video Alchemist's application scenarios
- Short video creation: Individual users can transform creative stories and fantasy scenes into videos, produce unique short videos to share on social platforms, and showcase their personality.
- Animation production: Creators can use Video Alchemist to generate animated characters and backgrounds and quickly produce animated short films without complex animation production software and skills.
- Historical events: Teachers can generate videos of historical events to help students better understand historical backgrounds and event processes.
- Script scenes: Producers and directors can generate preliminary video samples of script scenes for team communication and to present project concepts to investors.
- Character actions: Character actions and expressions can be generated to help actors and directors better understand the character's performance requirements.