What is Phantom
Phantom is a framework launched by ByteDance's Intelligent Creation team for subject-consistentvideo generation(Subject-to-Video, S2V). Based on cross-modal alignment technology, it combines text and image prompts to extract subject elements from reference images and generate video content consistent with the text description. Phantom is based on existingtext-to-video(T2V) and image-to-video (I2V) architectures. It redesigns the joint text-image injection model and learns cross-modal alignment based on text-image-video triplet data. The Phantom framework supports single-subject and multi-subject references, particularly emphasizing subject consistency in human generation tasks, covering existing identity-preserving video generation tasks and providing enhanced advantages.
Phantom's Main Features
- Extract subject elements from reference images: Identify and extract the subject (such as people, animals, objects, etc.) from the image as the core content for generating the video.
- Generate videos based on text prompts: Users control the content and style of the video based on text instructions, enabling highly customized video generation.
- Multi-subject video generation: Supports processing multiple subjects simultaneously and generating complex interaction scenes, such as multi-person interactions and human-pet interactions.
- Identity preservation (ID-Preserving): When generating videos, preserve the subject's identity features (such as faces, clothing, etc.), especially suitable for scenarios such as virtual try-on and digital human generation.
- High-quality video output: The generated videos perform excellently in visual effects, subject consistency, and text responsiveness, comparable to existing commercial solutions.
Phantom's Technical Principles
- Data structure design: Phantom constructs a text-image-video triplet data structure for training the model to understand the relationships between different modalities. The data is divided into In-paired (image and video subjects consistent) and Cross-paired (cross-video matching) types, avoiding the model simply copying the input image.
- Model architecture: Based on existing text-to-video (T2V) and image-to-video (I2V) architectures, the joint text-image injection model is redesigned. The model is divided into an Input Head and a trainable DiT module. The Input Head is responsible for encoding video, text, and reference images, and the DiT module is responsible for cross-modal alignment and video generation.
- Cross-modal alignment: After the reference image is encoded by a specific visual encoder (such as VAE and CLIP), it is concatenated with video features and text features respectively, and input into the visual and text branches of the DiT module.
- Identity preservation technology: When processing identity features such as faces, the similarity between the generated video and the reference image is evaluated based on facial recognition models (such as ArcFace) to ensure consistency of the subject's identity.
- Optimization and training: Based on large-scale triplet data training, learn how to balance the dual prompts of text and images when generating videos. The model inherits the weights of the base model in the pre-training stage and is further fine-tuned based on cross-modal data to achieve high-quality video generation.
Phantom's Project Address
- Project official website：https://phantom-video.github.io/Phantom/
- GitHub repository：https://github.com/Phantom-video/Phantom
- HuggingFace model library：https://huggingface.co/bytedance-research/Phantom
- arXiv technical paper：https://arxiv.org/pdf/2502.11079
Phantom's Application Scenarios
- Virtual try-on: Generate dynamic clothing display videos to help users preview the effect.
- Digital human generation: Create virtual characters with specific appearances for scenarios such as virtual streamers.
- Advertising video production: Quickly generate product advertisements based on images and text, improving production efficiency.
- Film and television animation: Generate character animation prototypes to assist creative validation and reduce production costs.
- Education and training: Generate teaching videos such as scientific experiments and historical scenes to enhance interactivity.