What is ConsiStory
ConsiStory is a training-free text-to-image generation method jointly developed by researchers from NVIDIA and Tel Aviv University, which can rapidly and naturally extend images to different scenes while following different text prompts, all while keeping the style and subject unchanged. The core idea of ConsiStory is to achieve subject consistency by sharing the internal activations of a pretrained text-to-image model during the image generation process. This method does not require any form of optimization or pretraining of the model, thereby greatly simplifying the process of generating consistent images.
ConsiStory Official Website Entry
- Official Project Homepage:https://consistory-paper.github.io/
- Arxiv Research Paper:https://arxiv.org/abs/2402.03286
- GitHub Source Code Repository: Coming Soon
Main Features of ConsiStory
- Training-Free: ConsiStory does not require any form of optimization or personalized training of a pretrained text-to-image (T2I) model, meaning users can directly use existing models to generate consistent images, greatly saving time and resources.
- Consistent Subject Generation: This method can generate a series of images that maintain the same subject identity under different text prompts, such as the same person, animal, or object. This is very useful for applications that require consistent visual elements (such as storybooks, character design, virtual asset creation, etc.).
- Cross-Frame Consistency: Through internal activation sharing and attention mechanisms, ConsiStory ensures that the generated images remain consistent in subject features, even under different backgrounds and situations.
- Layout Diversity: To increase the diversity of generated images, ConsiStory employs techniques such as attention dropout and query feature mixing to avoid excessive consistency in image layout.
- Compatibility: This method is compatible with existing image editing tools (such as ControlNet) and can be used in combination to achieve more complex image control.
- Fast Generation: Since no training step is required, ConsiStory can generate images quickly, about 20 times faster than existing state-of-the-art (SoTA) technologies.
Technical Principles of ConsiStory
- Subject Localization: At each step of the generation process, ConsiStory first localizes the subject in each generated image. This is done by analyzing the model's cross-attention features, which help identify regions in the image that may contain the subject.
- Subject-Driven Shared Attention: ConsiStory extends the self-attention mechanism, allowing a query in one image to attend not only to features of its own image but also to subject-related features in other images. In this way, different instances of the same subject can influence each other during the generation process, thereby maintaining consistency. To limit consistency in background and layout, ConsiStory uses subject masks to ensure that only subject-related features are shared.
- Layout Diversity Enhancement: To maintain the diversity of generated images, ConsiStory adopts two strategies: first, mixing features from non-consistency sampling steps with generated features; second, introducing random attention dropout during the shared attention process to reduce excessive consistency between different images.
- Feature Injection: To further improve subject consistency, particularly in details, ConsiStory introduces a feature injection mechanism. By constructing dense correspondence maps across images (using DIFT features), ConsiStory can precisely align and blend features between different images to enhance subject consistency.
- Anchor images and reusable subjects: To improve computational efficiency, ConsiStory can select a portion of generated images as "anchor images". In the shared attention step, only the anchor images share and receive features from other images. This not only reduces the computational burden but also improves generation quality and allows the same subject to be reused in new scenes.
- Multi-subject consistent generation: ConsiStory can handle images containing multiple subjects. By simply taking the union of all subject masks, the consistency of multiple subjects can be maintained within a single image.