What is AnyStory
AnyStory is an innovative text-to-image generation framework developed by Alibaba Tongyi Lab, achieving high-fidelity personalization for single and multiple subjectsImage Generation. It models the subject personalization problem through an "encoding-routing" approach. In the encoding stage, AnyStory combines a powerful ReferenceNet and a CLIP vision encoder to encode subject features with high fidelity, capturing rich details and semantic information. ReferenceNet supports high-resolution input and is aligned with the feature space of the denoising U-Net, providing a solid detail foundation for generated images; the CLIP vision encoder is responsible for extracting the coarse concept of the subject, ensuring that the generated images are closely aligned with the text description. In the routing stage, the decoupled instance-aware subject router can accurately perceive and predict the position of the subject in the latent space, guiding the injection of subject conditions, effectively avoiding the subject mixing problem commonly seen in multi-subject generation, so that each subject can maintain its unique features and details in the generated image.
Main Features of AnyStory
-
High-Fidelity Single-Subject Personalization: AnyStory can generate high-fidelity images with a specific subject, capturing rich details and semantic information, making the generated images closely aligned with the text description.
-
Multi-Subject Personalization: In scenarios involving multiple subjects, AnyStory can accurately perceive and predict the position of each subject in the latent space, guiding the injection of subject conditions, avoiding the subject mixing problem commonly seen in multi-subject generation, so that each subject can maintain its unique features and details in the generated image.
-
Text Description Alignment: AnyStory ensures that the generated images are closely aligned with the text description, extracting the coarse concept of the subject through the CLIP vision encoder to ensure consistency between the generated images and the text description.
-
Detail Preservation: Based on ReferenceNet, AnyStory can preserve the fine details of the subject, providing high-quality image generation support.
-
Flexible Subject Condition Injection: Through the decoupled instance-aware subject router, AnyStory can flexibly inject subject conditions into the latent space, achieving precise control over the generated images.
Technical Principles of AnyStory
- Encoding Stage
-
ReferenceNet: It is a general and powerful image encoder that supports high-resolution input and can extract fine details of the subject. The feature space of ReferenceNet is aligned with the denoising U-Net, making it easy for the U-Net to directly extract subject features at different depths and scales. ReferenceNet encodes the reference image through a variational autoencoder (VAE), and then extracts its features through a network with the same architecture as the U-Net.
-
CLIP Vision Encoder: The CLIP vision encoder is responsible for extracting the coarse concept of the subject, ensuring that the generated images are closely aligned with the text description. CLIP's features are mainly semantic; although the resolution is lower, they can provide features that align well with text embeddings, triggering subject generation.
-
- Routing Stage
-
Decoupled Routing Mechanism: The position of the subject in the latent space is predicted through independent branches. Given a series of segmented subject images, routing features are obtained through the CLIP image encoder and an additional single-query QFormer respectively. These features model the router through an image segmentation decoder, generating a coarse routing map, which is then refined through a masked cross-attention module to guide the injection of subject information.
-
Instance-Aware Routing Regularization Loss: To facilitate router learning and distinguish different subject instances, an instance-aware routing regularization loss is introduced. This loss function ensures that the router can accurately route subject features to the corresponding regions, reducing the impact on irrelevant regions.
-
Routing-Guided Subject Information Injection: For CLIP-encoded subject representations, decoupled cross-attention is used to integrate them into the U-Net, adding routing-guided localization constraints. For ReferenceNet-encoded subject features, the original reference attention is adopted, adding attention masks induced by the routing map to ensure that subject features are accurately injected into the target regions.
-
- Training Stage
-
Subject Encoder Training: Train the subject QFormer, ReferenceNet, and the corresponding key and value matrices in the attention blocks. Initialize ReferenceNet with pretrained U-Net weights, and collect a large amount of paired data that maintains subject identity consistency but shows variation in background, pose, and viewpoint for training to prevent subject overfitting.
-
Router Training: Fix the subject encoder and train the router. The main training data includes additional unpaired multi-person images. The router can effectively generalize to general subjects, attributed to the strong generalization ability of the CLIP model and the highly compressed single-token routing features.
-
AnyStory Project Address
- Project Website：https://aigcdesigngroup.github.io/AnyStory
- arXiv Technical Paper：https://arxiv.org/pdf/2501.09503v1
AnyStory Application Scenarios
-
Creative Painting: Artists can use AnyStory to transform ideas in their minds into visual works. For customizing personalized portraits for specific clients, AnyStory can provide high-quality image generation support and shorten the creation cycle.
-
Concept Design: In the early design stage, AnyStory can help artists and designers quickly generate concept images, providing references for subsequent detailed design.
- Character Concept Art: Game designers can use AnyStory to quickly generate concept art for multiple characters based on the game's background story and character settings.
-
Comic Creation: Provide comic creators with tools to quickly generate characters and scenes, accelerating the creation process and improving the visual effect of the works.
- Personalized Advertising: Based on different target audiences and marketing themes, quickly generate images containing specific products or brand elements, improving the appeal and effectiveness of advertisements.