Purpose: PhotoMaker is used for personalized text-to-image generation, focusing on generating realistic portraits of people. It takes multiple photos of the same identity as input, extracts character features, and then generates a new, personalized character image based on a text description, with styles that can cover photos, paintings, avatars, and more.
Main capabilities:
- Encode any number of input ID images into a stacked ID embedding as a unified ID representation, which can both comprehensively encapsulate the features of the same ID and accommodate the features of different IDs for subsequent integration.
- Support diverse applications: including recontextualization (generation under basic prompts), bringing characters from artwork/old photos into reality, changing age or gender (by replacing category words such as man and woman), and identity blending (input images of different IDs and integrate features to form a new ID, with the blending ratio adjustable by controlling the proportion of identity images in the input image pool or the prompt weight).
- Compared with methods based on test-time fine-tuning, it has better ID preservation ability, while significantly improving speed, with high generation quality and strong generalization ability.
Usage:
- The user provides multiple input images of the same identity, and the system converts them into a stacked ID embedding.
- During inference, the images that make up the stacked ID embedding can come from different IDs.
- First, obtain text embeddings and image embeddings from the text encoder and image encoder respectively.
- Then, by merging the corresponding category embeddings (e.g., man and woman) with each image embedding, extract the fused embeddings.
- Next, concatenate all fused embeddings along the length dimension to form the stacked ID embedding.
- Finally, feed the stacked ID embedding into all cross-attention layers to adaptively merge the ID content in the diffusion model.
- Note: During training, images of the same ID with the background masked are used, but during inference, images of different IDs can be directly input without background distortion to create a new ID.
Applicable scenarios:
- Personalized character photo generation, such as creating photos, paintings, or avatars for anyone.
- Bring characters from artistic paintings, sculptures, or old photos into the modern era and generate "photographed" photos.
- Stylize while preserving ID attributes.
- Change age or gender while maintaining the original identity.
- Identity blending, integrating features of different IDs to form a new ID.
Limitations:
- More information about fees, pricing, platform, effects, or specific usage steps is not provided for now.
- The project page borrowed from DreamBooth.
- The paper discusses the ID-oriented data construction pipeline, but the specific details are not expanded here.
