What is SynCD
SynCD (Synthetic Customization Dataset) is a high-quality synthetic training dataset launched by Carnegie Mellon University and Meta, used to enhance the customization capabilities of text-to-image models. SynCD contains multiple images of the same object under different lighting, backgrounds, and poses, ensuring object consistency across different images based on Masked Shared Attention and 3D asset guidance (such as Objaverse). SynCD uses a language model (LLM) to generate detailed object descriptions and background scenes, combined with a depth-guided text-to-image model to generate coupled images. SynCD solves the problem of difficulty in collecting multi-view, multi-background object images at scale in the real world, providing rich training resources for tuning-free model customization, and significantly improving the image quality and identity preservation capabilities of models when generating specific objects in new scenes.
Main Features of SynCD
- Provide Diverse Training Samples: Based on generating images from multiple perspectives and backgrounds, it increases the model's visual understanding of objects.
- Enhance Object Consistency: Based on Masked Shared Attention and 3D asset guidance, it ensures objects remain consistent across different images, avoiding drift in object features in generated images.
- Improve Generation Quality: Using high-quality synthetic data, it improves the image quality and identity preservation capabilities of models in customization tasks.
- Support Tuning-Free Customization: It provides data support for tuning-free methods, avoiding expensive optimization steps for each new object.
Technical Principles of SynCD
- LLM-assisted Prompt Generation：
- Use a language model (such as LLama3) to generate detailed object descriptions and background scene descriptions. For rigid objects, use object descriptions provided by Cap3D; for deformable objects, generate detailed descriptions directly from category names.
- Generate multiple background descriptions based on the LLM, combine object descriptions with background descriptions, and generate prompts for multiple images.
- Masked Shared Attention (MSA)：
- When generating multiple images, based on the Masked Shared Attention mechanism, share the features of foreground object regions to ensure object consistency across different images.
- In the attention module of the diffusion model, the features of each image not only focus on itself but also on object features in other images, ignoring background regions based on a mask.
- 3D Asset Guidance：
- For rigid objects, use 3D assets in Objaverse for multi-view rendering to generate depth maps and corresponding images.
- Based on depth guidance and multi-view correspondences, it further enhances the 3D consistency of objects. It supports pixel-level cross-view correspondences, "warping" features from one image to other images, ensuring consistent shape and color of objects under different viewpoints.
- Data Filtering and Quality Control：
- Use aesthetic score and object similarity (computed through the DINOv2 feature space) to filter out low-quality or inconsistent images, ensuring the quality of the final dataset.
- Based on automated filtering steps, remove images that do not meet requirements, and ultimately generate a high-quality synthetic dataset.
SynCD Project Address
- Project Website：https://www.cs.cmu.edu/~syncd
- GitHub Repository：https://github.com/nupurkmr9/syncd
- arXiv Technical Paper：https://arxiv.org/pdf/2502.01720
SynCD Application Scenarios
- Personalized Content Generation: Users upload photos of personal belongings or pets, and combine them with text prompts to generate new images of them in different scenarios or styles, meeting personalized needs.
- Creative Design and Artistic Creation: Designers and artists quickly generate concept images, validate design ideas, or create artworks with specific styles, improving the efficiency of creative expression.
- Virtual Scene Construction: In virtual reality (VR) and augmented reality (AR), generate specific objects in virtual scenes, such as placing virtual characters in different environments, enhancing immersion.
- Advertising and Marketing: Brands generate images of products in different usage scenarios or target audience environments, used in advertising and promotion to improve appeal and persuasiveness.
- Education and Training: In the education field, generate teaching materials, such as placing historical artifacts in ancient scenes, helping students better understand knowledge and improving learning outcomes.