What is SynthLight
SynthLight is a portrait relighting technology based on diffusion models jointly launched by Yale University and Adobe Research. It is trained by simulating synthetic data under different lighting conditions, and can re-render portrait photos into images with entirely new lighting effects, such as adding highlights, shadows, or adjusting the overall lighting atmosphere. The core lies in generating synthetic datasets based on a physically-based rendering engine, and through multi-task training and classifier-guided diffusion sampling strategies, it bridges the gap between synthetic data and real images, achieving high-quality relighting of real portraits.
Main features of SynthLight
- Portrait relighting: Relights portraits through environment lighting maps, generating realistic lighting effects such as natural highlights, projections, and diffuse reflections.
- Multi-task training: Conducts multi-task training based on real portraits without lighting labels, improving the model's generalization ability and enabling it to handle various real-world scenarios.
- Inference-time sampling: Adopts a diffusion sampling procedure based on classifier-free guidance, preserving the details of the input portrait during the inference stage to ensure the generated lighting effects are natural and delicate.
- Physically-based rendering engine: Uses a physically-based rendering engine to generate synthetic datasets, simulating lighting transformations under different lighting conditions and providing high-quality lighting effects.
- Generalizability: Although trained only on synthetic data, SynthLight generalizes well to real-world scenarios, including half-body and full-body portraits.
Technical principles of SynthLight
- Treating relighting as a re-rendering problem: SynthLight defines image relighting as a re-rendering process, adjusting the rendering effect of pixels by changing environmental lighting conditions.
- Physically-based rendering engine and synthetic datasets: The technology is based on a Physically-Based Rendering Engine to generate synthetic datasets. By rendering 3D head models under different lighting conditions, it simulates pixel changes under real lighting conditions.
- Multi-task training strategy: SynthLight adopts a multi-task training method, using real portraits without lighting labels for training. By combining training on real images and synthetic images, it helps the model better adapt to real-world scenarios and reduces the domain gap between synthetic data and real images.
- Diffusion sampling based on classifier-free guidance: During the inference stage, SynthLight uses a diffusion sampling procedure based on Classifier-Free Guidance. It utilizes the details of the input portrait to better preserve the image's texture and features while generating realistic lighting effects.
- Advantages of diffusion models: Diffusion models generate images by gradually adding noise and learning the denoising process. Based on this characteristic, SynthLight generates high-quality lighting effects, including specular highlights, shadows, and subsurface scattering.
SynthLight project address
- Project official website：https://vrroom.github.io/synthlight
- arXiv technical paper：https://arxiv.org/pdf/2501.09756
Application scenarios of SynthLight
- Post-processing of portrait photography: SynthLight can perform relighting processing on real portrait photos, generating realistic lighting effects such as natural highlights, shadows, and diffuse reflections.
- Virtual scene rendering and synthesis: By simulating different lighting conditions, SynthLight can seamlessly blend real portraits with virtual scenes to generate high-quality composite images.
- Game Development and Character Design: In game development, SynthLight can be used to quickly adjust characters' lighting effects to adapt to different scenes and environments, saving time and cost.
- Advertising and Commercial Image Production: In advertising and commercial image production, SynthLight can quickly generate images under various lighting conditions, helping designers rapidly explore different visual effects and improving work efficiency.