What is SHMT
SHMT (Self-supervised Hierarchical Makeup Transfer) is an advanced self-supervised makeup transfer technology jointly developed by Alibaba DAMO Academy and Wuhan University of Technology and other institutions. The technology is implemented through a latent diffusion model and can naturally apply various makeup styles to target facial images without the need for paired training data. SHMT adopts a "disentangle-reconstruct" strategy, combining a Laplacian pyramid and an iterative dual alignment module to achieve flexible control of texture details under different makeup styles and dynamic correction of alignment errors, significantly improving the effectiveness and fidelity of makeup transfer. Its main advantage lies in its ability to handle complex facial features and expression changes and provide high-quality transfer results.
Main features of SHMT
- Efficient transfer:It maintains efficiency and high quality when handling multiple makeup styles, and can naturally and accurately apply diverse makeup styles to given facial images.
- Dynamic alignment correction:Through the iterative dual alignment (IDA) module, it dynamically adjusts the injection conditions at each denoising step, correcting alignment errors caused by the domain gap between content and makeup representations.
- Diverse applications:It is suitable for multiple fields such as image processing, style transfer, and computer vision, such as online makeup try-on on e-commerce platforms.
Technical principles of SHMT
- Self-supervised learning:SHMT adopts a self-supervised strategy for model training, follows the "disentangle-reconstruct" paradigm, and does not rely on poor-quality fake reference data, thereby avoiding the problem of misleading the model. Through disentanglement and reconstruction, the model can learn effectively without annotated data.
- Hierarchical texture detail processing:This technology decomposes the makeup process into multiple levels, including base makeup, eye makeup, lip makeup, etc. Different makeup details are decomposed hierarchically through the Laplacian pyramid method and then selectively integrated into the content representation of the face, allowing more flexible adaptation to various makeup styles.
- Dynamically correcting alignment errors:Through the iterative dual alignment module (IDA), it dynamically adjusts the makeup injection process in the diffusion model and gradually corrects the "mismatch" problem between facial content and makeup style. At each denoising step, IDA uses the noisy intermediate results to dynamically adjust the injection conditions and correct alignment errors.
Project address of SHMT
- Github repository：https://github.com/Snowfallingplum/SHMT
- arXiv technical paper：https://arxiv.org/pdf/2412.11058
Application scenarios of SHMT
- Social media beauty filters:SHMT can be applied to social media platforms to provide users with real-time beauty filter effects, allowing users to preview different makeup styles before posting photos. This increases user interactivity and fun and improves the platform's user experience.
- Virtual makeup try-on applications:In the e-commerce field, SHMT technology can be integrated into virtual makeup try-on applications, allowing customers to try various cosmetics online and improving the shopping experience. Users can upload their own photos, choose different makeup styles, and preview the effects in real time.
- Film and game character design:In the entertainment industry, SHMT can be used to quickly change or design the makeup of characters in films and games, improving production efficiency. This not only saves time and cost, but also provides more creativity and flexibility for character design.
- Personalized advertising production:The advertising industry can use SHMT technology to customize models' makeup according to the preferences of the target audience, making advertisements more appealing. Through personalized makeup design, advertisements can better attract the target audience and increase the conversion rate of advertisements.