ReSyncer - An AI Video Editing Tool Jointly Launched by Tsinghua University and Baidu

What Is ReSyncer

ReSyncer is an AI video editing tool jointly launched by Tsinghua University and Baidu. It generates high-quality lip movement videos synchronized with sound through audio-driven generation. ReSyncer uses Style-SyncFormer to analyze sound and create 3D facial models, combining them with the target video to generate synchronized virtual characters with rich expressions. ReSyncer supports personalized fine-tuning, speaking style transfer, and face swapping, making it suitable for scenarios such as virtual hosts, performer creation, and real-time live streaming. It delivers outstanding results in audiovisual facial information synchronization.

ReSyncer's Main Features

• Lip Sync: Generates lip movements synchronized with sound based on the given audio.
• Style Transfer: Transfers a specific speaking style or facial expression to the target video.
• Personalized Fine-Tuning: Quickly adjusts the generated facial animation to match the facial features of a specific person.
• Video-Driven Lip Sync: Uses the facial images of the target video to drive lip animation.
• Face Swapping Technology: Replaces one person's facial features with another's, for identity conversion or special effects production.

ReSyncer's Technical Principles

• 3D Facial Model Generation: Uses Style-SyncFormer, a deep learning model, to predict 3D facial dynamics based on voice features.
• Stylized Facial Dynamics: Learns stylized 3D facial dynamics through a Transformer structure to achieve precise synchronization of facial expressions and lip movements.
• Style-Based Generator: Combines the predicted 3D facial dynamics with the facial images in the target video to generate high-fidelity facial images.
• Facial Feature Fusion: During the generation process, fuses 3D facial mesh information with stylized features through a simple insertion mechanism to improve the quality and stability of lip synchronization.

ReSyncer's Project Address

ReSyncer's Application Scenarios

• Film and Video Production: In film and video production, ReSyncer can be used to achieve complex effects, such as face swapping or lip syncing, increasing visual appeal.
• Advertising Industry: In advertising production, the style transfer feature can be used to create unique visual effects and attract the audience's attention.
• Social Media and Content Creation: Content creators can use ReSyncer to enhance their video content, for example, by making interesting imitation videos through face swapping technology.
• Education and Training: In language learning or professional training, the lip sync feature can help learners better understand and imitate pronunciation.