Wav2Lip

Wav2Lip is an open-source lip-sync project corresponding to the paper "A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wild," published at ACM Multimedia 2020. This repository is maintained by xiaoou2 and upgrades the original code from Python 3.6 to 3.9, and updates the weights of the visual quality discriminator.

Purpose

Wav2Lip is used to perform high-precision lip-sync between any video and any target speech. According to the official description, it is applicable to any person identity, voice, and language, and also to CGI faces and synthetic speech. The project provides complete training code, inference code, and pretrained models, and provides a Google Colab Notebook for quick start.

Main capabilities

• High-precision lip-sync: align video lip movements with target speech.
• Support for multiple inputs: the audio source can be any audio-containing data file supported by FFMPEG, such as *.wav, *.mp3, or even video files (the code automatically extracts audio).
• Provide pretrained models and checkpoints, and samples, available in the Google Drive folder.
• Provide evaluation benchmarks and metrics (the repo's evaluation/ folder), and give instructions for calculating the metrics in the paper.
• Provide training and inference code, supporting training on the LRS2 dataset.

Usage

Environment preparation:

• Install ffmpeg: sudo apt-get install ffmpeg
• Install dependencies: pip install -r requirements.txt; a docker image can also be used (the repository provides instructions).
• Download the face detection pretrained model to face_detection/detection/sfd/s3fd.pth.

Inference (using pretrained models for lip-sync):

• Any video can be synchronized with any audio, and the result is saved by default at results/result_voice.mp4, or the output location can be specified via parameters.
• The --pads parameter can be used to adjust the detected face bounding box, for example --pads 0 20 0 0 adds padding at the bottom to include the chin area, which can often improve results.
• If there is misalignment of the mouth position or artifacts such as "two mouths," it may be that face detectionOver-smoothing, you can retry with the --nosmooth parameter.
• You can use the --resize_factor parameter to get lower-resolution videos; since the model is trained on lower-resolution faces, 720p videos sometimes work better than 1080p (in most cases 1080p is also usable).
• The Wav2Lip model without GAN usually requires more experimentation to achieve ideal results, and sometimes can also give better results.

Training:

• Training data preparation: The model is trained on LRS2, and the LRS2 file lists (train, val, test .txt files) need to be placed in the filelists/ folder.
• Training is divided into two major steps: (i) train the expert lip-sync discriminator; (ii) train the Wav2Lip model.
• Pretrained weights can be downloaded to skip the first step.
• Training can involve training only the base model (without the additional visual quality discriminator, < 1 day) or adding the discriminator (about 2 days). The former runs wav2lip_train.py, the latter runs hq_wav2lip_train.py; their parameters are similar, and both support resuming training; you can view details via python wav2lip_train.py --help, and you can also set less commonly used hyperparameters at the bottom of hparams.py.

Use Cases

Suitable for research, academic, and personal video lip-sync experiments, including synchronization for arbitrary people, voices, languages, as well as CGI faces and synthesized speech.

Limitations

• License restrictions: This repository may only be used for personal/research/non-commercial purposes. Since the model is trained on the LRS2 dataset, any form of commercial use is strictly prohibited; commercial requests must contact the authors directly. The authors provide a hosted API (https://synclabs.so/）及改进的新口型同步模型。
• Training on non-LRS2 datasets: Training on other datasets may require code modifications. Training/fine-tuning on a small amount of single-speaker data usually struggles to achieve good results (this is an unresolved independent research problem). You must first train the expert discriminator on your own dataset, then train Wav2Lip. Most datasets downloaded from the internet require sync correction. Pay attention to the FPS of dataset videos; changing FPS requires extensive code changes. The expert discriminator's eval loss should drop to about 0.25, and Wav2Lip's eval sync loss should drop to about 0.2 to achieve good results.
• The authors also mention an HD model trained on a dataset that allows commercial use, generating face sizes of 192 x 288.
• Commercial licensing: The repository is limited to personal/research/non-commercial use; for commercial requests, please contact rudrabha@synclabs.so or prajwal@synclabs.so.

Acknowledgements and Citation

Part of the code structure was inspired by a certain TTS repository; the face detection code comes from the face_alignment repository; and thanks to zabique for the provided tutorial Colab Notebook. Using this repository requires citing the original paper.