CodeFormer is the official implementation of the NeurIPS 2022 paper "Towards Robust Blind Face Restoration with Codebook Lookup Transformer," developed by the S-Lab team at Nanyang Technological University. The project targets the blind face restoration task and is capable of restoring and enhancing low-quality, degraded, or blurry face images.
[Main Capabilities]
- Face restoration: Quality recovery for cropped and aligned face images.
- Whole image enhancement: Enhancement of complete images containing faces, involving face-background blending during the process.
- Video enhancement: Supports video input (via --input_path [YOUR_VIDEO.mp4]) and can enhance faces in videos.
- Face colorization: Provides colorization for cropped and aligned faces.
- Face inpainting: Provides inpainting for cropped and aligned faces.
- Background image enhancement: Uses Real-ESRGAN to support background enhancement.
[Usage]
- Environment dependencies: PyTorch >= 1.7.1; other required packages are listed in requirements.txt.
- You need to download the facelib and dlib pretrained models from Releases, Google Drive, or OneDrive and place them in the weights/facelib folder; also download the CodeFormer pretrained model and place it in the weights/CodeFormer folder. They can also be downloaded via the command line.
- Place test images in the inputs/TestWhole folder for whole-image testing; to test cropped and aligned faces, place them in the inputs/cropped_faces folder.
- Cropped and aligned faces can be generated via command.
- The fidelity weight w ranges from [0, 1]: a smaller w usually produces higher-quality results, while a larger w gives higher fidelity.
- Results are saved in the results folder.
- If you need to compare with CodeFormer in a paper, it is recommended to use the --has_aligned parameter to process cropped and aligned faces, because the whole-image command involves face-background blending, which may damage hair textures at the boundaries and lead to an unfair comparison.
[Version Update Highlights]
- 2023.07.20: Integrated into OpenXLab, available for online experience.
- 2023.04.19: Training code and configuration files released.
- 2023.04.09: Added inpainting and colorization for cropped and aligned faces.
- 2023.02.10: Added dlib as a face detector option, which can produce more accurate face identities.
- 2022.10.05: Supports video input.
- 2022.09.14: Integrated into Hugging Face.
- 2022.09.09: Integrated into Replicate.
[Applicable Scenarios] Suitable for image/video processing scenarios such as face image restoration, old photo restoration, video face enhancement, face colorization and inpainting, and can also be used for comparative experiments in related academic research.
[Limitations and Notes]
- The project uses the NTU S-Lab License 1.0, and distribution and use must comply with this license.
- The official maintainers only maintain the demos on Hugging Face Space, Replicate, and OpenXLab; other demos, APIs, applications, websites, and integrations are third-party and unrelated to the authors.
- The official statement notes that some websites use their models without authorization, reminding users to verify legality and avoid potential financial losses.
- Face-background blending during whole-image processing may damage hair textures at the boundaries; for paper comparisons, it is recommended to use the cropped and aligned face mode.
- For training commands, see the English or Simplified Chinese documentation.
[Acknowledgments and Citation] The project is based on BasicSR, with some code from Unleashing Transformers, YOLOv5-face, and FaceXLib, and uses Real-ESRGAN to support background enhancement. If you have any questions, you can contact shangchenzhou@gmail.com.
