Real-ESRGAN: An Image Super-Resolution Model Launched by Tencent

What Is Real-ESRGAN

Real-ESRGAN (Real-World Blind Super-Resolution with Pure Synthetic Data) is an open-source deep learning model launched by Tencent's ARC Lab, specifically designed to process and enhance the quality of low-resolution images to high-resolution images. The characteristic of this model is that it can be trained through a synthetic degradation process without real high-resolution images as references, thereby simulating image degradation in the real world. This method is called "blind super-resolution" because it does not need to know how an image degrades from high resolution to low resolution.

Features of Real-ESRGAN

• Image Quality Enhancement: Converts low-resolution images into high-resolution images while maintaining or enhancing image details and textures and reducing blur and noise.
• Artifact Removal: During image upscaling, Real-ESRGAN can identify and reduce common image artifacts, such as ringing and overshoot.
• Simulating Real-World Degradation: Through a high-order degradation model, Real-ESRGAN can simulate various image degradation processes in the real world, such as camera blur, sensor noise, sharpening, JPEG compression, and more.
• No Need for Real High-Resolution Images: The training of Real-ESRGAN does not rely on real high-resolution images, but instead generates training data through a synthetic degradation process, which enables the model to be trained without real high-resolution images.
• Enhancing Image Details: While increasing resolution, Real-ESRGAN can enhance local details in images, such as textures, edges, and contours, making the upscaled image clearer and more natural.

Official Entry Points for Real-ESRGAN

• Official GitHub Repository:https://github.com/xinntao/Real-ESRGAN
• Research Paper:https://arxiv.org/abs/2107.10833
• Replicate Running Address:https://replicate.com/xinntao/realesrgan
• Google Colab Running Address:https://colab.research.google.com/drive/1k2Zod6kSHEvraybHl50Lys0LerhyTMCo?usp=sharing
• Arc Version Anime-6B:https://arc.tencent.com/zh/ai-demos/imgRestore

How Real-ESRGAN Works

The working principle of Real-ESRGAN is based on the framework of deep learning and generative adversarial networks (GANs), and specifically includes the following key steps:

1. Data Synthesis: Since Real-ESRGAN does not use real high-resolution images during training, it first needs to synthesize training data. This is achieved by simulating real-world image degradation processes, including blur, downsampling, noise addition, and JPEG compression. These processes simulate the degradation path from high-resolution images to low-resolution images. To increase diversity, Real-ESRGAN adopts a high-order degradation model, that is, generating training samples by repeating these degradation processes multiple times.
2. Model Architecture: Real-ESRGAN uses a generator network similar to ESRGAN, which is a deep convolutional neural network (CNN) containing multiple Residual-in-Residual Dense Blocks (RRDBs). This network is responsible for restoring high-resolution images from low-resolution images. To improve the capability of the discriminator, Real-ESRGAN also adopts a U-Net discriminator with Spectral Normalization (SN).
3. Training Process: Training is divided into two stages. First, a model oriented toward peak signal-to-noise ratio (PSNR) (Real-ESRNet) is trained using the L1 loss function. Then, this model is used as the initialization of the generator, and Real-ESRGAN is trained by combining the L1 loss, perceptual loss (based on feature maps of the VGG network), and GAN loss (adversarial loss). This combined loss approach aims to improve image details while preserving the realism of the image.
4. Spectral Normalization: To stabilize the training process and improve the performance of the discriminator, Real-ESRGAN uses spectral normalization in the U-Net discriminator. This technique helps prevent instability and overfitting during training, while also providing more accurate gradient feedback to help the generator learn how to produce more realistic images.
5. Generating High-Resolution Images: After training is completed, the generator network of Real-ESRGAN can take low-resolution images as input and output corresponding high-resolution images. During this process, the model attempts to restore the details of the image while reducing artifacts introduced by the degradation process.
6. Evaluation and Optimization: After training is completed, the performance of Real-ESRGAN is evaluated by testing on multiple real-world datasets. This includes the use of no-reference image quality assessment metrics (such as NIQE) and visual comparisons. Based on the evaluation results, the model can be further optimized and adjusted.

Application Scenarios of Real-ESRGAN

• Digital Image Restoration: In the field of digital media, Real-ESRGAN can be used to improve the quality of old photos, scanned documents, compressed images, etc., and restore lost details.
• Video Enhancement: In video production and post-processing, Real-ESRGAN can be used to improve the resolution of videos, making the image clearer when played on large screens.
• Social Media Content Processing: Images and videos on social media platforms often suffer from quality degradation due to compression, and Real-ESRGAN can help users restore and improve the quality of this content.
• Surveillance Video Analysis: In surveillance systems, Real-ESRGAN can be used to improve the image resolution captured by surveillance cameras, helping to better identify details, which is crucial for security monitoring and incident analysis.
• Medical Image Processing: In the medical field, Real-ESRGAN can be used to enhance medical scan images, such as MRI, CT scans, etc., helping doctors observe and diagnose more clearly.