Image Super-Resolution (ISR) is an open-source project whose goal is to upscale low-resolution images and improve their quality. The project includes multiple Residual Dense Networks implemented in Keras for Single Image Super-Resolution (ISR), as well as scripts for training these networks using content loss and adversarial loss components.
In terms of main capabilities, ISR implements the following networks: the Residual Dense Network for Image Super-Resolution (RDN) based on Zhang et al.'s 2018 paper "Residual Dense Network for Image Super-Resolution"; the Residual in Residual Dense Network for Image Super-Resolution (RRDN) based on Wang et al.'s 2018 paper "ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks"; a multi-output version of the Keras VGG19 network for deep feature extraction in perceptual loss; and a custom discriminator network based on Ledig et al.'s 2017 paper "Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network" (SRGANS).
In terms of usage, the project provides Docker scripts and Google Colab notebooks for training and prediction. It also provides scripts to train in the cloud on AWS and nvidia-docker with a few commands. ISR is compatible with Python 3.6 and is distributed under the Apache 2.0 license. There are two installation methods: install from PyPI (recommended) or install from GitHub source. After loading and preparing images, a pretrained model can be loaded and prediction can be run. For large-image prediction, to avoid memory allocation errors, the by_patch_of_size option of the predict method can be used. During training, a model can be created and a Trainer object can be created with the desired settings, passing in the model (f_ext and discr are optional).
In terms of pretrained networks, there are currently 4 models available: RDN's psnr-large, psnr-small,noise-cancel, and RRDN's gans. Network parameters are selected automatically. RDN models are PSNR-driven; when creating an RDN model, choose weights='psnr-large' or weights='psnr-small'. RRDN models are trained using adversarial and VGG feature losses; when creating an RRDN model, choose weights='gans'. RDN models are trained using adversarial and VGG feature losses; when creating an RDN model, choose weights='noise-cancel'.
RDN pretrained weights: the weights of the RDN network trained on the DIV2K dataset are located at weights/sample_weights/rdn-C6-D20-G64-G064-x2/PSNR-driven/rdn-C6-D20-G64-G064-x2_PSNR_epoch086.hdf5. The model uses C=6, D=20, G=64, G0=64 as parameters, was trained for 86 epochs, and each epoch contains 1000 batches, with each batch containing 8 32x32 augmented patches extracted from low-resolution images. The artefact cancelling weights obtained by training with a different dataset and a combination of VGG19 and GAN perceptual losses are located at weights/sample_weights/rdn-C6-D20-G64-G064-x2/ArtefactCancelling/rdn-C6-D20-G64-G064-x2_ArtefactCancelling_epoch219.hdf5. It is recommended to use these weights only when compression artifacts need to be removed.
In terms of network architecture, the main parameters of the RDN network architecture include: D is the number of residual dense blocks (RDBs); C is the number of stacked convolutional layers within an RDB; G is the number of feature maps for each convolutional layer within an RDB; G0 is the number of feature maps for convolutions outside the RDBs and for the output of each RDB. The main parameters of the RRDN network architecture include: T is the number of residual-in-residual dense blocks (RRDBs); D is the number of residual dense blocks (RDBs) within each RRDB; C is the number of stacked convolutional layers within an RDB; G is the number of feature maps for each convolutional layer within an RDB; G0 is the number of feature maps for convolutions outside the RDBs and for the output of each RDB.amount.
Regarding training suggestions, when training your own model, start with only the PSNR loss (more than 50 epochs, depending on the dataset), and then introduce GANS and feature loss, which can be controlled through the loss weight parameter. If you encounter problems when loading your own weights or pretrained weights (AttributeError: 'str' object has no attribute 'decode'), you can try the corresponding solution.
The project welcomes various contributions, including models trained on different datasets, new model architectures and/or hyperparameter combinations, to improve the performance of the currently released models. The performance of new models will be published in this repository. For details, see the contribution guidelines. If you need to upgrade the version, you can use the corresponding command. If this work is helpful for research, please cite it in your publication. The authors are Francesco Cardinale (github: cfrancesco) and Dat Tran (github: datitran). See LICENSE for details.
