DAIN (Depth-Aware Video Frame Interpolation) is a video frame interpolation model published at CVPR 2019. The paper authors are Wenbo Bao, Wei-Sheng Lai, Chao Ma, Xiaoyun Zhang, Zhiyong Gao, and Ming-Hsuan Yang. Its core idea is to explicitly detect occlusion by exploring depth cues, and it develops a depth-aware optical flow projection layer for synthesizing intermediate optical flow, making sampling favor closer objects over farther ones. This method achieved state-of-the-art performance on the Middlebury dataset at the time. The project is based on the authors' previous TPAMI work MEMC-Net, in which an adaptive warping layer was proposed.
Main capabilities include: video frame interpolation, slow-motion generation (by setting time_step to 0.25, 4x slow motion can be generated; 0.125 or 0.1 generate 8x and 10x slow motion respectively, and even 100x slow motion is possible), downloading interpolation results on the UCF101, Vimeo90K, and Middlebury datasets, and training new models. The repository also provides a Google Colab demo that can perform a complete interpolation demo from a source video to a target video.
In terms of usage, the project needs to satisfy the following environment dependencies: Ubuntu 16.04.5 LTS, Python 3.6.8 (Anaconda3 4.1.1), CUDA 9.0 and Cudnn 7.0, PyTorch 1.0.0 (custom layers such as the depth-aware optical flow projection require the ATen API of PyTorch 1.0.0), GCC 4.9.1 and nvcc 9.0 compilers, and an NVIDIA GPU (the authors used a Titan X Pascal, compute capability 6.1, supporting compute_50/52/60/61 devices). Usage steps include: download the repository, build the PyTorch extension, generate the Correlation package required by PWCNet, create the model weights directory and the Middlebury dataset directory, download the pretrained models and the Middlebury dataset, and then run the test. Interpolation results are saved in the MiddleBurySet/other-result-author/[random number]/ directory. Training a new model requires downloading the Vimeo90K triplet dataset and the pretrained MegaDepth and PWCNet models. After running the training script, the optimized model is saved in the model_weights/[random number] directory. After replacing the pretrained model, it can be tested. Usage steps for the Google Colab demo: download the Colab_DAIN.ipynb file, visit Google Colaboratory, choose upload and upload the file, and run the cells one by one according to the instructions.
Applicable scenarios include video frame interpolation research, slow-motion video generation, video quality improvement, and so on. In terms of limitations, the project has clear requirements for the environment and hardware, requiring specific versions of Ubuntu, Python, CUDA, Cudnn, PyTorch, GCC, and an NVIDIA GPU, and the custom layers depend on the ATen API of PyTorch 1.0.0. The license is MIT.
