Splatting-based Synthesis for Video Frame Interpolation

Frame interpolation is an essential video processing technique that adjusts the temporal resolution of an image sequence. While deep learning has brought great improvements to the area of video frame interpolation, techniques that make use of neural networks can typically not easily be deployed in practical applications like a video editor since they are either computationally too demanding or fail at high resolutions. In contrast, we propose a deep learning approach that solely relies on splatting to synthesize interpolated frames. This splatting-based synthesis for video frame interpolation is not only much faster than similar approaches, especially for multi-frame interpolation, but can also yield new state-of-the-art results at high resolutions.

1. Introduction

Video frame interpolation is becoming more and more ubiquitous. While early techniques for frame interpolation were restricted to using block motion estimation and compensation due to performance constraints [8], [20], modern graphics accelerators allow for dense motion estimation and compensation while heavily making use of neural networks [36], [44], [45], [47]. These developments enable in-runtime in seconds teresting new applications of video frame interpolation for animation inbetweening [31], video compression [62], video editing [39], motion blur synthesis [3], and many others.