Video Frame Interpolation With Many-to-Many Splatting and Spatial Selective Refinement

In this work, we first propose a fully differentiable Many-to-Many (M2M) splatting framework to interpolate frames efficiently. Given a frame pair, we estimate multiple bidirectional flows to directly forward warp the pixels to the desired time step before fusing any overlapping pixels. In doing so, each source pixel renders multiple target pixels and each target pixel can be synthesized from a larger area of visual context, establishing a many-to-many splatting scheme with robustness to undesirable artifacts. For each input frame pair, M2M has a minuscule computational overhead when interpolating an arbitrary number of in-between frames, hence achieving fast multi-frame interpolation. However, directly warping and fusing pixels in the intensity domain is sensitive to the quality of motion estimation and may suffer from less effective representation capacity. To improve interpolation accuracy, we further extend an M2M++ framework by introducing a flexible Spatial Selective Refinement (SSR) component, which allows for trading computational efficiency for interpolation quality and vice versa. Instead of refining the entire interpolated frame, SSR only processes difficult regions selected under the guidance of an estimated error map, thereby avoiding redundant computation. Evaluation on multiple benchmark datasets shows that our method is able to improve the efficiency while maintaining competitive video interpolation quality, and it can be adjusted to use more or less compute as needed.

I. Introduction

Video frame interpolation (VFI) aims to increase frame rates of videos by synthesizing intermediate frames in between the original ones [1], [2]. As a classic problem in video processing, VFI contributes to many practical applications, including slow-motion animation [3], video editing [4], video compression [5], etc. In recent years, a plethora of techniques for video frame interpolation have been proposed [6], [7], [8], [9], [10], [11], [12], [13], [14]. However, frame interpolation remains an unsolved problem due to challenges like occlusions, blur, and large motion.