Reference Picture Synthesis for Video Sequences Captured with a Monocular Moving Camera

Inter-frame prediction plays an important role in video coding by predicting the current frame from previously encoded pictures, called reference pictures. In the case of camera motion, the content of a current frame could be very different from its reference pictures and may consequently lead to a more difficult Motion Compensation (MC). The main idea of this paper is to process the input 2D video sequence in order to estimate the 3D geometry of the scene and then employ this data to virtually synthesize "geometrically compensated" reference pictures. Since these virtual reference pictures are more similar to the current frame, motion estimation and consequently coding efficiency could be enhanced. The proposed method is tested over six different video sequences and around 11% bitrate reduction is achieved compared to the High Efficiency Video Coding (HEVC) standard.

I. Introduction

In the recently developed High Efficiency Video Coding (HEVC) standard, already reconstructed frames are put into Reference Picture Lists (RPLs) to serve as references for inter-frame prediction. In the case of camera motion, the current picture is usually very different from its reference pictures, leading to extremely time consuming Motion Estimation (ME) or even complete ME "failure" [1]. The main idea of this paper is to exploit the 3D geometry of the current scene as well as the camera motion information in order to synthesize geometrically compensated reference pictures and use them for motion compensation.