Virtual View Synthesis for Free Viewpoint Video and Multiview Video Compression using Gaussian Mixture Modelling

High quality virtual views need to be synthesized from adjacent available views for free viewpoint video and multiview video coding (MVC) to provide users with a more realistic 3D viewing experience of a scene. View synthesis techniques suffer from poor rendering quality due to holes created by occlusion and rounding integer error through warping. To remove the holes in the virtual view, the existing techniques use spatial and temporal correlation in intra/inter-view images and depth maps. However, they still suffer quality degradation in the boundary region of foreground and background areas due to the low spatial correlation in texture images and low correspondence in inter-view depth maps. To overcome the above-mentioned limitations, we use a number of models in the Gaussian mixture modeling (GMM) to separate background and foreground pixels in our proposed technique. Here, the missing pixels introduced from the warping process are recovered by the adaptive weighted average of the pixel intensities from the corresponding GMM model(s) and warped image. The weights vary with time to accommodate the changes due to a dynamic background and the motions of the moving objects for view synthesis. We also introduce an adaptive strategy to reset the GMM modeling if the contributions of the pixel intensities drop significantly. Our experimental results indicate that the proposed approach provides 5.40-6.60-dB PSNR improvement compared with the relevant methods. To verify the effectiveness of the proposed view synthesis technique, we use it as an extra reference frame in the motion estimation for MVC. The experimental results confirm that the proposed view synthesis is able to improve PSNR by 3.15-5.13 dB compared with the conventional three reference frames.

I. Introduction

Fvv has attracted considerable attention in recent years as it provides freedom to users to observe a scene from different angles or viewpoints [1]–[3]. A large number of views with a small baseline are required to facilitate this luxury, which increases transmission bandwidth and storage data significantly. Depth image based rendering (DIBR) is a practical way to reduce storage and transmission bandwidth for multiview videos from color textures and their corresponding depth maps [1], [2]. However, in the DIBR technique, certain portions of the virtual view are not visible due to being blocked by the front objects, which are termed as occlusions. Occlusions may create holes in the synthesized video [3]–[6]. Moreover, warping process from different views cause another source of error due to rounding of the pixel position coordinates.