Virtual View Color Estimation for Free Viewpoint TV Applications Using Gaussian Mixture Model

Free Viewpoint TV applications extensively use view synthesis technologies to generate virtual views. Mismatch in color and brightness between the original views captured by far apart cameras in this setting, is a real challenge as they result in visual discontinuities in the synthesized views and affect the overall viewer's quality of experience. In this paper, we present a novel color estimation algorithm specifically designed for FTV applications. Our method is based on Gaussian Mixture Model (GMM) histogram approximation and estimates the color of virtual views based on their relative position in space and the color of the real views. Subjective tests confirm that our approach significantly reduces the color related artifacts in the virtual views, which results in increased overall visual quality.

I. Introduction

Current Free Viewpoint TV Applications (FTV), such as Super Multiview (SMV) and Free Navigation (FN), use content captured by multiple cameras that surround the scene. View synthesis that utilizes captured views to create new virtual views is used for expanding the coverage or closing the gaps between existing real camera views, depending on the type of FTV application, i.e., Super Multiview or 2D walk-around-scene-like (FN) immersive experience [1] [2]. In these implementations, it is common for the majority of the cameras to have dissimilar radiometric lens characteristics and be pointed to the scene from a different viewpoint, often yielding significant luminance and chrominance discrepancies among the captured views [3]. As a result, synthesized views may have visual artifacts, caused by incorrect estimation of missing texture in occluded areas and possible brightness and color differences between the original real views.