Depth image-based rendering with spatio-temporally consistent texture synthesis for 3-D video with global motion

Depth image-based rendering (DIBR) techniques are advanced tools in 3-D video (3DV) applications that are used to synthesize a number of additional views in a multiview-video-plus-depth (MVD) representation. The MVD format consists of video and depth sequences for a limited number of original camera views of the same scene. An inherent problem of the view synthesis concept is given by image information that is occluded in the original views and becomes visible, in the extrapolated views. To handle these disoc-clussions, we propose a DIBR algorithm with advanced inpainting methods. Our renderer enhances visual experience by taking spatial and temporal texture consistency problems into account. In order to compensate the global motion in a sequence, image registration is incorporated into the framework. The proposed method shows objective and subjective gains compared to the state-of-the-art.

1. INTRODUCTION

In recent years, 3-D video (3DV) technology has reached the wide market. The 3-D Blue-Ray and stereo displays entered the shops and the living rooms. Despite the high standard that 3DV has reached today, there is still room for improvements. Moreover, new technologies such as autostereoscopic multiview displays require further research. The common representation format used for autostereo-scopic displays is multiview-video-plus-depth (MVD). The MVD format consists of video and depth sequences for a limited number of original camera views of the same scene. In order to generate additional virtual views depth image-based rendering (DIBR) techniques are used. A significant problem in DIBR techniques is that texture areas become disoccluded (holes) in the virtual views, especially in the extrapolated views beyond the viewing range of the original cameras. Three general methods have been proposed in the literature to handle such holes. First, the depth maps are preprocessed in a way that no disocclusions occur. Usually, the depth map is smoothed, using a symmetric [1] or an asymmetric filter [2], to lower gradients. This method gives good results, when small baselines have to be compensated. Nevertheless, foreground and background textures are distorted, especially in foreground-background transitions, during the warping procedure. The second way to close disocclusions is to cover them with plausible, known image information. Suitable filling techniques such as line-wise filling [3], inpainting methods [4], [5], bilateral filtering [6], and texture synthesis [7] are often used. Alternatively, image domain warping can be utilized to compute the virtual views. Applying the latter method, holes are covered by distorting non-salient image regions [8].