New Coding Tools for Illumination and Focus Mismatch Compensation in Multiview Video Coding

We propose new tools for multiview video coding (MVC) that aim to compensate for mismatches between video frames corresponding to different views. Such mismatches could be caused by different shooting positions of the cameras and/or heterogeneous camera settings. In particular, we consider illumination and focus mismatches across views, i.e., such that different portions of a video frame can undergo different illumination and blurriness/sharpness changes with respect to the corresponding areas in frames from the other views. Models for illumination and focus mismatches are proposed and new coding tools are developed from the models. We propose a block-based illumination compensation (IC) technique and a depth-dependent adaptive reference filtering (ARF) approach for cross-view prediction in multiview video coding. In IC, disparity field and illumination changes are jointly computed as part of the disparity estimation search. IC can be adaptively applied by taking into account the rate-distortion characteristics of each block. For ARF, the disparity fields are used to estimate scene depth, such that video frames are first divided into regions with different scene-depth levels. A 2-D filter is then selected for each scene-depth level. These filters are chosen to minimize residual energy, with the goal of compensating for focus mismatches. The resulting filters are applied to the reference frames to generate better matches for cross-view prediction. Furthermore, we propose a coding system that combines IC and ARF. Adjustments are made so as to maximize the gains achieved by using both coding tools, while reducing the complexity of the final integrated system. We analyze the complexity of all proposed methods and present simulation results of IC, ARF and combined system for different multiview sequences based on the H.264/AVC reference codec. When applying the proposed tool to cross-view coding we observe gains of up 1.3 dB as compared to directly using an H.264/AVC codec to perform predictive coding across views.

I. Introduction

Multiview video systems are used for simultaneously capturing scenes or objects with multiple cameras from different viewpoints. From the video sequences corresponding to each view it is possible to extract 3-D information, e.g., the scene depth can be estimated using the correspondence of objects from different views. Multiview video coding systems are being proposed for new multimedia services, e.g., 3-D cinema/TV, free viewpoint video and immersive virtual reality.