Visual pertinent 2D-to-3D video conversion by multi-cue fusion

We describe an approach to2D-to-3D video conversion for the stereoscopic display. Targeting the problem of synthesizing the frames of a virtual 'right view' from the original monocular 2D video, we generate the stereoscopic video in steps as following. (1) A 2.5D depth map is first estimated in a multi-cue fusion manner by leveraging motion cues and photometric cues in video frames with a depth prior of spatial and temporal smoothness. (2) The depth map is converted to a disparity map with considering both the displaying device size and human's stereoscopic visual perception constraints. (3) We fix the original 2D frames as the 'left view' ones, and warp them to "virtually viewed" right ones according to the predicted disparity value. The main contribution of this method is to combine motion and photometric cues together to estimate depth map. In the experiments, we apply our method to converting several movie clips of well-known films into stereoscopic 3D video and get good results¹.

1. INTRODUCTION

2D-to-3D video conversion is necessary for 3D video industry for two main reasons. (1) Indespite of many stereoscopic 3D movies produced in the recent few years, the amount isnot enough, especially for 3DTV industry. For instance, the Sky TV's 3D Channel could not start its officially broadcasting until it is able to supply more than 6 hours 3D programs per day. As many broadcasting companies plan to start 3DTV Channel in the near future, the desire for 2D-to-3D conversion technology is urgent. (2) 3D reproduction of the conventional well-know 2D movies or TV programs is appealing both to the potential audience and the producers, and also this will gives these films renascence after years of the premiere. Figure 1: Converting Titanic into stereoscopic 3D video