Stereo Conversion with Disparity-Aware Warping, Compositing and Inpainting

Despite of exciting advances in image-based rendering and novel view synthesis, it is still challenging to achieve high-resolution results that can reach production-level quality when applying such methods to the task of stereo conversion. At the same time, only very few dedicated stereo conversion approaches exist, which also fall short in terms of the required quality. Hence, in this paper, we present a novel method for high-resolution 2D-to-3D conversion. It is fully differentiable in all of its stages and performs disparity-informed warping, consistent foreground-background compositing, and background-aware inpainting. To enable temporal consistency in the resulting video, we propose a strategy to integrate information from additional video frames. Extensive ablation studies validate our design choices, leading to a fully automatic model that outperforms existing approaches by a large margin (49-70% LPIPS error reduction). Finally, inspired from current practices in manual stereo conversion, we introduce optional interactive tools into our model, which allow to steer the conversion process and make it significantly more applicable for 3D film production.

1. Introduction

Live-action feature films are typically not filmed in stereo apart from a few notable exceptions. Still, a significant amount of high profile productions are available in stereo. This is possible through a post production process referred to as stereo conversion. Although there is some automation in parts of the conversion process, it still heavily relies on manual work, which makes converting even a single feature film extremely expensive. A fully automatic conversion pipeline could hence significantly reduce costs while enabling studios to make large amounts of legacy content available to new audiences in stereo.