Inpainting of Wide-Baseline Multiple Viewpoint Video

We describe a non-parametric algorithm for multiple-viewpoint video inpainting. Uniquely, our algorithm addresses the domain of wide baseline multiple-viewpoint video (MVV) with no temporal look-ahead in near real time speed. A Dictionary of Patches (DoP) is built using multi-resolution texture patches reprojected from geometric proxies available in the alternate views. We dynamically update the DoP over time, and a Markov Random Field optimisation over depth and appearance is used to resolve and align a selection of multiple candidates for a given patch, this ensures the inpainting of large regions in a plausible manner conserving both spatial and temporal coherence. We demonstrate the removal of large objects (e.g., people) on challenging indoor and outdoor MVV exhibiting cluttered, dynamic backgrounds and moving cameras.

1 Introduction

Multiple viewpoint video is becoming commonplace in film and music video production, where shots are captured using multiple, wide-spaced, synchronised cameras for later editing. Frequently it is desirable to edit such footage to remove objects; e.g., to remove an actor in a motion-capture suit to be later replaced with an animated virtual character.