PAR2Net: End-to-End Panoramic Image Reflection Removal

In this article, we investigate the problem of panoramic image reflection removal to relieve the content ambiguity between the reflection layer and the transmission scene. Although a partial view of the reflection scene is attainable in the panoramic image and provides additional information for reflection removal, it is not trivial to directly apply this for getting rid of undesired reflections due to its misalignment with the reflection-contaminated image. We propose an end-to-end framework to tackle this problem. By resolving misalignment issues with adaptive modules, the high-fidelity recovery of reflection layer and transmission scenes is accomplished. We further propose a new data generation approach that considers the physics-based formation model of mixture images and the in-camera dynamic range clipping to diminish the domain gap between synthetic and real data. Experimental results demonstrate the effectiveness of the proposed method and its applicability for mobile devices and industrial applications.

I. Introduction

When taking photos in front of semi-reflectors like glass windows, photographers prevailingly attempt to capture transmission scenes behind the glass, while reflection contamination often degrades the image quality. Consequently, single-image reflection removal has become an attractive topic in computational photography [4], [5], [15], [22], [47], [64], which aims at removing undesirable glass reflections and recovering the clear transmission scene from a contaminated mixture image. The mixture image (denoted as $\mathbf {M}$) can be considered as the combination of two components: the transmission scene (denoted as $\mathbf {T}_\mathrm{S}$) and the reflection layer (denoted as $\mathbf {R}_\mathrm{L}$) [4], [22]. The major challenge of such an ill-posed layer separation problem is that both transmission scenes and reflection layers are from natural scenes, whose image content can be arbitrary, arousing the difficulty of differentiating the dominant content for mixture images. We call it content ambiguity in this article.