Multi-Modal Reflection Removal Using Convolutional Neural Networks

Although color images are easily interfered by glass, depth images captured by infrared sensors are robust to reflection. In this letter, we propose multi-modal reflection removal using convolutional neural networks (CNNs). We build a multi-modal CNN for reflection removal to separate transmission from reflection using depth information. The proposed network consists of two sub-networks: image restoration and depth adaptation. Image restoration sub-network (IRN) recovers transmission layer from the input image with reflection, whereas depth adaptation sub-network (DAN) guides reflection removal of the IRN. Moreover, to extract image details for reflection removal, we present a multi-scale loss function that penalizes non-similarity for multi-scale outputs. Experimental results demonstrate that the proposed method is robust to dominant reflections and outperforms state-of-the-art methods in terms of both peak signal-to-noise ratio (PSNR) and structural similarity.

I. Introduction

When people take photos through glass, the captured image contains reflections that affect performance in computer vision and pattern recognition. Thus, it is required to separate transmission from reflection. Assume that there is little sensor noise in imaging. If a plane of glass is located at digital cameras, the captured images contains two layers: transmission and reflection. Denote the captured image with reflection as . can be modelled by a linear combination of transmission layer and reflection layer , i.e., . Since the number of its solutions are infinite, reflection removal is an ill-posed problem. Moreover, both reflection and transmission layers are natural images that have similar characteristics, which makes the problem intractable.