Deep Joint Rain Detection and Removal from a Single Image

In this paper, we address a rain removal problem from a single image, even in the presence of heavy rain and rain streak accumulation. Our core ideas lie in our new rain image model and new deep learning architecture. We add a binary map that provides rain streak locations to an existing model, which comprises a rain streak layer and a background layer. We create a model consisting of a component representing rain streak accumulation (where individual streaks cannot be seen, and thus visually similar to mist or fog), and another component representing various shapes and directions of overlapping rain streaks, which usually happen in heavy rain. Based on the model, we develop a multi-task deep learning architecture that learns the binary rain streak map, the appearance of rain streaks, and the clean background, which is our ultimate output. The additional binary map is critically beneficial, since its loss function can provide additional strong information to the network. To handle rain streak accumulation (again, a phenomenon visually similar to mist or fog) and various shapes and directions of overlapping rain streaks, we propose a recurrent rain detection and removal network that removes rain streaks and clears up the rain accumulation iteratively and progressively. In each recurrence of our method, a new contextualized dilated network is developed to exploit regional contextual information and to produce better representations for rain detection. The evaluation on real images, particularly on heavy rain, shows the effectiveness of our models and architecture.

1. Introduction

Restoring rain images is important for many computer vision applications in outdoor scenes. Rain degrades visibility significantly and causes many computer vision systems to likely fail. Generally, rain introduces a few types of visibility degradation. Raindrops obstruct, deform and/or blur the background scenes. Distant rain streaks accumulate and generate atmospheric veiling effects similar to mist or fog, which severely reduce the visibility by scattering light out and into the line of sight. Nearby rain streaks exhibit strong specular highlights that occlude background scenes. These rain streaks can have various shapes and directions, particularly in heavy rain, causing severe visibility degradation. Figure 1.

An example result of our proposed rain removal method that removes heavy rain streaks and enhances the visibility significantly. Top: The raw image with rain streaks (left) and the output image of our method (right). Bottom: Closer looks at specific regions (blue and red) for more details.