Direction and Residual Awareness Curriculum Learning Network for Rain Streaks Removal

Single-image rain streaks’ removal has attracted great attention in recent years. However, due to the highly visual similarity between the rain streaks and the line pattern image edges, the over-smoothing of image edges or residual rain streaks’ phenomenon may unexpectedly occur in the deraining results. To overcome this problem, we propose a direction and residual awareness network within the curriculum learning paradigm for the rain streaks’ removal. Specifically, we present a statistical analysis of the rain streaks on large-scale real rainy images and figure out that rain streaks in local patches possess principal directionality. This motivates us to design a direction-aware network for rain streaks’ modeling, in which the principal directionality property endows us with the discriminative representation ability of better differing rain streaks from image edges. On the other hand, for image modeling, we are motivated by the iterative regularization in classical image processing and unfold it into a novel residual-aware block (RAB) to explicitly model the relationship between the image and the residual. The RAB adaptively learns balance parameters to selectively emphasize informative image features and better suppress the rain streaks. Finally, we formulate the rain streaks’ removal problem into the curriculum learning paradigm which progressively learns the directionality of the rain streaks, rain streaks’ appearance, and the image layer in a coarse-to-fine, easy-to-hard guidance manner. Solid experiments on extensive simulated and real benchmarks demonstrate the visual and quantitative improvement of the proposed method over the state-of-the-art methods.

I. Introduction

The rainy weather severely degrades the visual quality of the images, including the nearby rain streaks and distant veiling artifacts [1]. The rain would unexpectedly reduce the visibility of outdoor vision systems and bring negative effects to high-level vision tasks, such as scene classification [2], object detection [3], and semantic segmentation [4]. Numerous methods for different tasks such as the rain streaks’ removal [5] and jointly dehazing and deraining [6], [7], [8] have been proposed in recent years. In this work, we focus on the single-image rain streaks’ removal via a simplified linear model $$\begin{equation*} \boldsymbol {Y} = {\boldsymbol {X}} + {\boldsymbol {B}} + {\boldsymbol {N}}\tag{1}\end{equation*}$$ where is an observed rainy image, denotes the number of row and column, respectively, represents the desired clean image, means the rain streaks, and is the random noise.