SAPNet: Segmentation-Aware Progressive Network for Perceptual Contrastive Deraining

Deep learning algorithms have recently achieved promising deraining performance–s on both the natural and synthetic rainy datasets. As an essential low-level preprocessing stage, a deraining network should clear the rain streaks and preserve the fine semantic details. However, most existing methods only consider low-level image restoration. That limits their performances at high-level tasks requiring precise semantic information. To address this issue, in this paper, we present a segmentation aware progressive network (SAPNet) based upon contrastive learning for single image deraining. We start our method with a lightweight derain network formed with progressive dilated units (PDU). The PDU can significantly expand the receptive field and characterize multiscale rain streaks without the heavy computation on multiscale images. A fundamental aspect of this work is an unsupervised background segmentation (UBS) network initialized with ImageNet and Gaussian weights. The UBS can faithfully preserve an image s semantic information and improve the generalization ability to unseen photos. Furthermore, we introduce a perceptual contrastive loss (PCL) and a learned perceptual image similarity loss (LPISL) to regulate model learning. By ex-ploiting the rainy image and ground-truth as the negative and the positive sample in the VGG-16 latent space, we bridge the fine semantic details between the derained image and the ground-truth in a fully constrained manner. Comprehensive experiments on synthetic and real-world rainy images show our model surpasses top-performing methods and aids object detection and semantic segmentation with considerable efficacy. A Pytorch Implementation is available at https://github.com/ShenZheng2000/SAPNetfor-image-deraining.

1. Introduction

¹Rain is typical weather that degrades the visibility of images and videos. Especially in heavy rain, the combination of rain streaks and accumulation has a severe adverse impact on computer vision tasks, such as image classification, object detection, and semantic segmentation [23]. Therefore, it is crucial to remove rains and to recover the rainy images. Since 2017, deep learning deraining methods, based upon CNN [7, 48, 24, 38, 18, 36], or GAN [9, 36, 31], have attracted significant attention due to their outstanding accuracy, capacity, and flexibility.