SC-PNN: Saliency Cascade Convolutional Neural Network for Pansharpening

In many remote sensing tasks, different types of regions or targets differ in requirements for spectral and spatial quality. The discrepancy reveals that a uniform pansharpening strategy applying to the entire image may not fulfill the varying demands of different regions appropriately. From this aspect, we resort to saliency analysis to distinguish regions with different spatial and spectral requirements and then propose a new saliency cascade convolutional neural network for pansharpening (SC-PNN). SC-PNN is composed of two parts: a dilated deformable convolutional network (DDCN) for saliency analysis and a saliency cascade residual dense network (SC-RDN) for pansharpening. DDCN is a fully convolutional network based on hybrid dilated convolution and deformable convolution, aiming to separate salient regions, such as residential areas from nonsalient areas, including mountains and vegetation areas, with well-defined boundaries and integrity. In the fusion process, SC-RDN is specially designed with the help of saliency analysis. We first construct a deep regression network to estimate a primarily sharpened image and subsequently leverage the saliency map produced by DDCN to develop a saliency enhancement module. In this module, the quality of salient and nonsalient areas is further improved by two independent deep residual dense networks. Thus, a precise fused image can be predicted. Experiments on SPOT5, GeoEye-1, and WorldView-3 data sets reveal that, compared to state-of-the-art pansharpening methods, our proposal has a superior ability to improve the spatial quality and preserve spectral information. The effectiveness of the saliency enhancement module is also validated in the experiment.

I. Introduction

Optical satellites sensors today, such as SPOT, GeoEye, and WorldView, simultaneously capture two types of remote sensing imagery with discrepant spatial and spectral characteristics, i.e., panchromatic (PAN) images and multispectral (MS) images. Due to physics constraints and bandwidth limitations, there is an evident and inevitable tradeoff between spectral diversity and spatial resolution [1]. PAN images provide high-quality geometrical details but contain limited spectral information, while MS images are characterized by several narrower spectral bands with a lower spatial resolution. As high-resolution MS (HRMS) images are often required in a variety of remote sensing applications and commercial products, pansharpening techniques incorporating PAN images with multichannel low-resolution MS (LRMS) data have gathered increasing attention in the remote sensing community [2]. The key point in pansharpening is that, while ensuring the spatial increment, the detail injection implemented should preserve the unified spatial–spectral fidelity for fusion products.