A Dual-Path Fusion Network for Pan-Sharpening

Most existing deep learning-based pan-sharpening methods own several widely recognized issues, such as spectral distortion and insufficient spatial texture enhancement. To address these challenges in pan-sharpening, we propose a novel dual-path fusion network (DPFN). The proposed DPFN includes two major components: 1) the global subnetwork (GSN) and 2) the local subnetwork (LSN). In particular, GSN aims to search similar image blocks in panchromatic (PAN) space and multispectral (MS) space and exploits HR textural information from the PAN space and spectral information from the MS space for the fine representation of pan-sharpened MS features by employing a cross nonlocal block. Meanwhile, the proposed LSN based on a high-pass modification block (HMB) is designed to learn the high-pass information, aiming to enhance bandwise spatial information from MS images. HMB forces the fused image to obtain high-frequency details from PAN images. Moreover, to facilitate the generation of visually appealing pan-sharpened images, we propose a perceptual loss function and further optimize the model based on high-level features in the near-infrared space. Experiments demonstrate the superior performance of the proposed method quantitatively and qualitatively compared to existing state-of-the-art pan-sharpening methods. The source code is available at https://github.com/jiaming-wang/DPFN.

I. Introduction

With the rapid development of satellite sensors, remote sensing images have been widely used in various academic fields, and civil and military domains. However, as recognized by many, remote sensing sensors own the intrinsic tradeoff between spatial resolution and spectral resolution [1]. As a result, for a certain remote sensing sensor, the captured gray panchromatic (PAN) images are generally with a finer resolution than multispectral (MS) images. Typically, the goal of pan-sharpening is to integrate the texture details of the high-resolution (HR) PAN images with the spectral information from low-resolution (LR) MS images for the purpose of generating HR-MS images, therefore breaking the technological limits. Given the above utility, pan-sharpening tasks are often regarded as a crucial preprocessing step for numerous remote sensing data applications [2], such as the extraction of urban impervious surfaces [3], land-cover classification [4], and change detection [5].