Cross-Modality Interaction Network for Pan-Sharpening

Pan-sharpening seeks to generate a high-resolution multispectral (HRMS) image by merging the high-resolution panchromatic (PAN) image and its low-resolution multispectral (LRMS) counterpart. The main challenge lies in enhancing modality-aware features and efficiently integrating complementary information between PAN and MS pairs. To achieve desired fusion results, it is crucial to fully utilize both intramodality characteristics and intermodality relationships. Current research often overlooks the exploration of cross-modality relationships and neglects the enhancement of modality-aware features in pan-sharpening. In this work, we introduce an innovative pan-sharpening framework, named cross-modality interaction network (CMINet), which comprises three core designs: a modality-aware feature enhancement (MAFE) module to enhance the feature representation of both modalities, a cross-modality attention (CMA) module that effectively extracts the intramodality features and fully leverages the intermodality complementary information, and a modality alignment (MA) module to address modality-aware misalignment issue during fusion. Extensive experiments are conducted to verify the effectiveness of our proposed network and showcase its superior performance in comparison to other state-of-the-art approaches.

I. Introduction

Advanced remote sensing technology has led to the widespread adoption of remote sensing images in a multitude of fields, including agricultural development, environmental surveillance, scene interpretation, and mapping applications. Nevertheless, simultaneously capturing both high-spatial resolution and multispectral images poses a challenge for imaging devices, primarily due to physical and technological constraints. As an alternative, satellites are commonly equipped with two distinct types of sensors: one for acquiring high-resolution PAN images and another for capturing LRMS images. To tackle this issue, pan-sharpening has evolved as a critical technique for integrating PAN and LRMS images to generate the desired HRMS images. Considerable interest has been drawn to this field, leading to numerous research endeavors.