A Novel Coarse-to-Fine Deep Learning Registration Framework for Multimodal Remote Sensing Images

Multimodal remote sensing images with large rotation transformation (RT) are challenging to be registered. It needs to deal with the global geometric deformation caused by great RT and significant local appearance differences caused by different imaging mechanisms. Existing deep learning methods mainly use a single deep descriptor learning (DDL) network to extract invariant features for identifying matching samples and discriminative feature descriptors for separating nonmatching samples. However, it is difficult to extract local invariant feature descriptors to RT and modality change through a single DDL network. This article proposes a novel coarse-to-fine deep learning image registration framework for multimodal remote sensing images based on two task-specific deep models. Specifically, in the coarse registration stage, this article designs an effective deep ordinal regression (DOR) network for rotation correction, which can reduce the difficulty of multimodal image registration and boost image registration. The proposed DOR network transforms the rotation correction task into a rotation ordinal regression problem, which can exploit the potential relationship between the rotation ordinals to improve the accuracy of rotation estimation. In the fine registration stage, we adopt the DDL network to deal with the image modality change based on the rotation-corrected images. Extensive experimental results on multimodal image datasets demonstrate the significant advantages of the proposed coarse-to-fine deep learning registration framework. The DOR network achieves higher rotation correction accuracy, which can significantly improve the multimodal image registration performances.

I. Introduction

Multimodal image registration aims to align images that are acquired from the same scene, but by different sensors [1], [2], [3]. Due to multimodal images containing complementary information, multimodal image registration is crucial in various remote sensing image applications, such as change detection [4], [5], [6], object detection [7], [8], image fusion [9], [10], [11], targets tracking [12], image classification [13], [14], and image segmentation [15]. However, there are significant differences between multimodal images, which increase the challenges of multimodal remote sensing image registration.