Progressive Multi-Iteration Registration-Fusion Co-Optimization Network for Unregistered Hyperspectral Image Super-Resolution

Existing fusion-based hyperspectral image super-resolution (fusion-based HSI-SR) methods usually reconstruct high-resolution hyperspectral image (HR-HSI) by integrating the complementary information of low-resolution hyperspectral image (LR-HSI) and high-resolution multispectral image (HR-MSI). However, most of such methods rely on accurately registered images or consider registration and fusion as a two-stage task, which means that fusion must tolerate the accumulation of errors due to misregistration. In this article, we propose a progressive multi-iteration registration-fusion co-optimization network (PMI-RFCoNet) for unregistered HSI-SR, which progressively refines the registration and fusion result over multiple levels to reconstruct registered HR-HSI. To achieve registration-fusion co-optimization, the registration-fusion co-optimization block (Co-RFB) is designed to iterate continuously over multiple levels. We embed the interactive registration module (IRM) and the spectral recalibration and fusion module (SRFU) in Co-RFB, which can facilitate the network utilizing spatial and spectral features at different levels to generate more accurate HR-HSI. Specifically, IRM generates deformation field based on spatial correlations captured at long distances to repair non-rigid pixel offsets, and SRFU further performs adaptive high-fidelity spectral correction and spatial information fusion on the registration results. We conduct experimental verification on four widely used datasets, and the results show that PMI-RFCoNet can flexibly cope with different types and degrees of non-rigid deformation and achieve superior performance. Code is available at
https://github.com/Jiahuiqu/PMI-RFCoNet
.