GAN-based End-to-End Unsupervised Image Registration for RGB-Infrared Image

Image registration is a pre-processing step used in various computer vision applications. This paper presents an unsupervised image registration for a given pair of RGB and infrared images, with the RGB image used as the reference for infrared. This method exploits the GAN architecture with a spatial transformer module, to synthesize the transformed image using an unsupervised loss criterion. This loss used for error backpropagation taken as a linear combination of adversarial loss; Mean Squared Error (MSE) loss between the input RGB image and image synthesized by the generator; KL divergence loss between the IR image and the synthesized image; and another MSE loss estimated using features maps extracted from pretrained VGG-16. The adversarial loss forces the generator to output an IR like image, with the input IR image and the generated IR image labelled as real and fake respectively. The other three losses are backpropagated through the generator network to learn the transformation as well as to preserve the structure and resolution of the generated image. This unsupervised learning process is stopped after a specified number of iterations based on a validation set. A supervised method has also been developed for comparison with the presented method. The SSIM and PSNR values estimated between predicted registered image and its ground truth has been used as evaluation criteria. The unsupervised method has scored 0.8351±0.06 and 35.2723±0.68 for SSIM and PSNR respectively, while supervised scored as 0.7620±0.08 and 15.8978±2.21.

I. Introduction

Registration of images plays a fundamental role in many computer vision applications, for example, medical image processing or remote sensing. It is required to combine imaging data captured with different sensors, to have all the information on a single platform. One of the images is referred as a fixed while other as moving image.