Infrared and Visible Image Fusion based on CNN and Saliency Detection

In this paper, a novel image fusion framework is proposed for fusing the infrared and visible images based on convolutional neural network (CNN) and saliency detection. The proposed framework overcomes the bottleneck of previous pixelbased CNN methods in that it preserves the intensity and texture homogeneity within the saliency regions. First, an initial fused image was obtained using a two-channel CNN. Then a saliency map is derived based on the difference in structural similarity between the infrared and visible source images with the initial fused image. Finally, a weighted fusion rule is applied to generate the fused image. The proposed method was validated on a combined dataset of CIFAR-10 dataset and the TNO image fusion dataset. The results show that the fused images with CNN and saliency detection present good infrared target features as well as clear visible background, and effectively preserve the visual details and infrared features of the target regions. Our method showed improved performance over the state-of-the-art image fusion methods in qualitative and quantitative evaluations.