Fusion of Thermal and RGB Images for Automated Deep Learning Based Marble Crack Detection

Research is constantly turning towards the development of image-based inspection tools that leverage deep learning models to automate surface crack detection in materials. Yet, most efforts involve color imaging which may lead to poor models’ performance in different lighting variations. Thermal images are more constant in changes of lighting, however, lack crispness of details. In this work, thermal-RGB image fusion techniques are investigated for accurate marble crack detection. The proposed method uses a Feature Pyramid Network (FPN) combined with three different backbone architectures and an augmented dataset of thermal-RGB image pairs. The performances of different image modalities, i.e., RGB, thermal and fused images, are comparatively evaluated, to test the impact of image type, fusion techniques, deep networks, and feature extraction backbones, on the segmentation results, considering two different experimental evaluation approaches: a model-based and a backbone-based approach. Results verify the initial hypothesis that fused images are richer in spatial information. Model-based evaluation resulted in scores of up to 85.07% in terms of mean Intersection over Union (mIoU), which is 2.9% and 4.75% higher than by using separately the corresponding RGB (82.67% mIoU) and thermal (81.21% mIoU) images, respectively. Backbone-based evaluation highlighted seresnext50 as the optimal feature extraction network combination for FPN, reporting an mIoU of 87.04%.