Narrowing the Synthetic-to-Real Gap for Thermal Infrared Semantic Image Segmentation Using Diffusion-based Conditional Image Synthesis

Semantic segmentation is the task of assigning a semantic class to each pixel in an image. Due to the high annotation efforts for fully supervised learning of Deep Neural Networks (DNNs) for this task, only rather few comprehensive public datasets exist. This is particularly the case for thermal infrared imagery. To overcome this lack of training data, we propose to utilize conditional image synthesis in the thermal infrared spectrum. Existing semantic segmentation maps are used to condition the image generation process using pretrained text-to-image diffusion models. Therefore, we use the recently published ControlNet and retrain it to synthesize thermal infrared images for given semantic maps. In this way, we can generate large numbers of synthetic images that we can directly use together with the related segmentation map to train reference semantic segmentation approaches in the thermal infrared spectrum. Our experiments demonstrate that we achieve near state-of-the-art performance with pure synthetic training data on the recently published Full-time Multi-modality Benchmark (FMB) dataset and that our trained model shows better generalization ability across datasets. We provide code at https://github.com/HensoldtOptronicsCV/TIRControlNet.

1. Introduction

Semantic segmentation is a subtopic of image segmentation [32]. The task is to assign a semantic class to each pixel in an image. Semantic segmentation is used in several application areas such as automotive [9], remote sensing [63], and medical imaging [40]. In this paper, we focus on automotive applications. While modern deep learning-based approaches provide convincing results, they still require a large amount of training data. This is specifically the case for fully supervised learning schemes, where each pixel in an image has to be annotated [37]. As a result, only rather few comprehensive public datasets exist that provide dense labels for semantic segmentation [9], [33], [43], [62]. However, each of those datasets contains just a few thousand annotated images, which is by far less compared to public datasets annotated for tasks such as object detection [28] or image classification [42]. That is why recent research aims to reduce the annotation complexity for image segmentation [1], [7], [12], [23]. Procedurally generated synthetic data [38], [41], [58] that comes with precise annotation for free can be an alternative to real data but there is still a domain gap regarding scene and object appearance [36].