Differential Feature Awareness Network Within Antagonistic Learning for Infrared-Visible Object Detection

The combination of infrared and visible videos aims to gather more comprehensive feature information from multiple sources and reach superior results on various practical tasks, such as detection and segmentation, over that of a single modality. However, most existing dual-modality object detection algorithms ignore the modal differences and fail to consider the correlation between feature extraction and fusion, which leads to incomplete extraction and inadequate fusion of dual-modality features. Hence, there raises an issue of how to preserve each unique modal feature and fully utilize the complementary infrared and visible information. Facing the above challenges, we propose a novel Differential Feature Awareness Network (DFANet) within antagonistic learning for infrared and visible object detection. The proposed model consists of an Antagonistic Feature Extraction with Divergence (AFED) module used to extract the differential infrared and visible features with unique information, and an Attention-based Differential Feature Fusion (ADFF) module used to fully fuse the extracted differential features. We conduct performance comparisons with existing state-of-the-art models on two benchmark datasets to represent the robustness and superiority of DFANet, and numerous ablation experiments to illustrate its effectiveness.

I. Introduction

The videos captured by a single sensor or under a single shooting setting can only describe the imaging scene from a limited perspective, which restricts algorithms from understanding the semantics. Some works enhance feature mapping by combining the multimodal complementary images obtained from various sensors or different shooting settings, and have proven their effectiveness. Among all the multimodal object detection scenarios, infrared and visible images are the two most commonly used signals. Infrared images capture thermal radiation emitted from objects, which can effectively highlight salient objects but lack texture details. In contrast, visible images usually contain rich detail information but are easily affected by the complex background to lose objects. These complementary features can facilitate algorithm learning to obtain desired results containing rich texture details and accurate semantic information. Owing to the excellent characteristics of the detection performance, infrared and visible object detection has been widely applied in video surveillance [1], vehicle inspection [2], [3], pedestrian detection [4], [5], autonomous driving [6], [7], and many other domains [8], [9].