Frequency Mining and Complementary Fusion Network for RGB-Infrared Object Detection

In recent years, object detection on visible (RGB) and infrared (IR) has gained significant attention as a promising solution for robust detection in complex scenarios, especially in low-light conditions. With the help of IR images, object detectors have become more reliable and robust in practical condition by combining the RGB and IR information. Despite significant progress in this field, current methods ignore the distinct characteristics of the two modalities when extracting features. RGB images contain detailed texture and color information, which means they have many high-frequency signals. Meanwhile, IR images have smoother textures and edges but clear shapes, indicating a significant amount of low-frequency information. We must consider the differences between the two modalities when extracting corresponding features. To address this issue, we propose a novel network architecture: the frequency mining and complementary fusion network (FMCFNet), which accounts for the intermodal variability. Our network contains two critical modules: the frequency feature extraction (FFE) module and the complementary fusion (CF) module. The FFE module utilizes filters of varying kernel and pooling sizes to extract features with diverse frequency information and then adaptively selects the most responsive frequency component. The CF module uses the similarity scores generated by cross attention to model the interactions between two modalities. Comprehensive experimental results demonstrate that our method can effectively combine RGB-IR complementary information, achieving robust detection results.

I. Introduction

Object detection, as a core task in the field of computer vision, has been widely used in various scenarios, such as video surveillance, resource exploration, and autonomous driving. However, most of the existing object detection methods [1], [2], [3], [4] are designed for RGB images, which cannot get robust detection results under different weather conditions, especially low-light conditions. To address this difficulty, some studies [5], [6], [7], [8] take infrared sensors as a viable alternative for object detection in the absence of light. Infrared sensors can detect the infrared radiation of an object and are insensitive to changes in ambient light. The use of multiple modalities in object detection offers a more comprehensive visual representation compared with unimodal detection, enabling mutual compensation for their respective limitations.