Weakly Aligned Multimodal Flame Detection for Fire-Fighting Robots

Flame detection is a key module of fire-fighting robots, especially for autonomous fire suppression. To effectively tackle the fire-fighting tasks, fire-fighting robots are usually equipped with multimodal vision systems. On the one hand, cameras of different modalities can provide complementary visual information. On the other hand, the differences in installation position and resolution between different cameras also result in weakly aligned image pairs, that is, the positions of the same object in different modal images are inconsistent. Directly fusing the image features of different modalities is difficult to meet the accuracy and false alarm requirements of fire-fighting robots. Therefore, we propose a multimodal flame detection model based on projection and attention guidance. First, we use projection to obtain the approximate position of the flame in the thermal image and employ a neighbor sampling module to detect flames around it. Second, we design an attention guidance module based on index matching, which applies the attention map generated by the thermal modality to optimize the regional feature of the color modality. Experiments on multimodal datasets collected by an actual fire-fighting robot validate that the proposed method is effective in both fire and nonfire environments.

I. Introduction

Fire fighting is a risky work. Sending robots to fight fire instead of human fire fighters is a key trend [1]. Fire-fighting robots can be remotely controlled into extremely dangerous fire sites or used for quick response to fire alarms in the early stages. However, while the development of fire-fighting robots appears to be rapid, it is still an extremely challenging task for robots to extinguish fire autonomously. For autonomous fire fighting, flame detection is a fundamental step and essential task. For a mobile fire-fighting robot, flame detection can provide target location information, thus the robot knows where the fire is.