Multi-modal Target Detection Based on Edge Computing

Since single visible and single infrared target detection is often susceptible to factors such as light and weather, it has the problem of poor imaging effect and low image resolution in situations such as low light and bad weather. However, by using both IR and visible light to image and fuse together, it is possible to obtain fused images with prominent targets and rich background information at night and in poor weather conditions, thus improving the accuracy of target recognition and detection. Therefore, this paper proposes a multi-modal video fusion algorithm based on multi-scale decomposition and fusion based on the edge heterogeneous SoC board ZCU102 as the processing platform to achieve real-time co-awareness of visible-IR cameras. To address the problem of slow detection speed of traditional video monitoring platforms, a light-weight target detection algorithm based on improved YOLOv4 is proposed, and FPGA acceleration and deployment in a heterogeneous SoC is performed to achieve intelligent detection of target objects. Experimental results show that the image detection accuracy of the proposed method is significantly improved compared to RGB and IR single-mode images, with a 2.9 percentage point improvement in mAP compared to IR image-based and 7.2 percentage points improvement compared to visible images, outperforming the traditional method.