Enhance Public Safety Surveillance in Smart Cities by Fusing Optical and Thermal Cameras

The recent advancements in the Internet of Video Things (IoVT) and Edge-Fog-Cloud Computing paradigm make smart public safety surveillance (SPSS) a realistic solution for an effective public safety service in smart cities. Typically, a fully functional SPSS system requires multiple sensory inputs for situational awareness (SAW). As an essential component in the context of highly complex, dynamic, and heterogeneous smart city operations, SPSS is expected to be environment-resilient. Personal safety is among the top concerns of the residents in smart cities, and correspondingly pedestrian detectors are critical. Contemporary pedestrian detectors use optical cameras, whose accuracy is diminished in low-light environments, and they are rendered ineffective when obstacles block the direct line of sight to the camera. Complementary imaging sensors such as infrared have shown promise. This paper presents a full-spectrum, environment-resilient surveillance platform as an ultimate solution, which consists of multiple imaging units to cover a wide sensing spectrum. The initial hybrid pedestrian detection (HYPE) scheme is based on the fusion of data obtained from an IoVT network equipped with optical and thermal cameras. We demonstrate that training the YOLOv5 object detection model on a dataset of infrared images improves its accuracy in the detection of humans present in thermal surveillance images. A 41% decrease in objectness loss is achieved after transfer learning is performed.