Multispectral Information Fusion With Reinforcement Learning for Object Tracking in IoT Edge Devices

With recent advances in sensor technology, multispectral systems are becoming increasingly attractive for intelligence, surveillance, and reconnaissance applications. Fusing information from multiple imaging modalities is a major task for such systems. Combining feature maps obtained from multiple deep neural network pipelines demonstrates promising performance for object detection and tracking. However, feature fusion using multiple deep networks is computationally intensive and therefore not suitable for resource-constrained IoT edge devices. In this paper, we propose a novel method to fuse the input space to enable processing of multispectral data via a single deep network. We use task-driven feedback as a reward signal for our reinforcement learning-based multispectral input fusion. Proposed approach not only improves tracking accuracy but also maximizes modality-specific information as intended by the user.