Multimodal Data Augmentation for Visual-Infrared Person ReID with Corrupted Data

The re-identification (ReID) of individuals over a complex network of cameras is a challenging task, especially under real-world surveillance conditions. Several deep learning models have been proposed for visible-infrared (V-I) person ReID to recognize individuals from images captured using RGB and IR cameras. However, performance may decline considerably if RGB and IR images captured at test time are corrupted (e.g., noise, blur, and weather conditions). Although various data augmentation (DA) methods have been explored to improve the generalization capacity, these are not adapted for V-I person ReID. In this paper, a specialized DA strategy is proposed to address this multimodal setting. Given both the V and I modalities, this strategy allows to diminish the impact of corruption on the accuracy of deep person ReID models. Corruption may be modality-specific, and an additional modality often provides complementary information. Our multimodal DA strategy is designed specifically to encourage modality collaboration and reinforce generalization capability. For instance, punctual masking of modalities forces the model to select the informative modality. Local DA is also explored for advanced selection of features within and among modalities. The impact of training baseline fusion models for V-I person ReID using the proposed multimodal DA strategy is assessed on corrupted versions of the SYSUMM01, RegDB, and ThermalWORLD datasets in terms of complexity and efficiency. Results indicate that using our strategy provides V-I ReID models the ability to exploit both shared and individual modality knowledge so they can out-perform models trained with no or unimodal DA. GitHub code: https://github.com/art2611/ML-MDA.

1. Introduction

Real-world monitoring and surveillance application (e.g., individuals in airport, and vehicles in traffic) rely on challenging tasks, like object detection [55], [51], tracking [29], and re-identification (ReID) [24], [50]. The aim of person ReID is to recognize individuals over a set of distributed non-overlapping cameras. State-of-art systems for person re-identification (e.g., deep Siamese networks) typically learn an embedding through various metric learning losses, which aim at making similar image pairs (with the same identity) closer to each other and dissimilar image pairs (with different identities) more distant from each other. Despite the recent advances with deep learning (DL) models, person ReID remains a challenging task due to the non-rigid structure of the human body, the different view-points/poses with which a person can be observed, image corruption, and the variability of capture conditions (e.g., illumination, scale, contrast) [3], [31].