Graph-Based Progressive Fusion Network for Multi-Modality Vehicle Re-Identification

Vehicle re-identification (Re-ID) is a critical task in intelligent transportation, aiming to match vehicle images of the same identity captured by non-overlapping cameras. However, it is difficult to achieve satisfactory results based on RGB images alone in darkness. Therefore, it is of great importance to consider multi-modality vehicle re-identification. Currently, the proposed works deal with different modality features through direct summation and fusion based on heat map, which however ignores the relationship between them. Meanwhile, there is a huge gap between the different modalities, which needs to be reduced. In this paper, to solve the above two problems, we propose a Graph-based Progressive Fusion Network (GPFNet) using a graph convolutional network to adaptively fuse multi-modality features in an end-to-end learning framework. GPFNet consists of a CNN feature extraction module (FEM), a GCN feature fusion module (FFM), and a loss function module (LFM). Firstly, in FEM, we employ a multi-stream network architecture to extract single-modality features and common-modality features and employ a random modality substitution module to extract mixed-modality features. Secondly, in FFM, we design an efficient graph structure to associate the features of different modalities and adopt a progressive two-stage strategy to fuse them. Finally, in LFM, we use GCN-aware multi-modality loss to constrain the features. For reducing modality differences and contributing better initial mixed-modality features to FFM, we propose random modality substitution as a data enhancement method for multi-modality datasets. Extensive experiments on multi-modality vehicle Re-ID datasets RGBN300 and RGBNT100 show that our model achieves state-of-the-art performance.

I. Introduction

Vehicle re-identification (Re-ID) has very important applications in video surveillance and intelligent transportation, aiming to match the vehicle images of the same identity across non-overlapped camera views [1], [2], [3], [4], [5], [6], [7], [8]. Compared with person re-identification, it is more challenging due to the small inter-class similarity and large intra-class variation of vehicles [9]. Currently, vehicle re-identification methods based on deep learning have made great advances, but most of them focused on RGB single modality alone [10], [11], [12], [13]. In environments lacking ambient light, it is difficult to obtain effective information only with RGB cameras, which largely limits the scalability of Re-ID models in real-world scenarios. Fig. 1 shows the comparison of vehicle images of three modalities. Generally, Near Infrared (NIR) and Thermal Infrared (TIR) images are robust to lighting variations and provide complementary information to RGB images. With the complementary of NIR and TIR to RGB, we can effectively solve the problem of insufficient lighting. In addition, as the cost of capturing NIR and TIR images has decreased, it is easy to obtain NIR and TIR modality information. However, most current vehicle re-identification methods are unable to handle other modality information except RGB, and thus are insufficient to solve the RGB, NIR, and TIR modalities i.e. multi-modality vehicle re-identification task. Therefore, it is necessary to study the multi-modality vehicle re-identification. Fig. 1.

Comparison of vehicle images of different modalities. (a) RGB modality. (b) NIR modality. (c) TIR modality. (d) mixed modality.