Multimodal Biometric Recognition Neural Networks with Branch Attention Mechanisms

It is generally known that compared to unimodal systems, the multi-modal biometric systems can improve recognition accuracy by utilizing the complementary features of multiple biometric. Nevertheless, the modalities used may not always play equally important roles in multi-modal recognition. Especially, in practical application scenarios, it often occurs that some of the biometric samples are contaminated by noise and corruption. In this context, the multi-modal recognition systems should pay more attention to the high-quality modality than the ones with low-quality samples, so as to maintain high accuracy. In this paper, we proposes a multi-branch deep learning network for multi-modal recognition, each branch using one biometric sample as input. To highlight the more important biometric modalities, we propose to embed a branch attention module between branches before multi-modal feature fusion. The branch attention module includes fuse, select, and calibrate steps, which can dynamically calculate the weight of each branch based on its importance. The proposed multi-modal recognition model is tested on a chimeric bimodal dataset with the face and the ear modalities. Our proposed method still achieves 92.76% of rank-1 on the most contaminated dataset; the rest of the experimental results are shown in the Experiment and Performance Analysis section. Our experimental results clearly demonstrate that the proposed branch attention method can improve the multi-modal recognition performance.