H-vectors: Hybrid Transformer and ResNet Based TDNN for Speaker Verification

Recently, the ResNet-based and Transformer-based Time Delay Neural Networks (TDNN) have achieved tremendous success in the area of speaker verification. However, ResNet-based TDNN models cannot capture global interaction information, while Transformer-based TDNN models struggle to explore multi-scale hierarchical pyramid convolutional features. This paper proposes a novel hybrid Transformer- and ResNet-based TDNN to learn speaker embeddings, named H-vectors, which integrates Transformer and ResNet via a macaron structure, fully exploiting global interaction and local temporal correlations. Specifically, a combination of Multi-Head Self-Attention and SE-Res2block is sandwiched between a pair of feed-forward modules. Additionally, the Gated Relative Position Bias is used to provide position information for Multi-Head Self-Attention, so that the relative position information between frames can be adaptively calibrated according to speech content. Extensive experiments on the VoxCeleb1 benchmark demonstrate that the proposed model achieves an EER of 0.79 and a MinDCF of 0.103, respectively, surpassing most of the state-of-the-art ResNet or Transformer-based TDNN models.