Multimodal Synchronization Detection: A Transformer-Based Approach Using Deep Metric Learning

The temporal correspondence among audio and video is crucial for understanding the lip movements and speech content. Learning this temporal correlation between video and audio has multiple use cases in downstream tasks such as detection, classification or generation. So, this task of detecting, if both the modalities are temporally synchronized or not, has a very strong impact on solving multiple tasks. Our focus lies in understanding the temporal correlation or mapping between a speaker’s speech and the lip movements, especially in scenarios with unconstrained and extensive vocabulary. Our approach involves modelling the frame sequence through an encoder network and employing cross-attention between the frame sequences and audio to jointly learn embeddings. We employ deep metric learning to learn temporal synchronization, aiming to distinguish between embeddings of two modalities with different alignments while enhancing similarity for those that are temporally aligned. Our model has been trained on the LRS2 dataset allowing us to assess synchronization tasks in a natural, unconstrained setting. After extensive evaluation using qualitative metrics, we found that our approach outmatch the state-of-the-art model for Acappella and LRS2 dataset.