DiVAS: Video and Audio Synchronization with Dynamic Frame Rates

Synchronization issues between audio and video are one of the most disturbing quality defects in film production and live broadcasting. Even a discrepancy as short as 45 milliseconds can degrade the viewer's experience enough to warrant manual quality checks over entire movies. In this paper, we study the automatic discovery of such issues. Specifically, we focus on the alignment of lip movements with spoken words, targeting realistic production scenarios which can include background noise and music, intricate head poses, excessive makeup, or scenes with multiple individuals where the speaker is unknown. Our model's robustness also extends to various media specifications, including different video frame rates and audio sample rates. To address these challenges, we present a model fully based on Transformers that encodes face crops or full video frames and raw audio using timestamp information, identifies the speaker and provides highly accurate synchronization predictions much faster than previous methods.

1. Introduction

Audio-Video (AV) synchronization is a basic expectation to anyone that is consuming video, whether through streaming, social media, cable television, theaters or any other form of media. From the lens of the camera to the eye of the consumer, there are many instances where errors can be introduced, such as during content mastering, third party modifications, content encoding, or client playback. Studies show that the viewer experience can be negatively affected by a mere 45 millisecond (ms) discrepancy [7]; this is equivalent to a delay of a single frame in a 90 minute film at 25 frames per second (fps). While commercial solutions [1] exist, their scale and capabilities are insufficient for production. Thereby, detecting and identifying the origin of synchronization issues remains a significant burden for quality control teams, as it is largely a manual process. Thus, there is a pressing need for an automated detection system that can accurately identify and resolve AV synchronization issues before they reach the viewer.