Low Resource Speech-to-Speech Translation of English videos to Kannada with Lip-Synchronization

The introduction of online education has revolutionized the way people acquire knowledge. With resources from top universities like Stanford and MIT, students have access to an immense wealth of educational videos and other material. However, the majority of these resources are in English, creating a language barrier that prevents non-native speakers from utilizing them effectively. In addition, Indic languages have very limited educational resources available online and manual dubbing of videos is time-consuming and labour-intensive. Automated subtitling is not a feasible alternative either, as reading Kannada subtitles alongside English videos can be strenuous. To address these issues, a system is needed that can dub English videos into a specific target language. The focus of this work is on developing such a system for Kannada. The proposed solution is an exhaustive audio-visual system that generates Kannada videos from English videos with accurate lip synchronization, even in dynamic environments, with any kind of face or voice. The system consists of four stages, namely speech-to-text, neural machine translation, text-to-speech, and lip-synchronization. Each component has been evaluated using different metrics, such as Word Error Rate (WER), BiLingual Evaluation Understudy (BLEU) scores, and Mean Opinion Score (MOS), based on surveys conducted to test the output. Extensive quantitative evaluations portray that the naturalness of the lip-synced output video is superior to manually dubbed videos, proved by an overall Mean Opinion Score (MOS) of 4.56. The proposed system has the potential to revolutionize the film industry by replacing the entire manual dubbing process and generating Pan-Global movies. It will also bridge the language gap for people with little or no understanding of English and allow them to access valuable educational resources online in their native language.