Connectivity Enhancing-Automated Lip Interpretation Using Deep Learning

The research paper exploits the advances in deep learning technology to build the convolutional deep learning model to extract text content from videos. We aim to understand the speech in the video clips by analyzing the lip movements of the person. To predict the sentence, videos are segmented into frames, and specifically, the mouth region is isolated and passed on to the convolutional neural networks to capture the important features. The Adam optimizer is used, and the learning rate is regularized. Bidirectional LSTMs are employed as they work extremely well with sequential data like speech. The Connectionist Temporal Loss Function is utilized in the back-propagation of the weights while the model is learning the outputs. Finally, the SoftMax activation is used to predict the sentence. The accuracy is 95% for the prediction during training and 92% for the test data, as the audios are synchronized with videos. The model provides higher robustness even in cases of variability in speech.