Dip Into: A Novel Method for Visual Speech Recognition using Deep Learning

Visual speech Recognition is a method of understanding speech that allows for visual interpretation of speech sounds. Though most people who have normal hearing process some voice information from the sight of the moving mouth, Visual Speech Recognition is most commonly employed by those who are deaf or hard of hearing. Additionally, conversing clearly in loud surroundings can be improved by comprehending the linguistic clues of visual speech recognition systems. By utilizing the inherent existence of audio and visual streams in videos, a model for identifying the effects of intermodal self-monitoring for speech reconstruction (video-audio) is proposed in this paper. The model, which does not require human annotation, utilizes an attention architecture and an autoregressive encoder-decoder. For the purpose of reconstructing speech, we directly map silent face movement sequences to mel-scale spectrograms.