A Hybrid Temporal Modeling Phoneme Recognized Network for Real-Time Speech Animation

Real-time speech animation is a compelling but challenging topic, where multimodal inputs such as audio, video or depth information, have been considered to leverage performance. However, in conditions that audio is the only available input, the quality of the outcome relies heavily on the real-time phoneme recognition. In this paper, we introduce a novel deep-learning based Realtime Phoneme Recognition Network architecture to leverage spatial, separate temporal and unified temporal modeling in the input audio data. Our network can achieve super performance in phoneme recognition. Our empirical results show that compared to the state-of-the-art algorithms, RealPRNet can achieve up to 20% PER improvement and 4% animation quality improvement based on the subject tested.