GSLip: A Global Lip-Reading Framework with Solid Dilated Convolutions

The mainstream lip-reading framework employs Residual Network (ResNet) for spatial feature extraction and Temporal Convolutional Network (TCN) for constructing the temporal model. Addressing the modes of single-frame spatial and multi-frame temporal information extraction, we analyze and propose respective improvements. When extracting single-frame lip information, it is crucial to consider not only local fine-grained information but also global coarse-grained information. Global features such as lip contour, size, and movement amplitude significantly contribute to extracting local fine-grained lip features. However, traditional convolution, a key component of ResNet, is primarily utilized to extract features based on sliding windows in small regions, making ResNet less effective at capturing global features. Additionally, we observe key motion changes appearing in only a few consecutive frames in lipreading task. For multi-frame temporal information extraction, TCN is found to be inadequate in handling local continuous temporal dependencies. Therefore, we develop GSLip, a global lip-reading framework with solid dilated convolutions. The key ideas of GSLip include: 1) The Residual Global Context Network (ResGNet), which addresses the deficiency of traditional convolution in extracting global features; 2) the Continuous Temporal Convolutional Network (C-TCN), that enhances the ability to extract continuous temporal information. With the incorporation of these modules, our method has demonstrated excellent results on the Lip Reading in the Wild (LRW) and LRW-1000 datasets.