Multi-Modality Deep Restoration of Extremely Compressed Face Videos

Arguably the most common and salient object in daily video communications is the talking head, as encountered in social media, virtual classrooms, teleconferences, news broadcasting, talk shows, etc. When communication bandwidth is limited by network congestions or cost effectiveness, compression artifacts in talking head videos are inevitable. The resulting video quality degradation is highly visible and objectionable due to high acuity of human visual system to faces. To solve this problem, we develop a multi-modality deep convolutional neural network method for restoring face videos that are aggressively compressed. The main innovation is a new DCNN architecture that incorporates known priors of multiple modalities: the video-synchronized speech signal and semantic elements of the compression code stream, including motion vectors, code partition map and quantization parameters. These priors strongly correlate with the latent video and hence they are able to enhance the capability of deep learning to remove compression artifacts. Ample empirical evidences are presented to validate the superior performance of the proposed DCNN method on face videos over the existing state-of-the-art methods.

1 Introduction

Video contents create well over sixty percent of data traffic on the Internet, and this percentage is still steadily climbing as more and more of people’s daily interactions are conducted on line. Communication bandwidths and data storages are under constant pressures due to rapid expansion and ubiquity of on-line video applications. As such, video compression has been and will continue to be an indispensable enabling technology in the modern digital world. For most users, video files have to be compressed by one of the popular video compression methods (e.g., MPEG-4 [1], H.264 [2], HEVC [3]) to a sufficiently small size to achieve an acceptable level of cost effectiveness. For high compression ratio or low bit rates, lossy video compression inevitably produces objectionable artifacts, such as blocking, blurring, ringing and jaggies. Recently quite a few deep learning methods are proposed to remove video compression artifacts. Compared with the pure end-to-end DCNN approach for video compression [4], [5], the methods of compression artifacts removal [6], [7] have the operational advantage of being compatible with existing video compression standards, as they are essentially a post-processing step of restoring already-decoded videos by the standards. We call this CNN-based video restoration strategy deep video decompression.