Motion-Vector-Driven Lightweight ROI Tracking for Real-Time Saliency-Guided Video Encoding

The huge computation burden of state-of-the-art video coding technologies can be mitigated with Region-of-Interest (ROI) techniques that limit the highest coding effort to salient regions. However, the complexity overhead of saliency detection can easily cancel out the speed gain of ROI coding. This work introduces a lightweight ROI tracking technique that can be used in place of compute-intensive ROI detection to guide a video encoder in inter coding. Low computational overhead is achieved by feeding motion vectors (MVs) of a video encoder back to our neural network that is trained for accurate estimation of ROI movement and size changes. The network training is carried out with our new dataset that is also released in this work to foster the development of head tracking techniques in applications like video conferencing. Our experimental results demonstrate substantial speedups with minimal accuracy tradeoffs over traditional salient object detection (SOD) methods. In scenarios, where a single ROI is tracked with a 64-frame detection interval, our solution obtains up to 50-fold speedup with accuracy of 87% and an average ROI center error of 16 pixels. These results confirm that our ROI tracking approach is a potential technique for low-cost and low-power streaming media applications.

I. Introduction

The skyrocketing growth of visual data consumption by humans and machines has led to an unprecedented surge in global video traffic. This trend, coupled with the advent of high-quality immersive media applications, calls for more sophisticated video compression technologies that are able to overcome the constraints imposed by existing network and storage capacities. The latest video coding standards, like High Efficiency Video Coding (HEVC/H265) [1] and Versatile Video Coding (VVC/H266) [2], are in place to mitigate video bandwidth demands, but their computational requirements are cumbersome to reach without optimizations, particularly in real-time streaming media domain.