Reinforcement Learning Driven Adaptive VR Streaming with Optical Flow Based QoE

With the merit of containing full panoramic content in one camera, Virtual Reality (VR) and 360° videos have arisen in the field of industrial cloud manufacturing and training. Industrial Internet of Things (IoT), where many VR terminals needed to be online at the same time, can hardly guarantee VR's bandwidth requirement. However, by making use of users' quality of experience (QoE) awareness factors, including the relative moving speed and depth difference between the viewpoint and other content, bandwidth consumption can be reduced. In this paper, we propose Optical Flow Based VR(OFB-VR), an interactive method of VR streaming that can make use of VR users' QoE awareness to ease the bandwidth pressure. The Just-Noticeable Difference through Optical Flow Estimation (JND-OFE) is explored to quantify users' awareness of quality distortion in 360° videos. Accordingly, a novel 360° videos QoE metric based on Peak Signal-to-Noise Ratio and JND-OFE (PSNR-OF) is proposed. With the help of PSNR-OF, OFB-VR proposes a versatile-size tiling scheme to lessen the tiling overhead. A Reinforcement Learning (RL) method is implemented to make use of historical data to perform Adaptive BitRate (ABR). For evaluation, we take two prior VR streaming schemes, Pano and Plato, as baselines. Vast evaluations show that our system can increase the mean PSNR-OF score by 9.5-15.8% while maintaining the same rebuffer ratio compared with Pano and Plato in a fluctuate LTE bandwidth dataset. Evaluation results show that OFB-VR is a promising prototype for actual interactive industrial VR. A prototype of OFB-VR can be found in https://github.com/buptexplorers/OFB-VR.