Predictive Adaptive Streaming to Enable Mobile 360-Degree and VR Experiences

As 360-degree videos and virtual reality (VR) applications become popular for consumer and enterprise use cases, the desire to enable truly mobile experiences also increases. Delivering 360-degree videos and cloud/edge-based VR applications require ultra-high bandwidth and ultra-low latency [1], challenging to achieve with mobile networks. A common approach to reduce bandwidth is streaming only the field of view (FOV). However, extracting and transmitting the FOV in response to user head motion can add high latency, adversely affecting user experience. In this paper, we propose a predictive adaptive streaming approach, where the predicted view with high predictive probability is adaptively encoded in relatively high quality according to bandwidth conditions and transmitted in advance, leading to a simultaneous reduction in bandwidth and latency. The predictive adaptive streaming method is based on a deep-learning-based viewpoint prediction model we develop, which uses past head motions to predict where a user will be looking in the 360-degree view. Using a very large dataset consisting of head motion traces from over 36,000 viewers for nineteen 360-degree/VR videos, we validate the ability of our predictive adaptive streaming method to offer high-quality view while simultaneously significantly reducing bandwidth.

I. Introduction

Recently, 360-degree videos and virtual reality (VR) applications have attracted significant interest in various fields, including entertainment, education, manufacturing, transportation, healthcare, and other consumer-facing services. These applications exhibit enormous potential as the next generation of multimedia content to be adopted by enterprises and consumers via providing richer, more engaging and more immersive experiences. According to market research [2], VR and augmented reality (AR) ecosystem is predicted to be an billion market by 2025, roughly the size of the desktop PC market today. However, several key hurdles need to be overcome for businesses and consumers to get fully on board with VR technology [3], such as cheaper price and compelling content, and most importantly a truly mobile VR experience, in line with the expectation and adoption of mobile experiences in almost all consumer and enterprise verticals today. Of particular interest is how to develop mobile (wireless and lightweight) head-mounted displays (HMDs), and how to enable VR experience on the mobile HMDs using bandwidth constrained mobile networks, while satisfying the ultra-low latency requirements.