Learning Automata Based Q-Learning for Content Placement in Cooperative Caching

An optimization problem of content placement in cooperative caching is formulated, with the aim of maximizing the sum mean opinion score (MOS) of mobile users. Firstly, as user mobility and content popularity have significant impacts on the user experience, a recurrent neural network (RNN) is invoked for user mobility prediction and content popularity prediction. More particularly, practical data collected from GPS-tracker app on smartphones is tackled to test the accuracy of user mobility prediction. Then, based on the predicted mobile users' positions and content popularity, a learning automata based Q-learning (LAQL) algorithm for cooperative caching is proposed, in which learning automata (LA) is invoked for Q-learning to obtain an optimal action selection in a random and stationary environment. It is proven that the LA based action selection scheme is capable of enabling every state to select the optimal action with arbitrary high probability if Q-learning is able to converge to the optimal Q value eventually. In the LAQL algorithm, a central processor acts as the intelligent agent, which allocate contents to BSs according to the reward or penalty from the feedback of the BSs and users, iteratively. To characterize the performance of the proposed LAQL algorithms, sum MOS of users is applied to define the reward function. Extensive simulation results reveal that: 1) the prediction error of RNNs based algorithm lessen with the increase of iterations and nodes; 2) the proposed LAQL achieves significant performance improvement against traditional Q-learning algorithm; and 3) the cooperative caching scheme is capable of outperforming non-cooperative caching and random caching of 3% and 4%, respectively.

I. Introduction

It is expected that the first fifth generation (5G)-network based solutions will be commercially launched by 2020 [2]. According to a recent report from Cisco [3], mobile data traffic has grown 18-fold over the past 5 years and will grow at a compound annual growth rate (CAGR) of 47 percent from 2016 to 2021. More particularly, mobile video traffic accounted for 60 percent of total mobile data traffic in 2016, and over three-fourths (78 percent) of the world’s mobile data traffic will be video by 2021. In the vast area of 5G wireless systems, an extraordinary variety of technological innovations was included to support a large number of devices over limited spectrum resources [4]–[7]. Investigation presented in [8] shows that backhaul consumes up to 50 percent of the power in a wireless access network. Proactive caching at the base stations (BSs) is a promising approach to dealing with this problem, by introducing caching capabilities at BSs and then pre-fetching contents during off-peak hours before being requested locally by users (see [9], and references therein).