An Efficient Deep Reinforcement Learning-Based Automatic Cache Replacement Policy in Cloud Block Storage Systems

With the popularity of cloud services, cloud block storage (CBS) systems have been widely deployed by cloud providers. Cloud cache plays a vital role in maintaining high and stable performance in cloud block storage systems. In the past few decades, much research has been conducted on the design of cache replacement policies. Prior work frequently relies on manually-engineered heuristics to capture the most common cache access patterns, or predict the reuse distance and try to identify the blocks that are either cache-friendly or cache-averse. Researchers are now applying recent advances in machine learning to guide cache replacement policy, augmenting or replacing traditional heuristics and data structures. However, most existing approaches depend on a certain environment which restricted their application, e.g., some methods only consider the on-chip cache consisting of program counters (PCs). Moreover, those approaches with attractive hit rates are usually unable to deal with modern irregular workloads, due to the limited feature used. In contrast, we propose a cloud cache replacement framework to automatically learn the relationship between the probability distribution of different replacement policies and workload distribution by using deep reinforcement learning. We train an end-to-end cache replacement policy based on the requested address with two efficient and stable cache replacement policies. Furthermore, by using prioritized experience replay and setting parameter constraints, our framework can accelerate the offline training process without affecting the cloud application. We have evaluated our proposed framework by using block-based I/O traces collected from Alibaba Cloud and Tencent Cloud, two of the largest cloud providers in the world, and several open-source traces. Experimental results show that our method not only outperforms several state-of-the-art cache methods in hit rate, but also reduces request latency and data traffic to the backend storage.