$Q$ -Value Prediction for Reinforcement Learning Assisted Garbage Collection to Reduce Long Tail Latency in SSD

Garbage collection (GC), an essential operation for flash storage systems, causes long tail latency which is one of the key problems in real-time and quality-critical systems. In this article, we take advantage of reinforcement learning (RL) to reduce long tail latency. Especially, we propose two novel techniques which are: 1) Q-table cache (QTC) and 2) Q-value prediction. The QTC allows us to utilize appropriate and frequently recurring key states at a small memory cost. We propose a neural network called Q-value prediction network (QP Net) that predicts the initial Q-value of a new state in the QTC. The integrated solution of QTC and QP Net enables us to benefit from both short-term (by QTC) and long-term (by QP Net) history of system behavior to reduce the long tail latency. The experimental results demonstrate that the proposed scheme offers significant (by 25%-37%) reductions in the long tail latency of storageintensive workloads compared with the state-of-the-art solution that adopts an RL-assisted GC scheduler.

I. Introduction

Garbage collection (GC) is essential in flash memory storage, namely, solid-state drive (SSD), owing to the erase-before-write nature. GC causes IO blocking that increases the latency of subsequent requests, up to more than 100 times the median latency at the 99th percentile [1], which does not satisfy the quality-of-service (QoS) in real-time and quality-critical systems, e.g., service level agreement (SLA) of the enterprise server systems. Therefore, it is very important to reduce long tail latency. In addition, as the flash memory architecture changes from a 2-D planner to a 3-D stack structure, the block size (the number of pages per block) increases, which increases the blocking time due to the page copy operation in GC and finally aggravates the long tail latency problem [2].