ConeSSD: A Novel Policy to Optimize the Performance of HDFS Heterogeneous Storage

HDFS(Hadoop distributed file system) is the core storage service of Hadoop, which stores and processes large datasets efficiently. Therefore, the performance of storage services significantly impacts big data processing applications. Using SSD to replace HDD devices can obtain better performance, but due to the high unit price, it is too expensive to replace all HDD with SSD in HDFS. Therefore, heterogeneous storage has been widely used in HDFS. However, the existing hierarchical strategy of HDFS is simple and cannot obtain better cost performance. This paper presents a novel heterogeneous storage policy named Cone_SSD, which fully uses SSD media on each node to get the IO performance to near SSD. Experimental results show that the proposed policy improves read and write performance by 48.11 % and 47.45%, respectively, and the fluctuation of reading performance decreases by 54.92 % compared with other policies under the same price-performance ratio.