Research on Automatic Online Analysis Method of Data Hotness in Big Data Scenario

With the development of Big Data technology, data volume has reached a massive level, and the types of data have become more complex, while the access characteristics of data by different applications vary widely, resulting in significant differences in the frequency of different data being accessed. In order to efficiently manage massive amounts of data, more and more data management and deployment systems are considering policies based on the hotness of data access. Therefore, how to define the hotness of data and extract the hotness of data online becomes a pressing problem nowadays. To address this problem, this paper proposes an online method to calculate data hotness, which takes into account both the creation time of data (called age in this paper) and the frequency of data access. In order to verify the above method, this paper focuses on the application scenario of Amazon product review data analysis. Through analyzing the review data on HDFS cluster, the universality of different hotness data distribution of HDFS cluster is verified, and it shows better advantages by comparing with other work.