Mass log data processing and mining based on Hadoop and cloud computing

With the rapid development of the Internet, SaaS applications delivered as services through internet become an important alternative of traditional software. While using the services, users need real time usage information, and they also need to dig out useful knowledge. As a result, data processing and data mining techniques are designed to cope with such problems, and using log data is an effective method to record the SaaS usage information in a standard format. However, as the size of data grows, traditional distributed log data processing systems are not able to processing massive log data from SaaS applications with millions of users. This paper proposes a mass log data processing and data mining methods based on Hadoop to achieve scalability and performance. The model, process, architecture, and implementation of the data processing and mining methods are proposed, and the experimental results is shown and analyzed to prove the effectiveness of the methods.

I. Introduction

With the rapid development of the Internet, SaaS applications delivered as services through internet become an important alternative of traditional software^{[1], [2]}. SaaS applications are deployed in dynamic data centers [3] with cloud computing technologies managing resources to achieve flexibility and scalability ^[4],[5]. Thus, the relevant usage data of SaaS applications which can be massive and complicated become a huge information space.