Implementation of an improved algorithm for frequent itemset mining using Hadoop

Searching frequent item-sets in large size heterogeneous databases in minimal time is considered as one of the most important data mining problem. As a solution of this problem, various algorithms have been proposed to speed up execution. Most of the recent proposed algorithms focussed on parallelizing the workload using large number of machine in distributed computational environment like MapReduce framework. A few of them are actually capable to determine the appropriate number of required computing computers, considering workload balancing and execution efficiency. But internally not capable to determine exact number of required iteration for any large size datasets in advance to find out the frequent item-set based on iterative sampling. In this paper, we propose an improved and compact algorithm (ICA) for finding frequent item-set in minimal time, using distributed computational environment. It is also capable of determining the exact number of internal iteration required for any large size datasets whether data is in structured or unstructured format.

I. Introduction

Frequent item-set mining is one of the most important techniques to find out frequent item-sets in data mining [1]. Industries use the extracted frequent item-sets in decision making or setting policies. For example a retail-sector company is interested to know customer buying habits in particular area to sell out their product. Here, frequent item-set mining helps the company to know customer buying habits. On the other hand, even government of nations use the frequent item-set technique to extract useful information that further help to provide better services to people. Frequent item-set mining is the part of frequent pattern mining where frequent pattern represents those subsequences and sub-graphs which are occurred many times frequently in a given data sets. Traditional data mining tools fails to extract frequent item-sets when the size of transactional database is too large to compute. In Big Data era, we need a new approach to compute frequent item-sets where data-sets consist of millions of records [2]. Researchers proposed various approach to deal with Big Data challenges, but all these approaches suffers from synchronization, work load balancing and fault-tolerance problem [3]. To overcome this problem MapReduce model come into existence, originally proposed by Google [4]. This MapReduce model supports parallelization of tasks under Hadoop architecture [5].