Data Deduplication Based on Hadoop

Efficient and scalable deduplication techniques are required to serve the need of removing duplicated data in big data processing platforms such as Hadoop. In this paper, an integrated deduplication approach is proposed by taking the features of Hadoop into acount and leveraging parallelism based on MapReduce and HBase so as to speed up the deduplication procedure. In our proposed approach, a new small-file aggregation scheme is proposed, and the new standard of Secure Hash Algorithm-3, Keccak is employed. The effectiveness of our proposed approach is demonstrated by extensive experiments on both artificial datasets with specific characteristics, as well as real world datasets.