Image Parallel Processing by Using MapReduce

In recent years, big data processing platform Hadoop and parallel computing model MapReduce have achieved good results in mass data processing. However, since Hadoop does not design the input data streams for image files and MapReduce does not define an interface to match the data types of image files, the technology that running MapReduce program on Hadoop cluster to process massive image files in parallel is not mature. In addition, as image characteristic data (such as pixel intensive) is often skewness, image parallel processing on heterogeneous Hadoop cluster triggers a large data transfer and load imbalance among nodes, so that reducing the efficiency of MapReduce job. To tackle aforementioned problems, this paper carried out the following work, (1) The Hadoop platform is optimized to efficiently process the massive image; (2) The MapReduce model is extended and the interface is designed to support parallel image processing. (3) A new partition, namely PIHA, is proposed to mitigate data skew in heterogeneous Hadoop 3.2.0. Experiments show that the method proposed in this paper can effectively improve the efficiency of parallel image processing using MapReduce on Hadoop.