A review of research on MapReduce scheduling algorithms in Hadoop

Big data has created an era of tera where bulk volume of data is being collected at escalating rates. Due to increase in storage capacities, processing power and availability of data, the size of global data is growing in zeta-bytes. Hadoop is one of the technologies in the big data landscape for analyzing the data through Hadoop Distributed File System and Map-Reduce. Job scheduling is an important activity for efficient management of cluster resources. Hadoop schedulers are pluggable components which assign resources to jobs. In a variety of schedulers, prominent are the default FIFO, Fair and Capacity schedulers. In this paper, a comprehensive survey of the various job scheduling algorithms has been performed. Also their comparative parametric analysis has been carried out by emphasizing the common key points in these schedulers.

I. Introduction

Big data [1] refers to a massive collection of large amount of data whose processing depends upon open-source frameworks like Hadoop and MapReduce. It cannot be processed using traditional data-processing tools like relational databases and Structured Query Language. Specifically Big Data refers to the creation, storage, retrieval and analysis of data in terms of five V's viz. volume, velocity, variety, veracity and value. According to a report [2], Facebook processes more than 500TB of data daily. Many other similar reports on big data statistics [3] throw light over the challenges of big data.