Performance Driven Experimental Analysis of Apache Spark Configuration Parameters

Big Data refers to huge volume of data in the range of peta bytes or higher. In order to store and process such large volume of data there is a need for distributed processing platforms as conventional stand-alone platforms cannot store or process bigdata. Distributed bigdata analytics frameworks helps in storage and processing of bigdata. The most popularly used open source bigdata analytics framework provided by Apache foundation includes Hadoop and Spark. Apache Spark is widely considered for processing bigdata because of its in-memory computation capability. The major challenges involved in big data analytics are managing the huge volume and variety of data originating from different sources in different formats. Apache Spark has around 180 configurable parameters and all of these parameters are set to some default values. It is very difficult for the normal user to identify the optimal values for these configuration parameters. And also, not all parameters may have impact on the application performance. Therefore, the proposed work aims at conducting an experimental research for identifying the subset of key apache spark configuration parameters that impacts on the execution time of an application and its optimal values to maximize performance by analyzing the effect of various parameters and its values. In the proposed work the Execution Time of the spark job/application is considered as the performance indicator. The result section of the proposed work presents the various key parameters and its optimal values.