Comparative Study of Big Data Frameworks

We are really living in ever growing volume of data production. The huge amount of data in terabyte and petabytes are generating in real word and it is a challenging task to access, storage, analysis of all structured, unstructured and semi structured heterogeneous and complex data, also traditional tools is not suitable towards distributed and real-time processing. We need an efficient framework for processing such heterogeneous data and transform it into optimized meaningful information. There are many frameworks for distributed computing has been developed to perform huge amount of data processing. Hadoop Map Reduce is the extensively used framework because of its scalability, security, latency and efficiency, and reliability. The intension of this paper is to relative study of common framework such as Hadoop, Spark, Flink, Samza and Storm.

I. Introduction

According to IBM Big Data Analytics 2019 about 294 billion emails sent every day, over 1 billion google search every day with 40 thousand search every second, trillion of sensors monitor, track and communicate with each other’s, more than 30 petabytes of user generated data stored, accessed and analyzed, more than 230 million tweets each day with 7000+ tweets per second are generated. By 2020 at least third of all data will passes through cloud [1]. Before the year 2000 data was relatively smaller that the disk however data computation was complex, all the data computation depends on the processing power of the computer, later when data has grow the solution is large memory and fast processor. In order to store and process huge amount of data there are many frameworks available for data analytics. The fast evolution of big data technologies and the ready acceptance of the concept by public and private sectors left little time for the discourse to develop and mature in the academic domain [2]. The challenges of Big Data include capture, curation, storage, search, sharing, transfer, analysis and visualization [3]. Big Data Technology is very much adequate for the accurate analysis of our big data which yields strong conclusion and prediction. Big Data also categorized in two-part Operation Big Data and Analytical Big data. Using different Bigdata Framework we can analyze different Big Data issues/Problems.90% of data has got generated in last few years back.