On Streaming Consistency of Big Data Stream Processing in Heterogenous Clutsers

There is currently a growing interest on Big Data Stream processing. With the increasing capabilities of Internet-based computing systems to generate, store and process Big Data Streams, various applications are benefiting from the information extracted in real time from data streams. Such families of applications comprise IoT-based monitoring systems, decision making systems, recommendation systems, security, etc. In all of them, the common ingredient is that of using real time data analytics for the purpose at hand. The foremost challenge of processing Big Data Streams is that of throughput performance, i.e., the computing system should be able to compute at high throughput to accommodate the high data stream rate generation in input. Additionally to this challenge, the processing should be such that the consistency of streaming is achieved. The consistency of streaming requires that the order of events in the original stream (in input) is preserved also in the output. In this paper we show that using a heterogeneous cluster for Big Data Stream processing could indeed incur into streaming inconsistency and one would indeed have to carefully tune the system so that streaming consistency is achieved. We exemplify the approach using the Yahoo!S4 for processing the Big Data Stream from FlightRadar24 global flight monitoring system.

I. Introduction

Big Data processing has emerged as one of the most promising solutions to next generation of analytics applications, most prominently, business analytics, monitoring systems, recommendation systems, etc. Inspite of being a new field of computing, Big Data processing comprises nowadays algorithmic solutions, especially after the Map-Reduce paradigm [6] as well as technological solutions based on Apache Hadoop. Yet, Big Data cannot address requirements of applications that need to consume data analytics in real time. This requirement can be addressed by Big Data Streams, that is, streams of data unbounded in time due a continuous data generation from a networked computing system. Indeed, by processing Big Data Streams we would be able to know in real time business analytics about business parameter performance, detect in real time fraudulent transactions, intrusion, etc. Examples of applications based on Big Data Stream processing include IoT systems, Virtual Campuses, Global Flight Monitoring, Online Transactions systems e.g. [4], [7] (refer to [18] for a description of Big Data Stream applications).