Improvement of Data Stream Processing using Adaptive Ingestion

Adaptive batch sizing algorithm offers several advantages over static sizing in various real-world scenarios and use cases especially Streaming Data Processing. The data ingestion layer in distributed stream processing system plays a crucial role in the overall stream processing system. It is responsible for accessing data from diverse locations and delivering it to the processing layer. In the realm of data ingestion, achieving low latency, high throughput, and scalability across multiple producers and consumers is an essential mandate. To elevate performance and availability during the data ingestion process dynamic batch interval control algorithm has been proposed. This algorithm adjusts the length of batch intervals dynamically based on data ingestion rates, thereby optimizing the data stream ingestion layer. It is implemented and operated within the Apache Flume to facilitating the transmission of data to Spark Streaming, Experimental results demonstrate that the proposed batch interval control scheme yields significantly higher throughput and adjustable processing time during the data stream ingestion phase compared to a static batch interval approach.