Latency-Oriented Elastic Memory Management at Task-Granularity for Stateful Streaming Processing

In a streaming application, an operator is usually instantiated into multiple tasks for parallel processing. Tasks across operators have various memory demands due to different processing logic (e.g., stateful vs. stateless tasks). The memory demands of tasks from the same operator could also vary and fluctuate due to workload variability. Improper memory provision will cause some tasks to have relatively high latency, or even unbound latency that can eventually lead to system instability. We found that the task with the maximum latency of an operator has a significant and even decisive impact on the end-to-end latency.In this paper, we present our task-level memory manager. Based on our quantitative modeling of memory and task-level latency, the manager can adaptively allocate optimal memory size to each task for minimizing the end-to-end latency. We integrate our memory management on Apache Flink. The experiments show that our memory management could significantly reduce end-to-end latency for various applications at different scales and configurations, compared to the Flink native setting.

I. Introduction

Streaming Processing Engines (SPEs) [13], [26], [34], [37] have emerged to provide quick analysis for applications that need real-time responses and whose input data is in the form of streams, such as quantitative finance, network monitoring, and alert triggering. These applications typically require consistently short end-to-end (E2E) latency, the time from the data generated to the result is output. Short E2E latency can bring about a smooth user experience, service-level agreements guarantee, and huge profits in the financial market. For example, autonomous high-frequency algorithmic trading [36] must make timely adjustments to market fluctuations, or traders may lose trading opportunities or even incur losses.