Dynamic Data Partitioning Strategy Based on Heterogeneous Flink Cluster

As a popular big data stream processing engine, Apache Flink has obvious advantages in stream data processing. The data partitioning strategy of Flink stream processing jobs ignores metrics such as heterogeneous cluster, real-time computational resources of nodes and communication delays between nodes, which can easily cause load imbalance problems. In this paper, we propose a dynamic data partitioning strategy for stream processing jobs based on heterogeneous Flink cluster for adjusting the data load among nodes. The strategy dynamically adjusts the ratio of data processed by each node according to the real-time resource information, communication delay and hardware information of each node, so that the data load among nodes is at a relatively balanced level after multiple rounds of feedback adjustment. The experimental results show that the strategy can effectively improve the throughput and reduce latency of the job