A Hybrid MTL Framework with LSTM and Attention for Predicting Concurrent CDN Traffic

Content Delivery Networks (CDNs) are widely used for their ability to provide highly concurrent services with low latency, and the large amount of log data generated by CDN operations helps information service providers (ISPs) optimize CDN resource allocation, evaluate performance, and analyze operational conditions. Unfortunately, CDN concurrency studies based on real runtime data have been neglected. In addition, current neural network-based time series prediction methods often degrade model prediction performance due to inter-task interference when applied to multi-task long-term data. Consequently, this paper addresses this gap by presenting a novel hybrid network concurrency prediction framework with Multi-Task Learning (MTL), Moving Average Extended Kalman (MAEK) filter, Long Short-Term Memory (LSTM) network, and the Attention mechanism, denoted as MMLA. In this framework, the MAEK filter is employed to eliminate noise from the original data. Subsequently, LSTM with MTL is harnessed to capture both long and short-term data dependencies while considering two key features: the day of the week and the type of service. Finally, the Attention mechanism assigns weights to critical time steps of various tasks, thus mitigating inter-task interference and enhancing prediction accuracy and robustness. Empirical assessments are conducted using a sizable dataset from a real ISP in a single province. The experimental results show that the proposed MMLA outperforms the existing solutions, in terms of root mean square logarithmic error (RMSLE) and mean absolute percentage error (MAPE).