Robust and Rapid Adaption for Concept Drift in Software System Anomaly Detection

Anomaly detection is critical for web-based software systems. Anecdotal evidence suggests that in these systems, the accuracy of a static anomaly detection method that was previously ensured is bound to degrade over time. It is due to the significant change of data distribution, namely concept drift, which is caused by software change or personal preferences evolving. Even though dozens of anomaly detectors have been proposed over the years in the context of software system, they have not tackled the problem of concept drift. In this paper, we present a framework, StepWise, which can detect concept drift without tuning detection threshold or per-KPI (Key Performance Indicator) model parameters in a large scale KPI streams, take external factors into account to distinguish the concept drift which under operators' expectations, and help any kind of anomaly detection algorithm to handle it rapidly. For the prototype deployed in Sogou, our empirical evaluation shows StepWise improve the average F-score by 206% for many widely-used anomaly detectors over a baseline without any concept drift detection.

I. Introduction

With the booming development of web-based software systems like search engine, online shopping and social networks, careful analysis of monitoring data is becoming increasingly important for software reliability. In general, operators of the above systems monitor millions of Key Performance Indicator (KPI, e.g. number of Page Views (PV), number of online users, average response time) streams [1]. Diverse types of detectors can be used to detect KPI anomalies, e.g., Moving Average (MA) [2] and Time Series Decomposition (TSD) [3]. In order to accurately detect KPI anomalies, operators carefully tune the parameters of detectors based on their domain knowledge [1], [4]–[6].