Experimental Study using Unsupervised Anomaly Detection on Server Resources Monitoring

Monitoring resources in a server environment is an essential and indispensable process that ensures a decent resource allocation to show better performance. Existing resource monitoring solutions or tools can be used but require additional staff to monitor resource usage. It is a high workload task when there are thousands of servers and each server with thousands of application instances. When the consumption surpasses an unusual limit, it is considered an anomaly record. Unsupervised machine learning algorithms can detect anomalies through clustering. This study developed an experimental analysis using three clustering techniques to assess the performance of clustering for discovering the anomalies pattern in a multivariate time series dataset with 25,918 data points. The procedure to get the optimal numbers of clusters is performed before the experiment, which can be considered between 3 and 4 numbers of clusters. Cluster pattern analysis has been done for selecting the best optimal number of clusters among both selections. The experiment has included two dimensionality reduction techniques for dataset compression to investigate their influences towards clustering performance. Finally, three performance metrics are used for evaluating the cluster quality. This study concluded that the combination of K-Means with Principal Component Analysis produced the best cluster quality among 9 combinations of clustering techniques. Finally, the anomalies behaviors are identified in the distinguishable cluster from all the monitoring items. Nonetheless, it is worth noting that the output of the anomaly detection is different when clustering is performed using different numbers of clusters.