Statistical feature analysis and preprocessing assisted artificial neural network for cleaning multi-type concurrent anomalies in time series data

Real and clean data is a prerequisite for effective implementation of infrastructure condition monitoring. Classic long short-term memory (LSTM) network revealed a good performance in detecting and repairing anomalies of long time series data. However, its performance is greatly interfered when multi-type concurrent anomalies are alternating appearance in the time series data. In this paper, a data preprocessing method based on statistical principles is proposed to assist LSTM network’s cleaning work of multi-type concurrent anomalies. It is verified by experiments that this method can significantly reduce the interference that LSTM network receives from non-missing anomalies when tackling with missing data repair. In the test dataset, this method resulted in 66.81% reduction in RSME for the repaired data.