Expect the Unexpected: Unsupervised Feature Selection for Automated Sensor Anomaly Detection

With the growth of IoT applications, sensor data quality has become increasingly important to ensure the success of these data-driven applications. Sensor data riddled with errors are redundant and affects the accuracy of decision-making results. Moreover, several constraints are inherent in anomaly detection for IoT applications such as limited manpower, time, bandwidth, computational resources and the lack of labelled datasets. Most machine learning algorithms also require a large training set to fit a satisfactory model. In this paper, we propose a fully automated anomaly detection framework, which combines systematic time series feature engineering with unsupervised feature selection. The unsupervised feature selection approach automatically selects time series features based on their predictive power with respect to the statistics of near future measurements. The proposed framework also only needs a short calibration phase with respect to the deployment phase to detect anomalies despite not having seen any in training, which is critical for real-world applications. Its feature selection reduces the amount of data required in the deployment phase on an IoT application. The selected features are suitable for building a reliable anomaly detection model while achieving a similar or better anomaly detection performance than established methods, which are operating on the raw data. Results of the evaluation on two publicly available environmental monitoring datasets show that our proposed unsupervised feature selection approach is a crucial step for having a more accurate anomaly detection while providing complex application-specific time series features, which are safeguarding the sensor system against unseen sensor anomalies.