Platform Design for IoT Data Quality Improvement through Flexible Preprocessing Pipeline Building

With the proliferation of IoT devices, real-time data collection and monitoring services are being offered to various industries. However, time series data, due to its temporal nature, presents numerous unforeseen challenges over time, often leading to data contamination. Such contaminated data make it difficult to effectively utilize in practical applications. This paper investigates the issues with time series data collected from sensors and introduces preprocessing techniques and a corresponding platform to overcome these challenges. In addition to established preprocessing methods, novel approaches for processing and transforming multiple time series data sets are applied, followed by performance testing within the framework of the proposed service platform.