Big Data Compression and Storage for Continuous Spatio-Temporal Monitoring of Power Transmission Cables with Distributed Fiber-Optic Vibration Sensor (DFOVS)

In this paper, an effective real-time compression method is proposed to compress the tremendous amount of data generated in continuous spatio-temporal monitoring of power transmission cables with the distributed fiber-optic vibration sensor(DFOVS). The proposed method mainly consists of anomaly detection by statistical methods and Lifting Wavelet Transform(LWT) with an adaptive OTSU threshold. The experiment results for real field data indicate that the average compression ratio of this method can reach 138.9. Therefore the proposed method can effectively solve the challenging problems of real-time storage and transmission for big data acquired from a high density spatio-temporal acquisition grid. Besides, LWT has a higher calculation efficiency than the conventional Discrete Wavelet Transform (DWT) in real applications. And the data storage time can be obviously saved by using the proposed method on the big data processing platform of HDFS and HBase. The overall compression time is generally less than the massive data generation time in long distance or large range monitoring with DFOVS, thus it can be used for real-time compression of similar applications with moderate sampling rate.