Big Data Analysis of Massive PMU Datasets: A Data Platform Perspective

The discovery of `event signatures' and useful insights from very large historical Phasor Measurement Unit (PMU) datasets is predicated on offline Big Data analysis approaches that rely on the generation of predictive features on a massive scale. This paper presents lessons learned from a data platform perspective towards reducing barriers to adoption of Big Data analytics against a real dataset of almost half a trillion data points drawn from over 400 PMUs distributed across the North American power grid. We demonstrate software abstractions and targeted performance optimizations that can lead to significant productivity gains for power systems researchers seeking to perform offline exploratory temporal analysis and modeling tasks, with a focus on feature generation. We describe how our optimized approach goes beyond a naive application of mainstream Big Data technologies, enabling feature generation tasks, that previously took days or even weeks, to now be completed in just a few hours.

I. Introduction

The advanced analysis of high-volume and/or high- velocity data gathered from an ever-evolving number and variety of sensors on the electrical power grid (e.g., Phasor Measurement Units and Advanced Metering Infrastructure) has many potential applications ranging from the identification of predictive signatures for faults and failure events in transmission and distribution networks to cybersecurity and grid resiliency modeling studies. Given the electrical grid's status as a critical infrastructure with cascading implications on overall national security, the power systems community is well-aware that Big Data analytics techniques - both online and offline in nature - are an increasingly important approach to detect and uncover insights from grid data at scale. While the viability of applying offline Big Data analytics techniques to grid data has been established in recent research studies [4] [5], there has not been a commensurate effort dedicated to the development of ‘enabling’ software technologies needed to translate this research into reality, particularly in cases of techniques that rely upon massive quantities of feature data [3].