Toward Fair Power Grid Control: A Hierarchical Multiobjective Reinforcement Learning Approach

Renewable energy’s inherent randomness and intermittency pose a significant challenge to current grid control and operation as the energy mix transitions to green and clean energy. While a body of reinforcement learning (RL)-based research on automated power grid control has shown great potential and exciting promise for improving grid operations’ economic efficiency and security, little work has focused on fairness issues. To this end, we propose hierarchical multiobjective Markov decision process (HMO-MDP), a power grid control and decision model considering the fairness between the power plants on the energy supply side grounded on a structurally hierarchical multiobjective markov decision process (MO-MDP) we suggested. Specifically, our model: 1) develops an innovative hierarchical multiobjective optimization function and a brand-new reward function to improve the fairness of long-term supply-side gains; 2) introduces the Generalized Gini Index (GGI), a fairness metric based on ranking from economics, to measure the fairness of long-term supply-side gains reasonably; and 3) leverages auto-encoder block to capture the stable representation of the high-dimensional state in low-dimensional space, to reduce the state space and thus enhance generalization of the model. Furthermore, with the proposed model implemented under the A2C framework, we present two algorithms, hierarchical multiobjective A2C (HMO-A2C) and hierarchical multiobjective proximal policy optimization (HMO-PPO), for fair automated power grid control. We comprehensively evaluate our algorithms on the publicly available, data-driven simulator. Experimental results show that our algorithms can effectively promote the fairness of grid control and operation on the supply side, while ensuring security and holding suitable environmental adaptability.