Interpretable Deep Reinforcement Learning With Imitative Expert Experience for Smart Charging of Electric Vehicles

Deep reinforcement learning (DRL) is a promising candidate for realizing online complex system optimal control because of its high computation efficiency. However, the interpretability and reliability problems limit its engineering application in smart grid energy management. This paper for the first time designs a novel imitative learning framework to provide a reliable solution for computation-efficient grid-connected electric vehicles (GEVs) charging management in smart grids. The optimal strategies are derived by a priors optimization model based on vehicle-to-grid (V2G) cost-benefit analysis. With better interpretability and ensured optimality, the derived strategies are used to construct an experience pool for configuring the learning environment. Then, a novel imitative learning mechanism is designed to facilitate the knowledge transfer between expert experience and reinforcement learning model. Further, a novel dual actor-imitator learning network to enable flexible scheduling of V2G power of GEVs. With the dual network structure, the expert experience can be effectively utilized to enhance the training efficiency and performance of the DRL-based V2G coordinator. The effectiveness of the developed method in improving V2G benefit and mitigating battery aging is validated on a demonstrative microgrid in the U.K.

I. Introduction

With the rapid growth of economy and industry, the environmental and ecological problems caused by fossil fuels have become the bottlenecks restricting social modernization and urbanization. In recent years, many countries have set ambitious carbon neutrality targets in realizing energy system transition. According to the Future Energy Scenarios carried out by the U.K. government, grid decarbonization and transportation electrification will be important pathways in pursuing net zero [1]. However, grid integration of renewable energy and electric vehicles (EVs) brings great difficulties to power system operation. The intermittence of renewable generation and the additional charging load of EVs directly impact grid operation stability [2], [3]. In recent years, it comes to the concept of vehicle-to-grid (V2G) that schedules the charging and discharging time, power, and even position of grid-connected EVs (GEVs) to improve the stability and economy of the power grid [4]. With V2G technology, GEVs can be used as distributed energy storage to consume excess renewable power generation by providing ancillary services.