Deep Reinforcement Learning-Based Controller for SOC Management of Multi-Electrical Energy Storage System | IEEE Journals & Magazine | IEEE Xplore

Deep Reinforcement Learning-Based Controller for SOC Management of Multi-Electrical Energy Storage System


Abstract:

The ongoing reduction of the total rotational inertia in modern power systems brings about faster frequency dynamics that must be limited to maintain a secure and economi...Show More

Abstract:

The ongoing reduction of the total rotational inertia in modern power systems brings about faster frequency dynamics that must be limited to maintain a secure and economical operation. Electrical energy storage systems (EESSs) have become increasingly attractive to provide fast frequency response services due to their response times. However, proper management of their finite energy reserves is required to ensure timely and secure operation. This paper proposes a deep reinforcement learning (DRL) based controller to manage the state of charge (SOC) of a Multi-EESS (M-EESS), providing frequency response services to the power grid. The proposed DRL agent is trained using an actor-critic method called Deep Deterministic Policy Gradients (DDPG) that allows for continuous action and smoother SOC control of the M-EESS. Deep neural networks (DNNs) are used to represent the actor and critic policies. The proposed strategy comprises granting the agent a constant reward for each time step that the SOC is within a specific band of its target value combined with a substantial penalty if the SOC reaches its minimum or maximum allowable values. The proposed controller is compared to benchmark DRL methods and other control techniques, i.e., Fuzzy Logic and a traditional PID control. Simulation results show the effectiveness of the proposed approach.
Published in: IEEE Transactions on Smart Grid ( Volume: 11, Issue: 6, November 2020)
Page(s): 5039 - 5050
Date of Publication: 21 May 2020

ISSN Information:

References is not available for this document.

I. Introduction

Power system frequency stability is related to keeping the balance between active power generation and demand. Traditionally, this power balance has been maintained by synchronous generators with sizeable amounts of energy stored in their rotating masses, i.e., inertia. In the event of a system frequency disturbance caused by a significant power imbalance between generation and demand, the synchronous generator’s natural inertial response slows down the rate of change of the frequency (ROCOF) and allows more time for other control actions to restore the frequency to its nominal value. The increased penetration of renewable energy technologies based on power electronic converters, due in part to environmental concerns, has effectively reduced the amount of total rotational inertia in modern power systems. Grid-connected electrical energy storage systems (EESSs) are a suitable solution to overcome the negative consequences of the reduction of the system’s rotational inertia. EESSs technologies such as a battery (BESS) or flywheel (FESS) can inject or absorb large amounts of active power from the grid in timescales of seconds (or even less) and thus provide a suitable mechanism to the transmission system operator (TSO) for controlling the system frequency [1]. To be of value to the TSO, the EESS must be capable of responding quickly by injecting or absorbing active power, but to do so, it must have adequate stored energy; consequently, the energy level becomes a critical aspect of the provision of frequency support services. Throughout this paper, the remaining energy level of the storage asset is referred to as its state of charge (SOC), even though for some technologies, e.g., flywheel, the storage medium is not electric charge. Also, operating an EESS outside its recommended state of charge (SOC) range exposes the asset to over-currents or over-voltages that may reduce its expected life [2]. The control of the SOC is a very delicate activity; keeping the SOC close to its maximum value would enforce the availability of the EESS to deliver under-frequency support. However, it would render the asset unable to absorb power and thus negate its benefits in case of an over-frequency event. Conversely, a depleted EESS cannot provide frequency services to the power grid during under-frequency events, which could lead to lower frequency excursions and the activation of under-frequency load shedding (UFLS) schemes and could trigger system-wide cascade failures [3]. The SOC of an EESS providing frequency services to the power grid must be managed adequately to ensure that the asset has enough energy stored to operate when required. There are few fast frequency response services designed to cope with the decline of the rotational inertia in modern power systems. One of these services is called Enhanced Frequency Response (EFR) [4] and was devised by National Grid, the TSO of Great Britain (GB), to deliver full active power response within one second. Furthermore, EESSs are very attractive for the provision of EFR as the service specification includes the definition of envelopes for SOC management.

Select All
1.
H. Silva-Saravia, H. Pulgar-Painemal and J. M. Mauricio, "Flywheel energy storage model control and location for improving stability: The Chilean case", IEEE Trans. Power Syst., vol. 32, no. 4, pp. 3111-3119, Jul. 2017.
2.
S.-J. Kwon, G. Kim, J. Park, J.-H. Lim, J. Choi and J. Kim, "Research of adaptive extended Kalman filter-based SOC estimator for frequency regulation ESS", Appl. Sci., vol. 9, no. 20, pp. 4274, 2019.
3.
H. Bevrani, Robust Power System Frequency Control, Cham, Switzerland:Springer Int., 2014.
4.
Enhanced Frequency Response, Nov. 2016, [online] Available: https://www.nationalgrid.com/uk/electricity/balancing-services/frequency-response-services/enhanced-frequency-response-efr?overview.
5.
B. M. Gundogdu, S. Nejad, D. T. Gladwin, M. P. Foster and D. A. Stone, "A battery energy management strategy for U.K. enhanced frequency response and triad avoidance", IEEE Trans. Ind. Electron., vol. 65, no. 12, pp. 9509-9517, Dec. 2018.
6.
M. Bahloul and S. K. Khadem, "Design and control of energy storage system for enhanced frequency response grid service", Proc. IEEE Int. Conf. Ind. Technol. (ICIT), pp. 1189-1194, 2018.
7.
J. Pahasa and I. Ngamroo, "Simultaneous control of frequency fluctuation and battery SOC in a smart grid using LFC and EV controllers based on optimal MIMO-MPC", J. Elect. Eng. Technol., vol. 12, no. 2, pp. 601-611, Mar. 2017.
8.
F. Sanchez, J. Cayenne, F. Gonzalez-Longatt and J. L. Rueda, "Controller to enable the enhanced frequency response services from a multi-electrical energy storage system", IET Gener. Transm. Distrib., vol. 13, no. 2, pp. 258-265, 2019.
9.
D. Ernst, M. Glavic, F. Capitanescu and L. Wehenkel, "Reinforcement learning versus model predictive control: A comparison on a power system problem", IEEE Trans. Syst. Man Cybern. B Cybern., vol. 39, no. 2, pp. 517-529, Apr. 2009.
10.
G. Cybenko, "Approximation by superpositions of a sigmoidal function", Math. Control Signals Syst., vol. 2, pp. 303-314, Dec. 1989.
11.
L. Qing et al., "Decentralized reinforcement learning collaborative consensus algorithm for generation dispatch in virtual generation tribe", Proc. IEEE Adv. Inf. Technol. Electron. Autom. Control Conf. (IAEAC), pp. 1197-1201, 2016.
12.
L. Yin, T. Yu, L. Zhou, L. Huang, X. Zhang and B. Zheng, "Artificial emotional reinforcement learning for automatic generation control of large-scale interconnected power grids", IET Gener. Transm. Distrib., vol. 11, no. 9, pp. 2305-2313, Jun. 2017.
13.
J. Duan et al., "A deep reinforcement learning based approach for optimal active power dispatch", Proc. IEEE Sustain. Power Energy Conf. (iSPEC), pp. 263-267, 2019.
14.
E. A. Jasmin, T. P. I. Ahamed and T. Remani, "A function approximation approach to Reinforcement Learning for solving unit commitment problem with photo voltaic sources", Proc. IEEE Int. Conf. Power Electron. Drives Energy Syst. (PEDES), pp. 1-6, 2016.
15.
Z. Yan and Y. Xu, "Data-driven load frequency control for stochastic power systems: A deep reinforcement learning method with continuous action search", IEEE Trans. Power Syst., vol. 34, no. 2, pp. 1653-1656, Mar. 2019.
16.
J. Duan et al., "Deep-reinforcement-learning-based autonomous voltage control for power grid operations", IEEE Trans. Power Syst., vol. 35, no. 1, pp. 814-817, Jan. 2020.
17.
Z. Wan, H. Li, H. He and D. Prokhorov, "Model-free real-time EV charging scheduling based on deep reinforcement learning", IEEE Trans. Smart Grid, vol. 10, no. 5, pp. 5246-5257, Sep. 2019.
18.
V.-H. Bui, A. Hussain and H.-M. Kim, " Double deep Q -learning-based distributed operation of battery energy storage system considering uncertainties ", IEEE Trans. Smart Grid, vol. 11, no. 1, pp. 457-469, Jan. 2020.
19.
N. Ebell, F. Heinrich, J. Schlund and M. Pruckner, "Reinforcement learning control algorithm for a PV-battery-system providing frequency containment reserve power", Proc. IEEE Int. Conf. Commun. Control. Comput. Technol. Smart Grids (SmartGridComm), pp. 1-6, 2018.
20.
R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction, Cambridge, MA, USA:MIT Press, 2018.
21.
R. Bellman, "A Markovian decision process", J. Math. Mech., vol. 6, no. 5, pp. 679-684, 1957.
22.
A. Géron, Hands-On Machine Learning With Scikit-Learn Keras and TensorFlow: Concepts Tools and Techniques to Build Intelligent Systems, Sebastopol, CA, USA, 2019.
23.
D. L. Poole and A. K. Mackworth, "Learning to act" in Artificial Intelligence: Foundations of Computational Agents, Cambridge, U.K.:Cambridge Univ. Press, pp. 549-578, 2017.
24.
V. Mnih et al., "Playing Atari with deep reinforcement learning", Proc. NIPS Deep Learn. Workshop, pp. 1-9, 2013.
25.
H. van Hasselt, A. Guez and D. Silver, "Deep reinforcement learning with double Q-learning", Proc. 30th AAAI Conf. Artif. Intell. (AAAI), pp. 2094-2100, 2016.
26.
S. Fujimoto, H. van Hoof and D. Meger, "Addressing function approximation error in actor-critic methods", Proc. 35th Int. Conf. Mach. Learn. (ICML), vol. 4, pp. 2587-2601, 2018.
27.
T. P. Lillicrap et al., "Continuous control with deep reinforcement learning", Proc. 4th Int. Conf. Learn. Represent. (ICLR) Conf. Track, 2016, [online] Available: https://arxiv.org/abs/1509.02971.
28.
H. van Hasselt and M. A. Wiering, "Reinforcement learning in continuous action spaces", Proc. IEEE Int. Symp. Approximate Dyn. Program. Reinforcement Learn., pp. 272-279, 2007.
29.
G. E. Uhlenbeck and L. S. Ornstein, "On the theory of the Brownian motion", Phys. Rev., vol. 36, no. 5, pp. 823-841, Sep. 1930.
30.
D. M. Greenwood, K. Y. Lim, C. Patsios, P. F. Lyons, Y. S. Lim and P. C. Taylor, "Frequency response services designed for energy storage", Appl. Energy, vol. 203, pp. 115-127, Oct. 2017.

References

References is not available for this document.