Deep Reinforcement Learning-Based Controller for SOC Management of Multi-Electrical Energy Storage System | IEEE Journals & Magazine | IEEE Xplore

Deep Reinforcement Learning-Based Controller for SOC Management of Multi-Electrical Energy Storage System


Abstract:

The ongoing reduction of the total rotational inertia in modern power systems brings about faster frequency dynamics that must be limited to maintain a secure and economi...Show More

Abstract:

The ongoing reduction of the total rotational inertia in modern power systems brings about faster frequency dynamics that must be limited to maintain a secure and economical operation. Electrical energy storage systems (EESSs) have become increasingly attractive to provide fast frequency response services due to their response times. However, proper management of their finite energy reserves is required to ensure timely and secure operation. This paper proposes a deep reinforcement learning (DRL) based controller to manage the state of charge (SOC) of a Multi-EESS (M-EESS), providing frequency response services to the power grid. The proposed DRL agent is trained using an actor-critic method called Deep Deterministic Policy Gradients (DDPG) that allows for continuous action and smoother SOC control of the M-EESS. Deep neural networks (DNNs) are used to represent the actor and critic policies. The proposed strategy comprises granting the agent a constant reward for each time step that the SOC is within a specific band of its target value combined with a substantial penalty if the SOC reaches its minimum or maximum allowable values. The proposed controller is compared to benchmark DRL methods and other control techniques, i.e., Fuzzy Logic and a traditional PID control. Simulation results show the effectiveness of the proposed approach.
Published in: IEEE Transactions on Smart Grid ( Volume: 11, Issue: 6, November 2020)
Page(s): 5039 - 5050
Date of Publication: 21 May 2020

ISSN Information:

Citations are not available for this document.

I. Introduction

Power system frequency stability is related to keeping the balance between active power generation and demand. Traditionally, this power balance has been maintained by synchronous generators with sizeable amounts of energy stored in their rotating masses, i.e., inertia. In the event of a system frequency disturbance caused by a significant power imbalance between generation and demand, the synchronous generator’s natural inertial response slows down the rate of change of the frequency (ROCOF) and allows more time for other control actions to restore the frequency to its nominal value. The increased penetration of renewable energy technologies based on power electronic converters, due in part to environmental concerns, has effectively reduced the amount of total rotational inertia in modern power systems. Grid-connected electrical energy storage systems (EESSs) are a suitable solution to overcome the negative consequences of the reduction of the system’s rotational inertia. EESSs technologies such as a battery (BESS) or flywheel (FESS) can inject or absorb large amounts of active power from the grid in timescales of seconds (or even less) and thus provide a suitable mechanism to the transmission system operator (TSO) for controlling the system frequency [1]. To be of value to the TSO, the EESS must be capable of responding quickly by injecting or absorbing active power, but to do so, it must have adequate stored energy; consequently, the energy level becomes a critical aspect of the provision of frequency support services. Throughout this paper, the remaining energy level of the storage asset is referred to as its state of charge (SOC), even though for some technologies, e.g., flywheel, the storage medium is not electric charge. Also, operating an EESS outside its recommended state of charge (SOC) range exposes the asset to over-currents or over-voltages that may reduce its expected life [2]. The control of the SOC is a very delicate activity; keeping the SOC close to its maximum value would enforce the availability of the EESS to deliver under-frequency support. However, it would render the asset unable to absorb power and thus negate its benefits in case of an over-frequency event. Conversely, a depleted EESS cannot provide frequency services to the power grid during under-frequency events, which could lead to lower frequency excursions and the activation of under-frequency load shedding (UFLS) schemes and could trigger system-wide cascade failures [3]. The SOC of an EESS providing frequency services to the power grid must be managed adequately to ensure that the asset has enough energy stored to operate when required. There are few fast frequency response services designed to cope with the decline of the rotational inertia in modern power systems. One of these services is called Enhanced Frequency Response (EFR) [4] and was devised by National Grid, the TSO of Great Britain (GB), to deliver full active power response within one second. Furthermore, EESSs are very attractive for the provision of EFR as the service specification includes the definition of envelopes for SOC management.

Getting results...

References

References is not available for this document.