Action-Dependent Heuristic Dynamic Programming With Experience Replay for Wastewater Treatment Processes

The wastewater treatment process (WWTP) is beneficial for maintaining sufficient water resources and recycling wastewater. A crucial link of WWTP is to ensure that the dissolved oxygen (DO) concentration is continuously maintained at the predetermined value, which can actually be considered as a tracking problem. In this article, an experience replay-based action-dependent heuristic dynamic programming (ER-ADHDP) method is developed to design the model-free tracking controller to accomplish the tracking goal of the DO concentration. First, the online ER-ADHDP controller is regarded as a supplementary controller to conduct the model-free tracking control alongside a stabilizing controller with a priori knowledge. The online ER-ADHDP method can adaptively adjust weight parameters of critic and action networks, thereby continuously ameliorating the tracking result over time. Second, the ER technique is integrated into the critic and action networks to promote the data utilization efficiency and accelerate the learning process. Third, a rational stability result is provided to theoretically ensure the usefulness of the ER-ADHDP tracking design. Finally, simulation experiments including different reference trajectories are conducted to show the superb tracking performance and excellent adaptability of the proposed ER-ADHDP method.

I. Introduction

The wastewater treatment process (WWTP) is an extremely significant component of pollution prevention and control, affecting the sustainable development of water resources and the health of organisms [1]. Presently, the activated sludge technique has achieves excellent effects in removing nitrogen and phosphorus for the WWTP [2]. For the activated sludge process, it is reported that the dissolved oxygen (DO) concentration plays a prominent role in microbial metabolism, which affects the elimination effect of nitrogen and phosphorus pollutants [3]. Therefore, various classical and intelligent control methods were employed to guarantee that the DO concentration remains at the well-designed point. In classical control methods, the on–off control [4], the feedforward control [5], and the proportional–integral–derivative (PID) control [6] have been widely recognized in practical applications. However, due to the complex operating conditions of WWTP, these traditional technologies lack the optimization and adaptive abilities, which has sparked exploration and research on more sophisticated intelligent control methods.