Conservative Q-Learning Based Furnace Temperature Control in Municipal Solid Waste Incineration Process

The nonlinear, time-varying, and lagging nature of the municipal solid waste incineration (MSWI) process presents challenges for ensuring controller safety. While offline reinforcement learning (RL) can ensure safety in furnace temperature (FT) control, its performance is hindered by extrapolation errors, making it unsuitable for direct application in the incineration environment. To address this, we propose a conservative Q-learning-based furnace temperature control strategy (CQL-FTC). This strategy involves two stages: online sampling and offline training. During the online sampling stage, the agent interacts with the environment to collect samples, building an experience replay buffer (ERB) and performing pretraining. In the offline training stage, we introduce the CQL method, adding constraint terms to the traditional Bellman equation to minimize extrapolation errors. After offline training, the agent is directly applied to the FT setpoint control in the incineration process. Simulation results using the actual MSWI process dataset demonstrate the effectiveness of the proposed method in complex industrial environments.

I. Introduction

The production of municipal solid waste (MSW) in China has sharply increased [1], leading to the phenomenon of “garbage surrounding cities” and causing significant pollution to urban soil, water bodies, and air. MSW incineration (MSWI) technology has gained global attention for its ability to reduce waste, utilize resources, and mitigate environmental harm [2]. In developed countries like Europe, the United States, and Japan, its development has significantly matured. The uncertain composition, large variations in calorific value, and significant fluctuations in moisture content of MSW in China [3] have made implementing automatic combustion control (ACC) systems challenging. Currently, MSWI processes in China mainly rely on manual control by domain experts. The randomness and varied experience levels among domain experts lead to challenges in maintaining stable MSWI operations over extended periods. Therefore, it is crucial to develop tailored furnace temperature control strategies that address China's specific circumstances.