Controlling Partially Observed Industrial System Based on Offline Reinforcement Learning—A Case Study of Paste Thickener

In the field of mineral processing, controlling the paste thickener is a highly challenging and critical task because of the high complexity, incomplete observation space, and excessive environmental noises. In this article, we propose an offline-data-driven controlling strategy to optimize the operational indices in the thickening system based on offline reinforcement learning (RL). Compared to common RL methods that rely on online interactive training, our approach ensures the safety of the production process by training the controller solely using offline datasets, thereby avoiding dangerous online exploration. In terms of offline dataset collection, this study utilizes the prior knowledge of the thickening mechanism to design a proportional–integral–derivative controller as the behavior policy to collect operational trajectories as the offline dataset. In addition, to tackle a critical issue in controlling the thickening system: constrained observation space, this article analyzes the dynamical properties of the thickening system and introduces a novel offline RL algorithm, temporal batch-constrained Q-learning (TBCQ). The algorithm and associated model framework are specifically developed for controlling partially observed Markov decision processes. The TBCQ and trained policy are evaluated in both a simulated thickening environment and a real industrial paste thickener in a copper mine. The real-world experiments demonstrate that the proposed controller outperforms the baselines and effectively reduces the tracking error of underflow concentration by over 12%. The successful application of our pipeline in paste thickener also offers an innovative perspective on addressing optimization problems in complex industrial systems: performing offline RL on a dataset sampled from a suboptimal policy.

I. Introduction

As a typical process industrial device, paste thickener, also named deep cone thickener, is widely installed in mines for concentrating the lowly concentrated slurry to produce highly concentrated underflow. The discharged underflow is further processed to prepare paste for backfilling underground mines [1]. Stable underflow concentration is an important controlling index, which is related with the paste quality and the cost of cement and other additives. The primary way for stabilizing the concentration is to set up an intelligent controller to adjust the rate of underflow and the dosage of flocculant, which are two critical controllable variables affecting the operation of the thickening process. Conventional controlling strategies, such as logic control, proportional–integral–derivative (PID) controller [2], and model prediction control (MPC), are inapplicable in the thickening process due to the inherent challenges, such as unknown system dynamics, high complexity, and excessive environmental noises [3]. Therefore, most paste backfilling plants continue to rely on semiautomatic control systems and rely heavily on experienced operators. Some import operational indices, i.e., underflow concentration and mud level, usually deviate from the given setpoint.