Applying a Deep Q-Network for Human Operator Behavioral Modeling and Decision Support in a Twin-Roll Casting Process

In many industrial processes, human operators continue to manually adjust operation setpoints. However, variations among operators can lead to inconsistencies in process operation and product quality. During highly transient periods of operation such as process start-up, system modeling is difficult and not all control objectives can be quantified. Thus, automating such a process using model-based methods is not trivial. In this paper, a modified deep Q-network (DQN) algorithm, aimed at determining the optimal policy for setpoint adjustments during highly transient process operations, is proposed with application to a commercial steel manufacturing process. The major contribution is the design of a reward function that blends explicit and implicit characterizations of the performance objectives, in part through clustering of multiple distinct human operator behaviors from the training data. Over 200 sequences of industrial process data from a twin-roll steel strip manufacturing plant are clustered based on the operators’ setpoint adjustment behavior with respect to the casting roll separation force. We show that the trained reinforcement learning agent can independently adjust the setpoint value of interest within the specified transient operation period as desired, and highlight the sensitivity of the trained agent to the design of the implicit and explicit performance components of the reward function.