Self-Adaptive Traffic Control Model With Behavior Trees and Reinforcement Learning for AGV in Industry 4.0

Automated guided vehicles (AGVs) are considered as an enabling technology to realize smart manufacturing in the upcoming Industrial 4.0 era. However, several challenges including efficiency, timeliness, and safety still exist in AGVs system in discrete manufacturing shopfloor. To address these challenges, a self-adaptive traffic control model combining behavior trees (BTs) and reinforcement learning (RL) is proposed to implement optimal decisions according to diverse, dynamic and complex situations in Industry 4.0 environments. A cyber-physical systems using multiagent system technology is designed in which components such as AGVs and traffic commander are defined as specific agent that cooperates autonomously with each other. Then, the behavior construction model is constructed by BTs to enumerate all the possible states in AGVs traffic control. An RL model is further developed based on the BTs. By using this approach, in this article, AGVs have the ability to adaptively choose the optimal rule-based strategy from existing optional strategies. The case study of the scenario avoiding collisions at intersections illustrates that the proposed model can enhance self-adaptive capability of AGVs traffic control and simultaneously guarantees efficiency, timeliness, and safety.

I. Introduction

Nowadays, the continuously changing global market with a growing demand for customized and complex products brought fierce competitive pressures to manufacturers [1]. Thus, manufacturing enterprises have to improve efficiency, reliability, and flexibility of production. Material handling, as an important ingredient strongly affects production, has aroused widespread attention [2]. Automated guided vehicles (AGVs) have been widely used in material handling owing to high degree of autonomy and flexibility [3]. Recently along with the trend of Industry 4.0, many research efforts have been conducted in material handling using advanced technologies like Internet of Things (IoT) [4], [5], cyber-physical system (CPS) [6] and multi-agent systems (MAS) [7], etc. These works enhanced the abilities of real-time information exchange, decentralization, and integration of the cyber process and physical activities. Despite the significant achievements, challenges still exist in AGVs control. One of the main tasks of AGVs control is to resolve conflicts between vehicles. Many rule-based strategies are frequently used to deal with AGV conflicts such as first come first out (FCFO), minimum remaining time first (MRT), comprehensive priority first (CP) [6], [17], [24]. These strategies are designed according to different situations, respectively. FCFO aims at reducing total passing time at collision area, MRT focuses on reducing the delay rate, and CP can dynamically respond to environment changes. Nevertheless, the requirements imposed by complex shopfloor environment and diverse production characteristics such as mass customization and complex products assembly have brought new challenges. In these production modes, not only the efficiency of material delivery, but also economic, timeliness, and safety should be considered. This means the situations in shopfloor will be more complicated and a single rule-based strategy could not greatly enhance the production performance compared to mixed rule-based strategy [34]. Therefore, it is necessary to design a self-adaptive mixed rule-based strategy that the suitable rule can be selected according to various collision situations for AGVs traffic control.