Efficient Reinforcement Learning for Autonomous Ship Collision Avoidance under Learning Experience Reuse

In this paper, a learning experience reuse - reinforcement learning collision avoidance (LER-RLCA) method is proposed, which can synthesize near-optimal collision avoidance policy with efficient sampling and good seamanship, to solve the local safety sailing of autonomous ship in a multi-obstacle environment. Lying on the general reinforcement learning (RL), using learning experience reuse, the hidden features of historical training data were mined. Meanwhile, a new reward function combining external revenue signal with internal incentive signal was designed to encourage search the environment with a low probability of state transition. We further applied LER-RLCA algorithm to the simulation of autonomous ship collision avoidance. The results show that the proposed LER-RLCA algorithm can well realize the collision-free and safe navigation of autonomous ships, to avoid falling into local iteration, greatly improve the convergence speed of the algorithm, and improve the performance of online collision avoidance decision-making.

I. Introduction

Ship autonomous navigation technology is a crucial technology for maritime safety guarantee, which integrates intelligent perception, anti-collision, decision-making, control, and communication. In recent years, with the development of artificial intelligence technology, intelligent learning methods have been gradually applied to the fields of robots, drones, and unmanned vehicles, in the fields of intelligent optimization scheduling, decision planning, trajectory following, and forecasting [1]–[4]. RL is an artificial intelligence-based optimization learning method. Compared with traditional optimization or planning algorithms, this method does not rely on prior knowledge and supervision information, through “trial and error” interacting with the environment, balancing exploration, and utilization, learning optimization and planning are finally realized. According to this advantage, it has received more and more attention and research about autonomous ship decision-making, planning, and control [5]–[8].