A Comparative Study of Reinforcement Learning-based Collision Avoidance for Maritime Autonomous Surface Ships

The efficacy of reinforcement learning has been substantiated in the development of intelligent modules for achieving autonomous collision avoidance in maritime autonomous surface ships (MASS). However, the performance of reinforcement learning algorithms with different configurations varies for making decisions. The evaluation and comparison of different configurations of reinforcement learning algorithms in this application remain challenging due to the absence of standardized or consensually adopted testing methodologies. In light of this, we proposed a simulation-based evaluation framework with three hierarchical metrics, namely, collision-free achievement, deviation angle for path-following, and avoidance time consumed, to enable the evaluation of reinforcement learning-based collision avoidance approaches. Comparative experimental analyses were conducted on six configurations of reinforcement learning algorithms with distinct reward designs across three typical vessel encounter scenarios in a simulated environment. Results indicated that by employing a potential-based design in intermediate rewards, specifically through the calculation of the course deviation, a notable enhancement in path-following can be achieved. Additionally, the weighted sum approach in the final reward design has also been demonstrated to effectively enhance the respective performance. The evaluation framework proposed, and our comparative experiments provided valuable insights and reference for evaluating collision avoidance algorithms in unmanned ship navigation and the reward designs employed in reinforcement learning within this specific application scenario.