Improved 1vs1 Air Combat Model With Self-Play Soft Actor-Critic and Sparse Rewards

The within-visual-range (WVR) air-to-air combat problem, also known as dogfight, has been of great interest to researchers and is known to be challenging to address with reinforcement learning due to its extremely nonlinear dynamics. The previous work on dogfight has required the design of complicated reward functions to control high-dimensional state and action spaces and the design of suitable adversaries to effectively engage any opponent. In this paper, we present a self-play soft actor-critic (SPSAC) framework that is specialized for training one-on-one dogfight. It combines self-play (SP) using the proposed league configuration and dense-to-sparse (D2S) reward based on the recurrent soft actor-critic (RSAC) algorithm. We built a simulated air combat environment to evaluate our method, focusing on whether the trained models are robust against various opponents. SPSAC responds to all opponents more robustly than models trained by a straightforward self-play algorithm and greatly surpasses models trained using conventional reinforcement learning in terms of training performance. Moreover, the inherent nature of SPSAC eliminates the risk of converging to a suboptimal policy after the complex reward function is designed or the risk of overfitting by encountering fewer opponents.