Target Tracking Control of UAV Through Deep Reinforcement Learning

This study presents an innovative reinforcement-learning-based control algorithm for a vertical take-off and landing (VTOL) aircraft under wind disturbances. In our approach, the tracking control problem of the VTOL aircraft is formulated as a Markov decision process, and the appropriate system state, reward function, and soft update method are presented. To improve the control accuracy under wind disturbances, three kinds of wind fields were added in the learning environment to expand the exploration space and simulate the effect of wind disturbances on the flight control. Moreover, to ensure the tracking accuracy and the practical implementation, a quantum-inspired experience replay strategy was developed based on quantum computation theory. In this strategy, the preparation operation scheme was designed to encourage the exploration and speed up the convergence. The depreciation operation method was developed to enrich the sample diversity, which increased the robustness of the controller and allowed the control strategy learned in the numerical simulations to be directly transferred into real-world environments. Numerical simulations, hardware-in-the-loop experiments, and real-world flight experiments were conducted to evaluate the performance and merits of the proposed method. The results demonstrated high accuracy and effectiveness and good robustness of the proposed control algorithm in terms of standoff target tracking control and flight stability.

I. Introduction

With the rapid development of intelligence and network technology in the military field, unmanned aerial vehicles (UAVs) are playing an increasingly important role in reconnaissance, surveillance, security patrols, and target monitoring [1], [2], [3]. In the commercial field, UAVs are also widely used in traffic analysis, logistics and transportation, resource exploration, and many other applications. In recent years, as the research on the autonomous capabilities of UAVs has progressed rapidly, researchers have begun to consider increasing the scope of UAV autonomy [4], [5], [6], which has created many new application areas including UAV surveillance, reconnaissance, tracking flight, convoy protection, and precision guidance of drones. For the application of drones in these situations, it is often inseparable from a very imperative technology, that is, UAV target tracking control technology. In a UAV standoff tracking control mission, a controller is designed to be able to allow the vehicle to approach a target that is moving or static and ensure the UAV maintains a certain distance from the target. UAV target tracking control technology can provide the vehicles with intelligent consciousness. This greatly enhances the autonomous capability of the UAV, enabling it to complete more kinds of tasks and adapt to more complex working conditions. This paper will focus on a tracking control system for vertical take-off and landing (VTOL) aircraft through reinforcement learning methods.