Reinforcement Learning For Waveform Design

In this paper we propose the use of a deep neural network (DNN) trained using deep deterministic policy gradient (DDPG) reinforcement learning (RL) to solve a simplified radar waveform design problem. The problem is to select phases for a phase-coded waveform with a power spectrum (PS) containing a low-power notch to support spectrum sharing. The intention is to explore how DDPG may be applied in radar signal processing rather than develop an improved waveform notching method. The initial approach provides shallow notches averaging $\mathcal{O}(11\;{\text{dB}})$ deep. Further investigation using supervised learning, rather than RL, show that the actor neural network from DDPG can learn to produce waveforms with notches of $\mathcal{O}(120\;{\text{dB}})$ depth, and that the associated critic network can learn to accurately score the actor's performance. It is concluded that the DDPG algorithm is highly sensitive to limitations in the reward metric used to quantify the notches, and may well be a suitable method for radar signal processing when used with appropriate metrics.