Active Queue-Management Policies for Undersea Networking via Deep Reinforcement Learning

Future undersea networks require effective network resource management strategies able to support the increasing demand for undersea data. Due to the limited bandwidth and high delays characteristic of underwater acoustic communications, queueing management policies for in-transit data are expected to support the quality-of-service requirements that software applications need while helping to manage network congestion. This work develops a queue management policy for in-transit data inspired by weighted-fair queueing. Our policy dynamically allocates portions of the available transmission bandwidth to different traffic types flowing through the queues. The policy uses user-defined traffic prioritization, queueing-delay requirements, and the history of queue-occupancy levels to define the allocation of bandwidth. We use soft actor-critic, a type of deep reinforcement learning algorithm, to train a learning agent. Additionally, we include an active-queue management policy tuned to identify congestion for individual traffic types via the instantaneous queue sojourn-time experienced by packets. The active-queue management policy interacts with the learning agent and with the congestion-control protocol used by the source to mitigate congestion. The performance of our proposed algorithm is illustrated via numerical simulations using an environment developed in OpenAI Gym.