DDPG-Based Joint Time and Energy Management in Ambient Backscatter-Assisted Hybrid Underlay CRNs

Ambient backscatter (AB) communications and radio frequency (RF)-powered cognitive radio networks (CRNs) address the concerns of energy and spectrum scarcities from different perspectives, and the integration of them has potential benefits for throughput. Motivated by this fact, we study the RF-powered AB-assisted hybrid underlay CRN (ABHU-CRN), and optimize the long-term secondary throughput. Based on the channel states, secondary users choose to perform different actions, and two action spaces are accordingly designed. Due to dynamic and unpredictable environment states, we jointly control the time scheduling and energy management of secondary users, and propose two algorithms, i.e., adjusted-deep deterministic policy gradient (A-DDPG) and combination of A-DDPG and convex optimization (C-ADCO), for the long-term secondary throughput. A-DDPG, a deep reinforcement learning algorithm with continuous spaces, is extended from DDPG to adapt to the design of two action spaces. C-ADCO utilizes the convex optimization that can find the optimal solution to assist A-DDPG to accelerate the convergence. In simulations, the ABHU-CRN under A-DDPG and C-ADCO achieves higher throughput than the optimal throughput of AB-assisted overlay CRN and AB-assisted underlay CRN, which indicates the advantage of the hybrid transmission mode in the ABHU-CRN.

I. Introduction

Along with the development of wireless communication technologies, more and more wireless devices emerge into our surroundings, making our daily lives more convenient. Meanwhile, this fact has also led to a significant growth of spectrum demands. Nowadays, spectrums are extremely scarce, and most of them have been allocated to licensed users (primary users, PUs) [1]. However, the majority of licensed spectrums keep idle at a certain time and location, so that the spectrum utilization ratio is exceedingly low [1], [2]. Therefore, it is challenging to provide available spectrums for increasing wireless devices without causing harmful effects on PUs. In addition, there are difficulties in the energy supply of wireless devices in some scenarios [3]. For instance, most wireless devices are too small to carry high-capacity batteries [4], and many wireless devices are installed in places that are not suitable to replace their batteries frequently, such as the interior of the body. Based upon the aforementioned facts, spectrum scarcity and energy insufficiency are major concerns for wireless communication networks [3], thus how to address these concerns is of practical significance.