Deep Reinforcement Learning Optimal Transmission Algorithm for Cognitive Internet of Things With RF Energy Harvesting

Spectrum scarcity and energy limitation are becoming two critical issues in designing Internet of Things (IoT). As two promising technologies, cognitive radio (CR) and radio frequency (RF) energy harvesting can be used together to improve both energy and spectral efficiency. In this paper, an optimal transmission problem in a cognitive IoT (CIoT) with RF energy harvesting capability is investigated, where the optimization problem is formulated as a Markov decision process (MDP) without any priori-knowledge. Considering that the channel activity states of primary user network (PUN), RF energy arrival process and channel information are not available in advance, a deep reinforcement learning (DRL) based deep deterministic policy gradient (DDPG) algorithm is proposed to deal with the dynamic uplink access, working mode selection and continuous power allocation to maximize a long term uplink throughput. The simulation results show that the proposed algorithm is valid and efficient to achieve better performances when compared with deep ${Q}$ -network (DQN) based, myopic and random algorithms.

I. Introduction

The Internet of Things (IoT) technique has emerged as a promising communication networking paradigm which can realize the ubiquitous connection between things and things, things and people [1], [2]. With the explosive growing numbers of wireless devices connected to the Internet, to solve spectrum scarcity and energy scarcity problems are the two important challenges before us. Cognitive radio (CR) can be used to mitigate the spectrum scarcity problem in IoT while radio frequency (RF) energy harvesting can supply energy to wireless networks and increase energy efficiency [3]–[6]. In addition, to improve the efficient utilization of spectrum and energy resource in cognitive IoT systems, it is also a key problem to find an optimal transmission strategy. •

Cognitive radio has been considered as a best candidate technology to address spectrum demands of IoT systems through spectrum sharing with primary users (PUs). There are three types of spectrum sharing in cognitive radio: overlay, underlay and interweave [7]. The underlay mode and overlay mode belong to the aggressive spectrum sharing scheme, where secondary users (SUs) have right to coexist with PUs under the condition that the interference received by PUs is lower than a specified threshold. For the interweave mode, which is a protective spectrum sharing scheme, SUs are only allowed to access the idle channels that are temporarily not used by PUs. This interweave paradigms is considered in this article.

•

RF energy harvesting has ability to convert received RF signals from radio environment into electricity for operating devices and transmitting data [8]. Compared with traditional energy supply from a battery which needs to be physically charged or replaced, RF energy can provide sustainable and green power to wireless devices [9]. Hence, supplying power for cognitive radio networks (CRNs) with RF energy is an effective solution to improve spectrum and energy efficiency in IoT systems.