OptRISQL: Toward Performance Improvement of Time-Varying IoT Networks Using Q-Learning

In order to support the recent explosive growth in the applications of Internet of Things (IoT), networking technologies are evolving, resulting in high data throughput, low-latency data transfer, and improved lifetime of Internet of Things Devices (IoDs). These technologies work fairly for static conditions as the devices have fixed locations. However, in several practical IoT networks, including intelligent transportation networks and mobile health monitoring systems, the devices change their locations with time, resulting in time-varying network topologies. Dynamic networks generally operate on multi-hop data transmission schemes. However, due to its dynamic nature, these networks are susceptible to poor performance as a consequence of the inaccurate selection of relay IoD. In this context, the selection of optimal relay IoD towards data transfer is an important problem in time-varying IoT networks. To address such a critical issue, in this work, we consider a dynamic IoT network in which the devices select an optimal relay IoD at various discrete time instants to improve network performance. Thereafter, a novel reinforcement learning-based data routing algorithm in the time-varying multi-hop IoT network is proposed for optimum data routing. The proposed algorithm, Opt imal ${R}$ elay ${I}$ oD ${S}$ election Using Q-L earning (OptRISQL), selects the optimum relay IoD for data routing using Q-learning. The proposed method maximizes the aggregate reward value between specified device-gateway pairs by adjusting the network’s Q-matrix at discrete time instants to identify optimal relay IoD. The proposed method’s applicability and effectiveness are demonstrated using a simulated IoT testbed and real-field datasets. Moreover, when compared to various existing methods, the acquired findings indicate the proposed method’s improved network performance in terms of Energy-Efficiency (EE) and Quality-of-Service (QoS).

I. Introduction

In recent years, there has been an explosive growth and development in the Internet of Things (IoT) related technologies due to increase in the use-cases, such as, smart health, Augmented and Virtual Reality (AR-VR), smart home, Unmanned Aerial Vehicles (UAVs)-aided sensor networks, underwater transmissions, and industrial automation [1]. IoT has the capability to facilitate the connection establishment and data exchange among huge number of devices. These IoT networks can either be static or time-varying in nature depending upon the application scenario. Existing research on IoT mainly focuses on static devices like Radio-Frequency Identification (RFID) sensors in buildings, traffic cameras on the road, security systems, and smart agriculture equipment. In addition to static devices, mobile devices have emerged as an essential part of IoT networks as mobile phones and automobiles are embedded with smart sensors [2]. The applications of IoT have now grown up to help many real-time systems such as intelligent transportation, smart mobile healthcare, and cognitive wearables.