Deep Reinforcement Learning Enables Joint Trajectory and Communication in Internet of Robotic Things

Internet of Robotic Things (IoRT) emphasizes the integrated robotic, artificial intelligence computing, and communication technologies, enabling more sophisticated operations and decision-making. As a crucial element of IoRT, mission-critical applications, such as industrial manufacturing and emergency services, impose stringent requirements on ultra-reliable and low-latency communication (URLLC). The paper focuses on addressing URLLC challenges in the context of IoRT, particularly when autonomous mobile robots (AMRs) coexist with static sensors. We prioritize safe and efficient AMRs’ travel through trajectory design and communication resource allocation in IoRT systems without the need of any prior knowledge. To enhance network connectivity and exploit diversity gains, we introduce the flexible decoding and free clustering as the next-generation multiple access technologies in spectrum-limited downlink IoRT system. Then, aiming at minimizing the decoding error probability and travel time, we formulate a long-term multi-objective optimization problem by jointly designing AMRs’ trajectory and communication resource. To accommodate the inherent dynamics and unpredictability in the IoRT system, we introduce a multi-agent actor-critic deep reinforcement learning (DRL) framework, offering four distinct implementations, each accompanied by comprehensive complexity analyses. Simulation results reveal the following insights: 1) in terms of DRL implementations, off-policy algorithms with deterministic policies outperform their on-policy counterparts, achieving approximately a 67% increase in rewards; 2) In terms of communication schemes, our proposed flexible decoding and free clustering strategies under designed trajectories can effectively reduce decoding errors; and 3) In terms of algorithm optimality, our DRL framework shows superior flexibility and adaptability in communication environments compared to traditional A* search and heuristic methods.

I. Introduction

Driven by the next-generation wireless technologies and Industry 5.0 standards, the emergence of the Internet of Robotic Things (IoRT) has revolutionized the conventional manufacturing, paving the way for intelligent autonomous enterprises [1]. In contrast to traditional Internet of Things (IoT) systems, which predominantly involve static or stationary devices, IoRT integrates robotic systems with IoT technologies to achieve unprecedented levels of intelligence, real-time adaptability, and enhanced functionality [2]. Mission-critical applications within IoRT, such as quality control, machine maintenance, and inventory management, are essential for ensuring ultra-reliable data transmission, low end-to-end latency, and safe control [3]. Given these mission-critical applications, IoRT is entrusted with meeting the demanding criteria of ultra-reliable and low-latency communication (URLLC) [4]. Unlike traditional assumptions of infinite block-length packet in Shannon’s capacity theory, URLLC systems utilize short block-length packets to meet rigorous error-rate performance criteria while managing computational complexities [5]. Under the regime of short block-length packets, designing an IoRT system with URLLC requirements necessitates the efficient allocation of wireless communication resources, in order to achieve high autonomy, ultra-low latency, and ultra-high reliability.