Multi-Agent Collaborative Caching Strategies in Dynamic Heterogeneous D2D Networks

Device-to-Device (D2D) communication and edge caching are two promising approaches to relieve stress on core networks, and the dynamic topology and complex device types of D2D networks bring more challenges to the design of caching strategies. However, most of the existing research works on D2D caching focus on predictable user mobility without considering its randomness. In addition, they ignore the heterogeneity of user devices. In this paper, we focus on the caching strategies in sophisticated dynamic heterogeneous D2D networks with randomly moving users and heterogeneous user devices. We formulate the average user request delay expectation minimization problem which is proved to be NP-hard. To diminish the impact of coupling user strategies and make full use of the network resources, we propose the policy-network assisted multi-agent collaboration (PAMAC) caching strategy, in which the agents are placed on the user devices and trained collaboratively using a distributed approach. Specifically, in PAMAC, we first propose the collaborative training order optimization (CTOO) algorithm to schedule the training order of the user devices in the network. Then we propose the binary particle swarm policy network (BPSP) algorithm and introduce it to each agent to design the multi-agent collaborative caching scheme. Extensive simulations are conducted to show that PAMAC outperforms other existing caching strategies.

I. Introduction

With the widespread use of Internet of Things (IoTs), the network data traffic has increased exponentially. According to Cisco's estimation, there will be 5.3 billion Internet users by 2023, up from 3.9 billion in 2018, and the number of network devices will be more than 3 times of the global population [1]. The exponential growth of traffic puts a significant burden on network transmission, leading to the emergence of edge computing, which pushes cloud services from the network core to the network edge [2], and the combination of edge computing and content caching gives birth to edge caching technology. Edge caching stores contents at the edge of networks in advance, and directly transmits contents to edge devices when requested, greatly improving the quality of service (QoS) of networks [3], [4]. In addition, Device-to-Device (D2D) communication extends the application scenario of networks and becomes an effective means to reduce network transmission delay and increase network throughput [5]. Therefore, edge caching for D2D networks has attracted wide attention.