Flowlet-Level Routing Optimization with GNN-Based Multi-Agent Deep Reinforcement Learning

Traditional flowlet routing is a traffic distribution-based network routing algorithm that avoids packet disorder problems. In dynamic network environments, however, this approach may lead to suboptimal performance and network congestion. In this paper, we design a multi-agent flowlet level routing optimization (MAFRO) framework that combines multi-agent deep reinforcement learning (MADRL) and graph neural networks (GNN). MADRL uses multiple agents that interact with the network environment and learn through a trial-and-error process to optimize flowlet routing. Meanwhile, MADRL uses the properties of GNN to model the network topology, interacting and capturing the complex relationships between network nodes. MAFRO enables agents to make more informed and adaptive routing decisions based on the current state of the network, leading to better end-to-end delay and packet loss rate performance. Experimental results demonstrate that MAFRO achieves better performance than the baseline algorithm.

I. Introduction

In recent years, the need for efficient network management has posed significant challenges to the network community. A key aspect of network management is routing, which determines how packets are forwarded from their sources to their destinations. Routing can be classified into three levels of granularity: flow level, packet level, and flowlet level routing [1], [2]. Fig. 1 shows the queuing status at different granularities, at three different time instances (T₁,T₂,T₃). Fig. 1(a) depicts the queuing status of three senders transmitting elephant and mice flows to three receivers via a switch with three queues on each outgoing port. For the routing optimization based on flow granularity, it is easy to cause link congestion and path load imbalance when multiple long flows are assigned to be forwarded on the same path. As shown in Fig. 1(b), there is a mice flow in queue Q_l behind the elephant flow in Q_l, which causes the mice flow to experience a large queuing delay. The routing optimization based on packet granularity significantly improves resource utilization. However, when the links fail or have different link rates, the network topology becomes asymmetric and prone to disorder problems, causing trailing traffic delays [3]. Fig. 1(c) illustrates this problem, where the mice flow experiences disorder even though the packets are evenly distributed in each queue. The flowlet granularity-based routing policy, on the other hand, when the time interval between two packets in a flow is greater than the timeout of the flowlet, cuts the flow into flowlets consisting of multiple consecutive packets and selects a forwarding path for each flowlet according to the routing policy [4]. As shown in Fig. 1(d), flowlets are generated due to the occurrence of flowlet intervals in elephant flows. It can be seen that the routing policy based on flowlet granularity is more uniform compared to flow granularity, while no disorder occurs compared to packet granularity. In addition, flowlets are generated when the time interval between the arrival of adjacent packets is not less than the set flowlet timeout. If the flowlet timeout is set to a value greater than the maximum delay difference between equal-cost paths, packet disorder can be prevented [5]. This approach provides a balance between fine- grained packet-level routing and coarse-grained flow-level routing [6]–[8], leading to more efficient use of network resources. Fig. 1.

Switch queues at three different granularities