Efficient Provision of Service Function Chains in Overlay Networks Using Reinforcement Learning

Software-Defined Networking (SDN) and Network Functions Virtualization (NFV) technologies facilitate deploying Service Function Chains (SFCs) at clouds in efficiency and flexibility. However, it is still challenging to efficiently chain Virtualized Network Functions (VNFs) in overlay networks without knowledge of underlying network configurations. Although there are many deterministic approaches for VNF placement and chaining, they have high complexity and depend on state information of substrate networks. Fortunately, Reinforcement Learning (RL) brings opportunities to alleviate this challenge as it can learn to make suitable decisions without prior knowledge. Therefore, in this article, we propose an RL approach for efficient SFC provision in overlay networks, where the same VNFs provided by multiple vendors are with different performance. Specifically, we first formulate the problem into an Integer Linear Programming (ILP) model for benchmarking. Then, we present the online SFC path selection into a Markov Decision Process (MDP) and propose a corresponding policy-gradient-based solution. Finally, we evaluate our proposed approach with extensive simulations with randomly generated SFC requests and a real-world video streaming dataset, and implement an emulation system for feasibility verification. Related results demonstrate that performance of our approach is close to the ILP-based method and better than deep Q-learning, random, and load-least-greedy methods.

1 Introduction

Traditionally, network operators and administrators commonly leverage dedicated middleboxes such as Firewall (FW), Intrusion Detection System (IDS), and Performace Enhancement Proxy (PEP) for Service Function Chains (SFCs), which has a number of drawbacks such as complex management, poor efficiency, and vendor lock-in [1], [2]. With the rise of emerging cloud computing [3], Software-Defined Networking (SDN)[4], and Network Functions Virtualization (NFV) technologies [5], SFC provision is evolving to a new paradigm where Virtualized Network Functions (VNFs) as a service are outsourced to NFV-enabled clouds for Service Functions (SFs) instantiation and provided on-demand with lower cost and great flexibility [6], [7], [8]. Besides, such a paradigm can make vendors of SFs diversified to enhance the network resilience [9], [10].