CloudPlanner: Minimizing Upgrade Risk of Virtual Network Devices for Large-Scale Cloud Networks

Cloud networks continuously upgrade softwarized virtual network devices (VNDs) to meet evolving tenant demands. However, such upgrades may result in unexpected failures. An intuitive idea to prevent upgrade failures is to resolve all compatibility issues before deployment, but it is impractical to replicate all deployed VND cases and test them with lots of replayed real traffic for the VND developers. As a result, the operations team takes upgrade risk to test upgrades by gradually deploying them. Although careful upgrade schedule planning is the most common method to minimize upgrade risk, to the best of our knowledge, no VND upgrade schedule planning scheme has been adequately studied for large-scale cloud networks. To fill this gap, we propose CloudPlanner, the first VND upgrade schedule planning scheme aiming to minimize the VND upgrade risk for large-scale cloud networks. CloudPlanner prioritizes upgrading VNDs that are more likely to trigger failures based on expert knowledge and historical failure-trigger VND properties and limits the number of tenants associated with simultaneously upgraded VNDs. We also propose a heuristic solver which can quickly and greedily plan schedules. Using real-world data from production environments, we demonstrate the benefits of CloudPlanner through extensive experiments.

I. Introduction

Today, major cloud providers operate tens of regions and availability zones (AZs), serving millions of tenants. Of the components comprising a cloud platform, the cloud network plays a critical role in providing each tenant with a reliable, high-performance, and isolated network environment. The cloud network is a virtual multi-tenant overlay network that sits on top of physical networks, such as data center networks and WANs. It comprises a vast number of softwarized virtual network devices (VNDs), e.g., virtual switches, cloud firewalls, etc. Typically, a VND runs as one or more processes on a physical server. To meet the ever-evolving demands of tenants for new service features, cloud providers must frequently upgrade their VNDs. For instance, Alibaba Cloud, a major cloud vendor, conducts thousands of VND upgrades each month, involving hundreds of thousands of individual VNDs.