Dynamic GPU Scheduling With Multi-Resource Awareness and Live Migration Support

In clouds and data centers, GPU servers with multiple GPUs are widely deployed. Current state-of-the-art GPU scheduling policies are “static” in assigning applications to different GPUs. These policies usually ignore the dynamics of the GPU utilization and are often inaccurate in estimating resource demand before assigning/running applications, so there is a large opportunity to further balance the loads and improve GPU utilization. Based on CUDA (Compute Unified Device Architecture), we develop a runtime system called
${\sf DCUDA}$
which supports
“dynamic”
scheduling of running applications between multiple GPUs. In particular,
${\sf DCUDA}$
takes into consideration multidimensional resources, including computing cores, memory usage, and energy consumption. It first provides a real-time and lightweight method to accurately monitor the resource demand of applications and GPU utilization. Furthermore, it provides a universal migration facility to migrate
“running applications”
between GPUs with negligible overhead. More importantly,
${\sf DCUDA}$
transparently supports all CUDA applications without changing their source code. Experiments with our prototype system show that
${\sf DCUDA}$
can reduce 78.3% of overloaded time of GPUs on average. As a result, for different workloads consisting of a wide range of applications we studied,
${\sf DCUDA}$
can reduce the average execution time of general applications by up to 42.1%, and even up to 67% for memory-intensive tasks. Besides,
${\sf DCUDA}$
also reduces 13.3% of energy in light-load scenarios.