Hay: Enhancing GPU Sharing Performance With Two-Level Scheduling for Ray

Graphics Processing Units (GPUs) are extensively adopted in many clusters, providing computational services concurrently for applications from a wide spectrum of domains, especially deep learning (DL). To simplify DL training, a unified framework like Ray has been developed to deploy models on scaled clusters. Nevertheless, existing frameworks commonly choose to allocate GPUs to DL training tasks in an exclusive fashion to maximize performance. Inevitably, the dispatched tasks are incapable of occupying the ample GPU resources fully, and even worse the regular jobs are disallowed to co-locate to guarantee exclusiveness. Towards the issue, this paper proposes Hay, a resource-aware dynamic scheduler to cooperatively dispatch DL training tasks and regular workloads in GPU clusters. The design tracks the resource of all GPUs in the cluster, and models node capacity by a busyness score computed from corresponding GPUs’ utilization. Incoming tasks are processed by a two-phase heuristic policy to select the best node and GPU. Experiment results demonstrate that Hay remarkably reduces the interference between GPU-sharing tasks and achieves an average of 1.18x (up to 1.43x) performance improvement compared to the Ray scheduler.