Container Migration for Distributed Deep Learning Training Scheduling in Kubernetes

Distributed deep learning training is one of the most frequently used machine learning methods because it can speed up the training process and solve memory problems caused by overly complex models. One of the platforms used to run distributed deep learning training is Kubeflow, which runs on top of Kubernetes. Kubernetes is an orchestration tool that makes it easy to run and manage a containerized workload in a computer cluster. Some schedulers have been proposed for scheduling distributed deep learning training in Kubernetes, such as DRAGON, which runs on Kubeflow and focuses on locality awareness and gang scheduling capabilities, and SpeCon, which focuses on container migration capability. In DRAGON, the Locality Aware Task Placement strategy is used to place tasks from a training job into as few nodes as possible to minimize network overhead. As a result, a node may perform more tasks than other nodes, resulting in an uneven distribution of resource usage. To solve the problem, we proposed to combine DRAGON’s and SpeCon’s approaches to make a scheduler with container migration and gang scheduling capabilities. We use container migration to send training tasks in containers from one node to a better node without repeating the development of the training process so that the distribution of resource usage can be evenly distributed. Compared to the DRAGON scheduler, our approach successfully reduced the total elapsed time by 55.692% without any drawback in accuracy.