The Optimization of Model Parallelization Strategies for Multi-GPU Training

Data parallelism (DP) is most widely used among all the parallel methods in largescale network training but the speedup from leveraging DP begins to scale poorly as the number of devices in data parallel training grows. Therefore, model parallelism (MP) is needed for further accelerating. In this paper, an integer linear programming based tool, NetPlacer, is proposed to find optimal operation-to-device placement. Unlike existing methods using time estimation, NetPlacer discards estimating simulated execution time and uses strict memory and computing balancing constraints on the network to find a scheme with a balance between memory and computation on different devices. As far as we know, we are the first to use equipment equalization as a target for model parallel strategy optimization. A speedup of at least 20% is achieved through NetPlacer for Inception-V3 on a 2-GPU machine compared to what one single GPU alone can achieve.