Accelerating Distributed ML Training via Selective Synchronization (Poster Abstract)

In Bulk-synchronous parallel (BSP) or simply synchronous training, deep neural networks (DNNs) are launched across multiple workers concurrently and aggregate their local updates either by Parameter server (PS) [1] or via decentralized AllReduce [2] . Thus, aggregation step on every iteration is blocking, i.e., all workers must wait for reduction phase to complete before proceeding to the next step. ML accelerators like GPUs and TPUs have reduced computation times, but communication cost continues to increase with the growing size of DNNs. Even with weak scaling and Gustafson’s law [3] , distributed training does not scale linearly with the number of workers due to high synchronization overhead.