A Flexible Communication Mechanism for Pipeline Parallelism

Pipeline parallelism is a promising approach to utilizing increasing hardware resources, by orchestrating multiple CPU cores to run a giant program. One key component of pipeline parallelism is the communication mechanism which links different stages of a pipeline. Unfortunately, hardware support of the communication mechanism is still missing on modern commercial CPUs, forcing developers to adopt software-based queues as solution. Existing queues, however, focus mainly on the throughput between different pipeline stages. As a result, they suffer from performance degradation in real applications due to the fluctuation of incoming data and interruptions from other hardware components and applications. To solve the problem, this paper invented a novel software-based queue, which is efficient and flexible. The queue can not only shrink its queue size to keep its memory footprint tiny to utilize CPU cache better, but also enlarge its queue size when the pipeline is busy to avoid dropping data. Experiments show that the new solution is much more flexible than other queue implementations, and in a real testbed, where the data arriving rate fluctuates sharply, outperforms FastForward and MCRingBuffer (two of the best known solutions), by a factor of 2, when the batching size is 8,192.