Training Speech Recognition Models on HPC Infrastructure

Automatic speech recognition is used extensively in speech interfaces and spoken dialogue systems. To accelerate the development of new speech recognition models based on deep learning techniques, developers at Mozilla have open sourced a Speech-To-Text engine known as project DeepSpeech based on Baidu's DeepSpeech research. In order to make model training time quicker on CPUs for DeepSpeech distributed training, we have developed optimizations on the Mozilla DeepSpeech code to scale the model training to a large number of Intel CPU systems, including Horovod distributed training framework integration into DeepSpeech. We have also implemented a novel dataset partitioning scheme to mitigate compute imbalance across multiple nodes of an HPC cluster. We demonstrate training of the DeepSpeech model on the LibriSpeech clean dataset in 6.45Hrs on a 16-Node Intel Xeon based HPC cluster.