ASRDataset: A Multi-granularity Shuffle System for Preparing Large-scale ASR Training Data

Automatic Speech Recognition (ASR) is an essential task in the field of artificial intelligence. With the widespread application of deep learning (DL), end-to-end ASR systems have recently seen significant improvements. The first key step to train an ASR model is preparing the data for training, which involves several processing steps, such as audio data augmentation, feature extraction, batching, shuffling and more. Among these, shuffling data serves the purpose of reducing variance and ensuring that models remain general and overfit less. However, shuffling samples globally over a large-scale dataset that could be dynamic and appended new data is challenging. In this paper, we introduce a multi-granularity shuffle strategy and implement ASRDataset system that is explicitly designed to prepare training data for ASR tasks. The shuffle strategy deploys chunk-level and batch-level shuffling to achieve efficient multi-granularity shuffling. We processed 400k-hour ASR data with our system and verified its effectiveness and efficiency on several typical ASR training tasks.