Evaluating the Potential and Realized Impact of Data Augmentations

Data augmentations have been shown to improve predictive performance of machine learning models in many domains. Augmentations are typically used to improve classification performance, but augmentations can distort the intrinsic properties of the original data, thus reducing the utility of a model for real-world applications. Because augmentations directly affect the training data, and thus also affect the machine learning models trained with said data, intelligent selection of augmentations is as critical as the selection of input features and other options in the machine learning pipeline. Such an approach will enable greater transferability of trained models from the research lab to products and services. This paper presents two metrics to evaluate the potential and realized impact of data augmentations. The first metric, eff-score, assesses the relative efficacy of prospective data augmentations before model training. To observe augmentation effects on the intrinsic properties of the training data, the second metric, nirvana distance, measures the effect of data augmentations beyond overall predictive performance after model training. These metrics are tested with a well known multi-purpose audio data set and augmentations from the domain of environmental sound scene analysis. The relative eff-scores correlate with classification results from predictive models trained on the augmented data sets, and the distance components of the nirvana distance explain results observed but not previously understood from output confusion matrices. These results demonstrate promise for data-driven, efficient selection of data augmentations whilst exposing previously hidden impacts on machine learning models. Furthermore, since eff-score and nirvana distance are domain-independent, these metrics have widespread applicability.