Clarifying Myths About the Relationship Between Shape Bias, Accuracy, and Robustness

Deep learning models can perform well when evaluated on images from the same distribution as the training set. However, applying small perturbations in the forms of noise, artifacts, occlusions, blurring, etc. to a model's input image and feeding the model with out-of-distribution (OOD) data can significantly drop the model's accuracy, making it not applicable to real-world scenarios. Data augmentation is one of the well-practiced methods to improve model robustness against OOD data; however, examining which augmentation type to choose and how it affects the OOD robustness remains understudied. There is a growing belief that augmenting datasets using data augmentations that improve a model's bias to shape-based features rather than texture-based features results in increased OOD robustness for Convolutional Neural Networks trained on the ImageNet-1K dataset. This is usually stated as “an increase in the model's shape bias results in an increase in its OOD robustness”. Based on this hypothesis, some works in the literature aim to find augmentations with higher effects on model shape bias and use those for data augmentation. By evaluating 39 types of data augmentations on a widely used OOD dataset, we demonstrate the impact of each data augmentation on the model's robustness to OOD data and further show that the mentioned hypothesis is not true; an increase in shape bias does not necessarily result in higher OOD robustness. By analyzing the results, we also find some biases in the ImageNet-1K dataset that can easily be reduced using proper data augmentation. Our evaluation results further show that there is not necessarily a trade-off between in-domain accuracy and OOD robustness, and choosing the proper augmentations can help increase both in-domain accuracy and OOD robustness simultaneously.

I. Introduction

While computer vision models have been applied to different areas of research to this date, developing models that can perform well when perturbations are present in the input image is still a challenging quest. To use computer vision in practice, we need reliable models that are robust against major and minor image perturbations. Nevertheless, even the most commonly used models fail to perform well when minor perturbations in light, color, etc. are present in the input image [1]–[4]. Fig. 1 illustrates some examples that can easily fool a trained ResNet-50 [5] model with 89.06% in-domain accuracy. To push the boundaries of computer vision, more work still needs to be done to make models reliable enough to be used even in use cases, such as in autonomous vehicles, where a faulty decision can lead to disastrous consequences.