Are Deep Models Robust against Real Distortions? A Case Study on Document Image Classification

As deep learning models in the context of document image classification are reaching diminishing returns with nearperfect recognition scores, their robustness characteristics are poorly understood. In order to evaluate the robustness of existing state-of-the-art document image classifiers against different types of distortions that are commonly encountered in the real world, we present two separate benchmark datasets, namely RVL-CDIPD and Tobacco3482-D. The proposed benchmarks are generated by augmenting the well-known pre-existing document image classification datasets (RVL-CDIP and Tobacco3482) with 21 different types of distortions including varying severity levels. We leverage the proposed benchmark datasets to analyze the robustness characteristics of existing document image classification systems. Our analysis reveals that despite higher accuracy models exhibiting relatively higher robustness, they still severely underperform on some specific distortions, with classification accuracies dropping from ~90% to as low as ~40% in some cases. Interestingly, some of these high accuracy models perform even worse than the baseline AlexNet model in the presence of distortions, with the relative decline in their accuracy sometimes reaching as high as 300-450%. We envision these benchmarks to serve as a strong signal of progress in document image classification tasks, beyond the saturated accuracy metrics. The datasets and code to reproduce them is publicly available: https://github.com/saifullah3396/docrobustness.

I. Introduction

Deep learning has been applied to a number of problems in recent years and has exhibited exceptional performance in particularly challenging domains including computer vision [1], [2], [3], [4], natural language processing [5], and speech recognition [6]. However, despite these impressive gains in accuracy metrics, these models exhibit poor robustness to even minor shifts in the data distribution [7], [8], [9]. These minor distribution shifts, which are likely to occur in the real-world, pose a significant challenge to the deployment of these systems in the real world.