A Study of Oracle Approximations in Testing Deep Learning Libraries

Due to the increasing popularity of deep learning (DL) applications, testing DL libraries is becoming more and more important. Different from testing general software, for which output is often asserted definitely (e.g., an output is compared with an oracle for equality), testing deep learning libraries often requires to perform oracle approximations, i.e., the output is allowed to be within a restricted range of the oracle. However, oracle approximation practices have not been studied in prior empirical work that focuses on traditional testing practices. The prevalence, common practices, maintenance and evolution challenges of oracle approximations remain unknown in literature. In this work, we study oracle approximation assertions implemented to test four popular DL libraries. Our study shows that there exists a non-negligible portion of assertions that leverage oracle approximation in the test cases of DL libraries. Also, we identify the common sources of oracles on which oracle approximations are being performed through a comprehensive manual study. Moreover, we find that developers frequently modify code related to oracle approximations, i.e., using a different approximation API, modifying the oracle or the output from the code under test, and using a different approximation threshold. Last, we performed an in-depth study to understand the reasons behind the evolution of oracle approximation assertions. Our findings reveal important maintenance challenges that developers may face when maintaining oracle approximation practices as code evolves in DL libraries.

I. Introduction

Deep Learning (DL) techniques are widely applied to solve important real-world problems, such as image and voice recognition, and autonomous driving cars. Due to the complexity and wide application of deep learning techniques, practitioners build DL libraries to make DL techniques more accessible to application developers. Modern DL applications heavily depend on popular DL libraries, such as TensorFlow [1], PyTorch [2], Theano [3] and Keras [4]. The quality assurance practice of DL libraries is critical as it affects millions of applications that are built on top of them.