Optimizing Neural Network Testing with Test Theory

Since 2017, researches on neural network (NN) testing adequacy criteria have rapidly emerged, which characterize different aspects of NNs to define the quality of the tests. Most existing studies focus on structural coverage metrics, but these criteria have a low correlation with neural network quality, robustness, and other attributes; the test data generation based on these criteria is often incremental without considering the distribution of test data. Inspired by the item analysis (IA) process in test theory (TT), this paper designs four test item adequacy criteria for neural networks, including item difficulty coverage, difficulty block coverage, discrimination coverage, and model capability coverage, which firstly links the test theory with neural network test adequacy, which provides a new perspective for the design of test adequacy criteria and test set optimization. In this paper, we prepared more than 1500 trained neural network models for systematically analyzing the characteristics of neural network tests, which reveals the shortcomings of the existing test sets in item difficulty and differentiation. An optimization method for neural network testing is established based on project difficulty distribution. Experiments are conducted in this essay to verify the effectiveness of our scheme and the availability of the test item adequacy criteria.

I. Introduction

Deep neural network (DNN) testing is essential before the deployment. It involves generating test cases in a certain way, and checking and detecting defects of neural networks by observing the relationship between output and test oracle. It can make judgments on some attributes or abilities of neural networks, and verify their functionality and performance. Its main purpose is to discover potential threats or defects of neural networks [1] and objectively evaluate some attributes of neural networks, so as to better ensure the quality of neural network technology applications.