Python Coverage Guided Fuzzing for Deep Learning Framework

In recent years, deep learning (DL) applications have been widely used in both industrial and academic domains. Bugs in the DL framework have become one of the leading causes of DL model training and deployment failures. At present, fuzzing has emerged as one of the most effective and commonly used techniques to detect bugs in the DL framework. However, existing fuzzing techniques focus on testing the high-level APIs and use a random-based generation approach to explore the vast input space of API parameters. This kind of approach suffers low effectiveness because it cannot distinguish which lines in the lower-level component have been executed. We analyzed the structure of the most popular DL frameworks (e.g., TensorFlow and PyTorch) and found that line coverage information from the Python layer can be used to characterize the execution of the underlying layers. Therefore, we propose PCFuzz, a Python coverage-guided fuzzing technique, to address the above question. The main idea of PCFuzz is to perform guided fuzzing tests based on the Python line coverage of lower-level components. To collect Python coverage at a low cost, we designed a light-weighted Python instrumentation approach specifically for DL frameworks. On the basis of this instrumentation, PCFuzz then leverages coverage to optimize the mutation scheduling, thereby enhancing the overall effectiveness of DL framework fuzzing. We conduct experiments on the most popular DL frameworks TensorFlow and PyTorch and compare PCFuzz with the state-of-the-art model-level fuzzer LEMON and API-level fuzzer Free-Fuzz. The results show that PCFuzz is more efficient in covering the codes in the low-level Python libraries and raw operations. Specifically, within the same time cost, PCFuzz achieves on average 13.9x higher Python line coverage than LEMON. For FreeFuzz, PCFuzz achieves on average 52.56% higher Python line coverage. In addition, during our preliminary experiments, PCFuzz discovered 1 previously unknown bug in TensorFlow.

I. INTRODUCTION

The deep learning (DL) library is an important tool for implementing and running DL algorithms. It provides developers with convenient and efficient interface, enabling them to develop DL applications more easily. However, with the widespread use of DL applications, the security of DL libraries is a growing concern. Attackers may take advantage of the vulnerabilities of DL libraries to tamper, steal, maliciously inject, and other attacks on the models to make them fail to function properly or generate erroneous behaviors. Therefore, the security of these DL libraries is becoming increasingly important. Various testing methods have been proposed to improve the security of these DL libraries, such as differential testing [1], [2] and fuzzing [3], [4].