Test-Driving Static Analysis Tools in Search of C Code Vulnerabilities

Recently, a number of tools for automated code scanning came in the limelight. Due to the significant costs associated with incorporating such a tool in the software lifecycle, it is important to know what defects are detected and how accurate and efficient the analysis is. We focus specifically on popular static analysis tools for C code defects. Existing benchmarks include the actual defects in open source programs, but they lack systematic coverage of possible code defects and the coding complexities in which they arise. We introduce a test suite implementing the discussed requirements for frequent defects selected from public catalogues. Four open source and two commercial tools are compared in terms of their effectiveness and efficiency of their detection capability. A wide range of C constructs is taken into account and appropriate metrics are computed, which show how the tools balance inherent analysis tradeoffs and efficiency. The results are useful for identifying the appropriate tool, in terms of cost-effectiveness, while the proposed methodology and test suite may be reused.

I. Introduction

Static program analysis belongs to the class of problems that are undecidable [1]. In practice, it is implemented as an approximation of the program's behavior that inevitably restricts the analysis capability in correctly detecting actual code defects. Important considerations are: (i) the programming language, (ii) the targeted defects, (iii) the analysis effectiveness, i.e. the proportion of detected real defects and (iv) the analysis efficiency that affects the needed computing resources for code scanning.