Evaluating C/C++ Vulnerability Detectability of Query-Based Static Application Security Testing Tools

In recent years, query-based static application security testing(Q-SAST) tools such as CodeQL have gained popularity due to their ability to codify vulnerability knowledge into SQL-like queries and search for vulnerabilities in the database derived from the software. The industry has made considerable progress in building Q-SAST tools, facilitating their integration into the continuous integration (CI) pipeline, and sustaining an active community. However, we do not have a systematic understanding of their vulnerability detection capability in comparison to conventional SAST tools. We conduct the first in-depth study of Q-SAST to demystify their C/C++ vulnerability detectability. Our study is conducted from three complementary aspects. We first use a synthetic CWE test suite and a real-world CVE test suite, totaling almost 30 K programs with known CWE/CVE, to assess popular (commercial) Q-SAST and industry-leading SAST (requiring no queries). Then, we gather defect-fixing pull requests (PRs) since the release dates of three popular Q-SAST tools, characterizing historically-fixed defects and comparing them to pitfalls exposed in our CWE/CVE study. To enhance vulnerability detection, we design SAST-MT, a metamorphic testing framework to detect false positives (FPs) and false negatives (FNs) of Q-SAST. Findings of SAST-MT can be used to easily expose the root causes of Q-SAST's FPs and FNs. We summarize lessons from our study that can benefit both users and developers of Q-SAST.

I. Introduction

Ddiscovering vulnerabilities is a classic yet challenging problem. Constantly analyzing large amounts of source code, in particular, has been a major focus of software engineering research. Among the various paradigms of security analysis tools [17], [71], [140], static application security testing (SAST) tools [69], [91] have become one of the mainstream due to their high flexibility for analyzing software without running it and their typically greater comprehensiveness when compared to dynamic application security testing (DAST) tools [16], [140].