Benchmarking Static Analysis Tools for Web Security

Static analysis tools are recurrently used by developers to search for vulnerabilities in the source code of web applications. However, distinct tools provide different results depending on factors such as the complexity of the code under analysis and the application scenario; thus, missing some of the vulnerabilities while reporting false problems. Benchmarks can be used to assess and compare different systems or components, however, existing benchmarks have strong representativeness limitations, disregarding the specificities of the environment, where the tools under benchmarking will be used. In this paper, we propose a benchmark for assessing and comparing static analysis tools in terms of their capability to detect security vulnerabilities. The benchmark considers four real-world development scenarios, including workloads composed of real web applications with different goals and constraints, ranging from low budget to high-end applications. Our benchmark was implemented and assessed experimentally using a set of 134 WordPress plugins, which served as the basis for the evaluation of five free PHP static analysis tools. Results clearly show that the best solution depends on the deployment scenario and class of vulnerability being detected; therefore, highlighting the importance of these aspects in the design of the benchmark and of future static analysis tools.

I. Introduction

Web applications have the remarkable ability to be quickly deployable and instantly accessible to the millions of users. They bring competitive benefits to most business areas, and therefore, the demand for new web applications with complex features is increasing fast. Since their development is often carried out under tight schedules, this can lead to many bugs and security problems. Thus, with no surprise, an Acunetix survey states that 60% of the vulnerabilities found are in web applications [1]. When exploited, their impact may have severe consequences for organizations, including financial losses, liability problems, brand damage, and the loss of market share [2].