Searching for a Needle in a Haystack: Predicting Security Vulnerabilities for Windows Vista

Many factors are believed to increase the vulnerability of software system; for example, the more widely deployed or popular is a software system the more likely it is to be attacked. Early identification of defects has been a widely investigated topic in software engineering research. Early identification of software vulnerabilities can help mitigate these attacks to a large degree by focusing better security verification efforts in these components. Predicting vulnerabilities is complicated by the fact that vulnerabilities are, most often, few in number and introduce significant bias by creating a sparse dataset in the population. As a result, vulnerability prediction can be thought of us preverbally “searching for a needle in a haystack.” In this paper, we present a large-scale empirical study on Windows Vista, where we empirically evaluate the efficacy of classical metrics like complexity, churn, coverage, dependency measures, and organizational structure of the company to predict vulnerabilities and assess how well these software measures correlate with vulnerabilities. We observed in our experiments that classical software measures predict vulnerabilities with a high precision but low recall values. The actual dependencies, however, predict vulnerabilities with a lower precision but substantially higher recall.

I. Introduction

Software security is a critical part of the software development process. While there is a significant body of work on predicting defects, unfortunately little is known about the field of vulnerability prediction. Some recent work focused on this topic in the open source domain [9] [15] [22]. In this paper, we focus on vulnerability prediction for a proprietary commercial product (Windows Vista). We define a component to be vulnerable if it has been changed as part of a security update after it was released publically.