Text-Based Machine Learning Models for Cross-Domain Vulnerability Prediction: Why They may not be Effective?

Machine learning models are being utilized as applicable tools for source code vulnerability detection. However, the usage of models and techniques still vary in usability and prediction. In generalizing models to a more natural language approach, researchers have opted to train models on source code to identify existing and potential vulnerabilities. Exploratory research has been performed by treating source code as plain text. Additionally, models have been trained and tested within and across domains, such as projects. Based on a recent empirical study, text-based models are not effective across-domains. In this study, we examine results from this text-based approach to machine learning models. By providing a statistical analysis of data used and prediction results provided, we explore why the approach may not be sufficient for vulnerability prediction across domains.

I. Introduction

Distribution of vulnerability severity has grown overtime, even more so since 2016¹

https://nvd.nist.gov/general/visualizations/vulnerability-visualizations/cvss-severity-distribution-over-time

. In 2021 alone, over 40,000 vulnerabilities were recorded in the Common Vulnerabilities and Exposures (CVE) database²

https://cve.mitre.org/index.html

, Software vulnerabilities are neither limited to open-source or proprietary software nor devices or services. Vulnerabilities can have a multitude of consequences such as system exploitation which may lead to data leaks involving sensitive data including personal identifi-able information or intellectual property.