Breakthrough to Adaptive and Cost-Aware Hardware-Assisted Zero-Day Malware Detection: A Reinforcement Learning-Based Approach

In this paper, we have identified and addressed pressing challenges associated with online and cost-effective malware detection based on Hardware Performance Counters (HPCs) information. Existing Hardware-Assisted Malware Detection (HMD) methods guided by standard Machine Learning (ML) algorithms have limited their study on detecting known signatures of malicious patterns; thus, neglecting to address unknown (zero-day) malware detection at run-time which is a more challenging problem since the malware HPC data does not match any known attack applications’ signatures in the existing database. In addition, prior works have not presented a flexible and balanced solution that considers the trade-off between detection rate and implementation cost for adaptive selection of the best performing ML algorithms for online malware detection. In this paper, we first propose a unified feature selection method based on a heterogeneous feature fusion technique to effectively determine the most important HPC events for low-cost yet accurate malware detection. Next, we present Reinforced-HMD, a novel reinforcement learning-based framework for adaptive and cost-aware hardware-assisted zero-day malware detection based on desired performance metric and available hardware resources. To this aim, six classical and two reinforcement learning algorithms are implemented and their efficiency is thoroughly analyzed for detecting unknown malware using HPC events. Experimental results demonstrate that our Reinforced-HMD framework based on Upper Confidence Bound (UCB) learning approach achieves an accurate and robust detection rate with a 96% in both F1-score and AUC metrics for flexible and efficient zero-day malware detection while utilizing an optimal set of built-in HPC events.