The Process of Reverse Engineering GPU Malware and Provide Protection to GPUS

As malware authors keep devising new methods to breach the security of computers and hide their malware from antivirus and forensics tools, new techniques need to be developed to alleviate the threats posed by these innovative and advanced malware attacks. The purpose of this research is to reverse engineer a unique family of malware, namely the Win Jelly and the Demon keylogger, that escapes detection by utilizing AMD and NVIDIA Graphics Processing Units (GPUs) as a hideout. After the static and dynamic analysis of Win Jelly and Demon to gain a deeper understanding about the behavior of these malware and how they exploit the GPU, a new technique is developed using OpenCL to completely remove the malware from the GPU and to help avoid future threats. The proposed method and tool successfully removed the malicious files from the GPU without any drawbacks. This paper will raise the awareness for AMD and NVIDIA GPU users and help GPU developers to implement more security measures to the GPU vulnerability.