Explainable Artificial Intelligence for Improving a Session-Based Malware Traffic Classification with Deep Learning

In network security, applying deep learning methods to detect network traffic anomalies has achieved great results with various network traffic representations. A possible representation is the transformation of raw network communication to images to extract valuable information from the unmanageable amount of network traffic by applying representation learning. However, since deep learning models can result in black boxes for users, it is interesting to understand what valuable information is learned from network communication converted into images. This paper elaborates on that question using explainable artificial intelligence (XAI) methods to identify network packets that most influence the prediction and verify that packets in a malware communication containing malicious payloads have high influence on the prediction. We inspect the Grad-CAM and visualize the Integrated Gradients of Xception and VGG-19 model and investigate the attention heat maps of our Vision Transformer (ViT) model. In addition, we present a novel transformation of sessions to a new image representation to expand the informativeness of network communication. For multiclass classification, our best model Xception achieves an accuracy of 97.95%, whereas, for binary classification, Xception and VGG-19 achieve well above 99.50%. Our ViT model achieves a significantly lower performance with 95.86% for multiclass and 99.36% for binary classification. In particular, computing centers could benefit by examining their inbound and outbound traffic to detect malicious behaviors ahead of time.