MTSecurity: Privacy-Preserving Malicious Traffic Classification Using Graph Neural Network and Transformer

Encrypting network traffic is an effective means of safeguarding user privacy and sensitive information. However, it also introduces potential vulnerabilities that can be exploited by network attackers, posing significant security risks to the Internet. In response to the challenge of low accuracy in existing methods for classifying encrypted malicious traffic, we propose a novel approach named MTSecurity, which leverages Transformer and Graph Neural Network technologies. This method automatically extracts raw byte features and graph-based traffic interaction features from encrypted malicious flows, combining them to substantially enhance the classification accuracy of encrypted malicious traffic. Furthermore, we introduce a graph structure called the Malicious Traffic Interaction Graph (MTIG) for representing encrypted malicious traffic. MTIG is based on the client-server interaction process and incorporates multi-dimensional traffic features. Experimental results demonstrate that the proposed MTSecurity model consistently performs well across different datasets, surpassing state-of-the-art methods. It achieves an accuracy of 0.9946 and an F1 score of 0.9940 on the MCFP dataset, and an accuracy of 0.9948 with an F1 score of 0.9934 on the USTC-TFC dataset.