A Customized Protocol Cluster Analysis Method based on Reinforcement Learning

The network protocol defines the rules of communication between massive hosts on the Internet. As an indispensable means of information transmission in the network, the definition, structure and formulation of protocol are directly related to communication security, which influences a wide range of applications in malicious code detection, deep packet detection and protocol fuzzing. Therefore, the reverse analysis of unknown protocols is extremely important to the research and evolution of cyber security. In this paper, we propose a static analysis technology route based on network trace named RLPA, which is based on reinforcement learning to analyse custom protocols. This method takes advantage of its interaction with the environment and self-learning to learn and explore the structural rules in traffic texts, so as to improve the clustering effect and performance. For the experimental verification, we implemented the representative common clustering methods. The results show that the unsupervised model is a little worse than reinforcement learning and supervised model in analyzing and classifying customized protocols. Among them, the protocol classification based on decision has superior performance. At the same time, the reinforcement learning model can maintain a stable classification effect when faced with different unfamiliar data sets.