Instruction Clustering Analysis for Network Protocol's Abnormal Behavior

Protocol's abnormal behavior analysis is an important task in protocol reverse analysis. Traditional protocol reverse analysis focus on the protocol message format, but protocol behavior especially the abnormal behavior is rare studied. In this paper, protocol behavior is represented by the labeled behavior instruction sequences. Similar behavior instruction sequences mean the similar protocol behavior. Using our developed virtual analysis platform HiddenDisc, we can capture a variety of known or unknown protocols' behavior instruction sequences. All kinds of executed or unexecuted instruction sequences can automatic clustering by our designed instruction clustering algorithm. Thereby we can distinguish and mine the unknown protocols' potential abnormal behavior. The mined potential abnormal behavior instruction sequences are executed, monitored and analyzed on HiddenDisc to determine whether it is an abnormal behavior and the behavior nature. Using the instruction clustering algorithm, we have analyzed 1297 protocol samples, mined 193 potential abnormal instruction sequences, and determined 187 malicious abnormal behaviors by regression testing. Experimental results show that our proposed instruction clustering algorithm has high efficiency and accuracy, can effectively mine unknown protocols' abnormal behaviors, and enhance the initiative defense capability of network security.