Behavioral Analysis of System Call Sequences Using LSTM Seq-Seq, Cosine Similarity and Jaccard Similarity for Real-Time Anomaly Detection

With the advent of technology, sophisticated malware presents a significant threat to computer security. In this work, we propose anomaly detection techniques that learn three different behaviors of windows system-call sequences. We apply Long-Short-Term-Memory (LSTM) for temporal behavior, Cosine Similarity for frequency distribution behavior, and Jaccard Similarity for commonality behavior. The proposed framework monitors the processes in a hypervisor-based environment to detect compromised virtual machines. System call sequences of normal processes and malware-infected processes were extracted with memory forensic techniques. Our proposed anomaly detection techniques were able to learn the above three behavior of the system call sequences with 99% accuracy.