Threat Hunting as a Similarity Search Problem on Multi-positive and Unlabeled Data

We present a new similarity search method (called Random Separations) that helps threat analysts with identification of unknown variants of known malware in network traffic. The method assumes that for each hunted malware family, few samples of network communication are available to analysts (multi-positive) and others are hidden in abundant (unlabeled) network data. We demonstrate the method on large-scale real-world data, where it outperforms the unsupervised approach (Isolation Forest and Lightweight Online Detector of Anomalies), the supervised approach (Random Forest) and the traditional similarity search algorithm (kNN). The evaluation involves eight high-risk malware families under various known/unknown ratios.