LOTIC: A General Framework for High-Speed Network Traffic Processing

Network flow is an important data source in network security analysis system. However, some serious problems such as data distortion and low scalability exist in current software/hardware-based network flow derivation technology. In order to solve these problems, this paper presents LOTIC, a general framework for high-speed network traffic handling. LOTIC improves and optimizes transmission performance through high-performance packet capturing tools, and introduces message system to import high-speed network flow into real-time processing platform for distributed packet parsing and flow reorganization. By importing network flow into data storage module (Hive) in real time, LOTIC achieves accurate flow capturing and scalable processing in 10 Gigabit-flow, which lays a foundation for big-data analysis of network traffic. Experimental results show that LOTIC can process 12.6 million packets per second in 10 Gigabit-flow. Besides, LOTIC can be easily deployed in large-scale cluster with better scalability and flow statistics accuracy than NetStream.