High Speed Traffic Archiving System for Flow Granularity Storage and Querying

Archiving Internet traffic is a function essential for retrospective network event analysis. The state-of-art approach for network monitoring and analysis is storing and analyzing the statistics of network flows. However this approach loses much valuable information inside Internet traffic. With the advancement of commodity hardware, especially the volume of storage device and the speed of interconnect technologies used in network adapter card, now it is practical to capture 10Gbps real-time network traffic with a commodity computer. In this context, this paper presents the design and implementation of a novel system for archiving and querying network flows. A flow granularity indexing mechanism and storage are proposed, and the bitmap index database is utilized to store index information. Based on real network traces, we demonstrate that the system has a higher performance in storing and querying with respect to both time and space metrics compared with traditional methods.