NBench: A Comprehensive SDN Controller Benchmarking Suite

Performance evaluation tools are very essential for assessing the capabilities of Software-Defined Networking (SDN) controllers. These tools play a critical role in understanding and quantifying the performance characteristics of SDN controllers. While several open-source performance testing tools, such as CBench, have been developed and are currently in use for evaluating SDN controllers, they have limitations. Specifically, CBench only supports OpenFlow 1.0 and lacks the ability to measure key performance indicators such as topology discovery time, end-to-end link setup time, etc. Moreover, it falls short in terms of system resource utilization and the accuracy of simulated workload representation. In this paper, we introduce NBench, a novel performance evaluation tool designed for SDN controllers. Compared to the well-known CBench, NBench demonstrates significant improvements in system resource utilization and workload simulation accuracy. Additionally, NBench offers a finer granularity of logging and supports the latest versions of the OpenFlow protocol, whereas CBench is limited to version 1.0. Beyond measuring traditional metrics such as Packet-In message response latency and throughput, we have pioneeringly incorporated the measurement of topology discovery time, end-to-end link setup time and link failure recovery time in SDN controllers. To the best of our knowledge, NBench is the first open-source tool to implement these tests.