Performance Characterization of Using Quantization for DNN Inference on Edge Devices

Quantization is a popular technique used in Deep Neural Networks (DNN) inference to reduce the size of models and improve the overall numerical performance by exploiting native hardware. This paper attempts to conduct an elaborate performance characterization of the benefits of using quantization techniques-mainly FP16/INT8 variants with static and dynamic schemes-using the MLPerf Edge Inference benchmarking methodology. The study is conducted on Intel x86 processors and Raspberry Pi device with ARM processor. The paper uses a number of DNN inference frameworks, including OpenVINO (for Intel CPUs only), TensorFlow Lite (TFLite), ONNX, and PyTorch with MobileNetV2, VGG-19, and DenseNet-121. The single-stream, multi-stream, and offline scenarios of the MLPerf Edge Inference benchmarks are used for measuring latency and throughout in our experiments. Our evaluation reveals that OpenVINO and TFLite are the most optimized frameworks for Intel CPUs and Raspberry Pi device, respectively. We observe benefits of using quantization for these optimized frameworks with no loss in accuracy except for the static quantization techniques. For example, INT8-based quantized models deliver 3.3x and 3.7x better performance over FP32 using OpenVINO on Intel CPU and TFLite on Raspberry Pi device, respectively, for the MLPerf offline scenario. To the best of our knowledge, this paper is the first one that presents a unique characterization study characterizing the impact of quantization for a range of DNN inference frameworks-including OpenVINO, TFLite, PyTorch, and ONNX-on Intel x86 processors and Raspberry Pi device with ARM processor using the MLPerf Edge Inference benchmark methodology,

I. Introduction

The last decade has seen the emergence of Deep Neural Network (DNN) training as an important workload on parallel systems, including High-Performance Computing and Cloud hardware. DNNs have been found to be very useful in many applications, like Computer Vision and Natural Language Processing due to their high accuracy. Compare with significant successes [1]–[3] realized in training such DNNs, there is relatively less focus on deploying them for inference on edge devices. The deployment of these large models for inference on commodity servers, as well as resource-constrained environments, is vital for successful democratization of Artificial Intelligence (AI) models.