Hardware-Aware Model Optimization Tool for Embedded Devices

Designing deep neural network models for embedded devices is a challenging task since the models need to be lightweight, fast, and accurate. This paper proposes a hardware-aware model optimization tool (HOT) to optimize a given model in terms of latency or accuracy by replacing its existing operators with the best-performing operators for target hardware. The proposed tool finds optimal operators with high accuracy and low latency in a short searching time while keeping the existing model structure rather than finding an entirely new network architecture. The result shows that the HOT improves MobileNetV2 backbone based models by up to 13.93% in accuracy (mAP) or 37.84% in latency (ms) with cascaded pyramid network (CPN) (pose estimation) and 31.03% in mAP or 56.64% in latency (ms) for single shot multi-box detector (SSD) (object detection) on digital television.