Avoiding Training in the Platform-Aware Optimization Process for Faster DNN Latency Reduction

Many real-world deep learning applications are sensitive to the size and latency of the models, especially when inference is performed on low-power devices. In many scenarios, it is possible to optimize DNNs to meet latency and energy budgets without sacrificing too much accuracy. NetAdapt, a newly developed tool, tackles this issue by optimizing the network through the removal of layers based on practical latency measurements specific to the target device. In this work, we improve two stages of NetAdapt's algorithm. First, we enhance the candidate models' search procedure by improving the mechanism that interpolates latency measurements for configurations not available in the look-up table. Secondly and more importantly in terms of novelty, we study the potential benefits of integrating No-training NAS techniques into the hardware-aware optimization algorithm to avoid the fine-tuning and evaluation stages with minimum accuracy loss. The experimental evaluation shows that the optimizations applied can accelerate NetAdapt's algorithm up to 37× for a large network and 4× for a small one.