MACA: Memory-aware convolution accelerating for CNN inference on edge devices

Deep learning inference tasks develop towards the edge due to their latency requirements and privacy issues. However, edge devices are limited by their power consumption and size, and generally have limited resources. The convolutional neural networks (CNN) is commonly used in image processing tasks which contains a large number of convolutional layers, accounting for more than 95% of the calculation time in most general used CNN model. This paper proposes a general convolutional layer optimization method called MACA, we implement and optimize a variety of convolution operators and design a memory-aware convolution operator automatic selection strategy to select appropriate operator, without modifying user code. Finally, we integrate MACA into PyTorch and conduct extensive experiments. The results show that when memory resources are sufficient, MACA can effectively increase the inference by 36.10% on average, and can reduce memory usage by an average of 29.14% to complete inference when resources are tight. This paper provides an effective solution for deploying deep learning models on resource-constrained edge devices.

I. Introduction

In recent years, deep learning (DL) has achieved great success in various fields such as computer vision [1], natural language processing [2], and speech recognition [3]. Traditional DL tasks are widely supported by cloud servers, but due to limitations in latency, privacy issues, and reliance on network connection, the need to deploy models on edge devices has become more urgent [4]. DL is a computationally intensive task, and edge devices are limited by their power consumption and size [5]. Generally, they are not equipped with acceleration devices to improve computing efficiency, and their memory resources are also strictly limited. Inference tasks are latency-sensitive, and users expect these tasks to be performed efficiently and produce accurate predictions in a short period of time.