Energy-Efficient Inference With Software–Hardware Co-Design for Sustainable Artificial Intelligence of Things

The emerging field of Artificial Intelligence of Things (AIoT) is propelled by the remarkable success of deep learning and hardware evolution, which has a significant impact on our daily lives. However, because of their notorious computing resource intensity, the widespread deployment of AIoT devices requires substantial electricity consumption as support, inevitably escalating energy consumption, and ultimately leads to a significant carbon emissions. Existing research on neural network compression and acceleration struggles to achieve energy-efficient inference on resource-constrained AIoT devices. To address this issue, we propose a software-hardware co-design approach that integrates advanced neural network optimization techniques with hardware power management capabilities to enable energy-efficient inference and ultimately achieve sustainable AIoT. We introduce a lightweight split and refinement block that adaptively reduces redundant computation in both channel and spatial dimensions. Several early exit (EE) branches are added to the backbone, which are controlled by a policy-based EE predictor. With the predicted EE index, a curve-fitting-based frequency scaling algorithm is presented to calculate the optimal frequency that minimizes energy overhead while maintaining latency constraints. Extensive experiments on CIFAR and CINIC classification tasks validate that our proposed method consistently reduces energy consumption for neural network inference while outperforming other competitive methods.

I. Introduction

The integration of artificial intelligence with Internet of Things (IoT) devices significantly enhances operational efficiency and decision-making capabilities, showcasing a transformative effect across various domains, including smart healthcare, autonomous vehicles, and advanced manufacturing systems. However, these substantial advances have been heavily based on high-performance hardware and sophisticated software, such as deep neural networks (DNNs). Although DNN inference can achieve high accuracy, it comes at the expense of increased execution time and energy overhead. With the expected number of IoT devices exceeding 41.6 billion by 2025, the overall energy consumption of Artificial Intelligence of Things (AIoT) systems could contribute significantly to carbon emissions. In fact, the successful design and deployment of AIoT systems requires addressing several research challenges. In particular, the latency constraint and the sustainability of energy have emerged as significant concerns.