Model Compression for IoT Applications in Industry 4.0 via Multiscale Knowledge Transfer

Recently, Industry 4.0 has attracted much attention. It has close relations with the Internet of Things (IoT). On the other hand, convolutional neural networks (CNNs) have shown promising performance in many foundational services of the IoT applications. For the IoT applications with high-speed data streams and the requirement of time-sensitive actions, fast processing is demanded on small-scale platforms or even on IoT devices themselves. Therefore, it is inappropriate to employ cumbersome CNNs in IoT applications, making the study of model compression necessary. In knowledge transfer, it is common to employ a deep, well-trained network, called
teacher
, to guide a shallow, untrained network, called
student
, to have better performance. Previous works have made many attempts to transfer single-scale knowledge from
teacher
to
student
, leading to degradation of generalization ability. In this article, we introduce multiscale representations to knowledge transfer, which facilitates the generalization ability of
student
. We divide
student
and
teacher
into several stages.
Student
learns from multiscale knowledge provided by
teacher
at the end of each stage. Extensive experiments demonstrate the effectiveness of our proposed method both on image classification and on single image super-resolution. The huge performance gap between
student
and
teacher
is significantly narrowed down by our proposed method, making
student
suitable for IoT applications.