Training Strategies for Convolutional Neural Networks with Transformed Input

Convolutional Neural Network (CNN) are now considered the main tool for image classification. However, most networks studied for classification are large and have extensive computing and storage requirements. Their training time is also usually very long. Such costly computational and storage requirements cannot be met in many applications with simple devices such as small processors or Internet of things (IoTs) devices. Therefore, reducing the size of networks and input sizes become necessary. However, such reductions are not easy and may reduce the classification performance. We examine how domain transforms under different training strategies can be used for efficient size reduction and improvement of classification accuracy. In this paper, we consider networks with under 220K learnable parameters, as opposed to millions in deeper networks. We show that by representing the input to a CNN using appropriately selected domain transforms, such as discrete wavelet transforms (DWT) or discrete cosine transform (DCT), it is possible to efficiently improve the performance of size-reduced networks. For example, DWT proves to be very effective when significant size reduction is needed (improving the result by up to 9%). It is also shown that by tuning training strategies such as the number of epochs and mini batch size, the performance can be further improved by up to 4% under fixed training time.

I. Introduction

One of the most popular methods which has been frequently used is for image classification in different applications is Convolutional Neural Networks (CNNs). Most CNNs require training time within an acceptable range. Also, there are other CNNs such as Res-Net, VGG, Alex net. [9] [10] [11] that can also be used for image classification; however, the training time required for such networks is significantly more than that of CNN. In this paper, we examine how adding a transform layer at the beginning of a CNN and different training strategies can result in performance improvement. The improved performance can be traded off for network size reduction. Therefore, we can make the network smaller and maintain almost the same performance.