On Adaptive Learning Framework for Deep Weighted Sparse Autoencoder: A Multiobjective Evolutionary Algorithm

In this article, an adaptive learning framework is established for a deep weighted sparse autoencoder (AE) by resorting to the multiobjective evolutionary algorithm (MOEA). The weighted sparsity is introduced to facilitate the design of the varying degrees of the sparsity constraints imposed on the hidden units of the AE. The MOEA is exploited to adaptively seek appropriate hyperparameters, where the divide-and-conquer strategy is implemented to enhance the MOEA’s performance in the context of deep neural networks. Moreover, a sharing scheme is proposed to further reduce the time complexity of the learning process at the slight expense of the learning precision. It is shown via extensive experiments that the established adaptive learning framework is effective, where different sparse models are utilized to demonstrate the generality of the proposed results. Then, the generality of the proposed framework is examined on the convolutional AE and VGG-16 network. Finally, the developed framework is applied to the blind image quantity assessment that illustrates the applicability of the established algorithms.

I. Introduction

Since the proposal of the perceptron [37] and backpropagation [38], artificial neural networks (ANNs) have become a more and more powerful tool in artificial intelligence with successful applications in many branches of science and engineering [53]. In the past decade, as an improvement of traditional ANNs, deep neural networks (DNNs) have drawn considerable research interest from both academic and industrial communities [7], [30] and gained ever-increasing popularity in a variety of application areas, such as recognition, signal and information processing, and image processing and classification [20], [21], [41], [47], [49], [50]. The deep belief network (DBN) proposed in 2006 [19], which is now widely acknowledged as a breakthrough of DNNs, is essentially a greedy and multilayer formed learning model combined by a stack of restricted Boltzmann machines (RBMs). In DBNs, the proposed learning algorithm is the so-called layerwise training method, where the lower layer learns simple features that will be subsequently transmitted to the higher layer as inputs. In this sense, DBNs can be considered as the combination of simple learning modules and the layer-wise training method [16], [18], [25], [51]. It is worth noting that as an alternative of RBMs, autoencoders (AEs) could also be chosen as shallow modules [8].