MLPNN Training via a Multiobjective Optimization of Training Error and Stochastic Sensitivity

The training of a multilayer perceptron neural network (MLPNN) concerns the selection of its architecture and the connection weights via the minimization of both the training error and a penalty term. Different penalty terms have been proposed to control the smoothness of the MLPNN for better generalization capability. However, controlling its smoothness using, for instance, the norm of weights or the Vapnik-Chervonenkis dimension cannot distinguish individual MLPNNs with the same number of free parameters or the same norm. In this paper, to enhance generalization capabilities, we propose a stochastic sensitivity measure (ST-SM) to realize a new penalty term for MLPNN training. The ST-SM determines the expectation of the squared output differences between the training samples and the unseen samples located within their Q -neighborhoods for a given MLPNN. It provides a direct measurement of the MLPNNs output fluctuations, i.e., smoothness. We adopt a two-phase Pareto-based multiobjective training algorithm for minimizing both the training error and the ST-SM as biobjective functions. Experiments on 20 UCI data sets show that the MLPNNs trained by the proposed algorithm yield better accuracies on testing data than several recent and classical MLPNN training methods.

I. Introduction

Most multilayer perceptron neural network (MLPNN) classifier training methods minimize an objective function by selecting the architecture of the network and the connection weights. Minimizing the training error as the only objective function commonly results in an overfitted MLPNN. Moreover, neural networks achieving the same training error on a given set of training samples often produce different decisions on the same unseen sample. To overcome such problems, a regularization [1] or penalty term is added to the minimized objective function to enhance the smoothness of the MLPNN outputs to secure better generalization capability. Output fluctuations of an MLPNN can be estimated, for instance, by the squared norm of weights or the Vapnik–Chervonenkis (VC) dimension. However, the squared norm of weights can be misleading when there are several combinations of weights yielding the same squared norm. This will be discussed in Section III-B3. The VC-dimension-based error produces a loose error bound that may not be useful to distinguish individual MLPNNs with the same number of connection weights and the maximum weight magnitude among all weights.