Review of Adaptive Activation Function in Deep Neural Network

A biological inspired algorithm from human brain known as deep neural network (DNN) containing of multiple hidden layers often occurs vanishing gradient problem due to the saturation characteristic of activation function. Thus, the choice of activation function in DNN is crucial to boost up the DNN recognition performance. Unsaturated activation functions i.e. rectified linear unit is recently proposed to prevent vanishing gradient problem happened during the training process of DNN. In this paper, DNN performance is investigated with three categories of activation functions i.e. saturated, unsaturated and adaptive activation functions. The experimental results showed that the saturation problem of hyperbolic tangent activation function can be solved by adding two trainable parameters in its function. The trainable version of rectified linear unit i.e. parametric rectified linear unit (PReLU) obtained lowest misclassification rate among all types of activation function i.e. 1.6% misclassification rate on MNIST handwritten digit dataset. This is due to the adaptive activation functions allows the network to estimate a better solution by training the activation function parameters during the training process. Therefore, adaptive activation functions improves the generalization of the network to deal with the real-world application.