Impact and Performance Analysis of Various Activation Functions for Classification Problems

The human brain serves as the inspiration for artificial neural networks in deep learning, which have a similar structure and operation as the brain’s network of neurons. These networks have layers of cells that are connected and process information in a similar way to real neurons. An activation functions plays a important role in these deep learning based neural network architectures. This paper examines the impacts and performance of various activation functions: step function, sigmoid, Tanh, soft-sign, soft-max, Relu and its variants. These activation functions are tested on four publicly available biomedical data sets. This paper studies the impacts of these activation functions and finds the best activation function for binary and multi-class classification problems in deep artificial neural networks. The results show that sigmoid has good performance for binary classification and soft-max has good performance for multi-class classification tasks. However, building a successful neural network depends on choice of parameter setting but, in this study, all other parameters are kept unchanged.