A Performance Comparison of Loss Functions

Generally, the deep neural network learns by way of a loss function, which is an approach to evaluate how well given dataset is predicted on a particular network architecture (or network model). If the prediction deviates too far from real data, a loss function would generate a very large value. Progressively, with the help of some optimization function, the loss function lowers the prediction error by providing the network architecture with information that can control the weights of the network architecture. Thus, the loss functions plays an important role in training the network architecture.Recently, several researchers have studied various loss functions such as Softmax, Modified softmax, Angular softmax, Additive-Margin softmax, Arcface, Center, and Focal losses. In this manuscript, we propose a new and simple loss function that just adds the existing loss functions. In addition, we conduct experiments with the MNIST dataset in order to compare the performance between all loss functions including the proposed and the existing loss functions. Resultingly, the experiments show that the proposed loss function is visibly superior to the ability to classify digit images. The experimental results also indicate that Arcface loss and Additive-Margin loss functions satisfy predefined test accuracy most quickly under two and three dimensional embedding, respectively. The fast learning ability of the both loss functions has the advantage of providing relatively high accuracy even when the number of train data is small.