Fusion of Multi-Level CNN With LBP Features For Facial Emotion Recognition

A facial emotion recognition framework is proposed in this work. The convolutional neural network (CNN) has high ability in extraction of hierarchical spatial features from low level texture characteristics to high level contextual features. A simple CNN model with three layers is suggested in this paper where the features extracted in all layers containing multi-level features are activated and used for classification. In addition, the local binary pattern (LBP) descriptor is used to extract discriminative features from the spatial structure of the input image. Therefore, four feature sources are provided by multi levels of CNN and the LBP descriptor. Each feature source is used for facial emotion recognition by applying to the support vector machine (SVM) classier. Finally, the majority voting rule is used for decision fusion to provide the final emotional label of each given face image. The proposed method with 84% overall accuracy, 83% weighted F1-score and 81% kappa coefficient provides the best performance compared to LBP, multi-level CNN and two-dimensional principal component analysis (2DPCA) methods.