Vision Transformer and Parallel Convolutional Neural Network for Speech Emotion Recognition

Vision Transformer (ViT) is a new approach for image processing tasks. The vision transformer splits the image into patches and converts it into a sequence of vectors. This sequence is suitable for the transformer structure. This paper uses the ViT method for speech emotion recognition. Unlike ViT, which splits the image into square patches, time frames are used as patches. In addition to using the frame-based ViT to benefit from its ability to learn global features, a convolutional neural network is also used. The convolutional neural network extracts local features and focuses on the two-dimensional structure of the input. Mel-Frequency Cepstral Coefficients extracted from audio files are used as inputs for the proposed neural network. The proposed model is applied to the RAVDESS dataset and is shown to achieve an unweighted accuracy of 79.2%.