Multimodal Depression Recognition that Integrates Audio and Text

Depression is characterized by persistent low mood significantly affecting people’s daily lives and work. According to research, speech concentrates mostly on information like tone, speed, and loudness, making it challenging to directly ascertain semantic information. Although textual information is comprehensive and straightforward, it also includes drawbacks including ambiguity and bias. Speech and text fusion are utilized for depression recognition to increase the efficacy of emotion recognition. The extraction of speech features adopts Mel frequency cepstrum coefficient (MFCC) features, and then a bidirectional long short-term memory network is built using convolutional neural networks and Bidirectional Long Short-Term Memory Networks (BiLSTM) for long-term modeling. The attention mechanism is further used to enhance the weight of key emotional features in speech. Textual feature extraction uses Bert, and then two layers of LSTM networks are further used to extract textual features. Afterward, extracted speech and text features are fused at the feature level, and their recognition effects on the E-DAIC dataset are compared. The experimental results show that the recognition of depression by multi-modal fusion features is better than that by single-modality features.