Automatic Depression Detection Using Multi-Modal & Late-Fusion Based Architecture

Depression has become second most fatal disease after cardiac arrests. Recent lockdowns during COVID, Wars such as Ukraine-Russia and War-Like-Scenario such as Taiwan-China etc has contributed badly and people suffering from severe depression, mood swings and anxiety has gone all time high. Due to lack of opportunities to reach a real medical practitioner, an AI based, automatic depression detection system could be a saviour for millions. In the current work, authors have proposed a multimodal deep Convolutional Neural Network (CNN) based automatic depression detection system. Authors have fused two CNN networks, one Long Short Term Memory (LSTM) based CNN process Electroencephalogram (EEG) signal while other CNN process facial video data. Both the networks classify the subject to one among three classes Depressed, Mild Depressed or Not Depressed. EEG waves are taken from frontal lobe of the brain. The level of alpha component in the EEG signal is being used as bio marker for depression classification. On the other hand, frame by frame analysis is being performed on facial video. An attention-based CNN is being proposed to process each video frame and Facial Action Coding System (FACS) based Linear Binary Pattern (LBP) classifier has been developed for depression classification. Further the result of these two networks is being fed into a late fusion network. The final depression class is being decided by this final classifier. The proposed model has shown quite promising outcome. Authors have trained the EEG network on MODMA, 3-channel EEG dataset and Video A-CNN network on AVEC (Audio/Visual Emotion Challenge) 2018 dataset and tested this entire model on DEAP (a Database for Emotion Analysis using Physiological Signals). dataset and it has performed better than almost all state-of-the-art models.