Multimodal Fatigue Recognition State Based on Eyelid Features

Fatigue recognition of operators in a complex human-machine interface can effectively avoid human error. Aiming at improving the operator's vigilance in the digital control room of nuclear power plants, this paper proposes a multimodal fatigue state recognition technique. It combines deep learning and eyelid feature information to recognize the fatigue state of operators. Temporal Convolutional Network (TCN) is integrated with the 3D Residual Network (ResNet3D) to retain the temporal and spatial features of the eyelid. By using a custom cosine annealing learning rate decay algorithm to avoid the training process of the model falling into the local optimum. In this study, experiments were conducted on a variety of eyelid feature information for different timesteps. The UTA-RLDD and DROZY datasets were used to train and validate the model. The experimental results show that the fusion recognition of eyelid images and eyelid aspect ratio (EAR) using ResNet3D and TCN multimodal model can achieve better recognition results. With the increase of the timestep, there is a significant increase in the effect of recognition of mental states. In addition, the study set up an eyelid fatigue dataset to better fit the working environment of the digital main control room in a nuclear power plant. When the timestep is set to 160, the recognition recall of the model for this test dataset is 88%. The multimodal model can detect the state of low vigilance and drowsiness in real-time. Compared with other methods, the recognition accuracy of the model is improved by an average of 2.25%.