Multi-Scale Driver Behaviors Reasoning System for Intelligent Vehicles Based on a Joint Deep Learning Framework

The mutual understanding between driver and vehicle is critically important to the design of intelligent vehicles and customized interaction interface. In this study, a deep learning-based joint driver behavior reasoning system toward multi-scale and multi-tasks behavior recognition is proposed. Specifically, a multi-scale driver behavior recognition system is designed to recognize both the driver's physical and mental states based on a deep encoder-decoder framework. The system jointly recognizes three driver behaviors, namely, mirror-checking, lane change intention, and emotions based on the shared encoder network. The encoder network is designed based on a deep convolutional neural network (CNN), and several decoders for different driver states estimation are proposed with fully connected (FC), and long short-term memory (LSTM) based recurrent neural networks (RNN), respectively. The proposed framework can be used as a solution to exploit the relationship between different driver states for intelligent vehicles towards an efficient driver-side understanding. The testing results on the Brain4Car dataset show accurate performance and outperform existing methods on driver postures, intention, and emotion recognition.