Federated Multimodal Learning for Privacy-Preserving Driver Break Recommendations in Consumer Electronics

In recent years, driver distraction behaviors, such as eating, drinking, and making phone calls, have become more and more frequent, especially during continuous driving. This results in a significant increase in traffic accidents caused by those behaviors. Therefore, it is crucial to recommend driver breaks when frequent instances of driver distraction are detected. Despite numerous single modal-based approaches, such as computer vision, were proposed for driver distraction detection, driver break recommendations still suffer from low accuracy and mandatory requirement of preinstalled cameras. To make the full use of the power of ambient information including images, audios, and postures, we introduce Multimodal Learning (ML) to identify driver distraction. Our model is capable of utilizing information from images, audios, and postures through consumer electronic devices such as mobile phones. Moreover, transfer learning is utilized to take advantage of previously trained models, thereby significantly enhancing the accuracy. However, training an personalized model requires a substantial amount of data from entities and individuals which raises privacy concerns. Henceforth, we integrate Federated Learning into Multimodal Deep Learning framework to protect participants’ privacy while achieving better performance in recommending driver breaks. The experimental findings conclusively demonstrate that the proposed method not only outperforms other existing approaches in terms of providing effective driver break recommendations but also prioritizes the privacy of the individuals involved.

I. Introduction

Driver distraction behaviors such as drinking water, making phone calls, and sending text messages occupy the driver’s attention and may lead to the loss of vehicle control and decision-making ability. According to a report from police, about 25% of accidents can blame on driver distraction. McEvoy et al. [1] conducted a questionnaire survey on 1367 drivers who stayed in hospitals because of car accidents and collected additional data from the ambulance and medical records. The result shows that more than 30% of the drivers mentioned at least one kind of distraction behavior when a car accident happened. The research of Duan et al. [2] also shows that driver distraction reduces the efficiency of driving significantly. Therefore, it becomes an important matter to monitor drivers’ behaviors during driving. However, it doesn’t mean that if a distracting behavior appears then an accident happens. It varies from person to person. Reports also show that these behaviors become more frequent after longer driving, and having a short recess can significantly reduce distraction behaviors. Therefore, this paper tries to design a driver break recommendation based on distraction behavior detection. It requires to detect distraction behaviors accurately not only in the spatial domain but also in the temporal domain. To reduce false alarms, only when distraction behaviors appear more than a dynamic threshold will the recommendation be notified to the driver. The challenge lies in accurately providing personalized break recommendations while safeguarding individual privacy.