On the Use of Real-Time Multi-Object Detection and Classification for Automated Assessment of Operator Visual Attention Allocation

In human-machine teaming, machine capability to automatically check human teammates for their Situation Awareness (SA) will not only improve human-machine teaming performance, but it will also create opportunities to provide timely interventions. Such machine capability may benefit a wide range of applications, ranging from Advanced Driver Assistance Systems (ADAS) to highly complex warfare situations. This paper extends our previous work on operator SA assessment (FUSION 2022 paper) by leveraging state-of-the-art Computer Vision (CV) techniques on object detection, segmentation and multi-target tracking (or multi-object tracking) for identifying relevant Situation Elements (SEs) and extracting information necessary for calculating Probability of Attending for those SEs. First, YOLOv7 and StrongSort is used for automated detection and tracking of SEs identified from video streams. Then, instance segmentation is utilized for identifying relevant pixel information of SEs (i.e., location, size, color) for computing specific Visual Attention Allocation (VAA) values of probability of attending (P(SE)) and attention allocation proportion ($f_{i}$) via the use of the SEEV (i.e., Salience, Effort, Expectancy, and Value) SA model. Finally, operator SA is derived by analyzing the operator’s visual attention via the use of eye-tracking to check if the operator is paying attention to relevant SEs. This paper concludes with a discussion of our current progress, technical challenges being pursued and potential future extensions.