EEG Context Fusion for AI-Based Object Detection and Drone Navigation in Situationally Aware Brain-Computer Interfaces

We are interested in the utility that artificially intelligent mobile systems such as drones offer to personnel in fast-paced situations such as hostage rescue and disaster relief for real-time situational awareness. Ideally, these assistive systems place no additional cognitive or physical burden on their user; rather, they should respond to the user’s intent with minimal physical and cognitive burden. To this end, brain-computer interface (BCI) and in particular electroencephalography (EEG) offers a novel way to capture intent. EEG-based drone control has been explored, but typically relies on the domain of motor imagery (MI), which still requires somewhat-manual drone piloting for micro directional movement, adding cognitive burden. We propose leveraging the robust computer-vision based AI that exists on modern drones to use objects as waypoints and fly mostly autonomously, with EEG in an object recognition paradigm for passively selecting the waypoint. While existing techniques for EEG object recognition-including steady-state visually evoked potentials (SSVEP), rapid serial visual presentation (RSVP), and eye tracking - rely on “marked” input or a controlled environment, we need one that will work on passive objects. In this work, we propose merging available scene imagery in a vision network with deep-learning based EEG processing to achieve passive intent recognition. The proposed context-scene fusion for a situational awareness paradigm presents accuracy in the same ranges as existing object recognition without requiring SSVEP, RSVP, eye tracking, or MI techniques and serves as proof-of-concept for this approach to real-world BCI applications.