VErGE: A system for collecting voice, eye gaze, gesture, and EEG data for experimental studies

Collecting multimodal data in controlled experiments is often challenging, especially when there is the need to record fine observations and experimental data over multiple input channels. Self-reporting and direct observation, including video recording, are often employed with good results, but in many cases user behavior may be elusive or difficult to quantify and record. This problem is particularly even more difficult to address in the case of multimodal systems, where input is elicited from users with various disabilities such as motor impairments, due to many factors like increased calibration time, constraints on body postures and limited range of movements, and interferences occurring at a physiological level, such as fatigue or involuntary spasms. In this work, we describe VErGE, a multimodal input system designed to collect voice, gestures, eye gaze and EEG (electroencephalography) data in experimental settings. We detail the architecture of our system and discuss a representative use case for collecting multimodal input from users with motor impairments.