Behaviour-based detection of Transient Visual Interaction Obstacles with Convolutional Neural Networks and Cognitive User Simulation

The performance of humans interacting with computers can be impaired by several obstacles. Such obstacles are called Human Computer Interaction (HCI) obstacles. In this paper, we present an approach of detecting a transient visual HCI interaction obstacle called glare effect from logged user behaviour during system use. The glare effect describes a scenario in which sunlight shines onto the display, resulting in less distinguishable colors. For the detection of this obstacle one and two dimensional convolutional neural networks (1D convnets and 2D convnets) are utilized. The 1D convnet decides based on temporal sequences while the 2D convnet uses synthetic images created with those sequences. In order to increase the available training data a cognitive user simulator is used that implements a generative optimization algorithm to simulate behavioural data. Four ensemble-based systems are implemented, one each for 5, 10, 15 and 20 game rounds. The first two are based on 1D and the other two on 2D convnets. Each system consists of multiple models voting for the final prediction. The accuracies of these systems in the order of the number of rounds are 72.5%, 82.5%, 80% and 85%.