Adapting Complex Event Detection to Perceptual Domain Shifts

Human decision-making, as well as control of autonomous systems, have deployed deep learning models for detecting complex events from unstructured sensory data. However, the strong performance of these models is restricted to events with short intervals of time and space due to the limited context memory of their architectures. Thus, detecting events that transpire over long periods of time with multiple spatially distant sensor sources (known as complex events) remains challenging for these purely neural-based methods, particularly as environmental conditions and object appearances change. In recent years, neurosymbolic approaches have been proposed that use both neural-based perception and symbolic reasoning for capturing complex events. However, these approaches still face issues of adaptation to perceptual domain shift in complex events. We address these problems in the context of a prototype neurosymbolic system called DANCER, which performs Domain Adaptation and Neurosymbolic inference in Complex Event Reasoning. DANCER aims to provide domain adaptation in a post-deployment setting while minimizing runtime user burden for annotation. To enable training and evaluation of DANCER, we also provide a physics-based synthetic sensor data generator to create videos given complex scenario specifications. We evaluate DANCER on a dataset of generated synthetic data. We show that DANCER yields a 48% increase in accuracy of complex event detection using domain adaptation while significantly reducing the annotation time of our synthetic complex events by up to 2.7x, demonstrating DANCER’s ability to effectively detect complex events under perceptual domain shift.

I. Introduction

Many cyber-physical systems today are equipped with rich sensing capabilities transmitting high dimensional unstructured data, and powered by an ever-growing list of AI/ML solutions that process and analyze such data on-device. In the future, these systems must be capable of detecting and reacting to various events of interest occurring in their environment. Typical applications in such systems involve spatiotemporal reasoning between events observed by different sensors in order to coordinate actions by humans, robots, actuated sensors, and other devices. Examples of such applications include urban scenarios and military scenarios, ranging from crimes, coordinated terrorism attacks, and rescue missions.