Positional Tracking of Physical Objects in an Augmented Reality Environment Using Neuromorphic Vision Sensors

Immersive systems such as Augmented Reality (AR) devices are being integrated across multiple domains, from manufacturing to health, environment and education. The Industry 5.0 focus on human-centric interfaces to these systems and the desire to enable them for enhancing the skill-sets of the user, present new challenges. One of these is the accurate positional tracking of objects within the AR user's environment that is relevant in real-time applications. Multiple co-located frame based cameras are typical solutions proposed for object tracking in AR. The advent of commercial neuromorphic vision sensors (NVS) in recent years, introduce a novel, event-based approach to computer vision. Inspired by biological vision, these systems reduce the amount of data transferred and enhance power efficiency when tracking in real-time. This work leverages the strengths of AR and NVS technologies to develop an integrated system for tracking physical objects and providing users with required information to interact with these objects in a virtual environment. The computational pipeline from training on a deep neural network, coordinate transformations from image, to physical and virtual coordinate systems and testing the utility of the system for an AR system user is presented. The NVS system shows significant improvement over frame camera based systems in responding to varied lighting conditions.

I. Introduction

The detection and classification of objects in motion is a classical problem in many fields ranging across health, manufacturing, surveillance, and autonomous transport applications. Traditionally, this problem is addressed from an image processing perspective, where images rendered from various sensing modalities are processed using algorithms that differentiate background and foreground features [1]–[3]. Additionally, the idea of using a system that combines different sensor technologies to tackle this issue has been explored for intelligent vehicles [4], [5] and robotic systems [6].