Pixel Processor Arrays For Low Latency Gaze Estimation

We demonstrate the use of a Pixel Processor Array (PPA) vision sensor for achieving rates of over 10,000 Hz, with processing latency below 0.1 ms, in a gaze tracking application, performing image processing directly upon the sensor itself with minimal external computation. Each pixel processing element of the PPA sensor is capable of light sensing, data storage and computation, allowing various visual processing to be performed efficiently at the point of light capture. We demonstrate how information such as pupil location, size, and LED reflections upon the eye's surface can be extracted by the PPA, along with a simple gaze tracker using pupil location. By extracting such information directly on-sensor, the data needing to be transferred from sensor to external processing is reduced from entire images to a hand-full of contextual bytes, this provides significant saving in terms of power, time and allows for frame-rates far exceeding traditional camera sensors.

1 Introduction and Overview

When developing for mobile platforms, constraints such as compu-tational power and battery life often provide significant bottlenecks to overcome. Mobile Virtual Reality (VR) and Augmented Reality (AR) headsets are a particularly challenging area as visual rendering and 6DOF tracking tasks must be performed rapidly and consistently in order to provide a comfortable user experience, often placing hard constraints on the fidelity and complexity of content developed for such devices. It is speculated that gaze tracking is poised to become a core technology in VR and AR devices, as not only does it enable interaction via gaze leading to a more immersive experience, but also can be utilized to achieve significantly more efficient rendering via techniques such as foveated rendering [1].