Multi-sensor Energy Efficient Obstacle Detection

With the improvement in technology, both the cost and the power requirement of cameras, as well as other sensors have come down significantly. It has allowed these sensors to be integrated into portable as well as wearable systems. Such systems are usually operated in a hands-free and always-on manner where they need to function continuously in a variety of scenarios. In such situations, relying on a single sensor or a fixed sensor combination can be detrimental to both performance as well as energy requirements. Consider the case of an obstacle detection task. Here using an RGB camera helps in recognizing the obstacle type but takes much more energy than an ultrasonic sensor. Infrared cameras can perform better than RGB camera at night but consume twice the energy. Therefore, an efficient system must use a combination of sensors, with an adaptive control that ensures the use of the sensors appropriate to the context. In this adaptation, one needs to consider both performance and energy and their trade-off. In this paper, we explore the strengths of different sensors as well their trade-off for developing a deep neural network based wearable device. We choose a specific case study in the context of a mobility assistance device for the visually impaired. The device detects obstacles in the path of a visually impaired person and is required to operate both at day and night with minimal energy to increase the usage time on a single charge. The device employs multiple sensors: ultrasonic sensor, RGB Camera, and NIR Camera along with a deep neural network accelerator for speeding up computation. We show that by adaptively choosing the appropriate sensor for the context, we can achieve up to 90% reduction in energy while maintaining comparable performance to a single sensor system.

I. Introduction

The design of a portable real-time vision system which continuously monitors the surroundings is complex. Such a system must achieve an acceptable level of performance while making sure that the energy consumption is also within limits. Further, due to their always-on nature, these systems go through a lot more variations in the environment than a typical static system, which makes it imperative to employ a variety of sensors to perform efficiently under different conditions. Therefore, the system designers must classify various contexts, and choose the corresponding best performing and least power consuming sensor in that context, to ensure that the performance levels are met while also saving energy. Examples of such systems include robotic systems working in collaboration with humans, factory floor robots, assistive devices for visually impaired and even battery operated autonomous vehicles.