Real-Time Visual Inertial Odometry with a Resource-Efficient Harris Corner Detection Accelerator on FPGA Platform

Visual Inertial Odometry (VIO) is a widely studied localization technique in robotics. State-of-the-art VIO algorithms are composed of two parts: a frontend which performs visual perception and inertial measurement pre-processing, and a backend which fuses vision and inertial measurements to estimate the robot's pose. Both image processing in the frontend and sensor fusion in the backend are computationally expensive, making it very challenging to run the VIO algorithm, especially the optimization-based VIO algorithm in real time on embedded platforms with limited power budget. In this paper, a real-time optimization-based monocular VIO algorithm is proposed based on algorithm-and-hardware co-design and successfully implemented on an embedded platform with only 2.6W processor power consumption. In particular, the time-consuming Harris corner detection (HCD) is accelerated on Field Programmable Gate Array (FPGA), achieving an average 16 × processing time reduction compared with the ARM implementation. Compared with the state-of-the-art HCD accelerator provided by Xilinx, the hardware resource required of our accelerator is largely reduced without any compromise in speed, thanks to the proposed dedicated pruning and paral-lelization techniques. Finally, experiment on the public dataset demonstrates that the proposed real-time VIO algorithm on the FPGA-based platform has comparable accuracy with respect to the existing state-of-the-art VIO algorithm on the desktop, and 3 × faster frontend processing speed over the ARM-based implementation.