Breaking of Brightness Consistency in Optical Flow With a Lightweight CNN Network

The sparse optical flow method is a fundamental task in computer vision. However, its reliance on the assumption of constant environmental brightness constrains its applicability in high dynamic range (HDR) scenes. In this study, we propose a novel approach aimed at transcending image color information by learning a feature map that is robust to illumination changes. This feature map is subsequently structured into a feature pyramid and integrated into sparse Lucas-Kanade (LK) optical flow. By adopting this hybrid optical flow method, we circumvent the limitation imposed by the brightness constant assumption. Specifically, we utilize a lightweight network to extract both the feature map and keypoints from the image. Given the challenge of obtaining reliable keypoints for the shallow network, we employ an additional deep network to support the training process. Both networks are trained using unsupervised methods. The proposed lightweight network achieves a remarkable speed of 190 fps on the onboard CPU. To validate our approach, we conduct comparisons of repeatability and matching performance with conventional optical flow methods under dynamic illumination conditions. Furthermore, we demonstrate the effectiveness of our method by integrating it into VINS-Mono, resulting in a significantly reduced translation error of 93% on a public HDR dataset.