ISFP: Iterative Simultaneous Optical Flow Estimation and Interest Points Extraction Network

In this paper, we propose ISFP, an end-to-end deep learning method for robust feature point extraction and optical flow estimation simultaneously. We observe that the structure of the deep network for feature point extraction bears similarities with the feature extraction network used in optical flow. Leveraging this observation, we merge these two tasks into a unified approach, enabling the extraction of feature points and their corresponding optical flow displacements in an end-to-end manner. ISFP serves as a comprehensive deep learning solution for feature point tracking in monocular visual SLAM systems, seamlessly replacing the front-end tasks of feature point extraction and inter-frame tracking. Our network utilizes a pre-trained interest point generation network to automatically generate interest point labels on an annotated optical flow dataset, thus creating a new dataset with ground truth interest points for training. Extensive experiments demonstrate the effectiveness and practical applicability of our approach in real-world scenarios. Compared with RAFT, our method is trained on a single FlyingChairs data set. It improves by 0.3 on the Sintel final data set, maintains comparable accuracy in clean, and the inference speed is 1.5 times that of RAFT.