Loading [MathJax]/extensions/MathMenu.js
Learning to Reduce Scale Differences for Large-Scale Invariant Image Matching | IEEE Journals & Magazine | IEEE Xplore

Learning to Reduce Scale Differences for Large-Scale Invariant Image Matching


Abstract:

Most image matching methods perform poorly when encountering large scale changes in images. To solve this problem, we propose a Scale-Difference-Aware Image Matching meth...Show More

Abstract:

Most image matching methods perform poorly when encountering large scale changes in images. To solve this problem, we propose a Scale-Difference-Aware Image Matching method (SDAIM) that reduces image scale differences before local feature extraction, via resizing both images of an image pair according to an estimated scale ratio. In order to accurately estimate the scale ratio for the proposed SDAIM, we propose a Covisibility-Attention-Reinforced Matching module (CVARM) and then design a novel neural network, termed as Scale-Net, based on CVARM. The proposed CVARM can lay more stress on covisible areas within the image pair and suppress the distraction from those areas visible in only one image. Quantitative and qualitative experiments confirm that the proposed Scale-Net has higher scale ratio estimation accuracy and much better generalization ability compared with all the existing scale ratio estimation methods. Further experiments on image matching and relative pose estimation tasks demonstrate that our SDAIM and Scale-Net are able to greatly boost the performance of representative local features and state-of-the-art local feature matching methods.
Page(s): 1335 - 1348
Date of Publication: 28 September 2022

ISSN Information:

Funding Agency:

Author image of Yujie Fu
National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, Beijing, China
School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China
Yujie Fu (Graduate Student Member, IEEE) received the B.S. degree in automation from Northeastern University, Shenyang, China, in 2019. He is currently pursuing the Ph.D. degree in pattern recognition and intelligent system under the supervision of Prof. Yihong Wu with the National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, and the School of Artificial Intelligence, University...Show More
Yujie Fu (Graduate Student Member, IEEE) received the B.S. degree in automation from Northeastern University, Shenyang, China, in 2019. He is currently pursuing the Ph.D. degree in pattern recognition and intelligent system under the supervision of Prof. Yihong Wu with the National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, and the School of Artificial Intelligence, University...View more
Author image of Pengju Zhang
National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, Beijing, China
Pengju Zhang received the B.S. degree from China University of Petroleum in 2015, the M.S. degree from Beihang University in 2018, and the Ph.D. degree from the School of Artificial Intelligence, University of Chinese Academy of Sciences, in 2022. He is currently an Assistant Researcher with the National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, Beijing, China. His research i...Show More
Pengju Zhang received the B.S. degree from China University of Petroleum in 2015, the M.S. degree from Beihang University in 2018, and the Ph.D. degree from the School of Artificial Intelligence, University of Chinese Academy of Sciences, in 2022. He is currently an Assistant Researcher with the National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, Beijing, China. His research i...View more
Author image of Bingxi Liu
School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China
Bingxi Liu received the B.S. degree from the China University of Mining and Technology. He is currently pursuing the M.S. degree with the School of Artificial Intelligence, University of Chinese Academy of Sciences. His research interests include SLAM, dense reconstruction, and visual localization.
Bingxi Liu received the B.S. degree from the China University of Mining and Technology. He is currently pursuing the M.S. degree with the School of Artificial Intelligence, University of Chinese Academy of Sciences. His research interests include SLAM, dense reconstruction, and visual localization.View more
Author image of Zheng Rong
National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, Beijing, China
Zheng Rong received the B.S. degree in information engineering and the Ph.D. degree in electronics science and technology from the Beijing Institute of Technology, China, in 2010 and 2017, respectively. He was a Visiting Scholar at the Robotics Institute, Carnegie Mellon University, USA. He is currently an Assistant Researcher with the National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of...Show More
Zheng Rong received the B.S. degree in information engineering and the Ph.D. degree in electronics science and technology from the Beijing Institute of Technology, China, in 2010 and 2017, respectively. He was a Visiting Scholar at the Robotics Institute, Carnegie Mellon University, USA. He is currently an Assistant Researcher with the National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of...View more
Author image of Yihong Wu
National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, Beijing, China
School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China
Yihong Wu received the Ph.D. degree in applied mathematics from the Mathematics Mechanization of Research Center, Institute of Systems Science, Chinese Academy of Sciences, Beijing, China, in 2001. She is currently a Professor with National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences. Her research interests include image matching, camera calibration, camera pose determination, 3...Show More
Yihong Wu received the Ph.D. degree in applied mathematics from the Mathematics Mechanization of Research Center, Institute of Systems Science, Chinese Academy of Sciences, Beijing, China, in 2001. She is currently a Professor with National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences. Her research interests include image matching, camera calibration, camera pose determination, 3...View more

I. Introduction

Establishing pixel-level correspondences between two images is an essential basis for a wide range of computer vision tasks such as visual localization [1], [2], 3D scene reconstruction [3] and simultaneous localization and mapping (SLAM) [4]. Such correspondences are usually estimated by sparse local feature extraction and matching [5], [7], [8], [9], [10], [11], [12], [13], [14], [15], [16]. A local feature consists of a keypoint and a descriptor. But the scale invariance of both existing keypoint detectors and descriptors is not enough to deal with large scale changes [17]. Few inlier correspondences can be established by matching local features under the circumstances of large scale changes in images, which is called as the scale problem of local features in this paper. If the scale difference between two images is small, we call that the two images are at related scale levels in scale space [18], [19].

Author image of Yujie Fu
National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, Beijing, China
School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China
Yujie Fu (Graduate Student Member, IEEE) received the B.S. degree in automation from Northeastern University, Shenyang, China, in 2019. He is currently pursuing the Ph.D. degree in pattern recognition and intelligent system under the supervision of Prof. Yihong Wu with the National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, and the School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China. His research interests include image matching and visual localization.
Yujie Fu (Graduate Student Member, IEEE) received the B.S. degree in automation from Northeastern University, Shenyang, China, in 2019. He is currently pursuing the Ph.D. degree in pattern recognition and intelligent system under the supervision of Prof. Yihong Wu with the National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, and the School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China. His research interests include image matching and visual localization.View more
Author image of Pengju Zhang
National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, Beijing, China
Pengju Zhang received the B.S. degree from China University of Petroleum in 2015, the M.S. degree from Beihang University in 2018, and the Ph.D. degree from the School of Artificial Intelligence, University of Chinese Academy of Sciences, in 2022. He is currently an Assistant Researcher with the National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, Beijing, China. His research interests include visual localization, image retrieval, and learning based descriptor.
Pengju Zhang received the B.S. degree from China University of Petroleum in 2015, the M.S. degree from Beihang University in 2018, and the Ph.D. degree from the School of Artificial Intelligence, University of Chinese Academy of Sciences, in 2022. He is currently an Assistant Researcher with the National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, Beijing, China. His research interests include visual localization, image retrieval, and learning based descriptor.View more
Author image of Bingxi Liu
School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China
Bingxi Liu received the B.S. degree from the China University of Mining and Technology. He is currently pursuing the M.S. degree with the School of Artificial Intelligence, University of Chinese Academy of Sciences. His research interests include SLAM, dense reconstruction, and visual localization.
Bingxi Liu received the B.S. degree from the China University of Mining and Technology. He is currently pursuing the M.S. degree with the School of Artificial Intelligence, University of Chinese Academy of Sciences. His research interests include SLAM, dense reconstruction, and visual localization.View more
Author image of Zheng Rong
National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, Beijing, China
Zheng Rong received the B.S. degree in information engineering and the Ph.D. degree in electronics science and technology from the Beijing Institute of Technology, China, in 2010 and 2017, respectively. He was a Visiting Scholar at the Robotics Institute, Carnegie Mellon University, USA. He is currently an Assistant Researcher with the National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, Beijing, China. His main research interests include robotics, with a focus on perception with multi-sensor fusion, visual-inertial odometry/SLAM, 3D reconstruction, and embedded systems.
Zheng Rong received the B.S. degree in information engineering and the Ph.D. degree in electronics science and technology from the Beijing Institute of Technology, China, in 2010 and 2017, respectively. He was a Visiting Scholar at the Robotics Institute, Carnegie Mellon University, USA. He is currently an Assistant Researcher with the National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, Beijing, China. His main research interests include robotics, with a focus on perception with multi-sensor fusion, visual-inertial odometry/SLAM, 3D reconstruction, and embedded systems.View more
Author image of Yihong Wu
National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, Beijing, China
School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China
Yihong Wu received the Ph.D. degree in applied mathematics from the Mathematics Mechanization of Research Center, Institute of Systems Science, Chinese Academy of Sciences, Beijing, China, in 2001. She is currently a Professor with National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences. Her research interests include image matching, camera calibration, camera pose determination, 3D reconstruction, and SLAM.
Yihong Wu received the Ph.D. degree in applied mathematics from the Mathematics Mechanization of Research Center, Institute of Systems Science, Chinese Academy of Sciences, Beijing, China, in 2001. She is currently a Professor with National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences. Her research interests include image matching, camera calibration, camera pose determination, 3D reconstruction, and SLAM.View more

References

References is not available for this document.