Loading [MathJax]/extensions/MathMenu.js
MFT-GAN: A Multiscale Feature-Guided Transformer Network for Unsupervised Hyperspectral Pansharpening | IEEE Journals & Magazine | IEEE Xplore

MFT-GAN: A Multiscale Feature-Guided Transformer Network for Unsupervised Hyperspectral Pansharpening


Abstract:

Unsupervised learning, learning data distributions without needing labeled samples, is a particularly promising approach for solving the challenging task of hyperspectral...Show More

Abstract:

Unsupervised learning, learning data distributions without needing labeled samples, is a particularly promising approach for solving the challenging task of hyperspectral pansharpening. Inspired by the above, we introduce an innovative generative adversarial network framework (named MFT-GAN) that incorporates a transformer network and multiscale interaction technology. Specifically, MFT-GAN is constructed of one generator and two discriminators. The generator is composed of a multiscale feature guidance branch (MFGB) and a feature interaction fusion branch (FIFB). The former aims at extracting spectral and spatial information branches from low-resolution hyperspectral (LRHS) and panchromatic (PAN) images by convolutional sampling operations, respectively. The latter accomplishes the interaction of spectral and spatial information at different scales through multiscale interaction technology. Then, the multiscale interaction results are fused by level-by-level convolutional sampling and pixel summation operations to generate high-resolution hyperspectral (HRHS) images. The discriminators are composed of a spectral transformer discriminator and a spatial transformer discriminator, designed to maintain structural and parametric balance and learn long-range spectral and spatial correlation relationships. In addition, hybrid loss functions are used to complete the adversarial training of the MFT-GAN to improve its performance. The experiments on simulated and real datasets further demonstrated the effectiveness of the proposed MFT-GAN method.
Article Sequence Number: 5518516
Date of Publication: 16 May 2024

ISSN Information:

Funding Agency:

Author image of Yanli Shang
Jiangsu Provincial Engineering Laboratory for Pattern Recognition and Computational Intelligence, Jiangnan University, Wuxi, China
Yanli Shang received the B.S. degree in network engineering from Anyang Institute of Technology, Anyang, China, in 2020, and the M.S. degree in computer technology from Jiangnan University, Wuxi, China, in 2023. She is currently pursuing the Ph.D. degree in electronic information with the School of Electronic Information, Wuhan University, Wuhan, China.
Her research interests include hyperspectral image fusion and deep lea...Show More
Yanli Shang received the B.S. degree in network engineering from Anyang Institute of Technology, Anyang, China, in 2020, and the M.S. degree in computer technology from Jiangnan University, Wuxi, China, in 2023. She is currently pursuing the Ph.D. degree in electronic information with the School of Electronic Information, Wuhan University, Wuhan, China.
Her research interests include hyperspectral image fusion and deep lea...View more
Author image of Jianjun Liu
Jiangsu Provincial Engineering Laboratory for Pattern Recognition and Computational Intelligence, Jiangnan University, Wuxi, China
Jianjun Liu (Member, IEEE) received the B.S. degree in applied mathematics and the Ph.D. degree in pattern recognition and intelligence system from Nanjing University of Science and Technology, Nanjing, China, in 2009 and 2014, respectively.
From 2018 to 2020, he was a Post-Doctoral Researcher with the Department of Electrical Engineering, City University of Hong Kong, Hong Kong, China. He is currently an Associate Profess...Show More
Jianjun Liu (Member, IEEE) received the B.S. degree in applied mathematics and the Ph.D. degree in pattern recognition and intelligence system from Nanjing University of Science and Technology, Nanjing, China, in 2009 and 2014, respectively.
From 2018 to 2020, he was a Post-Doctoral Researcher with the Department of Electrical Engineering, City University of Hong Kong, Hong Kong, China. He is currently an Associate Profess...View more
Author image of Jingyi Zhang
Jiangsu Provincial Engineering Laboratory for Pattern Recognition and Computational Intelligence, Jiangnan University, Wuxi, China
Jingyi Zhang received the B.S. degree in computer science and technology from Henan Polytechnic University, Jiaozuo, China, in 2021. She is currently pursuing the master’s degree in electronic information with the School of Artificial Intelligence and Computer Science, Jiangnan University, Wuxi, China.
Her research interests include deep learning and hyperspectral image fusion.
Jingyi Zhang received the B.S. degree in computer science and technology from Henan Polytechnic University, Jiaozuo, China, in 2021. She is currently pursuing the master’s degree in electronic information with the School of Artificial Intelligence and Computer Science, Jiangnan University, Wuxi, China.
Her research interests include deep learning and hyperspectral image fusion.View more
Author image of Zebin Wu
School of Computer Science, Nanjing University of Science and Technology, Nanjing, China
Zebin Wu (Senior Member, IEEE) received the B.S. and Ph.D. degrees in computer science from Nanjing University of Science and Technology, Nanjing, China, in 2003 and 2008, respectively.
He is currently a Professor with the School of Computer Science, Nanjing University of Science and Technology. His research interests include hyperspectral image processing, high-performance computing, and computer simulation.
Zebin Wu (Senior Member, IEEE) received the B.S. and Ph.D. degrees in computer science from Nanjing University of Science and Technology, Nanjing, China, in 2003 and 2008, respectively.
He is currently a Professor with the School of Computer Science, Nanjing University of Science and Technology. His research interests include hyperspectral image processing, high-performance computing, and computer simulation.View more

I. Introduction

Hyperspectral pansharpening [1] is a technique that produces high-resolution hyperspectral (HRHS) images by integrating low-resolution hyperspectral (LRHS) images with panchromatic (PAN) images. Remote sensing imaging systems can provide only high spatial resolution images (e.g., PAN images) or high spectral resolution images (e.g., LRHS images), driving the development of sharpening techniques. Moreover, the information contained in images can be improved by combining hyperspectral information from LRHS images with high-resolution PAN information from PAN images, which helps analyze and identify target features more accurately. Pansharpening has contributed to many fields, such as classification [2], [3], [4], unmixing [5], [6], [7], [8], and detection [9], [10], [11].

Author image of Yanli Shang
Jiangsu Provincial Engineering Laboratory for Pattern Recognition and Computational Intelligence, Jiangnan University, Wuxi, China
Yanli Shang received the B.S. degree in network engineering from Anyang Institute of Technology, Anyang, China, in 2020, and the M.S. degree in computer technology from Jiangnan University, Wuxi, China, in 2023. She is currently pursuing the Ph.D. degree in electronic information with the School of Electronic Information, Wuhan University, Wuhan, China.
Her research interests include hyperspectral image fusion and deep learning.
Yanli Shang received the B.S. degree in network engineering from Anyang Institute of Technology, Anyang, China, in 2020, and the M.S. degree in computer technology from Jiangnan University, Wuxi, China, in 2023. She is currently pursuing the Ph.D. degree in electronic information with the School of Electronic Information, Wuhan University, Wuhan, China.
Her research interests include hyperspectral image fusion and deep learning.View more
Author image of Jianjun Liu
Jiangsu Provincial Engineering Laboratory for Pattern Recognition and Computational Intelligence, Jiangnan University, Wuxi, China
Jianjun Liu (Member, IEEE) received the B.S. degree in applied mathematics and the Ph.D. degree in pattern recognition and intelligence system from Nanjing University of Science and Technology, Nanjing, China, in 2009 and 2014, respectively.
From 2018 to 2020, he was a Post-Doctoral Researcher with the Department of Electrical Engineering, City University of Hong Kong, Hong Kong, China. He is currently an Associate Professor with the School of Artificial Intelligence and Computer Science, Jiangnan University, Wuxi, China. His research interests include hyperspectral image classification, super-resolution, spectral unmixing, sparse representation, computer vision, and pattern recognition.
Jianjun Liu (Member, IEEE) received the B.S. degree in applied mathematics and the Ph.D. degree in pattern recognition and intelligence system from Nanjing University of Science and Technology, Nanjing, China, in 2009 and 2014, respectively.
From 2018 to 2020, he was a Post-Doctoral Researcher with the Department of Electrical Engineering, City University of Hong Kong, Hong Kong, China. He is currently an Associate Professor with the School of Artificial Intelligence and Computer Science, Jiangnan University, Wuxi, China. His research interests include hyperspectral image classification, super-resolution, spectral unmixing, sparse representation, computer vision, and pattern recognition.View more
Author image of Jingyi Zhang
Jiangsu Provincial Engineering Laboratory for Pattern Recognition and Computational Intelligence, Jiangnan University, Wuxi, China
Jingyi Zhang received the B.S. degree in computer science and technology from Henan Polytechnic University, Jiaozuo, China, in 2021. She is currently pursuing the master’s degree in electronic information with the School of Artificial Intelligence and Computer Science, Jiangnan University, Wuxi, China.
Her research interests include deep learning and hyperspectral image fusion.
Jingyi Zhang received the B.S. degree in computer science and technology from Henan Polytechnic University, Jiaozuo, China, in 2021. She is currently pursuing the master’s degree in electronic information with the School of Artificial Intelligence and Computer Science, Jiangnan University, Wuxi, China.
Her research interests include deep learning and hyperspectral image fusion.View more
Author image of Zebin Wu
School of Computer Science, Nanjing University of Science and Technology, Nanjing, China
Zebin Wu (Senior Member, IEEE) received the B.S. and Ph.D. degrees in computer science from Nanjing University of Science and Technology, Nanjing, China, in 2003 and 2008, respectively.
He is currently a Professor with the School of Computer Science, Nanjing University of Science and Technology. His research interests include hyperspectral image processing, high-performance computing, and computer simulation.
Zebin Wu (Senior Member, IEEE) received the B.S. and Ph.D. degrees in computer science from Nanjing University of Science and Technology, Nanjing, China, in 2003 and 2008, respectively.
He is currently a Professor with the School of Computer Science, Nanjing University of Science and Technology. His research interests include hyperspectral image processing, high-performance computing, and computer simulation.View more
Contact IEEE to Subscribe

References

References is not available for this document.