Cross-Modal Matching and Adaptive Graph Attention Network for RGB-D Scene Recognition

Despite the significant advances in RGB-D scene recognition, there are several major limitations that need further investigation. For example, simply extracting modal-specific features neglects the complex relationships among multiple modalities of features. Moreover, cross-modal features have not been considered in most existing methods. To address these concerns, we propose to integrate the tasks of cross-modal matching and modal-specific recognition, termed as Matching-to-Recognition Network (MRNet). Specifically, the cross-modal matching network enhances the descriptive power of the recognition network via a layer-wise semantic loss. The recognition network obtains multi-modal features from a two-stream CNN: global features are obtained by a higher-layer of a CNN to preserve the semantic content, and local layout features are learned by the graph attention network, thus better capturing the key object regions and modelling their relationships. Extensive experiments results demonstrate the MRNet achieves superior performance to state-of-the-art methods, especially for recognition solely based on single modality.

1. INTRODUCTION

Image classification, particularly within the domain of computer vision research, has received substantial attention over the years. Recently excellent performance has been obtained with the development of deep learning methods. A key challenge in this area is how to effectively learn image representations. For general image classification, the category of each image is highly related to the object in the image, as shown in Figure 1. However, the category of the scene image is related to several typical objects and its spatial layout. And the diversity of spatial layouts and object co-occurrences between scenes may lead to intra-class differences and inter-class similarities. This reduces the accuracy of scene recognition. Thus, deep learning methods for common image classification are not well suited for scene classification. To address these problems, we propose a more robust and effective method to extract multi-modal features and model their relationships by a cross-modal matching procedure for scene recognition.