Incorporating Dynamic Temperature Estimation into Contrastive Learning on Graphs

Contrastive learning, a powerful self-supervised learning paradigm, has shown its efficacy in learning embed dings from independent and identically distributed (IID) as well as non-IID data without relying on label information. Since high-quality discriminative embeddings form a rich embedding space, which benefits model performance on downstream tasks, it is necessary to study how to improve the quality of contrastive node embeddings in graph contrastive learning. However, there has been limited research on this area. In this paper, we investigate how to generate high-quality contrastive node embeddings based on an in-depth analysis of graph contrastive losses. Firstly, we propose a novel and effective method, GLATE, for estimating the temperatures in three mainstream graph contrastive losses during the training phase. Secondly, we conduct the derivation of GLATE, and the derivation results reveal the specific relationship between the quality of contrastive node embeddings and tem-peratures. Finally, the extensive experiments on 16 benchmark datasets demonstrate that GLATE consistently outperforms the state-of-the-art graph contrastive learning models in terms of both model performance and training efficiency.

I. Introduction

Self-supervised learning provides a promising learning paradigm without relying on high-cost label information for many research fields such as computer vision [1]–[3], natural language processing [4]–[7], speech recognition [8]–[10], and recommender systems [11]–[13]. Contrastive-based methods have a prominent place among the landscape of self-supervised learning methods [14]–[17]. Contrastive learning leverages the inherent structure and relationships within unlabeled data to train encoder networks [18]–[20]. The core idea behind contrastive learning is to map positives (i.e., positive samples) closer in the embedding space while pushing negatives (i.e., negative samples) apart. This process encourages the encoder network to capture intricate patterns, semantic relationships, and underlying structures present in the data, making it particularly adept at learning useful embeddings from diverse and complex datasets. Recently, researchers have explored the graph contrastive learning (GCL) framework for self-supervised learning on graphs [21]–[25]. On benchmark datasets, the state-of-the-art GCL models have demonstrated competitive performance against supervised learning models, e.g., graph convolutional network (GCN) [26], in various graph-related tasks such as node classification [27]–[31], graph classification [32], [33], and link prediction [34]–[37].