Dynamic Denoising of Contrastive Learning for GNN-based Node Embedding

Contrastive learning is a powerful learning paradigm that can fully exploit cheap, self-supervised learning signals in the data. Crucial to its success is the choice of the positive and negative pairs, yet they are typically based on simple heuristics, and may generate low-quality or even harmful supervision. How to identify beneficial and harmful contrastive pairs? Most of the existing works make the decision based on the input samples directly, which is a static view and ignores how the learning process would respond to such learning constraints with varying level of quality and feasibility (or difficulty). In this paper, we propose instead to use the solution path of the contrastive learning process itself as a more dynamic guidance. Specifically, we use a step-wise reweighting on the contrastive loss of each sample pair, based on their representations that are being optimized in each step of the back-propagation as an instant feedback. We show that, when the re-weighting function is inversely proportional to the updated sample similarities, we can achieve a dynamic denoising of the contrastive loss, so that those "infeasible" and "accomplished" sample pairs are naturally discounted, while those "achievable yet unfinished" ones naturally emphasized. Our reweighting scheme, which we call DynaDeno (Dynamic Denoising), is a unified framework for continuously improving the allocation of the learning resources in contrastive learning, which takes into account both the different levels of uncertainty with positive and negative sample pairs, as well as how the learning process responds to such constraints. When applied to GNN-based node classification, our approach has demonstrated promising results on widely used benchmark datasets.