The Phenomenon of Correlated Representations in Contrastive Learning

Contrastive learning is widely considered to be an important domain of machine learning. Its main premise is to use contrasting samples of data in order to learn common features that allow for accurate data clustering in a representation space. The generation of such a representation or embedding space can be of great value, for instance for re-identification tasks. While working on one such task, we noticed that, paradoxically, decreasing training data resolution led to a considerably higher re-identification accuracy. Upon further analysis, we discovered that this occurs since during training, highly correlated features are learned that are ultimately redundant and limit the model’s performance. This effect is exaggerated with the use of high-resolution data, therefore eventually decreasing the obtained re-identification accuracy. In this contribution, we characterize this phenomenon, which we believe to be a novel problem in contrastive learning. We study the effects of various changes to a common neural network architecture on this phenomenon, linking it to the concept of bias, and propose a set of solutions to mitigate its effect.