Accelerating Self-Supervised Learning via Efficient Training Strategies

Recently the focus of the computer vision community has shifted from expensive supervised learning towards self-supervised learning of visual representations. While the performance gap between supervised and self-supervised has been narrowing, the time for training self-supervised deep networks remains an order of magnitude larger than its supervised counterparts, which hinders progress, imposes carbon cost, and limits societal benefits to institutions with substantial resources. Motivated by these issues, this paper investigates reducing the training time of recent self-supervised methods by various model-agnostic strategies that have not been used for this problem. In particular, we study three strategies: an extendable cyclic learning rate schedule, a matching progressive augmentation magnitude and image resolutions schedule, and a hard positive mining strategy based on augmentation difficulty. We show that all three methods combined lead up to 2.7 times speed-up in the training time of several self-supervised methods while retaining comparable performance to the standard self-supervised learning setting.

1. Introduction

Learning representations without manual human annotations that can be successfully transferred to various downstream tasks has been a long standing goal in machine learning [17], [48]. Self-supervised learning (SSL) aims at learning such representations discriminatively through pretext tasks such as identifying the relative position of image patches [14] and solving jigsaw puzzles [36]. The recent success of SSL methods [23], [6], [8] builds on contrastive learning where the representations are in a latent space invariant to various image transformations such as cropping, blurring and colour jittering. Contrastive learned representations have been shown to obtain on par performance with their supervised counterparts when transferred to various vision tasks including image classification, object detection, semantic segmentation [7], [22], and extended to medical imaging [1] as well as multi-view [45] and multi-modal learning [37].