Mask Consistency and Contrast Regularization for Prompt-Based Learning

This paper presents a new contrastive semi-supervised learning approach to data efficient language model where the fine-tuned mask language model is constructed with hard prompts for sentence representation. Importantly, the prompt-based pseudo labeling is merged in a mask language model (MLM) with data augmentation through a contrast loss which is utilized to pull together those similar samples and push apart the samples which do not belong to their augmentations. Furthermore, the mask consistency training is implemented for fine-tuned MLM to pursue the closeness between word prediction from weak and strong augmentations. Accordingly, this study develops a data efficient solution which improves the model generalization and leverages the rich information from unlabeled data for few-shot text classification. The experiments on natural language understanding in few-shot and semi-supervised settings show that the proposed method considerably improves the performance by using the contrastive prompt-based learning and the mask consistency training.