Not All Features Matter: Enhancing Few-shot CLIP with Adaptive Prior Refinement

The popularity of Contrastive Language-Image Pretraining (CLIP) has propelled its application to diverse downstream vision tasks. To improve its capacity on downstream tasks, few-shot learning has become a widelya-dopted technique. However, existing methods either exhibit limited performance or suffer from excessive learnable parameters. In this paper, we propose APE, an Adaptive Prior rEfinement method for CLIP’s pre-trained knowledge, which achieves superior accuracy with high computational efficiency. Via a prior refinement module, we analyze the inter-class disparity in the downstream data and decouple the domain-specific knowledge from the CLIP-extracted cache model. On top of that, we introduce two model variants, a training-free APE and a training-required APE-T. We explore the trilateral affinities between the test image, prior cache model, and textual representations, and only enable a lightweight category-residual module to be trained. For the average accuracy over 11 benchmarks, both APE and APE-T attain state-of-the-art and respectively outperform the second-best by +1.59% and +1.99% under 16 shots with ×30 less learnable parameters. Code is available at https://github.com/yangyangyang127/APE.

1. Introduction

The advent of contrastive visual-language pre-training has provided a new paradigm for multi-modal learning [16], [17], [22], [42]. Its popularity has been observed across diverse downstream vision tasks, including 2D or 3D classification [14], [39], [41], [9], segmentation [27], [48], [36], [44], and detection [38], [45], [29]. CLIP [26] is one of the most acknowledged contrastive visual-language models and has attained widespread attention for its simplicity and superiority. Pretrained by massive image-text pairs sourced from the Internet, CLIP exhibits remarkable aptitude in aligning visionlanguage representations with favorable zero-shot performance on downstream tasks. To further enhance CLIP in low-data regimes, many efforts propose few-shot learning techniques with additional learnable modules upon the frozen CLIP for new semantic domains.