Transferrable DP-Adapter Tuning: A Privacy-Preserving Multimodal Parameter-Efficient Fine-Tuning Framework

In recent years, multimodal large-scale pre-trained models have achieved tremendous success and become a milestone in the field of artificial intelligence, demonstrating the effectiveness of the pre-training and fine-tuning paradigm in the multimodal domain. Thus, multimodal pre-trained models have been widely applied in various fields of daily life, including some privacy-sensitive areas such as medical diagnosis, financial analysis, public safety, and social media management. Fine-tuning multimodal pre-trained models to adapt to specific tasks in these fields often requires first acquiring data from these downstream tasks and then conducting fine-tuning training. During this process, sensitive private information may be inadvertently learned and leaked. Therefore, we propose a privacy-preserving multimodal parameter-efficient fine-tuning framework: Transferrable DP-Adapter Tuning (TDPAT). In this framework, the model owner sends a lightweight adapter and a lossy compression emulator to the data owner, who then fine-tunes the adapter on downstream data with the help of the emulator. During the fine-tuning training of the adapter, DP-SGD (Differentially Private Stochastic Gradient Descent) improved based on the ideas of multimodal contrastive learning, is incorporated to achieve differential privacy protection. The fine-tuned adapter is then returned to the model owner, who plugs it into the entire multimodal pre-trained model. The TDPAT framework simultaneously achieves privacy protection in three aspects: data security, model security, and inference security. Moreover, by utilizing the multimodal parameter-efficient method Adapter, it reduces the fine-tuning costs and improves fine-tuning efficiency while ensuring fine-tuning performance.

1. Introduction

With the increasing power of multimodal pre-trained models, they have been widely applied in various multimodal tasks such as visual question answering, image-text generation, and image-text retrieval. Simultaneously, the parameter-efficient fine-tuning method, Adapter, has been extended from unimodal to multimodal contexts. VL-Adapter [1] tested the adapter-based method on multiple image-text and video-text tasks, demonstrating that adapters can efficiently learn fused information from visual and linguistic inputs. CLIP-Adapter [2] showcased the powerful few-shot performance of adapters in data-scarce downstream tasks. Tip-Adapter [3] achieved training-free adaptability for downstream tasks by utilizing a key-value cache to construct the adapter. Multiway-Adapter [4] proposed an innovative framework that enhances modal alignment, improving the transferability of large-scale multimodal models in specific tasks while reducing fine-tuning time and parameter count.