Co-FT: Effective Fine-tuning for Pretrained Language Model via Adding Collaborator

Dialogue understanding is a common application in many online webs, which recognizes the intents of customers via utterance classification. The state-of-the-art methods finetune a pretrained language model (PTLM) on a task-specific dataset, which is sub-optima because it makes the PTLM forget the knowledge learned in pretraining. To address it, we propose an effective finetuning method (Co-FT) via adding collaborators for PTLM. Instead of finetuning the whole PTLM, we add an extra collaborator utterance encoder (CoUE). The CoUE consists of independent tokenizer and network weights, which can be implemented via rich flexible network architectures. We aggregate the features extracted by PTLM and the CoUE, and then feed them to the intent predictor (a feed-forward classifier) for dialogue intent recognition. Given a domain-specific dataset, we only update the weights of CoUE via a 2-stage training method. Our experiments show that Co-FT outperforms the PTLM finetuning method and achieves 78.43% f1-score. Besides, Co-FT also shows a good capacity in the few-shot learning.