Word and Character Semantic Fusion by Pretrained Language Models for Text Classification

The utilization of fine-tuned pre-trained language models (PLMs) in text classification has become widespread and has achieved remarkable performance. However, a significant limitation of these models is that they tend to mark special symbols that are not registered in PLMs as [UNK], which ultimately reduces the text classification accuracy. Although some PLMs have attempted to overcome this limitation by learning subwords or using character input during model training, effectively fusing subword and character features remains a challenge. To address this challenge, we propose a hybrid network architecture called SemFusion that collaborates two PLMs to learn rich semantic information and improve classification accuracy. Specifically, we use two different Transformer Encoders to extract features for each subword and character in the text sequence. These features include the [CLS] feature, which represents the overall meaning of the text sequence, and the token feature, which represents each subword or character. We then utilize a multi-head self-attention mechanism to model the correlation between text sequences at different granularities for enhancing semantic expression, thereby achieving more accurate text classification. Our experimental results on four benchmark text datasets demonstrate that our proposed method outperforms the current state-of-the-art methods in the literature.