Combination and Knowledge Extension of Pre-trained Language Model for Offensive Language Detection

Nowadays, more and more offensive comments are posted on social media. Those offensive comments can seriously cause mental damage to other people. Therefore, those toxic and harmful comments should be detected timely and accurate. In this paper, we exploit the powerful pre-train language models (PLM) in detecting offensive language. We consider two PLMs in our paper: BERT and DeepMoji. We finetune their combination and evaluate the performance with two datasets: Ask FM and Curious Cat. We found that DeepMoji outperforms BERT. We analyze the result from two aspects and conclude that the task and data of pre-training are important to PLM. Seeing that BERT is less effective than DeepMoji, it is possible to improve the performance of BERT. We then propose a “Knowledge Extension” method to improve the performance of the BERT model. We find that the performance of PLM with high-quality extensional knowledge can be improved significantly.