Transfer Learning Approach to Multilabel Biomedical Literature Classification using Transformer Models

The recently developed transformer models have a significant influence on Natural Language Processing (NLP) research. Transformer-based models have attained innovative results on numerous NLP benchmarks in linguistics. Researchers have also investigated transformer models for a variety of applications in the biomedical field. To facilitate information classification, this work aims to thoroughly investigate four state-of-the-art transformer-based models (BERT, Roberta, SciBERT, and ClinicalBERT). Large language model training, however, requires a lot of time and processing resources. As a result, pre-trained language models like BERT are advantageous because they (1) offer cutting-edge performance, (2) The knowledge gained can be used to do several tasks, such as categorization, and (3) free up practitioners from having to gather enough resources (including the hardware, data and time) to train models. However, pre-trained models possibly will perform poorly in some domains since they are generic. Here, we inspect the case of multilabel classification for biomedical Literature, which is a domain that has received relatively minor attention in the literature assessing pre-trained language models. Based on our study, using SciBERT as the baseline model gave the best results compared to the rest of the models with an 89.35% micro F1 score. In the context of classifying Biomedical Literature, domain-specific pre-trained models tend to perform better compared to generic models.