Leveraging Transfer Learning from SkimLit to Improve Sentence Classification in NSF Grants and Awards Abstracts

Transfer learning has emerged as a pivotal technique in natural language processing, enabling the reuse of knowledge learned from one domain to enhance performance in another. In this study, we explore the application of transfer learning to the domain of research abstracts, specifically focusing on the task of sentence classification. Leveraging the PubMed 200k RCT dataset introduced by Franck Dernoncourt and Ji Young Lee, this study investigates the potential for transferring knowledge from machine learning models generated from this dataset to improve the classification of manually collected abstracts from NSF's grants and awards submissions. Our research demonstrates that transfer learning from the PubMed 200k RCT dataset model enhances the performance of a machine learning model when applied to the new dataset of NSF abstracts. The model trained on the combined knowledge from both datasets outperforms a model trained solely on the NSF dataset. The accuracy in classification improved by 28.36% for a transfer learning model with one re-trained layer and improved by 51.07% for a model with two re-trained layers. The time-to-run for the transfer models demonstrated significant improvement as well, while there were only minimal improvements in loss. The observed improvements in classification accuracy and predictive capability demonstrate the effectiveness of knowledge transfer between abstracts of different domains. Through this research, we provide valuable insights into the benefits of transfer learning for sentence classification tasks. Our findings not only validate the utility of pre-existing domain-specific datasets but also highlight the potential for broader application of transfer learning techniques in medical research and document classification. This study contributes to the growing field aimed at utilizing the power of transfer learning in text analysis and classification.