MIMIC III Text classification with the generalization of BERT transformer model synergized with XGBoost classifier

Natural Language Processing (NLP) is a field of machine learning in which machine learning models are trained to analyze and synthesize natural language and speech. Text classification is a popular application in NLP. Previous text classification algorithms that were implemented include the usage of the Naive Bayes Classifier and Recurrent Neural Networks (RNN). The introduction of Transformer based Neural Network architecture, which uses Attention heads to understand the text and its context, has shown state-of-the-art results in various NLP tasks. With the development of Artificial Intelligence (AI) in the medical field, many AI techniques and models have also developed, and the need for good performance is crucial in the medical field. One of the medical applications of text-based AI is classifying Electronic Health Records (EHRs). The data-set we have chosen is the Note events data-set from the MIMIC III (Medical Information Mart for Intensive Care) database. In this work, we are introducing an integrated framework that uses a BERT based transformer in synergy with an XGBoost classifier in its tail to perform text classification, which theoretically and practically gives more efficient results. This novel synergy of XGBoost with the BERT model resulted in a significant boost in test classification accuracy compared to using the BERT model alone. Specifically, the proposed hybrid model achieved an excellent improvement factor by boosting the classification performance of the training accuracy metric by a factor approaching 10 to 13% and the testing accuracy by a factor of 3%.