Cost-Effective Tweet Classification through Transfer Learning in Low-Resource NLP Settings

Recently, pre-trained Transformer-based Language Models (TLM) for specific task has led to significant advancements in Natural Language Processing (NLP) tasks, achieving state-of-the-art results. Despite the ability of TLM and its benefits for low-resource languages where labelled data is scarce. Limited research has been conducted on investigating the applicability of TLM in low-resource settings for French tweets classification. In this paper, our focus is on exploring the effectiveness of Transfer Learning using fine-tuned Transformer language models, specifically FlauBERT and XLM-RoBERTa applied to French tweet dataset. We investigate their ability to achieve higher performance in text classification tasks while requiring less training data compared to traditional text classification methods.Experiments conducted on our labeled dataset consisting of 4000 French tweets reveal that FlauBERT outperforms its rival, achieving an average F1-score of 0.84 compared to XLM-R’s 0.79. However, in scenarios with even fewer available data, XLM-R has the potential to surpass FlauBERT.Furthermore, the environmental sustainability is considered while highlighting the potential CO
2
emissions reduction through manipulating the layers of FlauBERT without compromising the performance.