SudaBERT: A Pre-trained Encoder Representation For Sudanese Arabic Dialect

Bidirectional Encoder Representations from Transformers (BERT) has proven to be very efficient at Natural Language Understanding (NLU), as it allows to achieve state-of-the-art results in most NLU tasks. In this work we aim to utilize the power of BERT in Sudanese Arabic dialect, and produce a Sudanese word representation. We collected over 7 million sentences in Sudanese dialect and used them to resume training of the pre-trained Arabic-BERT, as it was trained on large Modern Standard Arabic (MSA) corpus. Our model -SudaBERT- has achieved better performance on Sudanese Sentiment Analysis, this clarifies that SudaBERT works better in understanding Sudanese Dialectic which is the domain we are interested in.