Fine-Tashkeel: Fine-Tuning Byte-Level Models for Accurate Arabic Text Diacritization

Most of previous work on learning diacritization of the Arabic language relied on training models from scratch. In this paper, we investigate how to leverage pre-trained language models to learn diacritization. We fine-tune token-free pre-trained multilingual models (ByT5) to learn to predict and insert missing diacritics in Arabic text, a complex task that requires understanding the sentence semantics and the morphological structure of the tokens. We achieve state-of-the-art accuracy on the dia-critization task with minimal amount of training and no feature engineering, reducing WER (word error rate) by 40%. We release our fine-tuned models for the greater benefit of the researchers in the community.