Analysis of Subword Tokenization Approaches for Turkish Language

Various tokenization approaches have been proposed in natural language processing research. These approaches have further evolved from character- and word-level representations to subword-level representations. However, the impact of tokenizations on model performance has not been thoroughly discussed, especially for morphologically rich languages. In this paper, we comprehensively analyze subword tokenizers for Turkish which is a highly inflected and morphologically rich language and we propose a morphologically-based approach. Also, we examine how the tokenizer parameters like vocabulary and corpus sizes change the characteristics of tokenizers.