BnGraphemizer: A Grapheme-based Tokenizer for Bengali Handwritten Text Recognition

Bengali Handwritten Text Recognition (HTR) is a challenging task due to the text's cursive disposition and significant variations in the graphemic unit, the smallest written unit in alphasyllabary languages. These graphemic units, or simply graphemes, may consist of multiple characters and diacritics, and the order of characters or diacritics between textual and visual representations is only sometimes preserved. Moreover, Bengali conjunct characters, a subset of the graphemes, often have unique visual representations vastly different from the characters they comprise. Because of these inherent characteristics, a character-based tokenizer fails to tokenize the label text appropriately for BHTR in a supervised setting, making it challenging to adapt the existing sequence-to-sequence frameworks often used for Handwritten Text Recog-nition (HTR). To address this, this paper proposes a trie- based text tokenizer (called BnGraphemizer) that allows one to tokenize text into a set of graphemes instead of characters. Experimental investigations show that the BnGraphemizer-based BHTR system performs better regarding higher Word Recognition Rate (WRR) than its character-based counterpart in two of three Bengali handwritten datasets.