Small, Multilingual, Explainable Transformers for Online Handwriting Decoding

A compact Transformer architecture is shown to provide an effective end-to-end framework for sentence recognition based on online handwritten gestures representing glyph strokes. Despite its reduced size, the encoder exhibits impressive transfer learning capabilities, producing latent representations that can generalise to various target grammars and output vocabularies, while also accurately performing spatio-temporal segmentation tasks. Novel visualization techniques are introduced, generalizable to other architectures and tasks, providing valuable insights and qualitative assessments of the model’s learning. The model is suitable for real-time edge inference and achieves an average Levenshtein accuracy over 96% on all tested languages, even when the encoder is imported frozen from the English language.