LURAT: a Lightweight Unsupervised Automatic Readability Assessment Toolkit for Second Language Learners

In second language acquisition, assessing the readability of texts is essential for many educational applications. Hence, the development of artificial intelligence tools to automatically assess readability with little or no human supervision is required owing to the high cost of manual readability labeling by educational experts, who must carefully read and assess the texts to perform this task. Prior unsupervised approaches have manually searched textual features that correlate well with readability labels, such as perplexity scores of large language models. However, these features do not capture the language knowledge of second language learners as the target users. To this end, we propose a novel unsupervised approach that captures their knowledge. Our key idea is to establish word difficulty features that accurately capture the language knowledge of second language learners. We analyze vocabulary test results to obtain and utilize the probability that a typical language learner knows a given word. By sacrificing syntactically complicated textual features, our approach enables lightweight classifiers that reflect learners’ vocabulary knowledge. In the experiments, our assessor, which was trained on vocabulary tests without costly readability labels, outperformed the perplexity-based assessors of large neural language models.