Automatic True/False Question Generation using Template-based Framework

Indonesia is a developing country that is experiencing developments in the field of science and technology. To support this development, it is also necessary to improve Indonesian students’ way of thinking. In this paper, we propose a model of generating true/false questions automatically using a template-based framework so that true/false questions can be used as a method of evaluate the way of thinking of Indonesian students. We evaluate the model with qualitative and quantitative testing methods. We use 3 criteria on qualitative testing: fluency, relevance, and answerability. The results of human evaluation for model quality experiments show that the majority (>%) assess the model output to be in the range between ‘okay’ and ‘good’ for fluency, relevance, and answerability. On the other hand, quantitative evaluation results show that the model we made was able to generate up to 3 questions per input sentence, but the number of questions generated by the methods in the template-based framework model in this paper differs significantly. Specifically, the synonym/antonym substitution method in the model generates an excessive number of questions, while negation modification method generates an insufficient number of questions. We analyzed that there is still ample room for improvement especially in terms of the fluency of the generated questions.