An Interactive Framework of Balancing Evaluation Cost and Prediction Accuracy for Knowledge Tracing

The development of online intelligent educational systems has revolutionized personalized learning, presenting an opportunity for the integration of knowledge tracing (KT). KT is an essential task that leverages students’ historical interactions to model their knowledge states, enabling accurate predictions of their future performance. The application of deep learning models on KT tasks, also known as deep learning based knowledge tracing (DLKT) models, has accelerated the process of KT tasks in recent years. Although DLKT models have achieved promising results, there is a huge challenge to avoid models generating wrong estimations that yield bad guidance in educational contexts. Hence, in this work, we propose a simulation framework to explore the feasibility of minimizing the evaluation cost while guaranteeing prediction performance. In the framework, we initially design a simple yet efficient DLKT model that learns the actual process of students’ knowledge acquisition. We then select the reliable predictions generated by the proposed model and assign the unreliable ones to teaching professionals based on confidence estimations. We present the results of a proof-of-concept experiment on three real-world publicly available datasets to demonstrate that our framework can obtain the balance of human cost and automatic evaluation accuracy, which can be flexibly deployed to real-world educational contexts in the future. To encourage reproducible research, we make our code publicly available at https://github.com/gwbnwnwh/human-in-the-loop.