Estimating Student Grades through Peer Assessment as a Crowdsourcing Calibration Problem

There is a trend to move education into an online environment, especially when offline learning is restricted by time, space, availability, or is impacted by issues such as a public health incident. Evaluating students’ performance in online education has always been challenging. Objective questions, which can be graded automatically, could only assess certain aspects of students’ mastery of knowledge. A grading problem appears if subjective questions exist, primarily when the class is taught at scale. Many online education platforms have been using peer assessment to resolve this problem. Aside from that, peer assessment also improves interactions between students, instructors, and peers. While peer assessment has some inherent weaknesses, reviewers may not have the same ability or attitude toward reviewing others, and the feedback generated by them shall not be taken at face value. Many algorithms have been developed to evaluate annotators’ trustworthiness and generate reliable labels in the crowdsourcing industry. We proposed an algorithm under the same concept that could provide accurate automated grading, an overview of students’ weaknesses from peer feedback, and identify reviewers who lack an understanding of certain concepts. This information allows instructors to offer targeted training and create data-driven lesson plans.