Assessing Student Programming Process Using Automated Reasoning

This Innovative Practice Full Paper presents an automated reasoning approach to the assessment of a student's programming process as captured by an autograder. When students are assessed in computing courses, they are typically assessed on their completed work, such as a programming assignment. Although such an assessment measures a student's general ability, it provides only an indirect measure of the effectiveness of their programming (or problem-solving) process. Without the ability to examine the process by which a student completes an assignment, an instructor may have difficulty teaching problem-solving skills since they cannot measure how the student's skill develops. Some computing courses make use of an autograder which automatically assesses student homework. The trace captured by an autograder is a unique source of information about a student's problem-solving ability, which can be analyzed to measure properties of that learner's process. This provides a way to assess the process-based aspect of student learning, enabling instructors to not only teach problem-solving skills but to determine how well students acquire these skills. This enables teachers to evaluate the methodology they employ and to select instructional material based on student need. Automatically analyzing the trace captured by an autograder has two major challenges: the scale of the data makes traditional approaches computationally expensive, and the definition of a problem-solving process is often expressed in a commonsense way based on an expert's intuition. These challenges can be addressed by using an Answer Set Programming (ASP) approach. ASP is a declarative language that is intended for solving computationally hard problems, and through various extensions, supports human-style commonsense reasoning. Assessing a student is computationally hard due to the combinatorial nature of the different ways an assignment may be solved. In this paper, we discuss the design of an ASP-based system for automatically assessing a student's problem-solving process using the trace captured by an autograder, and present a preliminary evaluation of the system applied within a university-level course on data structures & algorithms.