Reassessing Assessment in Pre-Engineering Statistics Courses

In this research study, 371 participants were recruited from introductory pre-engineering statistics courses enrolled in large university in the southern United States across four different instructors and given an assessment on the topic of inferential statistics and hypothesis testing. The students were given two sections on the assessment: one that had single-response multiple-choice items and one that had multiple-select items. All test items were analyzed using Bayesian Item Response Theory (IRT) methods. Item analysis indicates high item difficulty parameters, likely due to collapsing scoring into a dichotomous model. Analysis and comparison of the individual participants' results was done via Bayesian hypothesis testing and regression models, which indicated that: 1.) there was a small but significant relationship between student performance on standard in-class examinations versus results on the instrument used in this study, which indicates that participant performance may be related to performance on course exams, but the relationship was not nearly as strong as initially hypothesized, and 2.) there was a strong relationship between instructor and performance on the multiple-select items, which indicates that these items may be more sensitive to instruction than other multiple choice items. The multiple-select items were deliberately constructed to inform us of particular misconceptions that students may have about the content in the course as well as how topics in a statistics course interrelate with one another. Additionally, since multiple-select items are multifaceted, they have the potential to provide instructors with the ability to see if a student has full, partial, or no understanding of a concept as opposed to traditional single response multiple-choice items.