The Potential of Large Language Models as Tools for Analyzing Student Textual Evaluation: A Differential Analysis Between CS and Non-CS Students

Research on the analysis of Student Textual Evaluation encounters ongoing challenges. Large language models, as emerging tools in natural language processing, have garnered extensive attention. This study explores the potential of large-scale language models as tools for analyzing student course evaluations on the Coursera platform and compares Computer Science (CS) and non-Computer Science (non-CS) course reviews to investigate variations in student sentiment and thematic content between these two domains. The study adopts a systematic approach to review and analyze student reviews, identifying common sentiments and patterns, and categorizing reviews into relevant evaluation themes. Additionally, the study assesses inter-annotator agreement to validate the accuracy of manual analyses. Experimental findings reveal a strong correlation between large language models and actual course ratings as well as human-analyzed results, suggesting their potential as tools for assessing student course evaluations. Results from the analysis of CS and non-CS course reviews indicate significant disparities in the distribution of thematic content between these two academic domains.