Improving Feedback on GitHub Pull Requests: A Bots Approach

Rising enrollments make it difficult for instructors and teaching assistants to give adequate feedback on each student's work. Our course projects require students to submit GitHub pull requests as deliverables for their open-source software (OSS) projects. We have set up a static code analyzer and a continuous integration service on GitHub to help students check different aspects of the code. However, these tools have some limitations. In this paper, we discuss how we bypass the limitations of existing tools by implementing three Internet bots. These bots are either open source or free for OSS projects and can be easily integrated with any GitHub repositories. One-hundred one Computer Science and Computer Engineering masters students participated in our study. The survey results showed that more than 84% of students thought bots can help them to contribute code with better quality. We analyzed 396 pull requests. Results revealed that bots can provide more timely feedback than teaching staff. The Danger Bot is associated with a significant reduction system-specific guideline violations (by 39%), and the Code Climate Bot is associated with a significant 60% decrease of code smells in student contributions. However, we found that the Travis CI Bot did not help student contributions pass automated tests.