On the Automation of Audio Plugin Testing

Test automation, i.e., the generating and executing test cases without human intervention, is inevitable for today's agile software engineering processes. However, there are still areas where test automation is not applied due to various reasons like in the domain of digital signal processing and audio software. In this paper, we focus on test automation of audio plugins used in digital audio workstations. In particular, we present a framework behind an automatic regression testing tool, intended to support developers in the course of the continuous integration. The framework allows integrating different test case generation methods and supports regression testing by comparing the rendered audio output of two different audio plugin versions. The corresponding tool supports combinatorial and random testing. Furthermore, we present experimental results obtained making use of several real-world audio plugins. The experimental results show that random testing requires less test cases to reach the maximum achievable mutation score.