Metamorphic testing of programs on partial differential equations: a case study

We study the effect of applying metamorphic testing to alleviate the oracle problem for numerical programs. We discuss a case study on the testing of a program that solves an elliptic partial differential equation with Dirichlet boundary conditions. We identify a metamorphic relation for the equation and demonstrate the effectiveness of metamorphic testing in identifying the error. The relation identified should also be applicable to other numerical methods that yield better approximations on the refinement of grid points or step sizes.

1. Introduction

Numerical programs are vital to our daily lives. They have been used not only in various theoretical disciplines, but also engineering and medical practices including mission-critical and safety-critical applications. Unfortunately, despite the importance of the quality of numerical packages, we are far from doing a good job [17]. Like other testing contexts, we usually assume that we can verify the actual outputs of numerical software against some expected results. We call the mechanism of checking the correctness of the test output as a test oracle [5], [15]. Developers of numerical software generally adopt the following mechanisms as test oracles: (a)

Comparing with analytical solutions, simulation results, tabulated values, or hand-calculations [15].

(b)

Verifying with standard mathematical libraries or reference software packages [7].