Automatically finding Metamorphic Relations in Computational Material Science Parsers

Software testing is an important part of the software life-cycle. Unfortunately, some software systems have the inherent problem, that it is not clear a priori what the expected outcome is. This is known in the literature as the Oracle Problem. Scientific software suffers from this problem in particular. Metamorphic Testing is a testing approach that mitigates the Oracle Problem, as it is based on identified relations between a program's in- and output pairs. In this study, we investigate the feasibility of automatically finding such metamorphic relations on a software package known as the exciting-NOMAD parser which is widely used in computational material science. We are able to show that it is indeed possible to automatically find metamorphic relations within the NOMAD parser for the density functional theory code exciting. We analyse the metamorphic relations found through our tool in terms of both quantity and relation quality. Furthermore, we also publish our developed tool, as well as used data alongside this study through our replication package.