Experience Report: How Do Metamorphic Relations Perform in Geographic Information Systems Testing

Geographic Information Systems (GIS) is a foundational application for different information systems, such as navigation system and global position system. However, due to the complexity of the system and algorithms, traditional testing methodologies confronted with the test oracle problem. Metamorphic testing (MT) can help resolve the problem by comparing metamorphic relations (MR) among multiple inputs and outputs, which have applied in many different domains. In this paper, we try to apply MT in GIS testing. We propose a semi-automated MT approach for GIS testing. To illustrate the effectiveness of the approach, we conduct a case study with a typical component of GIS: superficial area calculation program. In the empirical study, we construct six kinds of MRs with different properties and characters of the program or its algorithm. Our method could detect the target faults effectively without generating test oracles manually.

I. Introduction

When we test the components of Geographic Information Systems (GIS), it is usually not easy to verify the correctness of the outputs. This is regarded as “test oracle problem [1]”, which restricts software testing seriously. Most of the programs of GIS belong to this category, such as terrain calculation program. Recently, metamorphic testing (MT) [2] is considered as an effective method for alleviating oracle problem. It checks the metamorphic relations (MR) between original test cases and follow-up test cases to determine whether the program satisfies necessary properties. Based on traditional MR definition model, we provide a formal model for MR description and decomposition. For test case generation, a MT based Adaptive Random Testing (MT-ART) algorithm is also proposed in this paper. With these two improvements, we study MT in GIS testing and propose a semi-automated MT method for GIS testing. In addition to presenting our technique, we describe an empirical study, which performed on an industry application of superficial area calculation program. In the case study, we construct six kinds of MRs from different characters of the target program and its algorithm. In this paper, we provide the following unique contributions: (1) A methodology for experimental study of MT in GIS, in which an adaptive test case generation method is used in metamorphic test cases generation. (2) A detailed case study on how to construct MRs for GIS, which kind of MRs performs better than others in fault detection. Based on these contributions, we present an alleviation solution to the test oracle problem of the GIS testing.