Metamorphic Testing of Code Search Engines

Code search engines are widely used by software developers. However, due to the huge amount of data being processed as well as the lack of complete specifications, the testing of code search engines faces the oracle problem. Metamorphic testing (MT) is a well-known testing technique that is effective for alleviating the oracle problem. In this paper, we propose to apply MT to test code search engines. We first identify five metamorphic relations (MRs) by considering the characteristics of code search engines. Four MRs are defined based on the partial specifications of code search engines and thus are used for the purpose of verification, and the one MR identified from the users' perspective can be used to conduct validation. The approach is evaluated by conducting a series of experiments involving four popular code search engines (namely, Krugle, searchcode, sourcegraph and Zoekt), and one well-known code repository (namely, GitHub) that provides the code search services. The experimental results show that the abnormal behaviors of Krugle, searchcode, GitHub, and Zoekt have been successfully detected. By further inspecting the root causes of these abnormal behaviors, three critical issues related with the code search engines under investigation are identified and reported. These results demonstrate the effectiveness of our approach, and are also helpful for the users and developers to gain a deeper understanding about the code search engines.