A Systematic Study on Factors Impacting GUI Traversal-Based Test Case Generation Techniques for Android Applications

Many test case generation algorithms have been proposed to test Android apps through their graphical user interfaces. However, no systematic study on the impact of the core design elements in these algorithms on effectiveness and efficiency has been reported. This paper presents the first controlled experiment to examine three key design factors, each of which is popularly used in GUI traversal-based test case generation techniques. These three major factors are definition of GUI state equivalence, state search strategy, and waiting time strategy in between input events. The empirical results on 33 Android apps with real faults revealed interesting results. First, different choices of GUI state equivalence led to significant difference on failure detection rate and extent of code coverage. Second, searching the GUI state hierarchy randomly is as effective as searching it systematically. Last but not the least, the choices on when to fire the next input event to the app under test is immaterial so long as the length of the test session is practically long enough such as 1 h. We also found two new GUI state equivalence definitions that are statistically as effective as the existing best strategy for GUI state equivalence.

I. Introduction

Android [37] is the most widely deployed mobile operating system (OS). In early 2017, the number of Android applications (apps) in Google Play is about 2.4 million and in late 2017, almost another 1 million Android apps have been added to Google Play [39]. Owing to the competitions among apps of the same kind, developers are increasingly placing higher priorities to assure their Android apps against bugs.