GUI Component Detection Using YOLO and Faster-RCNN

The graphical user interface (GUI) is crucial for communicating with software users. The detection of GUI elements holds significant importance for various software test automation tasks. In this study, two different object detection models such as YOLOv8 and Faster R-CNN are used to address challenging GUI component detection problems in mobile applications. Two different datasets were used. The first dataset is the RICO data, which consists of 5 classes. The second dataset consists of 600 mobile application screenshots collected from the open-source. This dataset consists of 7 classes and has been labeled and trained on Roboflow. In addition, an automation system was developed to test the object detection models for errors after the version change in the mobile application. With this test automation, the locations of the errors in the mobile application in real-world scenarios were determined and reported. In this experiment, 2 different data were trained in 2 different models, and the data we labeled gave the best result with a mAP value of 0.81 in the YOLOv8 model.