To What Extent is Stress Testing of Android TV Applications Automated in Industrial Environments?

An Android-based smart television (TV) must reliably run its applications in an embedded program environment under diverse hardware resource conditions. Owing to the diverse hardware components used to build numerous TV models, TV simulators are usually not sufficiently high in fidelity to simulate various TV models and thus are only regarded as unreliable alternatives when stress testing such applications. Therefore, even though stress testing on real TV sets is tedious, it is the de facto approach to ensure the reliability of these applications in the industry. In this paper, we study to what extent stress testing of smart TV applications can be fully automated in the industrial environments. To the best of our knowledge, no previous work has addressed this important question. We summarize the findings collected from ten industrial test engineers who have tested 20 such TV applications in a real production environment. Our study shows that the industry required test automation supports on high-level GUI object controls and status checking, setup of resource conditions, and the interplay between the two. With such supports, 87% of the industrial test specifications of one TV model can be fully automated, and 71.4% of them were found to be fully reusable to test a subsequent TV model with major upgrades of hardware, operating system, and application. It represents a significant improvement with margins of 28% and 38%, respectively, compared with stress testing without such supports.

1. Acronyms

TV

Television.

CPU

Central processing unit.

MIC

Microphone.

ADB

Android debug bridge.

D-pad

Directional pad.

GUI

Graphical user interface.

API

Application programing interface.

OS

Operating system.

USB

Universal serial bus.

TTS

Text to speech.

SAPI

Speech application programing interface.

ANR

Application not responding.

TAST

Testing of Android-based Smart TVs.

ANOVA

Analysis of variance.