Joint AI Task Allocation and Virtual Object Quality Manipulation for Improved MAR App Performance

The emergence of modern mobile System on Chips (SoCs), featuring robust neural network accelerators such as GPUs, DSPs, and NPUs, has made on-device inference a compelling alternative to cloud-assisted inference. Typical mobile augmented reality (MAR) applications enable users to interact with virtual objects, leveraging diverse Artificial Intelligence (AI) capabilities facilitated by a range of deep learning models. Several studies seek to improve the performance of MAR apps. However, they often overlook the computational concurrency between the AR tasks necessary to render virtual objects and the AI tasks, which can influence virtual object quality and AI inference response time. In this paper, we present HBO, a framework for MAR apps that trades off between AR and AI task performance. HBO leverages Bayesian optimization and heuristic algorithms to jointly manipulate the virtual objects' triangle count and AI task allocation for optimized MAR app performance. We have implemented HBO on Android and tested it on real smartphones and with real users. Our results show that HBO helps reduce the average AI task latency by up to 3.5x and increase the average virtual object quality by up to 38.7% compared to several stateof-the-art baselines.