Exploring Conversations with AI NPCs: The Impact of Token Latency on QoE and Player Experience in a Text-Based Game

The recent improvements of artificial intelligence (AI) technologies present a revolutionary opportunity for a wide variety of services. One of the most impactful AI technologies, gaining hundreds of millions of users in mere months, are conversational agents based on Large Language Models (LLMs). An obvious application example of conversational agents is found in gaming, where players often engage in conversation with non-player-characters (NPCs). The use of contemporary AI in this context is expected to bring unprecedented levels of adaptability and freedom of expression to in-game conversations, but may come with its own set of drawbacks, such as increased response latency, which may impair user experience during such interactions. In this paper we present the results of a user study investigating the impact of two different types of latency — time to first token and time per output token — on the overall Quality of Experience and player experience in a conversational task with an LLM-powered NPC in a text-based fantasy role-playing game. Conducted with a sample of fairly regular AI users (N=20), our study provides valuable insights regarding the possible threshold for noticeable latency for this type of service.