Reinforcement Learning with Temporal Logic Specifications for Regression Testing NPCs in Video Games

Reinforcement learning (RL) is a promising strategy for the development of autonomous agents in various control and optimization contexts, including the generation of autonomous players in video games. However, designing these agents, and in particular their reward functions to perform sequential decision-making, can be challenging for most users and often require tedious trial-and-error processes until a satisfactory result is obtained. Consequently, these strategies are generally beyond reach for designers and quality control teams, who could potentially make use of them to generate automatic testing agents. This paper presents the application of reinforcement learning and behavioral descriptions given through a formal temporal logic task specification language (TLTL) for the design of NPCs that can be employed as surrogates for the player in such contexts. We argue that these techniques enable designers to naturally specify the way in which they would expect the final player to interact with a level and then generate a test that automatically verifies whether this strategy continues to be feasible throughout the development of the game. We include a series of experiments conducted on a custom 3D test environment developed in Unity3D that show that the proposed methodology provides a simple mechanism for training NPCs in settings that are commonly encountered in modern video games.

I. Introduction

This paper will focus mainly on the problem of automatic generation in video games of non-player characters (NPCs) that are able to follow a sequential specification of tasks. This is especially relevant within the scope of the testing and QA process, where typically a group of human testers have to replay the level walkthroughs to find bugs that they report to the programmers for correction. These walkthroughs usually follow a logic that tries to foresee possible ways for the player to solve a level, which in our approach we represent as a set of steps ("Move to a platform, pick up an object, from there advance to the exit, all while avoiding being killed by an enemy attack. Check that it is possible to perform this sequence of tasks while losing less than half of the player’s health points"). Our goal is to develop NPCs that are able to learn to follow these steps and test the game’s levels, even when those levels or their logic undergo changes, as usually happens during the development of a video game.