A Turing Test for Beatmap-Generation

This paper examines whether AI can generate game content that provides an experience indistinguishable from human-created content. Specifically, we evaluate beatmaps for the rhythm game Dance Dance Revolution produced by an LSTMbased neural network model. Through a human trial study of 13 participants, we compare player ratings for AI-generated vs. human-curated beatmaps over dimensions of fun, annoyingness, naturalness, and length. In addition, players attempt to identify the origin of each beatmap as either human or AI-generated. Safe and Sound, the song with the best rated AI-generated beatmap, had ratings that rivaled human-curated beatmaps, however, the other AI beatmaps showed there is still a gap. For origin identification, AI maps were only correctly identified $63 \%$ and Human maps were misidentified as AI $36 \%$ of the time. These results indicate that while the AI exhibits promising similarities to human performance, key differences still remain. Refinement of the generative model such as adding more advanced architecture like the transformer or further increasing the size of training data is needed to fully match human beatmap creation. This work represents an initial “Turing Test” for procedural content generation in games, and as baseline human results, laying groundwork and inspiration for rigorous human-centered evaluation of AI creativity and experience generation.