Sonifying motor skills with Pizzicato, a game for motor behavior research

Learning motor skills is essential to many different aspects of life, from big movements needed for sports to small and simple movements used in the rehabilitation of stroke patients. In recent years, sonification, i.e. using sounds as feedback for actions, has been researched as a promising technique for studying motor behavior. In particular, we explore how to use sonification to make the process of learning motor skills accessible and engaging. We posit that an interactive and gamified environment can increase the engagement in that process. Moreover, an enjoyable setting is more likely to stimulate repetition, an indispensable feature of any learning endeavor. We, therefore, designed and developed Pizzicato, a rhythm-based serious game that leads players to move their arms and hands to actively play music. The game uses a common webcam to track your hand movements: pinching one finger to the thumb at the right position and moment will play musical notes that pleasantly add up to a full musical track. Our player tests have shown that players find Pizzicato accessible and engaging, and report that playing the game gives them a strong sense of agency. Pizzicato was developed in collaboration with neuropsychology colleagues, who are now starting to use it as a flexible tool for motor behavior research, both for diagnostic and rehabilitation purposes.