LeMo: Supporting Collaborative Music Making in Virtual Reality

When engaged in music composition and improvisation, composers who are working together may want to communicate in modalities other than sound since that is the primary medium of the creative activity. This raises questions about the Sonic Interaction Design of Virtual Environments, specifically in terms of how to design user experiences which support collaboration without being detrimental to the product actually being created. In this paper we explore a multimodal approach to supporting creativity in collaborative music making in Virtual Reality. We outline the design and study of our two-person Virtual Reality step sequencer named LeMo, in which people are enabled to communicate via visual representations including free-form 3D annotations instead of spoken communication, leaving their full auditory sense to experiencing the joint creation of music. We studied how people used the annotations in LeMo to support their composition process and human-human interaction and identified five classes of use of annotation, three of which are particularly relevant to the future design of Sonic Interactions in Virtual Environments.