Exploring musical information through images: a method for leveraging musical iconography to the semantics of internet of things

Musical things are often referred to physical objects, but also rely on soft immaterial resources, such as cultures, codes, human perceptions, and creation of symbols in modern cultures. Furthermore, musical things interact with subjective human actions, gestures, and emotions, which are not exactly physical entities but can be felt as objects through our bodies. The internet is populated by a complex network of images of bodies performing musical actions without direct links with their original or imagined sounds. In this study, we propose a method to retrieve musical context from body postures depicted in bidimensional images. By leveraging cultural and human body’s homologies we propose a strategy to visualize the similarity of body postures as a map of probabilities of musical features. The method involves 3D annotation of the body in static images in comparison with motion capture recordings. The results show the application to illustrations of capoeira from the XIX century. Although not reconstructing full musical samples from the past, it enables exploring homologous musical cultures to trace networks of digital objects using music processing and analysis. The study demonstrates how the method can explore significant features from static images, which could impact various fields, particularly in immaterial heritage and networks of multimodal representations.