A Humanoid Robot Learning Audiovisual Classification By Active Exploration

We present a novel neurorobotic setup and dataset for active object exploration and audiovisual classification based on their material properties. In the robotic setup, a humanoid drops an item on a sloped surface and records the video image frames and raw audio of the collision of the surface and object. The novel dataset includes 32800 images and 1600 s of audio recording from 800 samples for 16 objects and will be made publicly available. We propose a novel neural architecture for the classification of the objects. A detailed analysis of results shows that different materials are easier classified either in the audio or the visual modality. As a main contribution, we can show that combining modalities can achieve an even higher classification accuracy of 90%.