Event-Specific Audio-Visual Fusion Layers: A Simple and New Perspective on Video Understanding

To understand our surrounding world, our brain is continuously inundated with multisensory information and their complex interactions coming from the outside world at any given moment. While processing this information might seem effortless for human brains, it is challenging to build a machine that can perform similar tasks since complex interactions cannot be dealt with a single type of integration but require more sophisticated approaches. In this paper, we propose a new simple method to address the multisensory integration in video understanding. Unlike previous works where a single fusion type is used, we design a multi-head model with individual event-specific layers to deal with different audio-visual relationships, enabling different ways of audio-visual fusion. Experimental results show that our event-specific layers can discover unique properties of the audio-visual relationships in the videos, e.g., semantically matched moments, and rhythmic events. Moreover, although our network is trained with single labels, our multi-head design can inherently output additional semantically meaningful multi-labels for a video. As an application, we demonstrate that our proposed method can expose the extent of event-characteristics of popular benchmark datasets.

1. Introduction

Real-world events around us consist of different multisensory signals and their complex interactions with each other. In-the-wild videos of real-life events and moments capture a rich set of multi-modalities and their complex interactions therein. Thus, it is essential to leverage multisensory information for better video understanding, but their diversity and complex nature make it challenging. For instance, even though audio and vision signals are congruent, the way they relate are different. All these events have different types of characteristics (such as uni-modal types of vision only and audio only, and multi-modal types of continuous, instant, rhythmic, etc., as shown in Figure 1) which we call them as event types. That is, understanding video contents requires to properly deal with such diverse and complex associations and relationships. However, surprisingly, this has been overlooked by prior audio-visual recognition research.