Vision + X: A Survey on Multimodal Learning in the Light of Data

We are perceiving and communicating with the world in a multisensory manner, where different information sources are sophisticatedly processed and interpreted by separate parts of the human brain to constitute a complex, yet harmonious and unified sensing system. To endow the machines with true intelligence, multimodal machine learning that incorporates data from various sources has become an increasingly popular research area with emerging technical advances in recent years. In this paper, we present a survey on multimodal machine learning from a novel perspective considering not only the purely technical aspects but also the intrinsic nature of different data modalities. We analyze the commonness and uniqueness of each data format mainly ranging from vision, audio, text, and motions, and then present the methodological advancements categorized by the combination of data modalities, such as Vision+Text, with slightly inclined emphasis on the visual data. We investigate the existing literature on multimodal learning from both the representation learning and downstream application levels, and provide an additional comparison in the light of their technical connections with the data nature, e.g., the semantic consistency between image objects and textual descriptions, and the rhythm correspondence between video dance moves and musical beats. We hope that the exploitation of the alignment as well as the existing gap between the intrinsic nature of data modality and the technical designs, will benefit future research studies to better address a specific challenge related to the concrete multimodal task, prompting a unified multimodal machine learning framework closer to a real human intelligence system.

I. Introduction

We perceive and communicate with the world through a multisensory human system, by seeing objects, hearing sounds, speaking languages, as well as writing and reading texts. The information from these varied sources is processed by different parts of the human brain, as indicated by [5], [20], [252]. For instance, the occipital lobe acts as the primary center for visual processing, interpreting the distance and locations of objects, while the temporal lobe processes auditory information, helping us understand sounds. Language comprehension, facilitated by Wernicke's area in the posterior superior temporal lobe, is crucial for decoding both written and spoken words. Other sensory information, such as touch and movement, is processed by distinct brain areas. These integrated yet distinct functions form a complex and harmonious human sensing system. The specialized divisions in human neural processing, which highlight both unique and shared characteristics across different modalities, inspire us to think about the multimodal machine learning problem in the light of data in this paper.