Multimodal Learning on Temporal Data

In recent years, multimodal learning has attracted an increasing interest. A special scenario of multimodal learning, learning on temporal data, is common but has not been well studied. In multimodal temporal data, not all modalities of a sample arrive at the same time. Because of that, different types of samples may have different importance in many use cases, where an early sample with significant modalities may be more valuable than a later one as early predictions can be made to speed up decision-making processes. Besides, sample correlations are very common in multimodal temporal data, as samples accumulate in time and a late sample may contain the same data existing in an earlier sample. Training without the awareness of the importance and correlation yields less effective models. In this work, we define multimodal temporal data, discuss key challenges and propose two methods that improve traditional multimodal training on such data. We demonstrate the effectiveness of the proposed methods on several multimodal temporal datasets, where they show 1% to 3% improvements over the baseline.

I. Introduction

Multimodal learning has attracted an increasing interest in recent years. As computer vision and language models are advancing rapidly, multimodal models start to provide breakthrough improvements, which unlocks new applications that naturally involve two or more modalities. From the healthcare field where fusing medical images and electronic health records shows improvements in performances when compared to models that used only single modalities [1], [2], to autonomous driving where intelligent systems are built to process various signals in different modalities [3], [4].