Multimodal signal fusion

Our daily life involves multimodal signals (e.g., visual, audio, text, and signals from various sensors). Multimodal signals are highly correlated. For example, researchers have been using audio activities for video summarization of ball games and violence detection in movies. Multimodal signals are complementary to each other. For example, in multimodal surveillance, audio may carry important information not available in video. Usually, people don't hire a blind or deaf person for surveillance duties, since a human naturally fuses multimodal signals for the best results.