Late Fusion-based Distributed Multimodal Learning

Multimodal artificial intelligence promises deeper insights by analyzing data from diverse sources such as text, images, audio and more. However, efficiently processing and fusing large multimodal datasets remains an open challenge. This paper presents a Spark-based approach to parallelize multimodal encoding tasks. A key aspect is the use of late fusion with frozen backbone encoders, allowing encodings to be processed independently across cluster nodes. The encoded vectors can then be used for a variety of supervised and unsupervised tasks, regardless of whether they are gradient-based or not. Experimental results on image, text and audio datasets show that Spark clusters can offer competitive performance compared to GPUs, especially for I/O-intensive modalities. While GPUs outperform the Spark cluster when sufficient CPU cores are available, Spark makes it possible to use already available commodity hardware. The presented architecture demonstrates how distributed computing platforms like Spark can be effectively “repurposed” for multimodal AI, enhancing scalability and making such systems more accessible.