Multimodal Fusion With Block Term Decomposition for Asynchronous Federated Learning

Federated learning (FL) has been extensively studied as a means of ensuring data privacy while cooperatively training a global model across decentralized devices. Among various FL approaches, asynchronous federated learning (AFL) has distinct advantages in overcoming the straggler problem via server-side aggregation as soon as it receives a local model. However, AFL still faces several challenges in large-scale real-world applications, such as stale model problems and modality heterogeneity across geographically distributed and industrial devices with different functions. In this article, we propose a multimodal fusion framework for AFL to address the aforementioned problems. Specifically, a novel multilinear block fusion model is designed to fuse various multimodal information, which serves as an enhancement for perceiving and transmitting the important modality and block during local training. An adaptive aggregation strategy is further developed to fully utilize heterogeneous data by allowing the global model to favor the received local model based on both freshness and the importance of the local data. Extensive simulations with different data distributions demonstrate the superiority of the proposed framework in heterogeneity scenarios, which exhibits significant merits in the improvement of modality-based generalization without sacrificing convergence speed and communication consumption.

I. Introduction

Federated learning (FL) is an innovative distributed machine learning technology that has emerged as an advanced paradigm for multiparty training across decentralized local end devices, such as industrial devices and mobile phones. In FL, decentralized devices are leveraged to train local models with their data, which are then aggregated by the cloud server to update the parameters of the global model. It has significant merits in tackling the risks of privacy leakage and data security that arise when data is centralized in the cloud system via cooperatively training a global model [1]. As a result, FL has gained popularity in various industrial Internet of Things (IIoT), including smart manufacturing, industrial automation, and intelligent factories [2].