Communication-Efficient Collaborative Learning of Geo-Distributed JointCloud from Heterogeneous Datasets

With the popularity of cloud computing, the use of services provided by the cloud is increasing. Due to the high communication cost, and data privacy issues, a new crosscloud collaborative computing model is demanded instead of a single giant cloud. Coping with federated learning and Joint-Cloud, we propose a federated learning-based collaborative learning framework, in which the distributed cloud entities are able to learn the same model collaboratively. As compared to traditional cloud-centric approaches, the framework for JointCloud can reduce the network bandwidth overhead and guarantee privacy. However, there are two crucial challenges in the federated manner: heterogeneity and high communication overhead. To address the heterogeneity, we propose a Teacher-Student mechanism, the key of which is a regularization term incorporated with the objective function so as to adjust the gradients from the clients among JointCloud with different data distribution. Then, based on the Teacher-Student mechanism, we further present a communication-efficient federated optimation approach via joint Identification-Verification to reduce the communication rounds. We conduct extensive experiments on Non-IID datasets. The experimental results demonstrate that the proposed framework can significantly reduce communication costs and improve performance.