Data Transfer for Balancing Model Convergence and Training Time in Federated Learning

Federated learning is a distributed machine learning technique that addresses the challenges of traditional centralized machine learning, such as high computational expenses and network congestion. In federated learning, a model is distributed to each server from a central server, and then each server trains the model using its own data. After training, the models are integrated at the central server, and the integrated model is re-distributed to the servers for further training. Federated learning can train a model based on all the data without collecting the data in one place, reducing the burden on the network and addressing the problem of training costs for extensive amounts of data. However, federated learning has two unique challenges: prolonged training time depending on the amount of data in each server, the processing capacity of each server, and the bandwidth of links in a network, and the accuracy of the trained model, which is influenced by the heterogeneity of the datasets in each server. We propose an optimization problem for the training time and the convergence behavior in federated learning. Specifically, we present a nonlinear programming problem to minimize the total training time while optimizing the data transfer route and destination of each router, subject to constraints on network congestion avoidance and convergence of the trained model. We evaluate the proposed method in an experiment using a virtual GPU cluster and show that the proposed method improves both the accuracy of the trained models and the training time compared to a prior method.

I. Introduction

Traditional centralized machine learning techniques encounter challenges in dealing with the high computational expense of processing large amounts of data collected at a single location and the burden on the network caused by the accumulation of voluminous data [1]. Federated learning [2], a form of distributed machine learning, has attracted attention as a potential solution to this issue. In federated learning, a model with a uniform structure is first distributed to each server from a central server, with each server training the model using only its own data. After several training cycles, each server transfers the trained model to the central server, which then merges the models received from the servers. The central server subsequently re-distributes the integrated models to the servers, and each server trains the transferred models using its own data again. By iterating through this process, federated learning can train models based on all the data without collecting the data owned by each server in one place. As mentioned above, federated learning resolves the problem of training costs for extensive amounts of data by distributing training across multiple servers. Federated learning also resolves the issue of the network load by indirectly integrating information on all of the training data through model exchanges without gathering data in a single location.