Evaluation and Optimization of Distributed Machine Learning Techniques for Internet of Things

Federated learning (FL) and split learning (SL) are state-of-the-art distributed machine learning techniques to enable machine learning training without accessing raw data on clients or end devices. However, their comparative training performance under real-world resource-restricted Internet of Things (IoT) device settings remains barely studied. This work provides empirical comparisons of FL and SL in real-world IoT settings regarding (i) learning performance with heterogeneous data distributions and (ii) on-device execution overhead. Our analyses in this work demonstrate that the learning performance of SL is better than FL under an imbalanced data distribution but worse than FL under an extreme non-IID data distribution. Recently, FL and SL are combined to form splitfed learning (SFL) to leverage each of their benefits (e.g., parallel training of FL and lightweight on-device computation requirement of SL). Our work considers FL, SL, and SFL, and mounts them on Raspberry Pi devices to evaluate their performance, including training time, communication overhead, power consumption, and memory usage with resource-restricted IoT devices. Besides evaluations, we apply two optimizations. First, we generalize SFL by carefully examining the possibility of a hybrid type of model training at the server-side. The generalized SFL merges sequential (dependent) and parallel (independent) processes of model training and thus is beneficial to a system with a large scale of IoT devices, specifically at the server-side operations. Second, we propose pragmatic techniques to substantially reduce the communication overhead by up to four times for the SL and (generalized) SFL.

1 Introduction

Due to its stunning performance, deep learning (DL) has enabled various applications ranging from image classification, object detection, speech recognition to disease diagnosis, financial fraud detection [1], [2], [3]. One major factor in achieving high accuracy is usually to leverage big data to learn high-level features. The intuitive means is to gather the data centrally and then perform the DL model training. However, data can often be highly private or sensitive. For example, data collected from medical sensors [4] and microphones [5] would be such cases. Consequently, users may resist sharing their data with service/cloud providers to build a DL model. In addition, the data aggregator must pay great attention to the data regulations such as General Data Protection Regulation (GDPR) [6], and California Privacy Rights Act (CPRA) [7]. On the other hand, the centralized data might be mishandled or improperly managed by service providers—e.g., incidentally accessed by unauthorized parties [8], or used for unsolicited analytic, or compromised through the network and system security vulnerabilities—resulting in data breach [9], [10]. Therefore, there is a demand for training DL models without aggregating and accessing sensitive raw data that reside in the client-side [11], [12], [13], [14], [15].