Gradient-Based Differential Privacy Optimizer for Deep Learning Model Using Collaborative Training Mode

Deep learning model based on artificial neural network is one of the greatest pushers to realize intelligence of information system unprecedentedly. However, the risk of leaking user data privacy by attacking deep learning model exists in the training process, especially when multi-users concurrently utilize the service of cloud. Motivated by this observation, in order to protect the privacy of deep learning model, this paper proposes a gradient-based differential privacy optimizer using collaborative training mode based on CPU-GPUs hybrid system in cloud. In gradient-based differential privacy optimizer, random sampling, gradient tailoring, gradient-based random perturbation, and advanced privacy budget statistics together guarantee the usability and privacy of the model. Specifically, the effects of privacy budget, noise scale, and privacy deviation parameters and their combinations on accuracy of model are experimentally studied in this paper. To further improve the training efficiency of model, the CPU-GPUs hybrid system is explored as a collaborative training mode for gradient-based differential privacy optimizer. The experimental results indicate that using publicly available dataset MINIST and implemented in TensorFlow is proved to be feasible and efficient. Specifically, our implementation and experiments demonstrate that we can obtain approximately 96% of accuracy under a modest privacy budget. Furthermore, we can achieve up to 20%-30% speed-up ratio in the training process based on gradient-based differential privacy optimizer.

I. Introduction

Along with the rise of deep learning concept and the corresponding training methodology first proposed by the Geoffery et al. [1], it is now customary to apply deep learning model to analyze complex information in various practical domains. Deep learning model builds deep neural network by simulating the way of human brain processing information. Deep learning model has made breakthrough development in applications of speech processing [2], [3], natural language processing [4]–[6], especially computer vision [7], [8] in recent years. However, many malicious users attack the deep learning model to obtain the privacy of training sample data. Therefore, how to protect the sensitive information of users in the process of training process becomes particularly important in cloud paradigm.