I. Introduction
During the last decade, Deep Neural Networks (DNNs) have achieved great success in many critical applications, such as autonomous driving, medical image recognition and face recognition [1], [2]. These successful applications significantly depend on the numerous training data and sufficient computing power, which are not always available for the users. To alleviate the manual workload and computing burden, users often turn to third-party outsourcing services for model training. For instance, the preparation of a dataset, especially the labor-intensive task of labeling, can be both heavy and time-consuming. Therefore, many users prefer outsourcing such tasks to third-party companies. Moreover, users with limited computing power can only train DNNs remotely on third-party platforms, such as Google Cloud. This exposes a potential security vulnerability for backdoor attackers [3], [4], [5].