Impact of Labeling Noise on Machine Learning: A Cost-aware Empirical Study

Since the emergence of large datasets, machine learning models have demonstrated excellent performance in a wide range of applications. This accomplishment was made possible by the availability of large amounts of labeled datasets. Finding high-quality labeled datasets, on the other hand, is difficult to obtain. Acquiring high-quality datasets with limited class label noise becomes an important task since noisy datasets can affect the performance and structure of machine learning models. However, it is extremely difficult to reduce label noise significantly in real-world datasets unless using expensive expert annotators. This work studies the influence of varying degrees of label noise on the complexity and accuracy of machine learning models, based on considerable testing and research. It also explores how to reduce labeling costs while maintaining the desired accuracy.

I. Introduction

Machine learning have shown outstanding performance in a variety of applications since the recent advent of largescale datasets. This success is contingent on the availability of large amounts of labeled data [1]-[3], which is both costly and time-consuming to gather [4]-[6]. Many techniques are introduced in the literature to reduce the high labeling cost by using non-expert annotators Amazon’s Mechanical Turk [4][8]; however, the use of nun experts often results in falsely labeled data also known as noisy labels [4]– [7]. In real-world datasets, the percentage of incorrect labels has been observed to range from 5% to 38% [8]– [12].