Quality Improvement of Image Datasets using Hashing Techniques

Image processing has become extremely important, with the consequences of real-time image processing failures being severe; thus, research and study in real-time image processing methods are extremely important. Some images contain incorrect information, requiring the use of techniques to improve the image and make it more understandable. Others require some pre-processing for the machine to understand and make important decisions about the image on its own, without any manual intervention. Part of pre-processing includes doing a clean-up of the dataset by removal of the bias that could appear. This paper presents the framework for detecting duplicates and near duplicates aiming towards making the dataset more efficient. This will in turn help in training the ML model to be better. This is achieved by implementing the Difference Hash (dHash), Average Hash (aHash), and Perceptual Hash (pHash). Hash functions are ideal to detect (near-)identical photos because of the robustness against minor changes, while also minimizing the number of false positive collisions. The proposed model is well-suited for various real-time image processing applications.