Optimizing File Similarity Search Techniques in Data Lakes using Parallel Processing and Big Data Frameworks

Smart cities generate a massive amount of disparate data from multiple sources, necessitating the use of Data Lakes for efficient storage and management, especially with the advent of Big Data. However, it is critical to use appropriate search techniques and governance to prevent the Data Lake from becoming a data swamp. The Data Lake encompasses a wide range of documents and topics, which makes finding similar datasets, e.g., through schema matching approaches, difficult and computationally intensive. In this paper, we propose an approach that leverages Data Lakes and Big Data frameworks to improve the search for related or near-identical structured textual datasets. We present a framework for re-implementing the early pruning algorithm “DS-Prox” within a Big Data framework using Spark. The framework involves prototyping with two machine learning models: a Multilayer Perceptron neural network and a Random Forest ensemble learner. The effectiveness of our approach is demonstrated with good accuracy levels, highlighting the benefits of managing heterogeneous Big Data in Data Lakes.