Semantic Similarity Analysis for Resume Filtering using PySpark

Technology is expanding and applicants are making their resumes look better by developing their skill set for the current environment. Each resume now looks very promising and manually screening resumes and finding out the potential candidate in time consuming and is not an efficient method. In this paper, we propose a method using Natural Language Processing (NLP) and Spark to filter relevant resumes. The work proposes a cosine similarity-based approach to measure text-similarity and hence find worthy candidates based on key information provided by the employer. The implementation of the proposed approach on a Spark platform helps in handling large-scale datasets, enabling efficient semantic similarity analysis in the context of job descriptions. The experiments conclude that the proposed method can efficiently speed up the process in short - listing worthy candidates and can effectively replace the process of hiring managers going through each resume. Our model demonstrates superior performance compared to a non-Spark-based resume filtering system, achieving an average reduction in processing time by 80%, translating to an average runtime decrease from 5 seconds to 1. This efficiency gain is scalable, presenting enhanced performance in candidate shortlisting, especially with larger datasets.