Combining local and global approaches to ascertain semantic similarity

Interviewing potential candidates is both time consuming and resource intensive. This is particularly prominent in organizations which go for large scale recruitment processes. In the current study, a client based application has been proposed for interviewing candidates for data science profiles where interviewee’s answers are scored using machine learning. Different approaches were tried with pretrained models but as the application was very much domain specific, those models did not provide good results. Hence a custom embedding layer was built on open source data science textbooks. These embeddings were used with Gated Recurrent Units (GRU) to capture a local approach (subject specific) in the interview answers. However, this neglected the nuances of the English language involved in critical reasoning. Hence Bi-directional Encoder Representation from Transformers (BERT) was employed to capture the global approach (interaction between words in the English language) in the interview answers. The similarity scores from these two approaches were ensembled into a machine learning model which allotted the final score to the interviewee’s answer. The proposed method outperformed the pretrained models with significant margin when tested with the validation data.