Generalized Named Entity Recognition Framework

The process of digitization of data through Optical Character Recognition has been most commonly done through means of Computer Vision. An attempt was made to replace this with a Natural Language Processing method called Named Entity Recognition for extraction of the required data values from the OCR output of the images of documents through means of spaCy. Named Entity Recognition is a technique used to automatically identify named entities in a text and classify them into predefined categories. It helps businesses easily analyze huge amounts of unstructured data. Text Annotation helps machines recognize the crucial words in a sentence making them more powerful. It is an essential requirement for making a dataset that can be used for NLP Models. Labeling the keywords in each statement is important to make the entire statement understandable to machine learning models. A generalized NER framework was developed by the authors which lets users build training models on top of the existing spaCy models to allow for named entity recognition on their text data. The framework takes a configuration file which contains model name, model size and hyperparameters, along with annotated data in JSON format as input, and returns a customized spaCy model as output. An annotation tool STAT was also built by the authors specifically for use with the framework which produced the required training data files. The model built by the authors is based on test configurations with drop as 0.2 and 30 iterations whose results are discussed in the Results Section.