Named Entity Recognition Based on Pre-training Model and Multi-head Attention Mechanism

When processing Chinese named entity recognition, the traditional algorithm model have been having the ambiguity of word segmentation and the singleness of the word vector, and the training consequence of algorithm models was not well. To solve this problem, a BERT-BiLSTM Multi-Attention (PMA-CNER) model was proposed to improve the accuracy of Chinese named entity recognition (CNER). This model used BERT model to embed words based on BiLSTM model, which can extract global context semantic features more effectively. Next, a layer of Multi-head attention mechanism was added behind the BiLSTM layer, which can effectively extract multiple semantic features and overcome the shortage of BiLSTM in obtaining local features. Finally, the experimental results on the CLUSER2020 dataset and the Yudu-S4K dataset show that the accuracy rate is significantly improved, reaching 93.94% and 91.83% respectively.

I. Introduction

Named Entity Recognition(NER) is a task to identify the boundaries of entities in text and classify them according to predefined types. NER is the basis of many natural language processing(NLP) tasks, such as relation extraction, information extraction, question answering system, information retrieval, knowledge map construction, etc, so it has received extensive attention and research in the field of NLP.