Research on Long Text Similarity Calculation Method Based on TextRank and BERT

Aiming at the problem that the text input length of the BERT model is limited to 512 characters, and the number of words in long text is large, it is impossible to input long text directly into the BERT model to better obtain the meaning of words, this paper proposes a new long text similarity calculation method based on TextRank and BERT. Firstly, the method uses TextRank algorithm to extract a certain number of key words and sentences from the long text, to reduce the length of the text and retain the most important semantic information as far as possible. Then, the vector representation of the extracted sentence information is obtained through the BERT model, and it is input into the network layer of Bi-LSTM and attention mechanism to further extract features. Finally, the feature representation of the two pieces of text is obtained and the similarity is calculated to determine whether it matches. Experiments show that the proposed method can effectively solve the problem that the number of words in long text cannot be input into the BERT model, and effectively reduce the amount of calculation of the model, and the model can also better extract deep semantic information. Experiments on the public Chinese long text News datasets CNSE (Chinese News Same Event) and CNSS (Chinese News Same Story) achieve 82.86%and 87.91%accuracy respectively.