Paraphrase Quality Assessment Metric and Russian Paraphrase Dataset Collection

The task of generating paraphrases is becoming increasingly popular across many languages. It is not only useful for creating various content for websites and social media but also is frequently used to improve the performance of machine learning methods based on natural language processing (NLP) via data augmentation and enrichment. Because existing metrics for NLP are not really suitable for paraphrase quality assessment, there is always a challenge of quality assessment of a generated paraphrase. In this work, we propose a new metric for paraphrase quality assessment based on the topic modeling for long texts; we call this metric Intersecting Topic Similarity (ITS). We have collected the first of the kind dataset of strong paraphrases for the Russian language, which is now publicly available for future research. We conducted a comparison with existing metric families like BLEU and ROUGE, and METEOR metric on the collected dataset. Our metric ITS showed a better ability to distinguish paraphrases and is much more reliable in assessing paraphrase quality.