Discrete Text Summarization (DeTS): A Novel Method Grounded in Natural Language Inference and Grammatical Sequence Scoring

Extracting information and insights from textual data is a challenge and there are many gaps in the natural language processing field. In this paper, we proposed a new unsupervised method for discrete text summarization (DeTS) that summarizes a whole text corpus using a novel combination of natural language inference and grammatical sequence scoring. Instead of creating one summary from the text corpus, we capture the main themes of the text by extracting a series of concise and independent key points. DeTS can be used for theme extraction and text summarization purposes across any type of textual data, and it is a bottom-up approach free from the data bias from which most unsupervised and supervised methods suffer. It is a fully automated method without a need for any human annotations, but with optional guardrails and configurations, including a new concept that we introduce called “startwords” which accelerates the computation speed by 50%. Notably, DeTS outperforms existing methods due to its detailed nature, paired with high levels of accuracy and coverage. Furthermore, it is an ensemble of open-source libraries that are available to the public. We analyzed employees’ comments from a large multinational organization to evaluate the performance of the new method. The results have shown that the new method can be used to extract meaningful and valuable insights at scale from large textual data.