Extraction of salient textual patterns: synergy between lexical cohesion and contextual coherence

Most current information retrieval systems rely solely on lexical item repetition, which is notorious for its vulnerability. In this research, we propose a novel method for the extraction of salient textual patterns. One of our major objectives is to move away from keywords and their associated limitations in textual information retrieval. How individual sentences in text fit together to be perceived as a salient pattern is identified. A text network that exhibits textual continuity, arising from a connectionist model, is described. The network facilitates a dynamic extraction of salient textual segments by capturing semantics from two different categories of natural language, namely lexical cohesion and contextual coherence. We also present the results of an empirical study designed to compare our model with the performance of human judges in the identification of salient textual patterns. The preliminary results show that our model has the potential for automatic salient patterns discovery in text.

I. Introduction

RECENT RESEARCH in language engineering has taken natural language processing beyond the sentence level, not only because textual analysis is necessary, but also because of the difficulties of textual knowledge extraction in the enormous information cyberspace where there is constant massive dissemination of written text. Textual knowledge extraction involves interpretation of the text at a reasonable level and identification of the relative importance of each word. While the techniques involved in full text summarization are far beyond the state-of-the-art in current linguistics, the great demands in the community of information retrieval (IR) which struggle to move beyond keywords has motivated the research on the extraction of salient textual patterns and the identification of textual regularities [36], [45], [49].