Term Extraction from Chinese Texts Without Word Segmentation

The paper is dedicated to the problem of automatic term extraction from natural language texts. One of the first steps in this topic is building a domain thesaurus. Well approved methods of terms extraction based on word frequencies exist for alphabetic languages. Direct application of these methods for hieroglyphic texts is challenged because of missing spaces between words. The sentence segmentation task in hieroglyphic languages is usually solved by dictionaries or by statistical methods, particularly, by means of a mutual information approach. Sentence segmentation methods, as well as methods of terms extraction, separately, do not reach 100 percent precision and recall, and their combination just increases the number of errors. The aim of this work is to improve recall and precision of domain terms extraction from hieroglyphic texts. The proposed method is to identify repetitions of the two, three or four symbol sequences in each sentence and correlation of occurrence frequencies for these sequences in the target domain and contrast documents collection. According to the research, it was stated that a trivial ranking of all possible symbol sequences allows extracting only frequently used terms. Filtering of symbol sequences by their ratio of frequencies in the domain and in the contrast collection gave the possibility to extract reliably frequently used terms and to find satisfactory rare domain terms. Some results of terms extraction for the “Geology” domain from a Chinese text are presented in this paper. A set of articles from the newspaper “Renmin Ribao” was used as a contrast collection and some satisfactory results were obtained.