New Word Discovery Algorithm for News

With rapid popularization of various mobile phone applications, many new words appeared on the Internet, resulting in the ineffective segmentation of these new words. This paper proposes a new method, which can effectively identify new words in the news. Firstly, the collected corpus is pre-processed and then proceed name entities recognition. At the same time, the candidate words are filtered through dictionary filtering, and finally the candidate words are filtered by mutual information and minimum branch entropy. Through experimental verification, it can be found that the algorithm for identifying new words proposed in this paper can well recognize new words on the Internet.